Reliability and validity along with reality check option
Reliability and validity along with reality check option
Hi!
Can anybody give me an easy example of checking validity and reliability in Vensim?
If I do check/compare the model result manually with the real output, can it be sufficient for reliability check?
If I check through "Model Check" and "Unit Check", is it sufficient to say my model is valid?
I am little confused. I have gone through few good article on reliability and validity but every where mentioned that there are no certain way to do check those.
Waiting for some valuable responses.
Can anybody give me an easy example of checking validity and reliability in Vensim?
If I do check/compare the model result manually with the real output, can it be sufficient for reliability check?
If I check through "Model Check" and "Unit Check", is it sufficient to say my model is valid?
I am little confused. I have gone through few good article on reliability and validity but every where mentioned that there are no certain way to do check those.
Waiting for some valuable responses.
M Shamsuddoha
Curtin University, Australia
Curtin University, Australia
-
- Super Administrator
- Posts: 4590
- Joined: Wed Mar 05, 2003 3:10 am
Re: Reliability and validity along with reality check option
No. If the model matches real data, then that is a good sign that the model is working correctly.If I do check/compare the model result manually with the real output, can it be sufficient for reliability check?
No. Units checks make sure that you are not doing silly things like using a number in days and calculating something based on months with it. A model check just ensures that it will run.If I check through "Model Check" and "Unit Check", is it sufficient to say my model is valid?
There is not. All you can do is say the model is dimensionally correct and show how closely it matches real data.I am little confused. I have gone through few good article on reliability and validity but every where mentioned that there are no certain way to do check those.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Re: Reliability and validity along with reality check option
Dear Admin
Thanks for your reply. In my first question, you have told me that it is a good sign that data matches with real data which means model is working correctly. At the same time, you have answered in my third question that model should dimensionally correct and show how closely it matches with real data. Does it mean that if my model is dimensionally correct and matches with real data is sufficient to tell my model is valid and data is reliable? In my case, data is varying up to plus minus 12%. Is it alright as standard variation?
Lastly, Can I have a very easy example of doing reliability check in Vensim?
Thanks for your help.
Thanks for your reply. In my first question, you have told me that it is a good sign that data matches with real data which means model is working correctly. At the same time, you have answered in my third question that model should dimensionally correct and show how closely it matches with real data. Does it mean that if my model is dimensionally correct and matches with real data is sufficient to tell my model is valid and data is reliable? In my case, data is varying up to plus minus 12%. Is it alright as standard variation?
Lastly, Can I have a very easy example of doing reliability check in Vensim?
Thanks for your help.
Administrator wrote:No. If the model matches real data, then that is a good sign that the model is working correctly.If I do check/compare the model result manually with the real output, can it be sufficient for reliability check?
No. Units checks make sure that you are not doing silly things like using a number in days and calculating something based on months with it. A model check just ensures that it will run.If I check through "Model Check" and "Unit Check", is it sufficient to say my model is valid?
There is not. All you can do is say the model is dimensionally correct and show how closely it matches real data.I am little confused. I have gone through few good article on reliability and validity but every where mentioned that there are no certain way to do check those.
M Shamsuddoha
Curtin University, Australia
Curtin University, Australia
Re: Reliability and validity along with reality check option
There's another class of test that's very important, which is whether your model behaves sensibly in test conditions, especially extreme conditions. See the chapter on Reality Checks in the User Guide.
Conformance of your model to data by itself is a fairly weak test - it's often easy to build a bad model that fits data (e.g., linear regression with lots of parameters).
If your model fits data, is dimensionally consistent, appears sensible to reviewers knowledgeable about the system, behaves plausibly in response to extreme inputs, and is dimensionally consistent, you're starting to have a good case for validity.
You might take a look at:
http://blog.metasd.com/2012/05/doing-qu ... -research/
and the links in it, especially,
http://blog.metasd.com/wp-content/uploa ... itique.pdf
Conformance of your model to data by itself is a fairly weak test - it's often easy to build a bad model that fits data (e.g., linear regression with lots of parameters).
If your model fits data, is dimensionally consistent, appears sensible to reviewers knowledgeable about the system, behaves plausibly in response to extreme inputs, and is dimensionally consistent, you're starting to have a good case for validity.
You might take a look at:
http://blog.metasd.com/2012/05/doing-qu ... -research/
and the links in it, especially,
http://blog.metasd.com/wp-content/uploa ... itique.pdf
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Reliability and validity along with reality check option
Hi!
I can't find this file in the below link. It seems that it has been removed.
http://www.blog.metasd.com/wp-content/u ... itique.pdf
Thanks
I can't find this file in the below link. It seems that it has been removed.
http://www.blog.metasd.com/wp-content/u ... itique.pdf
Thanks
M Shamsuddoha
Curtin University, Australia
Curtin University, Australia
Re: Reliability and validity along with reality check option
The link still works for me.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Sensitivity Analysis
Hi Dear
I am trying to do sensitivity analysis of my model but it is giving me error and did not go through as it giving below message:
ERROR: Unable to achieve desired accuracy in integration at time 90.8125.
ERROR: Unable to achieve desired accuracy in integration at time 98.875.
ERROR: Unable to achieve desired accuracy in integration at time 98.875.
ERROR: Unable to achieve desired accuracy in integration at time 98.875.
and also giving a message below then Vensim program freeze and hang
WARNING: Simulation starts before first time for exogenous variable of .....
Please help me to fix my problem.
Thanks
DOha
I am trying to do sensitivity analysis of my model but it is giving me error and did not go through as it giving below message:
ERROR: Unable to achieve desired accuracy in integration at time 90.8125.
ERROR: Unable to achieve desired accuracy in integration at time 98.875.
ERROR: Unable to achieve desired accuracy in integration at time 98.875.
ERROR: Unable to achieve desired accuracy in integration at time 98.875.
and also giving a message below then Vensim program freeze and hang
WARNING: Simulation starts before first time for exogenous variable of .....
Please help me to fix my problem.
Thanks
DOha
M Shamsuddoha
Curtin University, Australia
Curtin University, Australia
Re: Reliability and validity along with reality check option
The first errors are warnings from RK2 or RK4 auto integration. They may not be important. Do you get any of these warnings in an ordinary simulation? If not, you may have to try some parameter variations matching your sensitivity file in order to recreate the problem.
The "before first time..." warning surely occurs in an ordinary simulation, unless you are varying INITIAL TIME. Best to diagnose it there, rather than in a sensitivity run.
Can you post the model and sensitivity files that recreate the crash?
The "before first time..." warning surely occurs in an ordinary simulation, unless you are varying INITIAL TIME. Best to diagnose it there, rather than in a sensitivity run.
Can you post the model and sensitivity files that recreate the crash?
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Sensitivity Analysis
Thanks for your reply.
The initial time is 0. How can I post you crash screen shot? as it shows two option of "Close Vensin" and "Debug". If I click debug it is asking for Visio key.
Can you please explain step by step regarding "before first time..."? What steps can I take to solve it? By the way, normal simulation does not show this problem. There is another error message of Vensim Program error of "DCC43327". I have attached it in the below.
Thanks
The initial time is 0. How can I post you crash screen shot? as it shows two option of "Close Vensin" and "Debug". If I click debug it is asking for Visio key.
Can you please explain step by step regarding "before first time..."? What steps can I take to solve it? By the way, normal simulation does not show this problem. There is another error message of Vensim Program error of "DCC43327". I have attached it in the below.
Thanks
tomfid wrote:The first errors are warnings from RK2 or RK4 auto integration. They may not be important. Do you get any of these warnings in an ordinary simulation? If not, you may have to try some parameter variations matching your sensitivity file in order to recreate the problem.
The "before first time..." warning surely occurs in an ordinary simulation, unless you are varying INITIAL TIME. Best to diagnose it there, rather than in a sensitivity run.
Can you post the model and sensitivity files that recreate the crash?
- Attachments
-
- DCC error.png (16.41 KiB) Viewed 37948 times
M Shamsuddoha
Curtin University, Australia
Curtin University, Australia
Re: Reliability and validity along with reality check option
The "before first time" error just means that your model starts before the first data point for some data variable. Check to be sure that you have Tools>Options>Settings>Warnings set to "Display." I'll look into the other.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Reliability and validity along with reality check option
Hi Tom
Thanks for your reply. My waring setting is set as suppress. I have added the partial model and another error message as an attachment. If I want to do 'Monte Carlo', it just crash. Please help. If you have time, can you see the Monte carlo setting and analysis what I wanted to do.
Thanks
Thanks for your reply. My waring setting is set as suppress. I have added the partial model and another error message as an attachment. If I want to do 'Monte Carlo', it just crash. Please help. If you have time, can you see the Monte carlo setting and analysis what I wanted to do.
Thanks
tomfid wrote:The "before first time" error just means that your model starts before the first data point for some data variable. Check to be sure that you have Tools>Options>Settings>Warnings set to "Display." I'll look into the other.
- Attachments
-
- exp.mdl
- (42.57 KiB) Downloaded 339 times
-
- Error.png (46.29 KiB) Viewed 37939 times
M Shamsuddoha
Curtin University, Australia
Curtin University, Australia
Re: Reliability and validity along with reality check option
I think we can't debug the DCC43327 error without a copy of the model and sensitivity files - you can email them to us, vensim at vensim.com if you can't post them publicly.
Set your warnings to "Display" to debug the data first time and integration errors.
Set your warnings to "Display" to debug the data first time and integration errors.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Reliability and validity along with reality check option
Hi Dear
I have given short form of the model named exp.
Could you please check the attachment of my previous message?
Thanks
Doha
I have given short form of the model named exp.
Could you please check the attachment of my previous message?
Thanks
Doha
tomfid wrote:I think we can't debug the DCC43327 error without a copy of the model and sensitivity files - you can email them to us, vensim at vensim.com if you can't post them publicly.
Set your warnings to "Display" to debug the data first time and integration errors.
M Shamsuddoha
Curtin University, Australia
Curtin University, Australia
-
- Super Administrator
- Posts: 4590
- Joined: Wed Mar 05, 2003 3:10 am
Re: Reliability and validity along with reality check option
We will need to be able to recreate the runs, so we need the model and sensitivity files you are using.
Tony.
Tony.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Re: Reliability and validity along with reality check option
Also, we'll need any input data that the model requires to run - either a .vdf or .xls, per usage.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Reliability and validity along with reality check option
Hi!
Please see the attachment for excel file.
Please see the attachment for excel file.
tomfid wrote:Also, we'll need any input data that the model requires to run - either a .vdf or .xls, per usage.
M Shamsuddoha
Curtin University, Australia
Curtin University, Australia
Re: Reliability and validity along with reality check option
Hmm ... this runs fine for me. Can you post one more file - the savelist (.lst)? Thanks.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Reliability and validity along with reality check option
Please see the attachment.
tomfid wrote:Hmm ... this runs fine for me. Can you post one more file - the savelist (.lst)? Thanks.
- Attachments
-
- exp.lst
- (52 Bytes) Downloaded 326 times
M Shamsuddoha
Curtin University, Australia
Curtin University, Australia
Re: Reliability and validity along with reality check option
I still can't replicate any problems - this works fine for me. What Vensim version and operating system are you running?
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Reliability and validity along with reality check option
Hi Tom
My version is 6.0b. Can you confirm me one thing that is it must to do sensitivity for a research?
My version is 6.0b. Can you confirm me one thing that is it must to do sensitivity for a research?
tomfid wrote:I still can't replicate any problems - this works fine for me. What Vensim version and operating system are you running?
M Shamsuddoha
Curtin University, Australia
Curtin University, Australia
Re: Reliability and validity along with reality check option
I think it's important, but the quality of the model is of greater importance.
I can make sensitivity runs with your model without any problems on 6.0b on a Windows 7 machine, so I'm not sure what the problem is. Also, I don't get any warnings about integration errors, because the integration method is set to Euler. Possibly you changed it before posting the model?
I can make sensitivity runs with your model without any problems on 6.0b on a Windows 7 machine, so I'm not sure what the problem is. Also, I don't get any warnings about integration errors, because the integration method is set to Euler. Possibly you changed it before posting the model?
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Reliability and validity along with reality check option
Yes, you are right. Somehow it was set to R2auto. What is the difference between them? Can you send me back your trial on sensitivity over my file? thanks again.
tomfid wrote:I think it's important, but the quality of the model is of greater importance.
I can make sensitivity runs with your model without any problems on 6.0b on a Windows 7 machine, so I'm not sure what the problem is. Also, I don't get any warnings about integration errors, because the integration method is set to Euler. Possibly you changed it before posting the model?
M Shamsuddoha
Curtin University, Australia
Curtin University, Australia
-
- Super Administrator
- Posts: 4590
- Joined: Wed Mar 05, 2003 3:10 am
Re: Reliability and validity along with reality check option
There is a section in the help system on integration,
http://www.vensim.com/documentation/ind ... ration.htm
It's best if you run the experiments yourself. Try opening the model and clicking Model->Reform/Clean and then run the sensitivity tests.
http://www.vensim.com/documentation/ind ... ration.htm
It's best if you run the experiments yourself. Try opening the model and clicking Model->Reform/Clean and then run the sensitivity tests.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Re: Reliability and validity along with reality check option
Hi Msdoha
About your question on validity and reliability one must first define what is validity or reliability.
For me it means approximately the same thing. My definition will certainly not be the same as most other people definitionx, because I have never seen a published model that I consider as valid.
A model to be valid must mimic sufficiently well the system that you want to study, so as to give the possibility to solve the problem.
To make the model mimic sufficiently well the system I use massively Reality Checks that I build progressively step by step with the model, starting with a simplistic model first. I use of course unit checks using the strictest testing option available in model>setting>units equiv. Mass balances are performed in some of the Reality Checks.
The other way to valid the model is to use it sufficiently at each stage of the modeling process.
One should start with a simplistic model and add some more material to the model, only when one has fully understood the model previously built. What means ‘fully understood’. To test how deeply you have understood a model, you must be able to predict the general behavior of the model, whatever the value of the parameters used and without running the model. Of course when the model gets bigger, this is more and more difficult, but one should thrive towards this achievement and how well you perform is a good indicator of your understanding.
If you do this and in parallel build your Reality Check you will not only have a model that mimics well the reality, but you will understood how it works and be able to build sound policies.
A complex model that mimics well the reality butis not deeply understood may be totally useless.
So for me a model to be valid and reliable must be correct relatively to the reality studied but too be understood which is something different.
Coyle writes in his book the following: ‘The key to successful modeling is to keep one’s understanding of the model and what it says about the problem ahead of its size.’
This rule must be respected all along the modeling process and once you achieve it, you can go along and eventually add some more material to your model.
Of course all this has a price: the price of quality.
So when one talks about validity, one must consider the model and the person who uses it.
The model may be valid, but if the person using it does not understand it fully, the person and the model are not valid together.
Joined a paper by Peterson and Eberlein about Reality Checks. Unfortunately the last paragraphs do not make sense and are in contradiction with the independence of RC and the model advocated previously and are impossible to achieve.
I join too an example of a model I am building that has already about 50 RC and that will have certainly more than hundred once finished. I have erased all the views but the Reality Checks for confidential reasons. You can still run it and run the Reality checks.
Regards.
JJ
About your question on validity and reliability one must first define what is validity or reliability.
For me it means approximately the same thing. My definition will certainly not be the same as most other people definitionx, because I have never seen a published model that I consider as valid.
A model to be valid must mimic sufficiently well the system that you want to study, so as to give the possibility to solve the problem.
To make the model mimic sufficiently well the system I use massively Reality Checks that I build progressively step by step with the model, starting with a simplistic model first. I use of course unit checks using the strictest testing option available in model>setting>units equiv. Mass balances are performed in some of the Reality Checks.
The other way to valid the model is to use it sufficiently at each stage of the modeling process.
One should start with a simplistic model and add some more material to the model, only when one has fully understood the model previously built. What means ‘fully understood’. To test how deeply you have understood a model, you must be able to predict the general behavior of the model, whatever the value of the parameters used and without running the model. Of course when the model gets bigger, this is more and more difficult, but one should thrive towards this achievement and how well you perform is a good indicator of your understanding.
If you do this and in parallel build your Reality Check you will not only have a model that mimics well the reality, but you will understood how it works and be able to build sound policies.
A complex model that mimics well the reality butis not deeply understood may be totally useless.
So for me a model to be valid and reliable must be correct relatively to the reality studied but too be understood which is something different.
Coyle writes in his book the following: ‘The key to successful modeling is to keep one’s understanding of the model and what it says about the problem ahead of its size.’
This rule must be respected all along the modeling process and once you achieve it, you can go along and eventually add some more material to your model.
Of course all this has a price: the price of quality.
So when one talks about validity, one must consider the model and the person who uses it.
The model may be valid, but if the person using it does not understand it fully, the person and the model are not valid together.
Joined a paper by Peterson and Eberlein about Reality Checks. Unfortunately the last paragraphs do not make sense and are in contradiction with the independence of RC and the model advocated previously and are impossible to achieve.
I join too an example of a model I am building that has already about 50 RC and that will have certainly more than hundred once finished. I have erased all the views but the Reality Checks for confidential reasons. You can still run it and run the Reality checks.
Regards.
JJ
- Attachments
-
- rentndrop138_2_RC.vpm
- (285.18 KiB) Downloaded 281 times
-
- RC_justification.pdf
- (797.16 KiB) Downloaded 270 times
Re: Reliability and validity along with reality check option
Thanks JJ. I will get back to you once I will have some questions to ask you. By the way, do you have english version of the attached model?
M Shamsuddoha
Curtin University, Australia
Curtin University, Australia