Reliability and validity along with reality check option

[mdsdoha]

Hi!

Can anybody give me an easy example of checking validity and reliability in Vensim?

If I do check/compare the model result manually with the real output, can it be sufficient for reliability check?

If I check through "Model Check" and "Unit Check", is it sufficient to say my model is valid?

I am little confused. I have gone through few good article on reliability and validity but every where mentioned that there are no certain way to do check those.

Waiting for some valuable responses.

[Administrator]

If I do check/compare the model result manually with the real output, can it be sufficient for reliability check?

No. If the model matches real data, then that is a good sign that the model is working correctly.

If I check through "Model Check" and "Unit Check", is it sufficient to say my model is valid?

No. Units checks make sure that you are not doing silly things like using a number in days and calculating something based on months with it. A model check just ensures that it will run.

I am little confused. I have gone through few good article on reliability and validity but every where mentioned that there are no certain way to do check those.

There is not. All you can do is say the model is dimensionally correct and show how closely it matches real data.

[mdsdoha]

Dear Admin

Thanks for your reply. In my first question, you have told me that it is a good sign that data matches with real data which means model is working correctly. At the same time, you have answered in my third question that model should dimensionally correct and show how closely it matches with real data. Does it mean that if my model is dimensionally correct and matches with real data is sufficient to tell my model is valid and data is reliable? In my case, data is varying up to plus minus 12%. Is it alright as standard variation?

Lastly, Can I have a very easy example of doing reliability check in Vensim?


Thanks for your help.

If I do check/compare the model result manually with the real output, can it be sufficient for reliability check?

No. If the model matches real data, then that is a good sign that the model is working correctly.

If I check through "Model Check" and "Unit Check", is it sufficient to say my model is valid?

No. Units checks make sure that you are not doing silly things like using a number in days and calculating something based on months with it. A model check just ensures that it will run.

I am little confused. I have gone through few good article on reliability and validity but every where mentioned that there are no certain way to do check those.

There is not. All you can do is say the model is dimensionally correct and show how closely it matches real data.

[tomfid]

There's another class of test that's very important, which is whether your model behaves sensibly in test conditions, especially extreme conditions. See the chapter on Reality Checks in the User Guide.

Conformance of your model to data by itself is a fairly weak test - it's often easy to build a bad model that fits data (e.g., linear regression with lots of parameters).

If your model fits data, is dimensionally consistent, appears sensible to reviewers knowledgeable about the system, behaves plausibly in response to extreme inputs, and is dimensionally consistent, you're starting to have a good case for validity.

You might take a look at:
http://blog.metasd.com/2012/05/doing-qu ... -research/
and the links in it, especially,
http://blog.metasd.com/wp-content/uploa ... itique.pdf

[mdsdoha]

Hi!

I can't find this file in the below link. It seems that it has been removed.
http://www.blog.metasd.com/wp-content/u ... itique.pdf

Thanks

[tomfid]

The link still works for me.

[mdsdoha]

Hi Dear

I am trying to do sensitivity analysis of my model but it is giving me error and did not go through as it giving below message:
ERROR: Unable to achieve desired accuracy in integration at time 90.8125.
ERROR: Unable to achieve desired accuracy in integration at time 98.875.
ERROR: Unable to achieve desired accuracy in integration at time 98.875.
ERROR: Unable to achieve desired accuracy in integration at time 98.875.

and also giving a message below then Vensim program freeze and hang

WARNING: Simulation starts before first time for exogenous variable of .....


Please help me to fix my problem.

Thanks

DOha

[tomfid]

The first errors are warnings from RK2 or RK4 auto integration. They may not be important. Do you get any of these warnings in an ordinary simulation? If not, you may have to try some parameter variations matching your sensitivity file in order to recreate the problem.

The "before first time..." warning surely occurs in an ordinary simulation, unless you are varying INITIAL TIME. Best to diagnose it there, rather than in a sensitivity run.

Can you post the model and sensitivity files that recreate the crash?

[mdsdoha]

Thanks for your reply.

The initial time is 0. How can I post you crash screen shot? as it shows two option of "Close Vensin" and "Debug". If I click debug it is asking for Visio key.

Can you please explain step by step regarding "before first time..."? What steps can I take to solve it? By the way, normal simulation does not show this problem. There is another error message of Vensim Program error of "DCC43327". I have attached it in the below.

Thanks

The first errors are warnings from RK2 or RK4 auto integration. They may not be important. Do you get any of these warnings in an ordinary simulation? If not, you may have to try some parameter variations matching your sensitivity file in order to recreate the problem.

The "before first time..." warning surely occurs in an ordinary simulation, unless you are varying INITIAL TIME. Best to diagnose it there, rather than in a sensitivity run.

Can you post the model and sensitivity files that recreate the crash?

[tomfid]

The "before first time" error just means that your model starts before the first data point for some data variable. Check to be sure that you have Tools>Options>Settings>Warnings set to "Display." I'll look into the other.

[mdsdoha]

Hi Tom

Thanks for your reply. My waring setting is set as suppress. I have added the partial model and another error message as an attachment. If I want to do 'Monte Carlo', it just crash. Please help. If you have time, can you see the Monte carlo setting and analysis what I wanted to do.

Thanks

The "before first time" error just means that your model starts before the first data point for some data variable. Check to be sure that you have Tools>Options>Settings>Warnings set to "Display." I'll look into the other.

[tomfid]

I think we can't debug the DCC43327 error without a copy of the model and sensitivity files - you can email them to us, vensim at vensim.com if you can't post them publicly.

Set your warnings to "Display" to debug the data first time and integration errors.

[mdsdoha]

Hi Dear

I have given short form of the model named exp.

Could you please check the attachment of my previous message?

Thanks

Doha

I think we can't debug the DCC43327 error without a copy of the model and sensitivity files - you can email them to us, vensim at vensim.com if you can't post them publicly.

Set your warnings to "Display" to debug the data first time and integration errors.

[Administrator]

We will need to be able to recreate the runs, so we need the model and sensitivity files you are using.

Tony.

[tomfid]

Also, we'll need any input data that the model requires to run - either a .vdf or .xls, per usage.

[mdsdoha]

Hi!

Please see the attachment for excel file.

Also, we'll need any input data that the model requires to run - either a .vdf or .xls, per usage.

[tomfid]

Hmm ... this runs fine for me. Can you post one more file - the savelist (.lst)? Thanks.

[mdsdoha]

Please see the attachment.

Hmm ... this runs fine for me. Can you post one more file - the savelist (.lst)? Thanks.

[tomfid]

I still can't replicate any problems - this works fine for me. What Vensim version and operating system are you running?

[mdsdoha]

Hi Tom

My version is 6.0b. Can you confirm me one thing that is it must to do sensitivity for a research?

I still can't replicate any problems - this works fine for me. What Vensim version and operating system are you running?

[tomfid]

I think it's important, but the quality of the model is of greater importance.

I can make sensitivity runs with your model without any problems on 6.0b on a Windows 7 machine, so I'm not sure what the problem is. Also, I don't get any warnings about integration errors, because the integration method is set to Euler. Possibly you changed it before posting the model?

[mdsdoha]

Yes, you are right. Somehow it was set to R2auto. What is the difference between them? Can you send me back your trial on sensitivity over my file? thanks again.

I think it's important, but the quality of the model is of greater importance.

I can make sensitivity runs with your model without any problems on 6.0b on a Windows 7 machine, so I'm not sure what the problem is. Also, I don't get any warnings about integration errors, because the integration method is set to Euler. Possibly you changed it before posting the model?

[Administrator]

There is a section in the help system on integration,
http://www.vensim.com/documentation/ind ... ration.htm

It's best if you run the experiments yourself. Try opening the model and clicking Model->Reform/Clean and then run the sensitivity tests.

[LAUJJL]

Hi Msdoha

About your question on validity and reliability one must first define what is validity or reliability.

For me it means approximately the same thing. My definition will certainly not be the same as most other people definitionx, because I have never seen a published model that I consider as valid.
A model to be valid must mimic sufficiently well the system that you want to study, so as to give the possibility to solve the problem.

To make the model mimic sufficiently well the system I use massively Reality Checks that I build progressively step by step with the model, starting with a simplistic model first. I use of course unit checks using the strictest testing option available in model>setting>units equiv. Mass balances are performed in some of the Reality Checks.

The other way to valid the model is to use it sufficiently at each stage of the modeling process.

One should start with a simplistic model and add some more material to the model, only when one has fully understood the model previously built. What means ‘fully understood’. To test how deeply you have understood a model, you must be able to predict the general behavior of the model, whatever the value of the parameters used and without running the model. Of course when the model gets bigger, this is more and more difficult, but one should thrive towards this achievement and how well you perform is a good indicator of your understanding.

If you do this and in parallel build your Reality Check you will not only have a model that mimics well the reality, but you will understood how it works and be able to build sound policies.
A complex model that mimics well the reality butis not deeply understood may be totally useless.

So for me a model to be valid and reliable must be correct relatively to the reality studied but too be understood which is something different.

Coyle writes in his book the following: ‘The key to successful modeling is to keep one’s understanding of the model and what it says about the problem ahead of its size.’

This rule must be respected all along the modeling process and once you achieve it, you can go along and eventually add some more material to your model.
Of course all this has a price: the price of quality.

So when one talks about validity, one must consider the model and the person who uses it.
The model may be valid, but if the person using it does not understand it fully, the person and the model are not valid together.

Joined a paper by Peterson and Eberlein about Reality Checks. Unfortunately the last paragraphs do not make sense and are in contradiction with the independence of RC and the model advocated previously and are impossible to achieve.

I join too an example of a model I am building that has already about 50 RC and that will have certainly more than hundred once finished. I have erased all the views but the Reality Checks for confidential reasons. You can still run it and run the Reality checks.

Regards.

JJ

[mdsdoha]

Thanks JJ. I will get back to you once I will have some questions to ask you. By the way, do you have english version of the attached model?