about using or not using reality checks

[LAUJJL]

Hi

I do not remember having once seen any Vensim model in
the litterature, for instance those published at the SD annual conference, using explicitely reality checks.
Reality checks are not used in the Sterman's book because this feature is no implemented in the two other softwares.
I am asking two questions.
1. Do people use or not use reality checks, eventually using it but not mentionning it in their presentation, because it would be long to explain, although to my opinion it would help the understanding of how the system studied behaves and give credibility to the model.

2. If they do not use reality checks, is it simply because they feel it is not useful, or takes too much time, or are not used to do it, or prefer other ways to verify the quality of the model, or is it because the modellers have sufficient experience and using reality checks would not help them very much. In short the usefulness of reality checks is decreasing with the expertise of the modeler.

Regards.
JJ

[bob@vensim.com]

Hi JJ,

There are only a small number of people that I know of using Reality Check statements. Though it would be great to hear from others who are using them and whom I don'tknow about.

I think there are serval reasons for the low uptake:

1. They take time which almost always short
2. People are overconfident in their own ability to "see" the qaulity of models (checks are run mentally).
3. They are outside the languate of system dynamics, while at the same time not in a language easily accessible to people with domain knowlege.

I don't think it is true that more expert modelers don't need them, but it may be true that more expert modelers believe they don't need them which amounts to the same thing.

[LAUJJL]

Hi Bob

Thank you for your answer.
I ask that question in relation with the last trend on the SD mailing list, because I am convinced that the real reason of the slow increase of SD, is in fact that the end users that have tried SD have had more bad experiences than good ones. Companies are able to make difficult efforts as long as they expect a return on their investment. This is the case when implementing an E.R.P. like S.A.P. or implementing a quality system. The reason of the bad experiences is to my opinion not because of the method but because of the modelers. I think that everything possible must be done to
improve the quality of a model and its credibility and when I see so much models that do not use a built in feature that if applied improves the quality and the necessary credibility (every time I build reality checks equation, I make changes to my models and I get more confidents on their value) I have a serious doubts about the quality of the other necessary efforts that must have been done: data verification, expert opinions verification, quality of the qualitative analysis, knowledge of the field etc...
Regards.
JJ

[bob@vensim.com]

Hi JJ,

There is a broad mix of great, good, not so good and essentially misleading work done in practice. This a bit off topic but I find much of the discussion we have had on impact odd because good applications in SD are just good science, and good science has a strong foothold in "science" and is generally misapplied in both politics and business. So the real question in my mind is how to bring science into decision making and that requires stopping the turf wars, especially those about which methodology to use. So I worry that efforts to promote any one methodology (even SD) may just delay real progress - tricky question though.

I will say this abour Reality Checks - they are domain specific and methodology free. So I do agree with you that using them would be progress - but their container needs to be much more general than it now is. Simply put, reality checks are a precursor to science but at they are now implemented they are a subset of system dynamics - I guess that's my fault.

[LAUJJL]

Hi Bob

I agree completely with you about the necessary scientific mentality needed, that I would name
highly exacting and not suffering the least impasse. The economic and business world doest not
work that way. For this reason I do not think that there is a quick solution to the adoption of SD at least in the public or business world.
About Reality Checks, the fact that they are domain specific and methodology free is what interests me. It is closer to the problem and does not mind how it is solved.
About the container that is still to much influenced by SD, I am trying to put in a model as much
Reality Checks that I can imagine and there are a lot of them. I will try to see if the way it is formulated is influenced by SD methodology.
There should eventually be other formulations than the ‘if this happens then that should happen’.
I will try too to find some information about the expected behaviour that is not possible to implement in the actual sets of Reality Checks equations.
The ideal of course would be to have the possibility to thoroughly describe the expected behaviour
however it is initially formulated whether qualitative or quantitative and completely free from any solving methodology.
But is the fact that few are interested in using RC now due to an inherent mentality or is it due to the fact that it may have some limitations to describe real world behaviour?
The fact that few are interested will probably not influence your strategy of Vensim development
towards more RC functionality?
One more remark: why did not you give your opinion to the SD mailing list? I think that it is the
best I have read about the subject and on top of that very simply explained.
Regards.
JJ.