using reality checks to build models

[LAUJJL]

Hi

I am presently studying again the user's guide chapter on reality checks.
When I studied it years ago, I did not realize or I do not remember that it shows an interesting way to build a model, starting first by building the reality check equations, and progressively building a model that solves them.
Can one imagine to follow that way of building models systematically? That would offer the great advantage of having not to build reality check equations after the model is built, which it is very tempting not to do because of lack of time, plus many other advantages.
I remember that when I followed a distant course with Bob years ago, he spoke about using reality checks to build models being contradicted by Jim Hines who advocated another loops centered method (which I experimented and never worked).
That would add a very substantial advantage to the use of reality checks.
But I noticed that in the modelling guide, this method is not used at all, for the very simple reason that you have no reality checks in any of the models built.
Though at the conclusion of the chapter, the method of building method with reality checks is proposed as valid.
The question is: if there had been a causal description of the yeast problem and the model built from a cld qualitative diagram and then making it quantitative, would have been the resulting model, worse or better?
Is it possible to blend the two methods?
Working with a structural causality method and an estimated behaviour of reality at the same time?
I have started to test the method with a simple model.
An evident reality check is that at the beginning of a period a level is equal to the level at the beginning of the preceding period plus a rate.
I have represented this with the simplistic model joined.
Is there a simpler way to represent this?
Regards.
JJ



[Edited on 3-3-2010 by LAUJJL]

[Edited on 3-3-2010 by LAUJJL]

[Edited on 4-3-2010 by LAUJJL]

[mike]

re; JJ’s questions;

Is it possible to blend the two methods?
Working with a structural causality method and an estimated behavior of reality at the same time?


my 2 cents on JJ's post ;

reality check seems a more granular and segmented way to do sensitivity simulations

sensitivity simulations let you observe multi-variate interactions
based on some set of variables changing

but if for some reason these sensitivity simulation interaction state-spaces
lack some particular flavor of granularity or segmentation then i would think
you could take the sensitivity simulation state-spaces one step further
to whatever level of granularity and segmentation one desires using
‘reality check’ .

so i guess one response to JJ’s question above is that :
YES these two methods can be blended and in fact need to be used together
whenever you are exploring behavior that is more granular or segmented than those state spaces resulting from what is possible simply using sensitivity simulations alone.


If the Reality Checks and the sensitivity simulation state spaces are inconsistent
then there is some reconciliation to do or divergences to acknowledge.



Regards,

Mike

[LAUJJL]

Hi Mike

I have not noticed anything yet concerning improving models with a high level of granularity making sensibility analysis uneasy. But it may come later on.
I understand your point where the reality may behave in peculiar conditions in a rather peculiar way making it look like a patchwork. In these circumstances reality checks could well describe these peculiarities that may be difficult to track using conventionnal sensibility analysis.
I have taken a small model that I had built using a causal diagramming (I use Coyle's method which is rather different than Hine's one) and built again the model forgetting everytning about the first model and working the same way that is described in the chapter on reality checks of the Vensim user's guide. It is a very interesting experience that I should have experienced much sooner, but I was not really convinced of the utility of the reality checks, not seeing anybody using it. I will share my experience later on, it may take some time, when I will have enough practiced the method. The method makes think totally differently which is by itself a first important benefit.
Regards.
JJ

[Edited on 9-3-2010 by LAUJJL]

[bob@vensim.com]

Hi JJ,

Reality Checks are intended to embody knowledge that people with domain expertize have about behavioral relationships that exist between different model variables - If I don't maintain my cars then my customers will complain, If I don't clean my cars my customers will complain (even more). As such they work in parallel to any other technique used to construct a mode (be it loops centric, level centric or detail centric). There can certainly be interplay, just as you can mix up these different approaches to model building.

The reason the modeling guide does not use these is actually that most of those models were written before a useable language for embodying Reality Checks was part of Vensim. That is not to say that the models do not rely on Reality Checks, that kind of thinking is second nature to many people in SD and the attitude becomes - well it is already there anyway by bother for formalize it. We all know better than to believe we actually get things right - just look at units checking, but the human being is a stubborn creature.

As far as checks at the level you are writing them, these are really after the physics of the system. That is something we can reasonably confidently think we have right. It is, of course, one of the strengths of System Dynamics.

[LAUJJL]

Hi Bob
Concerning the checks of the rate and the level I agree that it is a bit exaggerated. But by experience I have already made mistakes that should not have been normally made.
About reality checks, what I like (I may still change my mind with experience) is that it solves the paradox of SD which is putting the cart before the horses. Working with causal diagrams one is obliged to tell the computer the solution before even knowing it. I do not like being obliged to have any preconceived ideas about the problem I want to solve and in fact the only thing I am sure is the behavior of reality, and I think that logically one should start with this.
About thinking reality checks without using formally, I did not do it at all, and I am using SD now for 8 years. From now two days of practice, I recognize that it forces me to think differently, having only to concentrate on how reality behaves, which is much more reassuring than being obliged to imagine how it works or should work which I honestly do not know precisely (fancy thinking) and which is stressing especially for me as I am at the same time modeling and using the model.
But experience will tell me how to use it precisely.
About the use of reality checks by the user guide, I understand the reasons of them not being used, but it does not explain why one never sees any ‘new’ models with reality check equations.
My idea is well explained at the beginning of the reality checks chapter in the Vensim user's guide:
'It can also focus discussion away from specific assumptions made in models onto more solidly held beliefs about the nature of reality.'
Regards.
JJ

[tomfid]

I think the idea of using RCs in a more automated fashion for model specification is interesting. In a way, the usual SD approach to specifying lookups is a bit like RCs. Good practitioners ask a series of questions: what happens at 0? what happens at high values? ... Those get at SME knowledge about the relationship. Perhaps one could specify the known attributes, then let Vensim pick a conforming functional relationship. That idea could probably be generalized to other areas, e.g. stock-flow relationships, but as soon as you get more macroscopic, trouble is likely to follow.

On a marginally related note, I've found Reality Checks to be very useful for user interface verification. A set of RCs can quickly and repeatedly exercise all of the parameters that will be connected to sliders in an interface - a task that I find boring but essential.

[LAUJJL]

Hi Tom
Fundamentally I try to increase the use of reality checks, because it increases the confidence I can have in my models and their conformance to my needs. And it is the main thing I am looking for. I think that it has always been the main problem of the field and everything that helps alleviate it should be done.
I think that it is difficult for a modeler to build a model that belongs to a field he is not a specialist in a way that is adapted to somebody who has not built the model and who will have to use it. By using I mean people who have the power to take the policy decisions the model is dealing with. Generally the models I have seen have been made by people who do not ‘use’ them after they have built them.
This makes a big difference. One of the big advantages of spreadsheets when they started at the end of the seventies is that it allowed at last, end users to build their own models, trying to get nearer their own needs.
SD is unfortunately much more difficult to master technically.
I think too that SD people should make a move towards a more general service consisting of delivering a solution to problems instead of delivering a model and forget a bit the mathematical side of the question and concentrate on the problem itself. This would need of course a much closer participation to the client’s problem and a much wider knowledge of it.
It would add a considerable added value to the service and permit SD’ers to ‘sit at the table’ as was somebody writing in an ancient thread of the SD forum.
Regards.
JJ