Ventana software support forum

(was Using Statistics in Dynamics Models, Do we need to Simulate to
Validate Models?)

The discussion of statistics and simulation has been very interesting
but I fear perhaps somewhat abstract and theoretical in nature. A number
of the points that Jim Hines has raised are fundamentally more practical
and boil down deciding how to spend the limited time we have to look at
a problem. We clearly will never have the time to do everything we want
to do, let alone everything we could do, to attack a problem. So how do
we decide what to do?

As I see it there are three types of things we need to do when going
after a problem (these may iterate):

1. Model the problem (formally or informally) and make sure the model
makes sense.
2. Analyze the model to understand why it does what it does and extend
that understanding to the real problem.
3. Present that understanding (or the answers) to others in a meaningful
and convincing way.

The discussion so far has been heavily focussed on the first of these -
making sure the model makes sense. Clearly if we only do number 1
nothing will come of our work. So the question becomes when do we move
on to #2 and how much effort do we expend there. If we look at some of
the most influential models, both in our field and in others, people
have spent a great deal of time working on #2, and used the resulting
understanding as a basis for #3. I know, from working with Jim, that
this is exactly what he likes to do. I also know that I tend to spend
more time on #1 and have to speed through #2 and #3.

In part, I do things the way I do because I am never satisfied that my
models make sense. One reason for this is that the data are always
pointing out shortcomings. Another is that consulting engagements tend
to require overpromising of the scope of the problem to be addressed.
Still, I do not know if my allocation of time is really the best way to
attack the problems I work with a given amount of effort. I suspect that
Jim would tell me that if I spent more time on #2 I would do a better
job even for overscoped problems with messy data.

>From working with more mainstream consultants I know that many have a
tendency to focus almost entirely on #3 without really ever making sense
of anything. That always scares me. But the question remains - what are
the practical guidelines for working a problem. Is it just a personal
thing, or is there an emphasis that would make all of us more effective
at what we do?

Bob Eberlein
bob@vensim.com

Bob Eberlein describes three types of work to do in system dynamics
consulting:
1. Model the problem (formally or informally) and make sure the
model makes sense.
2. Analyze the model to understand why it does what it does and
extend that understanding to the real problem.
3. Present that understanding (or the answers) to others in a
meaningful and convincing way.

Bob concludes with two loaded questions:
A. What are the practical guidelines for working a problem?
B. Is there an emphasis that would make all of us more effective at
what we do?

Answers are likely to be found in the accumulated experience of system
dynamics. If we reflect on what approaches have worked in the past and
develop a hypothesis, we would be able to conduct experiments to confirm or
disconfirm it.

Heres a conjecture: We do not invest enough time at the beginning of our
work to understand the client.

Heres my reasoning.

For the most part, we work on really tough, messy problems that have no
optimal solution. Often the clients best thinking went into creating the
mess in the first place. So we are doubly challenged. We are asked to find
a way to improve actual conditions when part of the cause of problematic
behavior is paying our fee. We then deploy an inscrutable methodology and
often work in isolation from our clients.

Our field has used the term mental model to describe what I take to be a
construct of ones thoughts that is used to make sense of ones experiences.


If that definition is acceptable, then it would follow that a goodly hunk of
our work is aimed at amending a clients mental model. How much time does
our methodology suggest we allocate to understanding the workings of our
clients mental models?

Even the group model building literature that Ive read gives this
consulting step short shrift. The facilitators tend to dive into the system
dynamics problem-solving pool before becoming sufficiently acquainted with
the client. How does the clients mind process the system dynamics?

Sterman and Sweeneys work indicates that clients are ill-prepared to work
with the sorts of simulation models that we create. (i) Do we try to
educate them? Alternatively, (ii) do we learn for ourselves how to put
things in terms that our clients can process, how clients can learn from the
experiences we create? The answers so far appear to be (i) Yes and (ii) No.
If thats right, it is a losing proposition.

We need to make a choice at the very start of the engagement: allocate time
to understanding how clients make sense of the world. If we improve our
understanding of how our clients make sense of things at the outset, we can
make better allocations to Bobs Four Type work flow.

Its Four Type because Bobs second point is better viewed as two points:
"2. (a) Analyze the model to understand why it does what it does and (b)
extend that understanding to the real problem." The word "and" connotes the
right sequence: the *client* needs to understand why the world does what it
does. Then the client will extend what the client understands to the real
problem.

We often ask the client to learn from or with a simulation model. Maybe the
client helped develop it, maybe not. But the operative word here is learn,
and it is the client who must learn for a model to have any effect.

Ill finish off with a proposal to add Type 0 to Bobs work types:
0. Understand how the client makes sense of his/her experiences.

Jim Thompson
Director, Economic & Operations Research
Cigna HealthCare
900 Cottage Grove Rd. A142
Hartford, CT 06152
jim.thompson@cigna.com
Tel. 860.226.8607
Fax 860.226.7898