Dialog between Wakeland & Sterman after comments re Nordhaus

[wakeland@ccgate.leupstv.com]

TO: John StermanFROM: Wayne Wakeland DATE: 2/29/96

Rather than respond publicly to your comments since it will
probably amount to more unsubstantiated generalizations, I
thought I would respond privately.

In my experience, which is admittedly mostly with amateur S-D
modelers in the business environment and in graduate classes
(where I am an adjunct teacher), I have seen myself and many
other people create structure based on mental models, without
proper reference to data.To some degree, this may be
justifiable due to time limitations, and it is perhaps
acceptable as long as all such unsubstantiated assumptions are
clearly documented, and the models will not be used to make
important decisions.

Since I have not worked with professional Systems Dynamics
Modelers, I do not know if they ever do such things.When I
made my posting, I was generalizing, as you pointed out, by
using the words "S-D modelers" rather than "amateur S-D
modelers".[Perhaps I will respond publicly with that
clarification.]

Never-the-less, it is my opinion that many of us (possibly even
some of the professionals) need to be reminded from time to time
to check the data or get the facts.Many of us have a tendency
to put more faith in our mental models than is deserved.
Virtually every time I "go and get the facts" I discover that my
memory and/or pre-conceived notions were at least partially in
error.

Thus, I feel the admonition may still be useful...even if it is
an unsubstantiated geneneralization.

What do you think?

--Wayne Wakeland (wakeland@leupstv.com)
----------------------------------------------------------------

TO: Wayne WakelandFROM: John Sterman DATE: 3/4/96

Thanks for your note.I think I agree with nearly everything
you said in your message, especially the need to get the facts,
and the tendency to pass off mental models as if they were
facts.These are indeed serious problems, not only for SD
modelers (amateurs or otherwise) but also all other modelers,
and indeed, all people.The all too human tendencies to confuse
our attributions with data, over rely on memory, revise our
beliefs in hindsight, etc. are grave impediments to learning,
and so any efforts to inject rigor into the modeling process are
welcome (this is something Ive written extensively about).

I think what concerned me was the impression I got that SD was
being singled out for being particularly bad in this regard.
Amateur modelers aside (I dont think the economists or
physicists would want to be judged by the efforts of the
amateurs in their fields either), the literature suggests that
SD is really about the same with respect to rigor and data as
other fields in social science, and there are many exemplary
studies that are solidly grounded in data, fully replicable,
carefully tested, and which meet the highest standards.There
are also studies where the empirical side is weak.One
difference between SD and other disciplines perhaps is the
explicit recognition in SD that data is more than numerical
data, and that interviews, obvservation, prior theory, and other
types of data are important sources of information to specify,
constrain, and test models.Though you cant run regressions on
interview data, such qualitative information sources are
critical in most real-life studies if we are to move beyond
unsubstantiated mental models, or worse, leaving out altogether
important effects (just as much a subjective judgment as a
guess).

Another possible difference is that often SD modeling is used in
an exploratory mode designed to discover what the consequences
of mental models might be, or to uncover the more subtle aspects
of peoples mental models.In this mode, the intent is not
detailed prediction of policy responses, but improvement of the
groups mental models.For this purpose, time is often limited
and the modeler works with the data at hand - i.e. with the data
as filtered through the participants mental models.These
exercises often lead to greater insight, but they are only a
first step, in my view, to further work.

I hope my note did not offend - I do feel strongly that good SD
sets a very high standard for empirical testing, but you are
quite right that not all work is good work.We all, even
professionals, need to be reminded of the importance of careful
testing.Thanks.

I would have no objection if you wanted to have this exchange
posted to the list - up to you.

John Sterman

Sloan School of Management
MIT, E53-351
30 Wadsworth Street
Cambridge, MA 02142
phone: 617/253-1951fax: 617/258-7579e-mail:
jsterman@mit.edu

[gallaher@teleport.com (Ed Gallah]

A brief comment about the dialog -process-:

>TO: John StermanFROM: Wayne Wakeland DATE: 2/29/96
>
>Rather than respond publicly to your comments since it will
>probably amount to more unsubstantiated generalizations, I
>thought I would respond privately.

This is probably a good example where an apparent disagreement between
individuals is better handled outside the list itself. i.e., every comment
does not need to be public, and probably will be resolved better via
private discourse.

*After* it is resolved, if agreeable to both (all) parties, the issue can
be posted.

I think this situation was handled just right, and provides useful insight.

Ed Gallaher
gallaher@mail.teleport.com

[CrbnBlu@aol.com]

irt: wakeland@ccgate.leupstv.com, Tue, Apr 16, 1996 4:37 PM EST

I found this set of exchanges most intersting and here are some thoughts that
came up during the reading. Im not sure what constitutes a "Professional S-D
Modeler," yet being someone who makes their livelihood from this activity I
would contend that I am very much an "Amature." Arent we all to an extent?

Regarding data and assumptions, there seems to be an interaction between me,
my models, and the real situation which work together to get to a point that
makes sense. I would agree that the initial models contain an endless set of
assumptions, yet painful experience has shown that every one of the
assumptions must be verified and validated, and most often corrected, if the
resultant model is produce any meaningful results when the simulation is run.

And yes, there are also those instances where it is simply beneficial to
learn about the implications of the assumptions, whether they are correct or
not. I often find that a simulation will tell me that the assumptions cant
possibly be correct because the simulation produces a result which is
absolutely absurd. Of course there are times when this is simply due to a
flaw in the strucutre of the model.

thanks for sharing the thoughts,
Gene Bellinger
CrbnBlu@aol.com