Reliable models

["Adrian Boucher"]

Hi Ed

Thanks for your stimulating reply and for more fully providing the
Washington et al reference. Unfortunately I dont have an e-mail
address for the authors, but only a snail-mail and tel no.

Dr Neena Washington
Department of Surgery
University of Nottingham Medical School
Queens Medical Centre
Nottingham
UK

Tel: (+44) 115 274 4158

Ill drop her a line and pass on your address as requested.

On the points you raise in your mail: I agree entirely with the
trailblazer, pioneer, settler analogy. Although I am a relative
newcomer to SD, I am convinced that it is a potentially extremely
powerful thinking tool which ought to be part of the core curriculum
for students of ALL ages. If only I had discovered it 20 years ago,
when my mind was more willing to accept new ideas and build on them.

I am currently trying to evangelise our part of the universe, but
its like pushing water uphill. In schools we have a so-called
National Curriculum which is deemed to contain all that is needed to
become suitably educated for the modern complex world.
Unfortunately, it doesnt address transferable analytical skills such
as those developed through systems thinking and system dynamics. The
curriculum is said to be overcrowded so that there is no room for
additional techniques such as SD. This, it seems to me, entirely
misses the point. Many of the processes in biology, physics,
chemistry, economics, ecology, (I could go on, but I guess you get
the point), may be modelled simply by generic SD structures, and the
similarity of these could be used as a powerful learning environment
which emphasises the TRANSFERABILITY of results across subject
boundaries.

{OK, I know there is a vigorous debate among
practitioners on whether one should undertake analysis through
archetypes, etc, but it seems to me that for students to gain a
potentially deeper understanding of the dynamics of processes might
benefit their understanding of many of lifes complexities,
particularly in economics, business and political issues.}

Enough rambling. I hope that the fact that my sig indicates that I
am located some distance from the biomedical end of things helps
prove precisely the point I am trying to make. Many of the dynamic
processes in economics and finance have features in common
with those in biological/medical systems. Ergo, we need to get the
teaching of these skills higher up the political and educational
agendas.

Best regards


Adrian


Adrian Boucher
Director
NatWest Financial Literacy Centre
Centre for Education & Industry
University of Warwick
Coventry
CV4 7AL
UK

Tel: +44 1203 524 234
Fax: +44 1203 523 617

e-mail: adrian.boucher@csv.warwick.ac.uk

URL: http://www.warwick.ac.uk/WWW/faculties/ ... index.html

"The e-mail of the species is deadlier than the mail."

[KATHLEEN TRUMAN]

In response to Paul Atkins:

Children dont have to LEARN systems thinking -- they already know it.
A baby is born into a complex social and natural environment and they
have to use all of their senses to learn to operate in this complex
system. But with the development of language skills and especially
through the educational system and learning our writtern language, they
are forced to communicate complex things with only linear tools. This is
what begins to change the way they think --- by forcing this change, you
encourage/force the children to think the world is linear.

This is what is so beautiful about Forresters k-12 program!! And why I
think it will be wonderfully successful if you can get it past the linear
thinking administrators!! You cre finally giving children back the tools
they need to communicate the complexity of the world -- hopefully, before
their mind is forced through the funnel of education into a linear format.

If you look at many traditional cultures who do not have a written
language, you find that they express many of their ideas about how the
world is organized as multi-dimensional circular systems. I think the
Australian indegenous peoples with their "dream world" may be a good
example of this although I have not done a detailed analysis. Also the
Hopi and the Navajo tend to depict their worlds in this way.

Certainly, in my own experience, discovering Senges archetypes and using
Causal Loop Diagrams was an absolutely liberating experience --- finally,
I didnt have to force my view of how the world is organized in to a
linear format -- I mean, how can you explain complex feedback processes
using only words and without a diagram?? With a CLD and a few words, the
complexity is immediately understandable.



Kathleen Truman
trumank@nevada.edu
Environmental Studies Program
University of Nevada Las Vegas
4505 Maryland Parkway
Las Vegas, NV 89154-4030

Tel: 702 895-4457
Fax: 702 895-4436

["Paul Atkins"]

Hi All,

I have been lurking for a week now, having just joined the group. I am
new to system dynamics so I was planning to just hang around and get the feel
of it but something Adrian said prompted me to write.

On 1 May 96 at 14:27, Adrian Boucher wrote:

>[stuff deleted]. Many of the processes in biology, physics,
> chemistry, economics, ecology, (I could go on, but I guess you get
> the point), may be modelled simply by generic SD structures, and the
> similarity of these could be used as a powerful learning environment
> which emphasises the TRANSFERABILITY of results across subject
> boundaries.
>
> {OK, I know there is a vigorous debate among
> practitioners on whether one should undertake analysis through
> archetypes, etc, but it seems to me that for students to gain a
> potentially deeper understanding of the dynamics of processes might
> benefit their understanding of many of lifes complexities,
> particularly in economics, business and political issues.}

I am interested in how people learn to understand dynamic
systems. In particular, if someone (incl. children) plays around with,
designs and builds computer simulations of dynamic systems, do they acquire
generalisable skills that would allow them to more immediately understand
processes occurring in other dynamic systems? If so, are these skills explicit
"rules" (for example "the whole is sometimes greater than the sum of the
parts") or are they something like an implicit understanding that is difficult
to verbalise? Adrians comment about "archetypes" seems to have something to
do with this but I might have misinterpreted it (Adrian - could you clarify?
Thanks).

So far, I have not found any literature addressing these issues
and I would be very interested in your comments or suggestions for references.
It seems likely to me that learning system dynamics will teach a child
something that will help them to better appreciate complexity in the world, but
what is the nature of that something?

Regards,
Paul.
--------------------------------------------------------------------------------
Paul Atkins Email: p.atkins@unsw.edu.au
School of Psychology Phone: +61 2 385 1461
University of New South Wales Fax: +61 2 385 3641
Sydney NSW 2052 Australia

["Paul Atkins"]

Hi,

Thank you to Kathleen and Gene for replying to my query regarding what exactly
it is that people learn as they become experts at SD. In my last post I may have
focussed too much on children learning about SD. What about adults? The
recent discussion on this group has discussed the spread of knowledge. What
does it mean to be an expert in this area? I would expect that there would be
some "knowledge" that can be easily taught and easily learned and there would
be other types of knowledge about SD that is more implicit and that can only be
gained by doing lots of modelling.

With other learning tasks (like driving a car) we typically go through a number
of stages from explicitly thinking about what we should be doing in response to
certain environmental condtions; to automatically responding to the environment
without conscious thought, even to the point where we can talk while driving.
Where initially we consciously processed all the relevant stimuli available to
us, after a while we learn to unconsciously process only the essential elements
that guide us to the correct response.

As practitioners in the field, do you feel that you have gone through stages
like this? Do you have any examples of concepts that you struggled with
originally and now apply automatically without thinking when faced with a
certain set of conditions or stimuli? Alternatively do you know of any
literature looking at skill acquisition in SD? My apologies if I am going over
old ground but I havent seen much on this in the reading I have done so far.

Thanks,
Paul
--------------------------------------------------------------------------------
Paul Atkins Email: p.atkins@unsw.edu.au
School of Psychology Phone: +61 2 385 1461
University of New South Wales Fax: +61 2 385 3641
Sydney NSW 2052 Australia

["Adrian Boucher"]

Hi Paul

Thanks for your comments. I apologise for not being clearer.

Lets start with the notion of helping really young learners to
appreciate what SD can offer. The Creative Learning Exchange
publishes a really excellent periodical which summarises classroom
use of SD and provides some really neat adventures into simple
modelling. Last year, for example, there was a super piece on
getting students to "discover" Newtons Law of Cooling, using a cup
of coffee as the motivation. The CLE has a great resource pack
entitled RoadMaps which takes the beginner (of any age) through the
first faltering steps of modelling toward some really profound ideas:
conservation of non-renewable natural resources such as fish (Fish
Banks Game). One example which really appealed to me was the problem
of finding the solution(s) to the interception times of two bodies
possessing different velocities. The context was a passenger running
to catch the train as it left the station. Conventional analysis
solves a quadratice equation. The SD solution was much more
insightful. More delights are included in the Road Maps series.
Contact: Lees Stuntz, Executive Director: stuntznl@tiac.net


My good friend David Riley at the University of North London, Dept
of Geography, 166-220 Holloway Road, London N7 8DB
(Tel: (+44) 171 607-2789)
has done some great work in modelling hydrological
processes with secondary school students in the UK (using a HyperCard
shell and STELLA. He also developed a STELLA/HC model of
the Gaia Hypothesis (Loveluck) as part of the Apple UK "Renaissance" Project a few years back. I dont have an
e-mail address for David (Sorry), but I can certainly call him to get
in touch with you.

The K-12 discussion list organised by the System Dynamics in
Education Project at MIT is a fantastic source of examples, good
ideas and good educational practice in using the SD and systems
thinking tools to foster understanding. Contact: Nan Lux:
nlux@mit.edu

There is (was?) an interesting attempt to get SD approaches adopted
in US schools, by the US Educational TEasting Service. The reports
are in:

Mandinach, E and H F Cline (1994): "Classroom Dynamics", Lawrenced
Erlbaum Associates, Hillsdale, NJ: ISBN: 0-8058-0555-9.

Now, what is the "something" that students learn? In my view, it is
that a number of dynamic processes have common features (eg
exponential growth/ decay) which occur in a range of contexts both
within an academic discipline, and (possibly more importantly) across
subject boundaries. What SD and ST are able to offer is a way of
looking at these in general (or do I mean generic?) ways and
comparing the strucutre of the processes rather than the specific
mathematics and/or problem parameters. Senge (The Vth Discipline,
1990) provided a simple taxonomy of common processes and archtypal
structures. {OK hes not the only one to do this, but for the sake
of brevity.}
Others have developed STELLA (and PowerSim) implementations:
For example Gene Bellinger : CrnBlu@aol.com

The debate in the SD literature is whether it is a good idea or not
to try and model real life complexity by relying on archetypes (as
laid out in the STELLA/ithink manuals), or whether it is better
practice to model from first principles. David Lane wrote a good
paper on this for the International SD Conference at Stirling in
1994. Sorry, cant lay my hand on the precise ref. The Proceedings
of that Conference also contain a paper by Wolstenholme and Corben on
the same point, but with a differing perspective.

Last, but not least, George Richardsons Excellent Survey of Systems
development (Richardson, G P (1991): "Feedback Thought in Social
Science and Systems Theory", University of Pennsylvania Press.
ISBN: 0-8122-3053-1 provides a tremendous insight into the
developments of the methods and their applicability in a wide range
of contexts. Hope this helps. Apologies for th elength.

Best wishes

Adrian

Adrian Boucher
Director
NatWest Financial Literacy Centre
Centre for Education & Industry
University of Warwick
Coventry
CV4 7AL
UK

Tel: +44 1203 524 234
Fax: +44 1203 523 617
e-mail: adrian.boucher@csv.warwick.ac.uk
URL: http://www.warwick.ac.uk/WWW/faculties/ ... index.html

[jimhines@interserv.com]

> ... This does not mean that one should only build models
> when there is good data, but rather that one should recognize
> that models built w/o proper data cannot be assumed to be as
> solid and reliable as those that are.

Insights seem to be pretty insensitive to having a model fit numerical data.
For example, one kind of insight is that a particular loop-hypothesis CANNOT
generate the reference mode. If one originally thought that the hypothesis
COULD generate the problem (and was prepared to act on that belief), learning
that it cant may be rather valuable.

Even more broadly, if you are exploring a hypothesis, you wouldnt expect a
close fit to numerical data. The world has a lot going on it; roughly speaking,
such a model has only have one thing going on it. Such a model can lead to deep
understanding of the hypothesis.

Fitting such a model to data would be a waste of time and, would in fact, be
misleading -- the fit would be achieved by warping parameters to take account of
structure that is not in the hypothesis. The ability to explore the hypothesis
would be diminished, not enhanced, by fitting the model to data.

Numerical prediction by the model might or might not be improved by warping the
parameters, insight will not be.


Jim Hines
jimhines@interserv.com

[kbs-fr@world-net.sct.fr (Michel]

> ... This does not mean that one should only build models
> when there is good data, but rather that one should recognize
> that models built w/o proper data cannot be assumed to be as
> solid and reliable as those that are.

Dear Sir,

I do not totally agree with your opinion concerning the need for data. What
counts is "behavior patterns". Data only give the past, and if one has a
forecastable change in behavior pattern, all past data become useless and
even misleading whereas analysing behavior can keep the model in the right path.
Models of psychosociological behavior (we have done quite a few of them),
based only on behavior pattern, are not less good -or bad - than models in
economy (we have also done quite a few of them) based of a lot of data which
often prove to be not very reliable (were you never confronted with the
phenomenon of "modified statistics ?).

Sincerely

Michel Karsky

************************************************************
Michel KARSKY
KNOWLEDGE BASED SIMULATION
287, rue Saint-Jacques 75005 Paris - FRANCE
Tel: (33 1) 43 54 47 96 Fax: (33) 1 44 07 00 59
E-mail:kbs-fr@world-net.sct.fr

*************************************************************

[gallaher@teleport.com (Ed Gallah]

>> ... This does not mean that one should only build models
>> when there is good data, but rather that one should recognize
>> that models built w/o proper data cannot be assumed to be as
>> solid and reliable as those that are.
>
>Michel Karsky adds:

>I do not totally agree with your opinion concerning the need for data. What
>counts is "behavior patterns". Data only give the past, and if one has a
>forecastable change in behavior pattern, all past data become useless and
>even misleading whereas analysing behavior can keep the model in the right
>path.



This ongoing, interesting, and valuable discussion appears to be addressing
the question of whether models with or without data are more solid and
reliable. In other words, it looks like a discussion of the merits of
modeling techniques per se.

I believe there is an aspect of this discussion that is being overlooked.
In part, the real issue is our ability to *know*, and modeling should be
looked at as just one tool in this endeavor. If something is "knowable",
we can build a disciplined, accurate, well-tested model, and compare it
with data. Or we can build a less-disiplined, squishy model, and devote
less time comparing it with data. The latter path might be a result of
time constraints, naivete, or laziness, but in any case, it seems clear
that the former approach is better in every respect. (see description of
heating system, below).

For example, our task is to model a reasonably stable system, which exists
in "nature", but is not changing while we are in the midst of the modeling
process. A simple example would be a model of a home heating system. We
may not understand this system as thoroughly as we like, but we can be
reasonably confident that the system is not changing while we are working
through the model building/testing process. Unless perhaps the furnace
filter is becoming rapidly clogged from one month to the next.

A more complex example might be a model of an airplane wing. This is
considerably more complex, but it can be tested over and over and compared
against the model behavior.

In such a case it makes sense to state that models built with proper data
will be more "solid and reliable" than models built w/o such data.

In the second situation, our task is to model a socio-economic "system",
which is much more poorly understood. Since the "real" system, in
"nature", may be very complex we really dont know what components need to
be added for detail, or aggregated for simplicity. We also dont have a
good feel for specific parameter values. In addition, we find that the
structure and/or behavior of the system is indeed changing, *during the
modeling process* (e.g. interest rates, consumer confidence, political
climate). It is very difficult to collect "data", because neither we nor
anyone else knows for sure what the "correct" data might be.

By comparison, the resulting model will be less firm and reliable than a
model of a home heating system. Some might criticize *modeling* for this
reason, when in fact the "culprit" is the difficult subject matter itself,

The better comparison should be: To what extent does modeling help us
understand this complex, changing system? And does it bring anything to
the table in undestanding this topic, that is better than *other available
methods* in the same arena? (This concept was of course presented by
Forrester in Principles of Systems).

My own bias, which is why I am so interested in SD modeling, is that yes,
it almost invariably provides us with additional understanding. Some
models will be more "solid" than others, because the -topic- is more solid.
Some models will be "weaker", not because of incompetence of the model
builder, or because modeling is not the "right" tool, but because the
-topic- is by nature difficult and poorly understood.

Taken in this context, both authors quoted above are correct. Modeling
will be stronger in some cases than in others, but not because of an
inherent problem in modeling.

Sorry this is so long-winded, but I guess thats what a discussion group is for.

ed gallaher
gallaher@teleport.com

[jsterman@MIT.EDU (John Sterman)]

I must respectfully disagree with my esteemed colleague and friend, Jim
Hines, regarding the role of data in modeling.

Jim says "Numerical prediction by the model might or might not be
improved by warping the parameters [to get a fit to the data], insight
will not be." This is just dead wrong. Many times it is through (and
often only through) the rigorous comparison of model to data that one
discovers that ones conception of the dynamics is fundamentally flawed
and must be revised -- that is, comparison of model to data is an
essential (but not the only) route to new insight.

I think we can agree that the proper use and role of data in a model
will depend on the purpose of the modeling effort, the time available,
and the likely benefits of additional effort spent on data collection
and model testing weighed against their costs. There certainly are
cases where the issues are unimportant enough, the time too short, the
costs of data collection too great, or the mental models of the clients
so undeveloped or erroneous that a model built without recourse to or
use of available or potentially available data can yield benefits. And
sometimes a speculative model can identify those areas of high
sensitivity where additional data collection effort is worthwhile (this
happens quite often in practice).

However, in most real cases, the issues are too important, and the
dynamics too subtle to deliberately ignore one of the principal means
available to test our models (mental and formal). I say one of the
means, for surely it is just as much an error to rely only on numerical
data and model fit as a test of model adequacy as it is to ignore data
and historical replication altogether. The literature of system
dynamics model testing stresses a wide range of tests required to build
confidence in ones model by confronting it with multiple challenges and
opportunities for error to be revealed.

Testing the ability of a model to replicate historical data (to the
accuracy appropriate for the purpose) is critical to developing,
testing, and refining the insights Jim speaks of. Failing to use data
in this manner breaks one of the principal feedback loops by which we
can improve our understanding of the dynamics. Comparing model output
to data is not an exercise in academic nit-picking, nor is it something
done to impress ones colleagues or clients. It is a way to discover
flaws in your dynamic hypothesis, in your conception of the issue, and
in your understanding of policy interventions - in short, a way to
generate insight.

Jim is of course correct that one is never justified in arbitrarily
tweaking or warping parameters to attain a good fit to data, no matter
whether the parameters are estimated judgmentally or econometrically.
However, when one cannot replicate the data to an acceptable degree of
accuracy while keeping all parameters in the ranges established by prior
knowledge and other data (i.e., without tweaking), the alarm bells
should go off: there is something wrong with your model. Further
investigation may show there were exogenous events in the real system
not captured in your model, and which you are comfortable ignoring.
Quite often, however, what one discovers is that the basic structure of
the model simply cannot explain the dynamics, and must be revised. Of
course, fitting the data is no guarantee the model is reasonable (other
tests are required).

A wonderful set of examples of this process is described in the recent
and excellent article by Jack Homer "Why we iterate: Scientific
modeling in theory and practice" System Dynamics Review, 12(1), 1996.

John Sterman

Sloan School of Management
MIT, E53-351
30 Wadsworth Street
Cambridge, MA 02142
phone: 617/253-1951 fax: 617/258-7579 e-mail: jsterman@mit.edu

[Shaun Tang]

Dear all

> > ... This does not mean that one should only build models
> > when there is good data, but . . .
> What counts is "behavior patterns". Data only give the past, and if one has a
> forecastable change in behavior pattern, all past data become useless ...

I would like to provide my concurrence to both of the above comments
from two writers. IMO, the definition of data should be first
defined particularly in the SD modelling contex).

Historical data are helpful in modelling to most cases but sometimes
they have limitations such as some sorts of useful data
might havnt been collected before in the past. Often new sources
(sometimes together with new management structure) developed to provide
required data, which havent been recognised before,
must be established. Most importantly, not all data are measurable in our
conventional perspectives of viewing systems.

Not all variables are measurable and quantifiable in order to generate data.
Some very useful variables data can be unknown/ non-measurable to all of us,
but we can still incorporate them into a informative/ meaningful SD model
(as cognitive mapping).

I would think that in order to test the reliability of a SD model,
only some (but not all) of the variables must be measurable with data.

Cheers, Shaun
Shaun TANG
Department of Business Systems
Monash University
Clayton, Victoria 3168
AUSTRALIA

Email stang@fcit.monash.edu.au
Fax: +61 3 99055159
Phone: +61 3 99055810

[CrbnBlu@aol.com]

irt: gallaher@teleport.com (Ed Gallaher), Wed, Apr 17, 1996 5:59 AM EST

Regarding Eds comment:

To what extent does modeling help us understand this complex, changing
system? And does it bring anything to the table in undestanding this topic,
that is better than *other available methods* in the same arena? (This
concept was of course presented by Forrester in Principles of Systems).

This does seem to be the crux of matter now doesnt it?

Ed, thanks for the very coherent perspective.

Gene Bellinger
CrbnBlu@aol.com

[David W Peterson]

I strongly agree with John Stermans position that data-consistency is
very important and almost always useful, even though it is not the only
test of model quality. However, I would offer one tiny correction to his
statement, "... when one cannot replicate the data ... alarm bells should
go off: there is something wrong with your model."

Whenever data and model disagree, alarm bells should indeed go off, but
the alarm should stimulate an investigation, not force a conclusion.
Sometimes, there is nothing wrong with the model, but something is wrong,
instead, with the data. One of the valuable applications of robust models
of the system dynamics type is to reveal errors in otherwise plausible
data.

David W. Peterson
Ventana Systems, Inc.
60 Jacob Gates Road
Harvard, MA 01451

dwp@world.std.com

[jsterman@MIT.EDU (John Sterman)]

I agree with David Petersons point that a clash between model and data
often reveals errors in the data. It is critical to identify such
errors, not only to reconcile the disagreement between model and data,
but also to revise the data-generating and collecting processes so that
the data become more useful in the future. This necessarily involves
revising ones model of the processes by which data are generated. So
while I agree with David about the low quality of data, the discovery of
data errors also leads us to revise our models, both mental and formal
(we might modify the model to more explicitly portray the process by
which the data are generated, averaged, measured, and reported. A
stunning example of this process is the ozone hole data from Antactica.
The following is an excerpt from my paper "Learning in and about complex
systems" which appeared in the SD Review in 1994:

"The first scientific papers describing the ability of
chlorofluorocarbons (CFCs) to destroy atmospheric ozone were published
in 1974 (Stolarski and Cicerone 1974, Molina and Rowland 1974). Yet
much of the scientific community remained skeptical, and despite a ban
on CFCs as aerosol propellants, global production of CFCs remained near
its all time high. It was not until 1985 that evidence of a deep `ozone
hole in the Antarctica was published (Farman, Gardiner and Shanklin
1985). As described by Meadows, Meadows, and Randers (1992, 151-152):
The news reverberated around the scientific world. Scientists at
[NASA]...scrambled to check readings on atmospheric ozone made by the
Nimbus 7 satellite, measurements that had been taken routinely since
1978. Nimbus 7 had never indicated an ozone hole.

Checking back, NASA scientists found that their computers had been
programmed to reject very low ozone readings on the assumption that such
low readings must indicate instrument error.

The NASA scientists belief that low ozone readings must be erroneous
led to them to design a measurement system that made it impossible to
detect low readings that might have invalidated their models.
Fortunately, NASA had saved the original, unfiltered data and later
confirmed that total ozone had indeed been falling since the launch of
Nimbus 7. Because NASA created a measurement system immune to
disconfirmation the discovery of the ozone hole and resulting global
agreements to cease CFC production were delayed by as much as seven
years. Those seven years could be significant: ozone levels in
Antarctica dropped to less than one third of normal in 1993, and current
models show atmospheric chlorine will not begin to fall until the year
2000, and then only slowly. Recent measurements show thinning of the
ozone layer is a global phenomenon, not just a problem for penguins.
Measurements taken near Toronto show a 5% increase in cancer-causing
UV-B ultraviolet radiation at ground level: ozone depletion now affects
the agriculturally important and heavily populated northern hemisphere."

In this fashion the discovery of data errors leads to significant
changes in our models.

John Sterman

Sloan School of Management
MIT, E53-351
30 Wadsworth Street
Cambridge, MA 02142
phone: 617/253-1951 fax: 617/258-7579 e-mail: jsterman@mit.edu

[Jim Hines]

David,

You "agree strongly" with John that whenever model output and data do
not agree alarm bells should go off.

But, say that I wish to explore how inventory policies might cause
oscillations in the economy. Should I be alarmed that my model does
not exhibit a growth mode?

Regards,
Jim Hines
jimhines@interserv.com

[jimhines@interserv.com]

Concerning John Stermans reference to the ozone hole data.

It is telling that the scientific model for the destruction of ozone was
developed before time series data from Antarctica was available. This is one
prominant instance where model-based insight did not depend on a close fit with
time series data.

Jim Hines
jimhines@interserv.com

[jimhines@interserv.com]

Much of what I agreed with in John Stermans reply was in parenthetical comments
and merits further development.

In putting forward his personal belief in the value of a model being able "to
replicate historical data", John parenthetically says the replication should be
"to the accuracy appropriate for the purpse". In my view the parenthetical
comment is the key point of the entire exchange. Too many people believe that
the appropriate level of accuracy is always "as accurate as possible".

It is now a common observation that models can be made to match historical time
series to any level of precision desired. Although the observation strictly
applies to models composed of polynomials of arbitrary length, it is essentially
also true of dynamic models. Getting such a fit can require a lot of sweat and
touch of creativity, but you can do it. The problem with doing it is two fold:

1) A fit to data is not a strong test of system dynamics models. What this
means is that a fit to data cannot distinguish a good model from a bad model.
Useful models can fail to fit the data and useless models can fit the data quite
well.

2) The large expenditures required to get the fit, can in practice starve other
activities. System dynamics was founded on two suppositions. First, managers
for the most part know what is going on locally. Among other things they have a
sense for the reference modes and for links in the system. And, second what
they lack is an understanding of how the links fit together to produce the
reference modes. If managers do for the most part have a good sense of local
reference modes and links, the need for large expenditures on data is not
necessary; we can rely on the managers to provide the small amounts of numerical
data and the large amounts of information about decision making we need. What
should guide the bulk of our efforts is the second idea -- that managers have
difficulty understanding how structure produces behavior. We should put our
resources into helping managers make sense of the voluminous information they
have; that is, we should put our efforts into analyzing models and articulating
the causes of their dynamics.

I know that neither John nor anyone else in this discussion supports inadequate
analysis. I do believe, however, that in many cases where data fitting is
pursued, the data fitting continues long after it has stopped yielding dynamic
insights; and the model analysis stops long before the real insights have been
achieved. We need less data fitting and more analysis.

Regards,
Jim Hines
jimhines@interserv.com
MIT and LeapTec

[jsterman@MIT.EDU (John Sterman)]

Jim Hines writes:

> It is telling that the scientific model for the destruction of ozone
> was developed before time series data from Antarctica was available.
> This is one prominant instance where model-based insight did not
> depend on a close fit with time series data.

The ozone hole data were not the only data available to test and refine
models of the chemical reactions by which chlorine catalyzes ozone
destruction. To suggest that Molina or Rowland (who won the Nobel prize
in chemistry this year for this work) or the other atmospheric chemists
working on these issues failed to use the data, were satisfied with
qualitative assessments of model adequacy, or dont believe their models
have to be tested against the data is a gross misconception of how
science works.

Furthermore, the ozone hole data (and more data being gethered by
researchers around the world) are currently being used to refine the
various models so that we can better understand the likely dynamics of
atmospheric ozone as CFCs are phased out. Without such painstaking
work, these models would never have been accepted, and there is little
likelihood that the nations of the world would ever have agreed to phase
out CFCs.

John Sterman
jsterman@mit.edu

[jsterman@MIT.EDU (John Sterman)]

Jim Hines writes:

> But, say that I wish to explore how inventory policies might cause
> oscillations in the economy. Should I be alarmed that my model does
> not exhibit a growth mode?
>

If your model does not include a growth trend, you should be using
detrended data to test your model, but it is not acceptable to say that
you dont have to match the data. You should quantitatively compare the
phase, amplitude, spectral density, correlations, and other attributes
of your economic model to the characteristics of the real economy, and
if there are discrepancies, you should track them all down to a
satisfactory resolution, either in a revised model, revised
understanding of the data, or both. Fit to time series data would be an
appropriate test if your model was driven by exogenous variables.

Further, the assumption that the long-term growth of the economy is a
separate mode of behavior which can be safely ignored in models of the
business cycle, though a plausible and common one (which we often have
made ourselves), is only an assumption, and ought to be tested. There
are many theories of the business cycle in which growth and fluctuations
are fundamentally entwined (such as the neo-Schumpeterian theories): to
discriminate among these different theories might require you to
incorporate growth in your business cycle model.

John Sterman
jsterman@mit.edu

["Teiling, Bernard"]

I enjoy very much the debate on reliable models.
It reminds me of the reflection of Copernicus about his cosmological model:
"...and yet she is revolving" (i.e. earth revolves around the sun).
Copernicus proposed that a rotating Earth revolving with the other planets
about a stationary central Sun could account in a simpler way for the same
observed phenomena of the daily rotation of the heavens, the annual movement
of the Sun through the ecliptic, and the periodic retrograde motion of the
planets.
BUT the acceptable model was the Ptolemaic representation of planetary
motion by means of complicated combinations of circles called epicycles.
When Galileo wanted to confirm the model (the HELIOCENTRIC WORLD SYSTEM),
based on a structure which was more appealing to him, he encountered the
problem that data about the location of planets were more precisely
predicted by the Ptolemaic/Copernican model than his model.
Can you believe it? the epicycloid-trajectories-around-the-earth model is
more precise than the circular-trajectories-around-the-sun model (at least
for the selected set of observation data at that time, in the 1620s or so).

It took hundred years to recognize that both models were wrong! (does it
remind you of something?), and that the Galilean model was more accurate if
not more precise.
Thanks to Newton.
Bernard Teiling
usgle000.teilinb@wcsmvs.infonet.com

[jimhines@interserv.com]

John writes:
"To suggest that ... atmospheric chemists working on these issues [of ozone
chemistry] failed to use the data, were satisfied with
qualitative assessments of model adequacy, or dont believe their models
have to be tested against the data is a gross misconception of how
science works"

Thats reading a lot of "suggestion" into what I said. I merely pointed out
that the original work was a prominant instance where model-based insight did
not depend on a close fit with time series data.

But, lets talk about how science works for a moment. The sciences to which John
refers are those like biology and chemistry (and not, say, geology). These
sciences do not procede by fitting large models (meaning hundreds or thousands
of equations) to time series from the world. Instead they procede by
experimentation -- very pure little, man-made dramas designed expressly to test
a logical argument. Every effot is made to eliminate aspects of the real world
that do not impinge on the agrument at hand.

How different this is from the use of time series by econometricians and by
those of us who try to use time series data. In these endeavors, the model is
fit to data that comes directly from the world with all its confounding
variables and influences. Worse, unlike an experiment, the world cannot be "run
again" to see if it turns out the same way and continues to "validate" the
model. Granted, out-of-sample testing can be and should be preformed, but its
simply not possible to rerun the world. (There are, of course, a number of
moments that I would prefer not to rerun --- like the time in sixth grade when I
asked Liz Tannenbaum to go steady and ..., well, anyway you cant rerun the
world).

Interestingly, though, it is possible to run experiments in business. Not all
the time, perhaps; but, much of the time: Many of the model-based insights we
derive from system dynamics could be tested experimentally. Recently, for
example, I concluded a consulting engagement where the principle insight was
that the way the company intended to increase product development just wouldnt
work. The result, which even contradicts initial beliefs of people on the
companys modeling team, is on the controversial side . No amount of fitting to
time series will be convincing to the people who need to set policy. But, what
would be valuable, and what would be convincing, would be to run the experiment
(even while continuing the search for better policies). Why not take several of
the teams, and actually implement the policy?

For some reason, companies dont usually run experiments. They are willing to
spend tens and hundreds of millions of dollars to change the entire
organization; risking not only the investment, but the future as well. Ill
suggest to my client that they run an experiment, and let you know how they
respond. But in the meantime: Why dont companies run more experiments?


Jim Hines
jimhines@interserv.com

[jimhines@interserv.com]

John Sterman writes:

"Without such painstaking work [i.e. fitting ozone models to data], these models
would never have been accepted"

Lets first be clear that this benefit of time_series data -- i.e. gaining
acceptance -- is separate from the idea of generating insight, which began this
discussion. Johns observation here does not bear at all on the proposition
that insight is commonly generated, as it was in this case, without a close fit
to time-series data.

Now to consider the use of time-series data to convince managers: Yes, getting
close fits to data is used by people in our field to convince managers. And,
yes sometimes its effective. But, often it backfires. The implication of the
close fit to data is that the model accurately captures the company. Obviously,
it doesnt and its a simple matter for even well intentioned managers to point
out that the model does not capture the company:

"What do you assume in your model about our shipping costs from North America to
Asia?"
"Well, actually weve aggregated the regions together."
"Oh" And thats the end of that presentation.

Regards,
Jim Hines
MIT and LeapTec
jimhines@interserv.com

[jimhines@interserv.com]

To Bernard Teiling:

Yes, perhaps we are "revolving" a bit, here.

But I did appreciate your Galileo example very much. It captures the
distinction between precision of fit to time-series data and structural
fidelity. The distinction is even more important for managers than it is for
astronomers. Managers, if they get the structure roughly right, can often find
a lever to move their world.

Regards,
Jim Hines
jimhines@interserv.com

[jsterman@MIT.EDU (John Sterman)]

Jim and I are in complete agreement regarding the critical role of model
purpose in determining the appropriate standards and tests for building
confidence in models, including the accuracy and precision of a models
fit to historical data. I included the remark that one replicates data
"to the accuracy appropriate for the purpose" as a parenthetical only
because the idea that one builds a model for a purpose, and that the
purpose determines the criteria for assessment, is fundamental to system
dynamics (and to other modeling methods), and should hardly require
repetition at all. For example, Forrester wrote as early as 1961
(Industrial Dynamics, p115) that "The validity (or significance) of a
model should be judged by its suitability for a particular purpose. A
model is sound and defendable it it accomplishes what is expected of
it...validity, as an abstract concept divorced from purpose, has no
useful meaning." The role of purpose has ever since rightly been
central to system dynamics texts and nearly all the SD literature on
model assessment.

Perhaps this discussion would be advanced if Jim could be more explicit
about the particular contexts and purposes in which he feels it is not
necessary to test the extent to which ones model replicates the
historical data. If he feels there are model purposes for which it is
appropriate to test a model against data, perhaps he can indicate how
close is close enough. Getting the discussion down to cases might
help clarify areas of agreement and disagreement.

John Sterman
jsterman@mit.edu

[jimhines@interserv.com]

Johns recent posting raises three points: One regards the use of data when
investigating a single mode; the second regards the use of data when
investigating a singly dynamic hypothesis; and the third relates to driving
models with exogenous inputs.

1) I think John and I agree that you dont want to match the raw historical
record in a case where you are investigating a single mode in the real world --
say, when you are exploring the business cycle in industrialized economies.

John suggests that you DO want to match detrended data. Here, Im afraid I
cannot agree, if John means anything beyond a rough check. Detrending data
usually means fitting a very simple exponential model of growth, a model which
is is not intended to be accurate structurally. The result is a rough
indication of what history might have looked like if the real growth structure
were not present. In some cases it can be very rough if the growth structure
departs seriously from the exponential. Detrending output from a structure that
produces s-shaped growth, for example, will produce a decline at the end.

It is simply not possible in practice to precisely extract from a time series
the relevant mode you want to match unless you have a complete model of the
entire time series, something that we normally do NOT have in time series data
of social systems.

Actually this is not so different from the physical sciences. We dont normally
have complete models of the time series generated by the natural world, either.
We dont for example have a complete model of how a particular leaf fell last
autumn. Which is why scientists use experiment where possible -- they dont try
to extract the effect of gravity from the leaf time series.

2) The Issue is even clearer when you are interested in exploring a single
hyptohesis. For example, perhaps your hypothesis is that over- and
under-building of inventories may produce an oscillation. John suggests that

"You should quantitatively compare the phase, amplitude, spectral density,
correlations, and other attributes of your economic model to the characteristics
of the real economy, and if there are discrepancies, you should track them all
down to a satisfactory resolution, either in a revised model, revised
understanding of the data, or both".

I just dont think this is reasonable. Other structures youre not interested
in contribute to "the business cycle" (i.e. what remains after you detrend the
data "correctly"). Bob Eberlien showed that the multiplier-accelerator
structure will affect frequency, and amplitude. There are almost certainly
other structures whose influence will also be felt involving for example,
interest rates, trade, etc. To track all these influences down means I need to
build all of these structures into my model. I would become very old, but not
very much wiser about inventory dynamics, if I attempted to track down all the
discrepancies.

3) John also mentions driving models with exogenous inputs in order to match
time-series. But, I think we want structure that can create the dynamics
**endogenously**.

Regards,
Jim Hines
jimhines@interserv.com

[Jim Hines]

I think that Khalid phrases the issue just right. The goal is

"a rigorous fit of the model to the problem addressed"

Jim Hines
jimhines@interserv.com