Can SD models be validated without being run?

This forum contains all archives from the SD Mailing list (go to http://www.systemdynamics.org/forum/ for more information). This is here as a read-only resource, please post any SD related questions to the SD Discussion forum.
"Kim Warren"
Junior Member
Posts: 14
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by "Kim Warren" »

Content-Transfer-Encoding: quoted-printable

=20
Please forgive me if the following question is na=EFve - my math is very =
rusty!=20
=20
I am currently struggling to persuade colleagues in my field (strategic =
management) of some serious implications arising from an SD view of how =
the real world functions. To convey this simply, I try to explain that =
the general specification of an SD model, comes in three parts ...=20
o The value of all variables at any specific moment is =
instantaneously estimable from the then-current level of all =
asset-stocks (if you could instantaneously halve customer-numbers, for =
example, sales would halve, ceteris paribus)
o The value for each asset-stocks in the next period is its =
value in this period, plus what is added, minus what is lost (customers =
next year =3D customers this year, plus those newly won, minus those =
lost)
o ... and crucially ... those rates of change in asset-stocks =
are themselves, like all other variables, dependent upon the current =
level of existing asset-stocks (new-customers this year =3D fn. of =
current remaining potential customers, size of sales force etc.)
=20
Feedback is merely an unavoidable artefact that results from these =
relationships
=20
I further suggest that, though this appears to result in a closed =
system, certain stocks will typically be exogenous to the system under =
study, e.g. the number of potential customers for a new product will be =
a function of such factors as general levels of spending power etc. that =
are nothing to do with the business system under study. This contrasts =
strongly with conventional views of strategic performance - that it =
depends largely on exogenous factors (market growth rates, numbers of =
rivals etc.)
=20
An important implication of this SD perspective (if I understand it =
correctly) is that regression-based estimations of performance outcomes =
are doomed to failure - if todays customer-base is identically equal to =
the sum of all customers ever won since time=3D0, minus all historic =
customers ever lost, then there is no possibility that todays customer =
numbers can be well-explained by todays marketing efforts, todays =
price or indeed any other variable at a specific moment. If todays =
stock-values cannot be well-explained by any current values of other =
variables, then neither can the values of any item that depends on those =
stocks, which include sales revenues, profits and all other financial =
performance outcomes ... which is rather a shame, since multi-variate =
regression analysis forms the basis of the vast majority of empirical =
research in strategy (?and other social sciences)
=20
All of which makes me wonder ... surely it must be possible to validate =
an SD model without ever running it ...=20
- the stock accumulation relationships with flows are totally =
deterministic - todays customers =3D yesterdays, plus those won, minus =
those lost
- all other relationships, being instantaneous, must be taken =
to be constant through time (or at least the time-horizon over which the =
simulation is taken to be valid)=20
- ... and these other relationships must include the current =
rate of flows (i.e. ... todays rate of customer acquisition =3D a =
function of todays potential customers, sales force, product =
functionality etc. )
- if those current relationships are valid, they should be =
discoverable and verifiable from regression analysis of the real-world =
situation ( although regression is unsafe whenever it crosses one or =
more stock/flow boundary, it could be safe wherever no such boundary is =
crossed)
- such relationships should apply with confidence not only for =
todays flow rates, but for the same flow-rates over all of the relevant =
history (if not, then something else must be varying through time, which =
implies we must have missed an accumulating stock)
- if all this is so, then the model must be valid if the =
equations determining every stocks in- and out-flows conform with these =
statistical findings ... all of which can be confirmed without ever =
running the model
... or have I missed something?
=20
Kim Warren
=20
From: "Kim Warren" <
Kim@strategydynamics.com>
"George Backus"
Member
Posts: 23
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by "George Backus" »

Kim appears certain that there must be some way to validate a model without
running it. He notes that SDers have the nasty penchant of always wanting
causal feedback in their models -- spiced with a bit of real-world
non-linearity. Like most of us, I start with a simple conceptual (one or
two loop model) and work up to producing the next, new (over-weight) mother
of all models. In 30 years, I have never met even a one or two loop model
that produced the results I expected. There are always subtle or
not-so-subtle behaviors that I didnt expect (okay, so I am a slow
learner...). These results always produce "am ever I glad I noticed this
problem early" response. If you dont run, you dont know.

And moving on to data validation, it is true that you cannot prove a model
is correct by using data, but you can prove it is wrong. And if you cant
understand why the model produces different results (behaviors) than the
data, you have NO basis for assuming the model is valid. Using data can
give confidence, but never truth. Without the use of data, however, the
model must be deemed invalid. In science, a model is guilty until proven
innocent. In a practical sense, "innocent" just means better than ANY
alternative. (Sorry, this digression is my favorite soap box...)

In my work, using regression to determine the parameters "between loops"
appears to produce robust, defendable results (as Kim notes) -- even if the
statistics seem shaky. This assertion relies on the "assumptions" the that
loop is properly specified/understood in terms of the variables (and their
definitions) and relationships that it contains. If the loop contains the
appropriate approximation of the problem process, then we can then argue
that we have the best of what we need. The "missing" parts that the
statistics are complaining about are not relevant to our purpose.
Nonetheless, the nasty non-linearity in the decision process (that often
drives the rate equations) can appear. The flows can be then changing
regimes without an added level. Hence, we are again faced with the reality
that until you run the experiment (run the model), you can not be confident
of the outcome.

George Backus
Policy Assessment Corporation
14602 West 62nd Place
Arvada, CO 80004
Bus: 303-467-3566
Fax: 303-467-3576
Mobile: 303-807-8579
Email:
George_Backus@ENERGY2020.com
C.GRUTTERS@jur.kun.nl
Newbie
Posts: 1
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by C.GRUTTERS@jur.kun.nl »

Kim raises an interesting question

All of which makes me wonder ... surely it must be possible to
validate =
an SD model without ever running it ...=20



- if all this is so, then the model must be valid if the =
equations determining every stocks in- and out-flows conform with
these =
statistical findings ... all of which can be confirmed without ever =
running the model
... or have I missed something?


No, I wouldnt say that youve missed something but I would say that you
are building an argument along a questionable line.
Let me give some arguments or rebuttals.

As Sterman convincingly states in his Business Dynamics (2000, p. 846),
the word validation is out of order in the world of models.
Models are impossible to validate nor can they be verified. (If they
could, than Kim would have a point.)
This also implies that a model can not be proved to be (the) correct
(one), since that would at least suggest that the model at hand is the
only one possible.
In other words, a model implies the existence of another - competing -
model, which shifts the burden of proof from the equations towards the
concepts enclosed in the model at hand.

So, a word that could be used instead is, for instance, credibility or
convincingness (also taken from Sterman 2000, p.846, referring to:
Greenberger, Crenson & Crissey 1976).


C.GRUTTERS@jur.kun.nl

And this convincingness of a model does not show from its equations, but
from its ability to build confidence in the way the problematic
(reference mode of behavior) is contested.
Just like one does in the legal domain: the outcome as such is less
relevant than the route of argumentation towards it.

==
dr Carolus Grütters
From: C.GRUTTERS@jur.kun.nl

Law & IT and
Centre for Migration Law (CMR)
Faculty of Law
University of Nijmegen
The Netherlands
http://www.jur.kun.nl
it
http://www.jur.kun.nl/cmr
===
as of september 1st, 2004:
the University of Nijmegen wil be named as
Radboud University Nijmegen
"Ray on EV1"
Member
Posts: 29
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by "Ray on EV1" »

Kim,

I hope we can more clearly understand the semantic difficulties around using
the work "validate". From Websters Dictionary, there are two
opportunities:
1) To mark with an indication of official sanction.
2) To establish the soundness of

Thus, a group of people can agree to the use of a model and we could
consider this an official sanction. Or, through experimentation or data
analysis, we might find that a particular model reproduces data accurately
and state that the data fidelity establishes soundness.

In no case was it implied that the model accurately represents specific
components of the subject system.

Model validation could be interpreted in a variety of ways. Two typical
aspects are fidelity of data (behavior) and model component correlation to a
specific architectural concept.

Validation? I would be beneficial to more precisely describe your
requirements.

Raymond T. Joseph, PE
RTJoseph@ev1.net

Aarden Control Engineering and Science
Bill Harris
Senior Member
Posts: 75
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by Bill Harris »

"JEAN-JACQUES LAUBLE" <JEAN-JACQUES.LAUBLE@WANADOO.FR> writes:
> A model is correct if its outcome generates more profit (financial for me)
> than its cost.

Oh. I like that one.

My only addition might be to try to separate out random occurences (a
flip of the coin is cheap, and it might select the correct one of two
alternatives half the time, but wed likely usually hope wed do better
with a "better" model).

> An interesting question would be:
> Is the second level of correctness generally increasing if the first one
> (the behaviorial one) is increasing?

I guess thats what I was trying to say, if I understand your question
correctly.

Bill
From: Bill Harris <bill_harris@facilitatedsystems.com>
--
Bill Harris 3217 102nd Place SE
Facilitated Systems Everett, WA 98208 USA
http://facilitatedsystems.com/ phone: +1 425 337-5541
Yaman Barlas
Member
Posts: 44
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by Yaman Barlas »

Hi Kim,
I am not sure I understood all of your questions, but I will reply
anyway with few things I know:
I- A good (and important) part of model validation is done WITHOUT
simulation: I call this direct structure testing. This means
establishing confidence in the validity of each equation one by one, in
isolation. Some most typical tests of direct structure are: Dimensional
consistency test, parameter/variable confirmation test (that no
parameter or variable is invented just to make the units match, that all
have some real meaning), and direct extreme condition tests (to test
the plausibility of each equation by assigning extreme values to
inputs). None of these important tests involve simulation.
(The classes of tests that DO involve simulation are tests of indirect
structure testing and tests of behavior validity).
II- Another (rare) type of direct structure testing is the one you
mention: using data (and regression) to test the validity of a flow
equation or an auxiliary equation). This is RARE, because the functional
relations that we write in these equations are ceteris paribus. But in
reality, it is rather hard to find such ceteris paribus data. In other
words, in real life the simulation IS running and all variables
influence each other and dynamically produce the data sets for each
variable. For the comparison to be fair, one must run the simulation
model and then test by regression (or somehow) if the covariances
between variables from simulation conform the covariance data from real
life. In a nut shell, variable correlations comparable to collected data
are typically OUTPUTS of good simulation models. (Nice demonstrations of
this can be found in some of early JWF responses to wrong criticims of
his models).
III- Since stock equations are fixed in form, you are right, if one were
able to validate each of the flow equations of a stock, then the stock
dynamics would be automatically validated, assuming that all the
(major) flows are included in the model and no non-existing flows are
invented. (That is, if the stock-flow STRUCTURE assumed in the model is
valid). But again, in validating the flow and and auxuliary equations in
a nontriivial feedback model by collected real data, you face the
ceteris paribus issue I describe above, and also collinearity
problems).

IV- As some others said, all this leads us to one conclusion: In any
given situation, we do the best we can (with data if we can or without
data: and with simulation and without simulation) to increase confidence
in the model. This is a process; we know that the model will NEVER
PROVEN (or even demonstrated) to be valid. It is not a binary
question. There are highly valid model, there are reasonably valid
ones, lousy ones and terrible ones (like mine). And then, there are
also beatiful models and ugly models, also another important
dimension of validity. (So validation seems to be a poor term).

(Are the above in any way related to your questions?)
best,
Yaman Barlas
From: yaman barlas <
ybarlas@boun.edu.tr>

A Ref: "Formal Aspects of Model Validity and Validation in System
Dynamics", System Dynamics Review, Vol.12, no.3, 1996, pp. 183-210.
From: yaman barlas <ybarlas@boun.edu.tr>
Bob Eberlein
Member
Posts: 49
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by Bob Eberlein »

Hi Everyone,

I have enjoyed this discussion but one very important thing has not been
emphasized. Kim asks why we dont simply validate mode relationships by
running regressions. As long as there are observations on all model
variables that is something that can be done and, if the observations
are actually good, will work very well. In practice, however, a
regression approach leads to writing equations that only contain
variables which have data collected. In fact, a standard way to do
things is to simply substitute one input for another if it seems like it
would be a good surrogate and there is data available.

It makes a lot more sense to model things as it seems they should be
then step back and see what there is data for. When we do that the only
equations that have all their data available tend to be accounting
identities and that makes standard regression techniques pretty
irrelevant. We are left with a partially observed set of dynamic
variables and we want to make some claim about the relavance of the
equations in our model to relationships in the real world. As Yaman
points out it may be possible to directly inspect the relationships and
point out their failings. If none are apparent we need to see if the we
can use the behavior they imply to find their failings (as George points
out all we can ever do is invalidate things). When not everything is
measured the only practical way to test behavior is simulation.

The answer to Kims question is no. Apparently valid system dynamics
models may need to be run to be invalidated.

Bob Eberlein
bob@vensim.com
George A Simpson
Junior Member
Posts: 11
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by George A Simpson »

An SD model is a theory - just like any scientific theory.

It is built on a body of evidence, and is useful for explaining
observations.

There are two questions one can ask about a theory:

1. - is it internally self-consistent (corresponding to verification in SW
development)?

2. How well does it accord with observations (corresponding to
validation)?
In business, this can be translated "how useful is it as a guide to
decision-making?"

Kims suggestion relates only to the first point.

Sterman in section 21.1 of "Business Dynamics" is right that the terms
verification and validation seem to imply something about objective truth.
However I think he goes a bit too far in stating that validation and
verification are impossible. (Perhaps the headline was more to shock
people into paying attention than meaning it literally.)

Each theory is an approximation to the truth - Newtonian gravity is pretty
good on the whole - certainly all you need for spaceflight. But
Einsteinian gravity is even better, and so far no discrepancy has been
detected.

So the notion of approaching truth as a limit needs to be considered. All
models are wrong, but they are not all equally wrong. Truth exists, and is
what our models are seeking to approach, in the limit. Of course our
process is imperfect and wandering, because we dont know exactly where
truth lies. Its a search algorithm.

Coming back to validation - there is a domain over which one performs
validation work, and a theory (model) together with the domain over which
it has been validated, needs to be considered as a whole. Thus a model
that has been validated by having made a useful prediction is accorded a
degree of reliability that is higher than one that has not.

...george...
From: George A Simpson <
gsimpso4@csc.com>

Dr. George Simpson, Principal Consultant, CSC
temporary address:
CSC House, Fleet, Hampshire GU51 2UY.
tel +44 1252 813930.

permanent address:
Royal Pavilion T3 floor 2, Wellesley Road, Aldershot, Hampshire GU11 1PZ
United Kingdom
office +44 (0)1252 536523, fax +44 (0)1252 534132, mobile 07814 623 518
=?iso-8859-1?Q?Jean-Jacques_Laub
Junior Member
Posts: 16
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by =?iso-8859-1?Q?Jean-Jacques_Laub »

Hi Bill

I am not sure to have well understood your remark about random occurrences.

I think that randomness has a lot to do with the question relating
behavioural correctness and utility.

I think that there are two kind of utilities or outcome: results or
understanding or both.

If the outcome is understanding which is a continuous process, the risk of a
wrong outcome for randomness is null. If the outcome is results then there
is a risk of random wrong result, and the seek of results will always push
toward a better model.

About the relation between correctness and utility I propose that it depends
on the subject and the user.

There are roughly three category of subject: purely physical, middle
physical and social, and very social. When I mean social, I mean ambiguous
data.

>From a purely physical you get results, middle you get results and
understanding, and purely ambiguous you get understanding.

Then the real problem is on what is waiting the user.

If the user waits for a result and the problem is purely social or
ambiguous, he will be disappointed.

So I would put that in a table the number is the average level of
satisfaction going from 1 to 9:



Kind of problem: physical both physical and ambiguous
ambiguous total


average


satisfaction

Kind of user



Looking for results 9 3
0 12



Looking for results 9 7
5 21

And understanding



Looking for 9 9
9 27

understanding





total average 27 19
14

satisfaction





Of course the number in this table can be largely discussed, but I think it
shows that people looking for understanding are the less susceptible to be
disappointed.



It shows too one of the problem of SD when applied to ambiguous problems: it
has a low level of satisfaction, because people are looking for results,
preferably quick: results are quick, understanding can be slow.



About the relation between behaviour and usefulness, the more it is physical
the more the behaviour can be close to reality and its usefulness will
depend on it, and the more it is ambiguous, the less it will be close to
reality and the less it will be necessary for the understanding.



The difficulty is that most problems are a mix of physical and ambiguous
conditions.

It is then necessary to make allowance for the results and the
understanding, one destroying often the other.



I think that it would be easier to propose SD in an ambiguous context like
business, with no results and only giving understanding. It would then not
be suspected to give wrong results, the thing that business people fear. In
this context it is the fact that IT CAN GIVE RESULTS that can be damaging.

Regards.



Jean-Jacques Laublé
From: =?iso-8859-1?Q?Jean-Jacques_Laubl=E9?= <JEAN-JACQUES.LAUBLE@WANADOO.FR>

Allocar rent a car company.
John Sterman
Senior Member
Posts: 117
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by John Sterman »

The discussion Kim began on model "validation" has been interesting.

A couple of comments.

George Simpson wonders whether my argument in Business Dynamics that
"validation and
verification are impossible" was designed "more to shock people into
paying attention than meaning it literally."

No, I do mean it literally. The primary dictionary definitions of
"validation," "verification" and similar terms relate to being
objectively true, and this is precisely what all models are not. Any
model that refers to the real world (that is, that is not only a
mathematical tautology) is, by definition, not the real world but a
selected, filtered, abbreviated, constructed representation. It
differs from the real world in multiple, indeed infinitely many,
ways. It is not correct, true, or valid. It is wrong. Get used to
it. The question is whether the model is appropriate and useful for
a particular purpose. Claims that models are "valid" or have been
"validated" are usually part of a rhetorical strategy the modeler or
others are using to legitimate their analysis or themselves, get
their policies adopted, or gain other advantage.

The notion that science (in our discussion, a model) asymptotically
approaches truth is naive. Yes, Newtonian physics is awfully good
(for large enough scales and small enough speeds), and general
relativity is better, but both are still models. General relativity
is still not integrated with the quantum world, and our comfortable
understandings continue to be overthrown -- as the example of dark
energy (now thought to make up two-thirds or so of the mass-energy in
the known universe) shows.

More important, the criteria for choosing among theories, for
deciding what tests to apply, are themselves part of the worldview we
each use to determine which models are best. The metaphor that
science is a search process, imperfectly but on average climbing the
mountain of truth, presumes that there is a mountain and that we all
agree on the definition of uphill. We dont. When such basic
differences in paradigms create conflict, the usual reaction is
frustration and anger that the other people just dont see reason.
If youve ever argued with people who believe in UFOs or ESP (or free
markets, rational expectations, Marxism, intelligent design or
whatever theories differ from yours), youve no doubt experienced the
problem that there is no common ground -- such folks do not accept
the very tests, criteria, and evidentiary standards you believe are
the sine qua non of the scientific method; hence they must be crazy.
They of course think you are stubbornly blind to the obvious. Often
the result of such a paradigm clash is that we categorize our critics
in such a way as to allow us to continue to believe in the "validity"
of our theory, because those who think it rubbish must be crazy, or
unreasonable, or unqualified to judge (they dont have a PhD or a PhD
from the right school), or are grinding some personal or ideological
axe.

None of this means it isnt important and necessary to test our
models as thoroughly and rigorously as we can, or that for each of us
personally all theories are equally good and all methods of testing
equally appropriate. Like everyone else, I find some people are
unreasonable, unqualified, and wield personal and ideological axes.
I dont believe reality is entirely subjective or socially
constructed. Though one might come to believe that reality is
illusory, one is ill advised to step in front of a speeding bus.

Thus Kims question of whether a model could be "validated" without
being run should be rephrased as "can we build confidence in a model
without running it?" Even better, since Kim asked the question out
his frustration in trying to persuade colleagues that his models are
appropriate or useful or better than other models (i.e., their
models), rephrase as "can we persuade skeptics that a model is
useful, appropriate, or better than others without running it?" The
answer is no.

Certainly, as Yaman pointed out, there are many tests one can carry
out on the structure of a model, and these are vitally important.
They often get shortchanged relative to the overemphasis on matching
historical data. More effort should go in to tests of structure, of
assessments of the appropriateness of the model boundary. But that
isnt enough. People are not generally able to detect all the
problems in a model by inspection of the equations or structure.
Feedback from the comparison of behavior to data and experience under
historical, hypothetical, and extreme conditions is essential in the
detection of problems, the determination of whether they matter to
the purpose, and improvement of your model.

Specifically, Kim suggests:

"o The value of all variables at any specific moment is
instantaneously estimable from the then-current level of all
asset-stocks (if you could instantaneously halve customer-numbers,
for example, sales would halve, ceteris paribus)
o The value for each asset-stocks in the next period is its
value in this period, plus what is added, minus what is lost
(customers next year = customers this year, plus those newly won,
minus those lost)
o ... and crucially ... those rates of change in
asset-stocks are themselves, like all other variables, dependent upon
the current level of existing asset-stocks (new-customers this year =
fn. of current remaining potential customers, size of sales force
etc.)"

These points are not self-evident truths but contingent on a host of
other concepts and assumptions including the idea that there is a
well-defined asset stock (in Kims example, the customer stock is the
sum of the firms customers and that all variables such as sales
depend only on these aggregate, scalar stocks, including those
outside the model boundary and therefore exogenous or constant).
Further, the equations for the evolution of stocks are not merely
matters of definition. Consider the obvious equation for a firms
inventory:

inventory = INTEGRAL(production - shipments)

The inventory equation is not a self-evident proposition, but
contains important auxiliary assumptions, specifically that the only
flows into the stock of inventory are production and the only
outflows are shipments. This means that losses due to spoilage or
shrinkage are implicitly assumed to be zero, and that other sources
of inventory such as purchases of finished goods from another
supplier are excluded (as often occurs in the auto industry, when one
dealer swaps cars with another to get the specific option combination
a customer desires). These flows may or may not be significant in
any particular case, but this determination is fundamentally both
empirical and judgmental: the modeler must investigate in the field
whether e.g. spoilage occurs and then make a judgment as to whether
the magnitude of the spoilage is significant enough to include in the
model. These judgments inevitably involve considerations that are
neither evidential nor a priori self-evident (in practice, they
always involve consideration of model purpose, audience, and other
social factors). Judging whether their inclusion or omission matters
to the purpose almost certainly requires running the model. Besides
the substantive value of considering model behavior, failure to do so
will certainly provoke (legitimate, in my view) skepticism and
rejection from precisely those people one seeks to influence.

Chapter 21 of Business Dynamics covers these issues in more depth and
provides references to key literature; see also reference in Sterman,
J. and J. Wittenberg (1999). "Path Dependence, Competition, and
Succession in the Dynamics of Scientific Revolution." Organization
Science 10(3): 322-341.

In particular, I recommend:

Meadows, D. H. (1980) The unavoidable a priori, in Randers, J. (ed.),
Elements of the System Dynamics Method. Waltham, MA: Pegasus
Communications.

Meadows, D. H. and J. Robinson (1985) The Electronic Oracle:
Computer Models and Social Decisions. Chichester, England: John
Wiley and Sons.

Oreskes, N., K. Shrader-Frechette, and K. Belitz (1994) Verification,
validation, and confirmation of numerical models in the earth
sciences, Science 263 (4 Feb), 641-646.

Cartwright, N. 1983. How the Laws of Physics Lie. Clarendon Press, Oxford.

Feyerabend, P. 1975. Against Method. New Left Books, London.

Kuhn, T. H. 1990. The road since structure. PSA 2(7) 3-13.

and of course,

Kuhn, T. (1970) The Structure of Scientific Revolutions, 2nd ed.
Chicago: University of Chicago Press.

John Sterman
From: John Sterman <jsterman@MIT.EDU>
"Jim Hines"
Senior Member
Posts: 88
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by "Jim Hines" »

Kim asks whether you can validate a model using multiple regression
techniques on the equations of rates and auxiliaries with the stocks as
independent variables. (He explains why he believes the technique would
fail using the stocks as dependent variables).

When Kim says "validate" I think he means to estimate the values of the
parameters of (rate and auxiliary) equations to see if you get
reasonable values. I know this isnt everyones definition of
"validity", and Im sure well have some interesting postings about
that. In the meantime here are some thoughts about the question of
whether regression can give you parameter estimates which you can then
evaluate for reasonableness (and if the estimates are not reasonable,
then you conclude the equation being estimated isnt very good).

In practice the answer sometimes seems to be "yes". But, in general, I
think the answer has to be "no" for at least the following reasons:

A. Practical
1) There is often too little data to estimate the parameters in an
equation.
2) The data itself are often wrong.
3) There may be no known statistical technique for estimating a
particular non-linear equation.

B. Theoretical/Philosophical
4) The purpose of a model may be to look at a particular mode of
behavior, in which case you wouldnt expect the equations to match any
actual historical time series (which would have all the modes in them).
5) The purpose of a model may be to represent a class of systems (e.g.
projects) -- in this case, too, you wouldnt expect the models
structure to be **exactly** correct for any particular actual project.
6) Regression (and many other statistical estimation techniques)
assumes the model is correct. But, we know most or all of the equations
in a model are incorrect (the world is simply not composed of algebraic
equations) -- so, the fundamental basis of (most) statistical techniques
are violated.

And, so Id like to ask the flip side of Kims question: Is there any
rational basis for having a high regard for statistical estimates beyond
the fact that these techniques are widely used and the anecdotal
evidence that they are useful?

Jim
From: "Jim Hines" <jhines@mit.edu>
"geoff coyle"
Senior Member
Posts: 94
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by "geoff coyle" »

Yaman RIGHTLY reminds us of what, and Jay Forrester pointed this out well
over 40 years ago, is the fundamental test - that the models defence rests
on justifying its equations, not on its comparison to observed data. To be
sure (and this was not Kims original question) it is comforting if the
simulation adequately resembles the data, but that is a second-order
consequence of the correctness of the equations, not an independent test
of the satisfactoriness of the model, let alone an indication that the model
gives reliable indications of policy actions. Jay also stated that the
equations and parameters should never be fiddled to fit the data.
Inability to fit observed data (where it is appropriate even to try to do
so) means that the model has serious weaknesses. In any case, data fitting
is usually dodgy to say the least. One significant reason is the difficulty
of setting up the initial conditions for time delays. Theres a further
problem in that the model will usually produce dynamics for vastly more
variables than are in the observed data. Can we rely on a test that matches
perhaps 3 model variables against data and ignores maybe 200 model variables
for which no data exist?

Another, and VITAL test on a running model is to verify its mass balances.
That is, for every separate entity in the model (whether hard, such as
money, or soft, such as customer satisfaction), the quantity in the system
at any time must be what was there to start with - the initial conditions -
plus cumulative, possibly exogenous, inputs, minus cumulative drains from
the model. This tests not only that the equations are correct, but also
that they have been correctly (without quotes) connected together and that
the model is neither gaining nor leaking mass. My 1996 book covers this.

Of course, however and however often we test a model, one cardinal point is
the extent to which it is capable of answering the questions for which it
was originally devised. That is the reason why listserv enquiries on the
lines of can anyone give me a model for domain X are fundamentally
misplaced. My model of some aspect of, say, the mining industry was devised
to answer specific questions and your questions about that domain might well
be entirely different from mine. Unless you know what my questions were, and
I know what yours are, my model is no help to you and will probably waste
your time, even if it does not mislead you.

None of this has anything to do with Kims question, of course.

Geoff
From: "geoff coyle" <
geoff.coyle@btinternet.com>
"Swanson, John"
Newbie
Posts: 1
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by "Swanson, John" »

This discussion is interesting, but in a grubby commercial world is
not very helpful.

I have always liked Forresters and Stermans views on model validity
and, I hope, understand them. However when I go to present ideas to
potential clients (I work in the field of transport and urban dynamics)
the very first response I get, usually before Ive even finished, is
how did you validate all that?

What they mean is, what data sets did you use and what clever mix of
econometric or statistical analysis did you use to estimate the
parameters for all those equations?

Now, I could try saying that validation is an illusion, or that all
you can really do is unvalidate, but I dont think that has the right
ring to it. I can explain that what matters is that the model is good
enough for the purpose, or good enough to risk using it (I think thats
one of Geoff Coyles expressions) but that isnt going to excite them
either, and anyway it begs the question of how you know what good enough
means. The fact is that the next person who comes in, my competitor, will
say, yes my model is all carefully validated using maximum likelihood and
the latest algorithms, it replicates historic data closely, and there are
lots of t-tests etc to prove it. And he will win the job.

To put it briefly, the SD position on this is probably right philosophically,
but it doesnt sell well.

I wonder what other peoples experience has been?

John Swanson
Associate
Steer Davies Gleave
28-32 Upper Ground, London SE1 9PD

[direct] +44 (0)20 7910 5542
[reception] +44 (0)20 7919 8500
[fax] +44 (0)20 7827 9850
[email]
j.swanson@sdgworld.net

www.steerdaviesgleave.com
"Jack Homer"
Junior Member
Posts: 3
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by "Jack Homer" »

"John Gunkler"
Member
Posts: 31
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by "John Gunkler" »

I agree with John Swanson -- we need to be able to justify that a model is
worthy of being used, not engage in sophism that goes deeply into the
philosophy of science. (Books have been written on the
verifiability/validation of scientific theories but citing them doesnt help
someone very much who has to decide whether to use a model to help make
decisions that will affect peoples live.)

Let me try to provide a real-life example of what concerns me:

One could create an SD model showing how "employee satisfaction" relates to
"productivity." The loops are fairly obvious and the "logic" of the
relationships seems quite self-evident. In fact, from the 1950s on many
people have tried to justify improving employee satisfaction by citing its
supposed effects on productivity, profitability, or other of their favorite
accounting measures. There were entire movements in business about this,
such as the Quality of Work Life initiatives, and in my experience many
people (and I include myself among them) want to believe that there is a
positive relationship between employee satisfaction and organizational
success.

I have not done this yet, but I believe one could relatively quickly put
together an SD model of this relationship, tweak the parameters (as were
not supposed to do), and put together a fairly convincing case that
improving employee satisfaction pays off on the bottom line. Everybodys
happy! ... Until we intervene and discover that the results are not what the
model leads us to expect.

Because the research results are far from clear on the relationship between
"employee satisfaction" and organizational results. Some studies have
purported to show a (correlational) relationship, many others have failed to
produce one.

Heres an anecdotal explanation of why the relationship may not hold in its
simple form:

Case A. A "highly satisfied" employee, dedicated to the purpose of
the organization, works her/his butt off.
Case B. A "highly dissatisfied" employee, so unhappy as to no
longer care about the organization, slacks off.

[These are the paradigm cases that those of us who want to see the
relationship validated typically think of. But then, there could also be
...]

Case C. A "highly satisfied" employee, doesnt care whether the
organization succeeds or fails as long as the paycheck keeps coming in,
doesnt see much relationship between his/her efforts and whether the
organization continues to be able to send the paycheck, and does just enough
to keep from being fired.
Case D. A "highly dissatisfied" employee, who cares very much about
the success of the organization but believes it is taking some wrong
directions, works his/her butt off to improve the organization.

As you can see, Cases C and D (should they exist), would throw off the
correlation we expect or want -- and, in a sense that is very meaningful to
me, would "invalidate" our SD model! In practice we would have to pay
attention to the "commitment" that employees have to the success of the
organization (and perhaps a few other variables) rather than instituting
wholesale interventions to improve "employee satisfaction." Satisfaction
would end up being one of several potential but not guaranteed ways to
improve "employee commitment."

HERES THE POINT: How do we prevent someone from "fooling" us in such a way
with a model? How do we know [better: How do we improve the likelihood]
that a model that matches historical data and "seems right" will suggest to
us interventions that will actually improve the results we desire? Thats
what I believe the plea for "model validation" is at least partly about.

From: "John Gunkler" <
jgunkler@sprintmail.com>
"Jim Hines"
Senior Member
Posts: 88
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by "Jim Hines" »

I think there may be an impression brewing that (1) everyone agrees
with what Jay said about validation and (2) Jay said validation isnt
possible.

This is not right on either count.

1) Jay never said that validation isnt possible. He said that
validation is a process of increasing confidence in a model. And, as
far as I know, he believes that building (some) confidence in a model is
not only possible but is always necessary.

2) Not everyone agrees with #1. It pains me to admit it, but Im one of
those doubters. I think there are some cases where validation is
irrelevant.

Heres the scoop:

Theres a continuum of model use: At one end of the continuum is the
model as "answer machine" and at the other end is the model as
"idea-generator". An answer-machine needs to be validated, an
idea-generator does not. Heres why:

ANSWER MACHINES MUST BE VALIDATED. An answer machine isnt very useful
if you can figure the answer out all by yourself without the machine.
So, you use an answer machine when you dont know the answer.
Unfortunately, this means youre going to face a terrible question
whenever the machine spits out its answer: How do you know that the
answer-machines answer is right, if you cant solve the problem
yourself? How do you know that 14.43 days really is the right amount of
inventory to hold? In two words, you dont. In three words, you never
will. This is a terrible position to be in, but one that is extremely
common. Youre in this same position whenever someone gives you an
answer that you cant figure out for yourself. A doctor may tell you
that the answer to your abdominal pain is to remove your appendix --
You can never be sure his answer is right. The most you can do is to
try to increase your confidence **in the doctor** enough to actually go
under the knife. You have to validate your doctor just like you have to
validate a model: Has the doctor (model) been right in the past? Does
the doctor (model) seem logical? Etc.

If anyone wants to build (and use) an answer machine, he will need to
validate the model at least to his own satisfaction. Further, people
who ask about validation of such model are right on target. Theyre
want to boost their confidence **in the model** to a point where theyll
be comfortable acting on what the model "says" (or disregarding what the
model says).

IDEA-GENERATORS DO NOT REQUIRE VALIDATION. At the other end of the
continuum are models that are used to help you think about something --
to enrich your mind. For example you might say "I think the positive
loop


Salespeople --> sales
. + + |
/| |
| |/
| +
salesforece <---Revenue
+

will generate growth". If you build a model of this loop you will
likely learn that the loop can in fact generate growth, but it can also
generate decline. The task now is not to validate the model, but to
understand how its structure cause the behavior. If you analyze the
model youll realize that you get growth if each salesperson brings in
more "salesforce-budget" than he costs, because in that case there will
be some money left over with which to hire another salesperson. You get
decline if each salesman costs more than he brings in. At this point,
you have an "answer" that you can create yourself. How you got this
idea is irrelevant -- it doesnt matter whether the idea came via a
model or whether you got this idea while staring at your morning cup of
coffee. And, validating the model is as silly as validating your coffee
cup. The key thing from this point forward is the idea, not the model.


---Final comments--->

I know what youre thinking: Dont you have to have confidence in the
model that you analyze? The answer is no. You can analyze it as a
formal exercise.

I know what else youre thinking: How do you know the generated idea is
right? You dont. And, you never will. You can validate the idea
(i.e. undertake a process to boost your confidence in it). That process
doesnt often require a computer simulation model -- observation is
sometimes sufficient; an experiment is extremely useful.

Jim Hines
jhines@mit.edu
=?iso-8859-1?Q?Jean-Jacques_Laub
Junior Member
Posts: 16
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by =?iso-8859-1?Q?Jean-Jacques_Laub »

Hi Fabian



I thank you for your suggestions.



But I fear that the message I wanted to deliver was not clear enough;
probably too long and too complicate.



I tried to express it more shortly so that it can eventually be better
understood.



What I wanted to say is that it may be more effective when dealing with
people accustomed to take decisions taking into account their experience,
their habits and the habits of others, their own selfishness and others
too, and lots of other ambiguous factors, to express ideas with words
instead of numbers. If you use numbers, people will ask questions about
their validity, and they will have more difficulty implementing them (the
numbers) in an ambiguous world. Words leave more freedom for adjustments;
leave to the listener some possibilities to enrich them with his own
experience, so that he may have the impression to have invented on his own
the policy he will follow.



This is shorter and hopefully more understandable!



Regards.



J.J. Laublé.
From: =?iso-8859-1?Q?Jean-Jacques_Laubl=E9?= <JEAN-JACQUES.LAUBLE@WANADOO.FR>

Allocar, rent a car company
=?iso-8859-1?Q?Jean-Jacques_Laub
Junior Member
Posts: 16
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by =?iso-8859-1?Q?Jean-Jacques_Laub »

Hi John (Swanson)

I am not a consultant, but have been dealing lots of time with consultants.

The reaction of clients about consultants comes from their very defaults.

They first do not take the time to understand what is really the clients
problem, but are too much concerned by looking clever, having worked a lot
on the subject, and using difficult and sophisticated methods to justify
their consulting.

I have the same problem with my associates in my business. I have lately
showed a simple SD model to my brother and he looked very cross, because he
found many occasions to show that the model was not accurate enough, and not
taking into account many other factors.

The object of the model, was too explain the overall importance of
increasing the quality of work in our business.

I was faulty. I should have explained that to my brother using words and not
numbers.

There were no equations any more, no data to justify, and mostly my brother
could add his own words, modify slightly the policy, and above all LOOK
CLEVER then me.

Forgetting his own selfishness and taking into account the clients one, is
one of the mandatory quality of a good consultant.

One could say that you did not make the client participate with the
modelling process from the start, and that you came at the end of the work,
with results that your client did not understand how they have been
generated. But it is not easy to make the client participate, especially if
he takes the decision to buy.

If you take this approach, at least, you avoid the competition and move the
battles field to your advantage.

Regards.

J.J. Laublé
From: =?iso-8859-1?Q?Jean-Jacques_Laubl=E9?= <JEAN-JACQUES.LAUBLE@WANADOO.FR>

Allocar, rent a car company
"Fred Nickols"
Newbie
Posts: 1
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by "Fred Nickols" »

Regarding Jack Homers post below, I can vouch for what Jack says. When I
was head of Strategic Planning and Management Services at Educational
Testing Service (ETS), Jack did some critical SD work for us -- and for
others at ETS later on. I knew enough to know that SD was relevant and
applicable but I couldnt then or now do it myself. So, at John Stermans
recommendation, I engaged Jack. I was right about SDs relevancy and
applicability to the issues we tackled back then and Jack is right about
businesspeople being less concerned with validation than he was -- at least
as far as this businessperson is concerned.

Regards,

Fred Nickols
nickols@att.net
www.nickols.us
"George Backus"
Member
Posts: 23
Joined: Fri Mar 29, 2002 3:39 am

Can SD models be validated without being run?

Post by "George Backus" »

Jim Hines argues that idea generators do not need validation (where I
think we all agree that by validation we mean "valid" confidence rather than
perfect and absolute "truth."). The exercise of a positive loop just tells
how generic parameters affect the output of a system of equations. You are
learning useful generic math to help you have confidence that you do
understand the model behavior when you do run it. By "creating the loop,
we still have no basis to believe the causal ideas we assign to this math
have any more real-world relevance than does "how many angels fit on the
head of a needle?"

I have (had) a (very) young friend who hypothesized that the sun rose
every day - even if it was cloudy. He made a good mental model of the
logic. The theory was clear enough that, with a few parameter assumptions,
you could to even make a mathematical model to describe it. And if we
watched each morning, we could occasionally observe the sun burn-off the
clouds.. But of course, the sun never rises. Never has. Never will. If we
compared the model to data we would have discovered this "truth." And, our
knowledge of the loop math would tell us the "error" was not due to
parameter uncertainty, but rather due to "idea" invalidation. Our
observations and morning "experiment" deceived us.

The scientific method does not prove a scientific theory. It just gives
us confidence in (or occasionally refutation of) the leading "idea"
contender. Our ideas are our hypotheses. Until "validated" by comparing
model results with data, models are mere, self-deceiving, conjecture.
Scientists often have to "correct" the data to take out what the theory/idea
is not testing. Their methods then explain why differences between the
"data" and the model do not (or do) relate to the support (or refutation) of
the hypothesis. Theil and cointegration statistics (among others) allow us
to understand whether the data support (give confidence) in the model (or
force us to rethink the relationships). Predictive perfection is a
red-herring issue where we do need to educate clients, but it is not
relevant to model "validation."

I just dont see how we do anybody any good (other than satisfying
needs for self-deception) by declaring something meaningful simply because
we want (need?) it to be true. Dont we need to test all ideas? In SD, we
RUN models to do that testing. We have no choice but to compare our runs to
the real world (i.e. data). If we dont use that process, how are any
better than a religious cult (e.g., classical economists :-)?



George Backus
Policy Assessment Corporation
14602 West 62nd Place
Arvada, CO 80004
Bus: 303-467-3566
Fax: 303-467-3576
Mobile: 303-807-8579
Email:
George_Backus@ENERGY2020.com
Locked