Robust Analysis

[Barongo]

Hi
I have a research interest in robust analysis, specifically in regard to
policy recommendations in SD models. I have tried to look around in most
SD literature that I can access and in the System Dynamics Bibliography,
(version 95), I cant find some previous related work.
Is there some one who knows of something done in this direction or could
point me to the relevant literature? Most if not all I have come across
concerning robust analysis (robustness) are in the fields of
Optimazation, Econometrics and Statistics addressing the issue of
parameter estimators.
Your advice and/or direction will be highly appreciated.

--
Mike
Mikeb@ifi.uib.no

[Tom Fiddaman]

There are some useful links to papers on model validity, including
robustness, in Liz Krahmers guide, "How to Conduct a Model Critique," which
I highly recommend. Theres a link to the article on my web pages at:

http://home1.gte.net/tomfid/sdbookmarks.html#Pubs

You might also look over Desert Island Dynamics, just above in the listing.
The section on conceptualizing, formulating, and validating models lists
some useful articles.

In the SD tradition, robustness typically refers to (1) the ability of the
model to behave plausibly when subject to parameter changes, external shocks
or extreme conditions, or (2) the ability of policy conclusions to withstand
alternate parameter or structural assumptions. Jay Forrester describes the
first sense of the term (though I dont think he actually uses it) in
Industrial Dynamics (Chapter 13.5).

Most if not all I have come across
>concerning robust analysis (robustness) are in the fields of
>Optimazation, Econometrics and Statistics addressing the issue of
>parameter estimators.

My guess is that the many of the papers youre encountering are focusing on
the second sense of robustness in a rather narrow way, e.g. parameter
estimates that are insensitive to distributional assumptions or optimal
solutions that dont lie on a spike in the payoff surface.

SD modelers have historically emphasized the first sense and criticized
econometric and analytical optimization models for lack of robustness of the
model. You cant say much about the robustness of policy conclusions without
first examining the model. One (or maybe all) of my former instructors liked
to tell about an econometric model of the leather market that conveniently
turned hides back into cows when subjected to certain inputs. Many
energy-economy models use price elasticities that imply that, to eliminate
US oil imports, oil prices would have to rise so high that oil expenditures
would consume the entire GDP, even in the long run (see John Stermans
dissertation). William Nordhaus model of the economics of climate change
contains an infinite sink for CO2. I could go on and on...

SD models are not immune to problems. In Morecrofts replication of
Forresters classic Market Growth model (and in the original, I believe)
theres no first-order control on the stock of production capacity on order,
which can go negative as a result. In the World3 model, economic output
takes several years to fall to zero even if the entire world population is
exterminated at once. (I think both models are on my web site if anyones
interested).

When faced with a new model, I usually visually inspect it for features that
can lead to problems, like lack of first-order negative feedback on the
outflow from a physical stock or table functions that exceed their input
range. Then I change the time step and integration method to see if the
behavior is a numerical artifact. Next I bang on the model by making large
parameter changes or subjecting it to STEP or PULSE test inputs.

In most software you can do multivariate sensitivity analysis and look at
the output for anomalies. In Vensim, you can use the optimizer to identify
sensitive parameters, or to actively search for parameter combinations that
cause undesirable behaviors (like negative inventory).

Vensims Reality Check language makes it easy to automate many tests of
robustness (e.g. "no workers means no production") and check them every time
a model is run, though to be honest I mostly rely on manual methods. You can
also use command files to automate a suite of tests to be run on a model.

A useful (but time consuming) approach is to convert a model to a game and
have a bunch of people try to "beat the system". It may take infinite
monkeys a long time to write Shakespeare, but a few players can often trash
a seemingly robust model in a few hours.

The point is not so much that defects exist. The goal of making a model
robust to all possible conditions is often in conflict with other goals like
preserving simplicity. Some kinds of robustness problems may not be relevant
given the purpose of the model.

Whats dangerous is that authors and users are often unaware of glaring
weaknesses. Levinthal & Marchs model of adaptive organizational change has
been cited about 40 times, without anyone ever mentioning that agents in the
model learn better in a noisy environment than a deterministic one, or that
some characteristic behaviors of the model are simulation artifacts.

Hope this rant is helpful.

- Tom

****************************************************
Thomas Fiddaman, Ph.D.
Ventana Systems http://www.vensim.com
34025 Mann Road Tel (360) 793-0903
Sultan, WA 98294 Fax (360) 793-2911
http://home1.gte.net/tomfid/ tomfid@premier1.net
****************************************************

[Tom Fiddaman]

By pure bad luck, Liz Krahmers guide to model critiques moved yesterday.
Ill update the link on my web pages soon, but in the meantime you can get
to it directly by:

http://jdmacomber.mit.edu/www/howto.htm

Thanks to Mike Barongo for noticing.

(Desert Island Dynamics is still in the same place)

- Tom

****************************************************
Thomas Fiddaman, Ph.D.
Ventana Systems http://www.vensim.com
34025 Mann Road Tel (360) 793-0903
Sultan, WA 98294 Fax (360) 793-2911
http://home1.gte.net/tomfid/ tomfid@premier1.net
****************************************************