sensitivity analysis: anyone heard about it?

["Andreas Hoernedal"]

Hello!

I am a Swedish M.Sci. student, currently doing my master thesis on a
simulation project, using the ithink software. The aim of the project
is to give an estimate of the performance of certain technical
systems in a war or crisis scenario, when accidents, sabotage, and
other mishaps are bound to happen.

My part of it, is to find out how one can estimate the sensitivity of
the total systems performance (expressed as a number) to different
aspects of the system, e.g. how fast the repair crew can fix a broken
down sub-system. This sensitivity estimate is then used to guide
trade-offs in resource allocation to the system, e.g. more
equipment vs. more training. Another problem is to find when system
performance changes sensitivity to one or many parameters.
Simulations indicate that system performance reaches a steady state
after some time, depending on certain constraining factors, and that
the magnitude of system performance in this steady state varies
greatly in sensitivity.

My question is: have there been any previous research on this kind of
problem? Does anyone consider this problem worthwhile, possible or
even interesting to solve? Is the solution even easier than i think?
Even harder? I have until now had a rather heuristic approach to this
problem, but I am curious to find out about some more advanced theory
that may yield a solution.

Since interest in deeper research in System Dynamics seems to be
non-existent here in Sweden, I havent yet found anyone with very
much experience or knowledge in advanced S.D. research.

My own field of study has been mostly in the general applied maths and OR
area, especially control theory, optimization, queueing theory, etc.

Suggestions, anyone?

Andreas Hoernedal, Apelbergsgatan 54
111 37 Stockholm, tfn. 08-202859
t92_hos@t.kth.se
If possible, please use MIME-coding.

[George Richardson]

This question is well answered by the System Dynamics Bibliography
available on disk or on line (http://www.std.com/vensim/SDBIB.HTM) for a
small fee.

...GPR

-----------------------------------------------------------------------
George P. Richardson G.P.Richardson@Albany.edu
Professor of public administration, public policy & information science
Rockefeller College of Public Affairs and Policy Phone: 518-442-3859
University at Albany - SUNY, Albany, NY 12222 Fax: 518-442-3398
-----------------------------------------------------------------------

["George Backus"]

Andy Ford (forda@mail.wsu.edu) has produced several papers on sensitivity
using HYPERSENS (a package we developed with his help). It uses
Latin-Hypercubic sampling to make the runs ( I believe you can get this
capability in the spreadsheet add-on packages @Risk and Crystal-ball. The
output can then be feed as input to IThink or other packages.

The inputs changes the model outputs and these results are stored. You can
then perform a regression to determine the impact of the input sensitivity
on the "system" output(s). You can obtain essentially all the information
you want with whatever caveats you need. The statistical nuances, however,
can get tedious and "design of experiment" considerations should be
reviewed. In the HYPERSENS package, the regression and other statistical
information is automatic. The system is using partial correlation
coefficients and these are seen to have some theoretical problems (that we
have not YET seen in practice). See the book "Elements of the Systems
Dynamics Method" for details.

With a thoughtful experimental design(s), you can determine the sensitivity
over the fully realizable range of the model. You can also use the
"experiments" to find the policies most robust to the "other" uncertainty.

If Andys papers dont help, contact me and I will send you our
documentation. (The use of HYPERSENS is a commercial issue and may not be
available to you directly for free. We may be able to link you to a
Scandinavian or other nearby organization who has HYPERSENS and knows how
to use it. You could then take advantage of their version for free - and
we would have no problem with that.)

George Backus
gbackus@boulder.earthnet.net
Policy Assessment Corporation
14604 West 62nd Place
Arvada, Colorado USA 80004-3621
Bus: +1-303-467-3566
Fax: +1-303-467-3576
----------

[jforestr@MIT.EDU (Jay W. Forrest]

The Vensim software includes facilities for doing automatic variation of
all parameters and ranking the results on a specified criteria.


Jay W. Forrester
Professor of Management, Emeritus
and Senior Lecturer, Sloan School
Massachusetts Institute of Technology
Room E60-389
Cambridge, MA 02139
tel: 617-253-1571
fax: 617-252-1998

email: jforestr@mit.edu

[Anthony Gill]

Suggest you look at *Powersim Solver* in connection with sensitivity
analysis. You can contact Powersim via the Phrontis web page or directly
through www.powersim.no

Prof Pal Davidsen at the University of Bergen in Norway *may* be a
source of information.
Good Luck

Anthony Gill phone: +44 (0)1295 812262
Phrontis Limited
Beacon House fax: +44 (0)1295 812511
Horn Hill Road
Adderbury email: t2@phrontis.demon.co.uk
Banbury
OXON. OX17 3EU URL: http://www.phrontis.com/
U.K.

[Jose J Gonzalez]

Jay W. Forrester wrote:
>
> The Vensim software includes facilities for doing automatic variation of
> all parameters and ranking the results on a specified criteria.
>
... and so does the Powersim Solver too
--
Prof. Jose J. Gonzalez
Dept. of Engineering
Inst. for Information Technology
Agder College
Groosvn. 36
N-4890 GRIMSTAD, Norway
Phone: +47 37 25 32 40 (office) Fax: +47 37 25 31 91 (office)
+47 37 04 70 29 (home) +47 37 04 70 74 (home)
+47 92 09 09 39 (cellular)
Email: Jose.J.Gonzalez@hia.no

[Stephen Wehrenberg]

Don ... the @RISK package does just what you say ... available from
Palisade ... see their web site at

http://www.palisade.com

Now if it could only interface with an SD modeling package ... sigh.

Steve

--
Stephen B. Wehrenberg, Ph.D.
Chief, Forecasts and Systems, US Coast Guard;
Administrative Sciences Program, The George Washington University;
wstephen@erols.com
Empowered at birth.

[James Melhuish & Jennifer Shearm]

Don,

I read in the System Dynamics list of your interest in graphical output
of sensitivity simulations. I work at Ventana Systems and the software
we make (Vensim) has the ability to do Monte Carlo and Latin Hypercube
sensitivity simulations and display the resulting output in graphical
form, either by multiple individual traces, or by confidence bounds.
Contact us or look at our web site http://www.std.com/vensim if you
have any questions.

James

--
James Melhuish
Ventana Systems, Inc.
82 Harvard Street, Newtonville MA 02160, USA
phone: 617 964 8621 email: jjms@world.std.com

["Douglas Lanz"]

Hi;
I thought Id reply to this one, as it touches on some stuff we have
incorporated in our industrial process control application.

You will find just checking the ideas behind Statistical Process Control,
ie: the Xbar and Sigma charts do a similar thing, though they call these
"control limits". We use a chart of our predictions combined with a chart
of our residuals to indicate this and some modified SPC control limits
(even though in our application they are confidence intervals as often as
not), but it is not as good as Id like. No graphing package yet gives
shading, which Id like to use to indicate confidence in a prediction, so
one just has to plot the %age lines in dotted form and the prediction in a
heavy line to get the idea across.

Others would find it useful if you got enough replies to post a summary of
what people are doing in this area.


Douglas P. Lanz,
Douglas_Lanz@mindlink.net
Advanced Process Control Ltd.,
#208 - 3190 St. Johns St.,
Port Moody, B.C. V3H 2C7
Tel: (604) 469-0092 Fax: 469-0071

["George Backus"]

For Don Taylor:

If you have performed a "full experiment" that changes all the salient
input variables (including those that switch in alternative structures),
then you can produce a variety of statistical plots. Many times the
distribution of the output uncertainty for a specific variable is not
gaussian. Nonetheless, you can produce parametric and non-parametric
confidence intervals around the mean and reference results (they can be
different). The non-parametric analysis simply counts the runs and produces
the 1, 5, 25, 50 75, 90, 95 and 99 percent (dynamic) confidence intervals,
for example. You can do the same parametrically if you assume (or test for)
a gaussian distribution. (As noted in an earlier note. Andy Ford describes
all this in his work using our HYPERSENS package.)

One can also make a plot of the partial correlation coefficients over time
for those input parameters contributing a cumulative 99% impact on the
uncertainty, for example. You can them see how certain parameters affect
results with a different intensity over time (and behavioral mode). Many
parameters only come in to play for small intervals/conditions.

A scatter plot of results is actually one of the most informative. When you
notice an "unusual" run, you can retrieve it and dissect it to see what
happened. Often you realize that a structure is faulty or that a mechanism
must be missing. In some cases the effect is real and a specific set of
input conditions knock the system into a new operating regime. You then
either need to control the system to stay out of that mode or take
advantage of the situation by "forcing" the system to stay in that
operating mode (if that is a more preferable state).

Lastly, the scatter plot (if its in lots of colors) can also show if chaos
kicks in anywhere and what its impact on the over system performance is
(often negligible for the major output conditions of interest). Often, but
not always, this chaos is unrealistic. The real world has stayed away from
it by evolutionary system changes or the parameters really dont vary by
the degree necessary to cause it . In the later case of input parameter
variance, a re-think often will indicate the that some of the input
parameters are not independent and that the selection of the parameter
values must come from a sampling of the interdependent ellipsoid-space
where one sample from it tells the values of multiple related input
parameters. (Andy Ford again discusses this issue.) Most often the
"experimental" chaos we see is due to this incorrect assumption of input
parameter variance independence.

George

George Backus
gbackus@boulder.earthnet.ne
Policy Assessment Corporation
14604 West 62nd Place
Arvada, Colorado USA 80004-3621
Bus: +1-303-467-3566
Fax: +1-303-467-3576

[Jim Hines]

This is in reply to Don Taylors recent posting about confidence bands.

Vensim will produce a pleasing visual display of confidence bands if you
specify the distributions for your uncertainty around parameters.

Jim Hines
LeapTec and MIT
JimHines@interserv.com

[saeed@ait.ac.th (Khalid Saeed)]

Besides having a software capability to implement sensitivity analysis, you
also need to carefully design an experimental process to conduct and
interpret it. Each prameter tested must be given a behavioral/policy
meaning, sensitivity functions must be carefully defined on the basis of
the purpose of the model, results of the analysis must be interpreted in
terms of their behavioral/policy implications.

Khalid

Khalid Saeed
Professor and Program Coordinator
Infrastructure Planning & Management Program
School of Civil Engineering
ASIAN INSTITUTE OF TECHNOLOGY
P.O. Box 4, Klongluang, Pathumthani, THAILAND 12120
phones: (66-2) 524-5681, (66-2) 524-5785; fax: (66-2) 524-5776
email: saeed@ait.ac.th

Visit our program website at: http://www.ipm.ait.ac.th/

["Richard G. Dudley"]

Don:

To date I have mostly played, rather than worked, with models (Stella 4)
but I usually put in a random plus or minus (or random value drawn from
a normal
distribution) around key inputs. Sensitivity analysis will tell you how
a system will respond to fixed changes in a input variables, but random
values, as you suspect, make the system more realistic. It is not hard
to set this up so the random inputs can be turned off and on. For
example
fish reporduction in a give year can be "predicted" by a given curve,
but it really has a very large random variation which is unlikely to be
fully explained by added variables.

I assume that some packages can automate / standardize / formalize /
simplify this same approach. But what I mean here is not changing input
x by +/- 1,2,3,4,5 for the run of the model but rather for each
iteration within the run x has a random value drawn from a distripution.

cheers,

ps: is your pdx.edu Portland State?
--
Richard G. Dudley
rdudley@indo.net.id
http://home.indo.net.id/~rdudley

["George Backus"]

I would suspect that all the packages with sensitivity analysis
capabilities perform actual sampling via a user defined (or a standard form
- normal, uniform, exponential, discrete, etc.) distribution.

One neat method of sensitivity analysis has been left out of this
discussion. At the 1994 (Scotland) International SD Conference, Henry
Neimeier of the Mitre Corporation presented a paper on "Analytic
Uncertainty Analysis." He simply assumed that the "real" parameters
considered in a model have real bounds (go between X and Y) rather than +/-
infinity. He then showed that the distribution could be approximated
adequately by a Beta distribution and that the full uncertainty for all
variables/parameters could be determined in a single model run! Quite a
time saver. (Other work in non-SD industrial engineering - Bruce Schmieser
of Purdue University - showed that if you have the mean, variance and
skewness matched, the type of distribution does not significantly change
the analysis. This gives more strength to the adequacy of using only the
Beta distribution.) Henry was working on a book presenting his methods but
I have heard nothing, yet. Maybe he will let the SD community use his
draft.

George Backus
Policy Assessment Corporation
14604 West 62nd Place
Arvada, Colorado USA 80004-3621
gbackus@boulder.earthnet.net
Bus: +1-303-467-3566
Fax: +1-303-467-3576

[Jian Li]

> This is in reply to Don Taylors recent posting about confidence bands.

SENECA2.0 can do the same thing.

John Li
PhD
jlee@kuentos.guam.net