Forecasting accuracy

This forum contains all archives from the SD Mailing list (go to http://www.systemdynamics.org/forum/ for more information). This is here as a read-only resource, please post any SD related questions to the SD Discussion forum.
Locked
Anthony Gill
Junior Member
Posts: 7
Joined: Fri Mar 29, 2002 3:39 am

Forecasting accuracy

Post by Anthony Gill »

My point was that you cannot forecast the future with any degree of
accuracy especially when you are talking about 10 or more years ahead.
Yes you may be able to correctly forecast a few months ahead.

We can begin to understand the behaviour of systems of TODAY where we
have good tools to manage complexity. The future is another story. What
do we know about incipient discontinuities? Are our paradigms
sufficiently adaptive to deal with situations that have yet to manifest
themselves? In an increasingly global society, are we not prisoners of
our own world views much of the time? Social and political dimensions to
model building are often excluded or at best ill defined with resultant
consequences. Then we have to deal with how different people, even
within the same community (unless they are overcome with group think)
interpret their worlds.

Take the Year 2000 problem. Some airlines plan to ground their aircraft
on the magical day. Why? The uncertainty of managing complexity is too
high. Lives are at stake.

If the UK Treasury gets it wrong at worst lifestyle is at stake. Also
how do you know whose forecasts to believe? Who decides this? The UK is
not alone. Every country has the same problem with Treasury models.
Models that ignore globalization are likely to be less useful. Was it
the Conant-Ashby theorem that stated that a regulator of the system
needs a good model of the system? For a SD model (or any other model) to
be a good model of the systems it must reflect the behaviour/working of
the system against a specified purpose. If the purpose is predictive
with a *high degree of accuracy for a decade or more ahead* then I
believe that the modeller has unreasonable expectations of SD (or any
other modelling approach).

Scenario planning (yes, SD models are an excellent tool to support this)
gives a reasonable way of trying to understand the impact of possible
futures but does not try to predict the future. This is the way to
tackle such problems. And forget about confidence limits/accuracy etc.

What about having to take decisions for long term projects? I accept
that managers and politicians have to take decisions but as always there
is are risks involved. That is where risk assessment comes in. Once a
long term project has started we better be aware of feedback loops
that negate the rational of the original decision and be prepared to
cut our losses.


Tony Gill phone: +44 (0)1295 812262
Phrontis Limited
Beacon House fax: +44 (0)1295 812511
Horn Hill Road
Adderbury email:
TonyGill@phrontis.com
Banbury
OXON. OX17 3EU URL: http://www.phrontis.com/
U.K.
Alex.Rodrigues@dsi.uminho.pt (Al
Junior Member
Posts: 4
Joined: Fri Mar 29, 2002 3:39 am

Forecasting Accuracy

Post by Alex.Rodrigues@dsi.uminho.pt (Al »

The accuracy with which an SD model can produce forecasts is an =
important issue -- both scientifically and commercially. However, one =
could say that this is yet an issue that remains to be clarified. For a =
certain model of a certain system, who can demonstrate, either logically =
or empirically, that forecasts within an x% of confidence band can be =
produced?=20

For commercial purposes, accurate forecasts are highly desirable. The =
manager relying on the model will use its results as the basis for =
decisions and negotiation -- most, if not all managers would like to =
have a crystal-ball. Point-prediction is particularly desirable in =
life-cycle systems, like projects. Here, it is often the final =
cumulative result of a certain dimension of behaviour that matters, as =
opposed to the way such behaviour pattern fluctuates over time. For =
example, what will be the final project cost? (regardless of how it =
evolves)

As the user-audience of SD models (both, modellers and end-users) =
increases, a good, clear, and shared understanding of the issue is =
critical for the methodologys credibility (and for particular =
applications of the methodology).

It is a long time now since Forrester stated that an SD model should not =
be used for point-prediction, but rather to identify the likely changes =
in the systems *mode of behaviour* and the *likely* magnitude of these. =
This "softer" perspective on quantitative forecasting has since then =
been advocated (e.g. Barlas 1987).

Obviously, a model will produce accurate forecasts as far as its =
underlying assumptions occur in the real world (including both =
parameters and structure). In other words, the models results will be =
accurate forecasts as far as the world will behave as represented in the =
model. Since an SD model captures and simulates, in many cases, =
managers taking decisions (according to policies), it is also obvious =
that in the real world managers will also have to take the same =
decisions (while following the same policies). Otherwise, the models =
forecasts may not be "accurate" (though they still may be, either by =
chance (i.e. by the wrong reasons), or because the difference happened =
to have no impact).=20

Proving that a model can produce accurate forecasts, a priori, is very =
difficult. First, one would need a considerable amount of time and =
effort to develop the model, produce the forecast, and then wait for a =
few years to check the accuracy... One would also need a high number of =
"similar" test-cases. Then, one would need to make the model totally =
transparent to the user, and subject it to thorough validation to ensure =
that "the right behaviour was reproduced for the right reasons". =
Finally, the dominant feedback structure that drives the system under =
study would have to remain constant over all those years, so that the =
model would also remain valid.

There are a few issues that follow the problem: (1) what are the =
critcial factors that affect the potential accuracy, and how? For =
example, the level of aggregation, accuracy of the estimated exogenous =
parameters, the type of system (steady-state, life-cycle), the =
environment, etc.; (2) what type of accuracy: point-prediction (e.g. how =
much staff in year 2010)? cumulative behaviour (total cost in year =
2020)? pattern characteristics (e.g. phase, amplitude)? There are =
certainly some types of forecasts easier to achieve accurately than =
others.

SD models are "models of social systems" and as such their object of =
study is complex and of large magnitude; it is generally an opened =
system highly interactive with the rest of world; and it is often =
vaguely understood (a good reason to build the model in the first =
place!). In many cases, there may be a low probability that the model: =
(1) will capture correctly how the real system works (i.e. it is the =
right "engine"), (2) it will not miss unforeseen events which will =
impact behaviour (i.e. no surprising external "shocks"), and (3) will =
consider assumptions that will occur and remain constant over the =
time-frame of the study (i.e. the system "working laws" will not change =
themselves).

It is unlikely that one succeeds in achieving accurate forecasts without =
acknowledging explicitly the factors that constraint the accuracy, and =
include these as part of the analysis. Perhaps what is important is to =
work on studying how the factors mentioned above affect and restrain =
accuracy. Such study would allow the modeller to produce a confidence =
band *as a function of the constraining factors*, and make this explicit =
to the user/manager. In this way, the manager will know what to expect =
from the model, and why. If the manager has scope to act on the =
constraining factors, then perhaps the band can be reduced. While not a =
perfect crystal ball, the "imperfection" of the SD model is acknowledged =
and justified, preventing false promises and false hopes. =20

Either as an alternative to an intuitive mental model, or as a =
complement to existing models (e.g. Rodrigues et all 1994;1997), the SD =
model will certainly prove useful with its unique perspective.

Alexandre Rodrigues

__________________________________
Alexandre J G P Rodrigues

Departamento de Sistemas de Informa=E7=E3o
Escola de Engenharia
Universidade do Minho
4800 Guimar=E3es
Portugal, EU

Department of Information Systems
The School of Engineering
University of Minho
4800 Guimar=E3es
Portugal, EU

Tel.: +351 53 510 149
Fax: +351 53 510 250
Email: Alex.Rodrigues@dsi.uminho.pt
Web: www.dsi.uminho.pt
j.swanson@sdg.co.uk
Junior Member
Posts: 5
Joined: Fri Mar 29, 2002 3:39 am

Forecasting accuracy

Post by j.swanson@sdg.co.uk »

My thanks to those who offered such thoughtful responses to my earlier=
=20
posting.
=20
Bill Buchanan asked Transport of what? - its about movements of=20
people and freight in an urban area in the UK.
=20
Bill Braun asked about the confidence limits. None were set - just=20
the text I quoted. I suspect all they really meant was very=20
accurate.
=20
What I was interested in was the apparent contradiction between the=20
requirement for a policy and/or strategic analysis tool and the=20
insistence on high precision point forecasts. This is not uncommon in=
=20
my experience (in transport), and it can be hard to explain to people=20
that it may not be a sensible thing to ask for - it is, after all,=20
something they feel they need. Of course, there are plenty of people=20
prepared to say sure, we can do that - the chances of anyone actuall=
y=20
checking in 2011 are pretty slim, after all.
=20
John Swanson tel: +44 (0)171-919-8500
Associate fax: +44 (0)171-827-9850
Steer Davies Gleave
32 Upper Ground
London
SE1 9PD
From: j.swanson@sdg.co.uk
www.sdg.co.uk
=20
j.swanson@sdg.co.uk
Junior Member
Posts: 5
Joined: Fri Mar 29, 2002 3:39 am

Forecasting accuracy

Post by j.swanson@sdg.co.uk »

=20
I have been asked to consider developing a strategic model to look at=20
transport policy. The brief asks for a "Stategic Assessment Tool" to=20
"assess the consequences of alternative transport and development=20
strategies". So far so good. Then it says the model must be able to=20
"predict accurately (within narrow confidence limits) to 2011."
=20
I wonder what the SD communitys advice would be to someone with that=20
ambition?
=20
John Swanson =20
Associate =20
Steer Davies Gleave
London
=20
From: j.swanson@sdg.co.uk
=20
"Antonio Barrsn Iqigo"
Junior Member
Posts: 5
Joined: Fri Mar 29, 2002 3:39 am

Forecasting accuracy

Post by "Antonio Barrsn Iqigo" »

Keith Eubanks
Junior Member
Posts: 2
Joined: Fri Mar 29, 2002 3:39 am

Forecasting Accuracy

Post by Keith Eubanks »

"I have been asked to consider developing a strategic model to look at
transport policy. The brief asks for a "Stategic Assessment Tool" to
"assess the consequences of alternative transport and development
strategies". So far so good."

So far not so good. No modeling endeavor should begin as "a strategic model
to look at transport policy" but rather as an exercise to understand and
emulate a mode of behavior. One does not build models of policies but
rather models of systems that control specified behaviors. Once a behavior
mode has been statisfactorily replicated, a model can then be used as a
platform for testing how various policies (differring structures and
parameters) might affect the behavior under study. Discussing accuracy
without a clear and focused objective (behavior mode) is a little premature.

"predict accurately (within narrow confidence limits) to 2011."

This statement is somewhat incompatible with the first. If the purpose of a
modeling project is to study how various "policies" might affect a
particular behavior (say passenger miles driven within the USA), then
"predictive accuracy within narrow confidence limits" is probably
unnecessary and, in fact, a little misleading. By definition, the modeler
has assumed that behavior will vary with differing policies. Given this,
what is the model predicting: policies or passenger miles? Every
prediction of passenger miles is founded on a set of assumptions on the
policies and structures that drive passenger miles, both in the past and
future.

The value of such an undertaking would be learning how the system that
controls passenger miles driven (for example) is affected by the policies in
question, not a point prediction in 2011. The behavior of the real world
will be driven by what policies and structures are actually in place.

Making a point prediction is a different undertaking than asking what would
be a good policy to produce a "desired" behavior. All predictions are based
on a set of assumptions of what policies and structures are actually in
place and how these might evolve over the time frame in question. This is
not a "what if" animal but rather a "I think the world works and will
continue to work this way" beast.


------------------------------------
Keith Eubanks
From: Keith Eubanks <
Keith.Eubanks@PA-Consulting.com>
PA Consulting Group
Pugh-Roberts Practice
41 Linskey Way
Cambridge, MA 02142
(617) 864-8880
Bill Braun
Senior Member
Posts: 73
Joined: Fri Mar 29, 2002 3:39 am

Forecasting accuracy

Post by Bill Braun »

This seems to be a project specific question that is in line with the
recent discussion on Validity of Models. Reading through that thread might
offer some preliminary insight. Your question appears to reflect that you
have accepted their requirements as the defacto framework for thinking and
responding. Is there any opportunity to reframe their requirement or at
least open some dialogue? Their requirement may be founded on their
assumption that simulation is capable of and/or routinely delivers accuracy
to the degree they would like to have available. As a matter of interest,
what are the confidence limits they set forth?

Bill Braun
From: Bill Braun <medprac@hlthsys.com>
"George Backus"
Member
Posts: 33
Joined: Fri Mar 29, 2002 3:39 am

Forecasting accuracy

Post by "George Backus" »

John Swanson wanted to know about forecasting with SD. We have hit this
topic on the server before but .....

Although forecasting is far from the best use of SD, it would appear that it
can
be as, if not more, accurate (valid) than any alternative. The biggest
problem is the inclusion of additional dynamics (equations) not relevant to
the "real" problem. SD is designed to solve a system problem and NOT to
broadly model a system. A SD model needs a defined purpose that determines
what aspects of a system to include in model -- George Richardson or John
Sterman, this is an invitation for you to get on your respective soap
boxes. These superfluous mechanisms (equations) simply add confusion and
complexity.

Second, the remaining, original-model, parameters can become distorted as
they are forced (statistically estimated) to make sure the model exactly
produces historical values. One has to really understand these
distortions -- whether it means there is an important missing mechanism that
is incorrectly "subsumed" in existing structure or whether the parameters
being adjusted are being set outside of a reasonable range because of bad
system definition (you are modeling a different system than that associated
with the data) or a bad system definition (you made a model that reproduces
the basic dynamics portrayed in the data, but you have some bad hypothesis
about what really is causing the historical response).

There is an econometric field of literature called cointegration that deals
with these issues. It provides the best econometric fits and is based on DQ
logic. The problem is that cointegration does not allow (assume) SD causal
logic (only what is called Granger Causality - that may or not be related to
SD causality.) In that SD attempts to add (force) causal logic, but can use
the cointegration methods, it provides a better forecast for the "closer to
right" reasons (it is seldom blind-sided).

As ridiculous as it may sound, errors out 10 years (looked backed upon
posthumously) are often in the 0.1% range (but usually only for those
variables of initial interest). These error bounds are also tight because
strong negative feedback is locking in the results. (Validity of the error
bounds is a different -- sore -- point.)

It should be noted that there is some mutual exclusivity here. A good
forecasting model is generally a BAD, BAD, VERY BAD strategic planning
model. It
contains so much "garbage" that a client cannot understand the system.
(Look at Erling Moxnes recent article on the massive misunderstanding when
just one trivial loop dominates.) We find clients attributing all sorts of
mythical reasons for the system behavior (a focus on some detail dear to
their heart) and cannot they be moved from their positions. If you try to do
a sensitivity analysis to prove the point, then you no longer get the "exact
forecast" and therefore "prove" to them their minor equation is important.
(A million $ change in a 10 billion dollar company is important to the
manger looking at it. The idea that this is only a 0.01% change just invokes
a quizzical look of "Are you so rich that a million dollars does not
matter?").

We find that the "forecasting" models can be "used" for scenarios, but even
this is problematical because you have to post mortem "correct" for the
parameters that have been distorted in the forecasting exercise. For
example, the model may show a 20% change via some policy, but you have
to argue with the client that the "real" answer would probably be,say, less
than the 20%. This is wonderful for generating massive mental chaos in the
client and revoking the "credibility" generated by the fictitious
"forecasting"
validation.

Bottom line. In my experience, make two "related" models (one for the
best --mindless -- forecast available and one for the most defensible
strategic analysis available) or only make a planning/strategy model OR a
forecasting model. But dont make a Frankenstein that attempts to be all
things to all clients for all times. Have a Happy Halloween....

G

George Backus, President
Policy Assessment Corporation
14604 West 62nd Place
Arvada, Colorado, USA 80004-3621
Bus: +1-303-467-3566
Fax: +1-303-467-3576
George_Backus@ENERGY2020.com
"George Backus"
Member
Posts: 33
Joined: Fri Mar 29, 2002 3:39 am

Forecasting accuracy

Post by "George Backus" »

Anthony Gil is concerned about the use of the forecast. This may be the most
important issue surrounding forecasting models. Most everyone who is
unfamiliar with SD and feedback thinks a forecast is a objective item; "just
give me the truth." How the forecast is to be used is what determines how
it will be produced and its biases. That use also determined the credibility
(facts are irrelevant), what can be considered (e.g., economic growth), and
what cannot be considered (e.g., benefits to one segment of society at a
severe cost to another) .

Because the forecast implies certain ramifications, viewed with horror by
one constituency and delight by another, it is a political (policy)
instrument in and of itself. Therefore, the "used" forecasting model is
often very
simple (defendable to the extent the finger can always be pointed elsewhere)
and purposefully wrong (it gives the required political spin but CANNOT be
used as the basis of the final decision). All the social/psychological
concerns Anthony raised are precisely why there can be no forecasting tool
acceptable to anyone but the decision maker dictating what that forecast
should look like.

Nonetheless, if the system is well defined and in tight feedback, and
detailed dynamics are only required for the first two quarters, then the
average forecasting ability of a SD model can be quite good over even a long
time horizon. That is, its error bounds although larger than hoped, do not
diverge and (retrospectively) has smaller errors than alternative
methodologies.

Relative to Jay Forresters comments, this does mean that the forecast is
not overly useful other than providing the REQUIRED but phony confidence
test for the policy model. It also means that the forecast is design to be
self fulfilling and the policy model has as its job the task to uncover the
folly of that position.

I do not fully disagree with Jay on the "momentum" comment, however. While
the
forecast is based on levels that contain past information, the formulation
of the forecasting (portion) of a model can allow for what appears
externally as bifurcations -- from loop dominance shifting. These shifts
are often founded in technological changes with positive feedback in market
acceptance. While uncertainty of timing for these "bifurcations" can be
problematic, events occur quickly (exponential growth driving a switching
decision) and timing errors are seldom off more than a quarter.

Still, the fact remains that a model that is "distorted" to provide a
prefect forecast is very likely to have compromised the model structure
that would have allowed it to be a valid (accurate) policy tool.
Unfortunately, if you want to work in the top part of the company, ideals
must erode, and a unholy compromise of forecasting and policy models seems
to be required. The justification to this sin is that with it you may
actually use those valid SD parts to some good end, otherwise (it seems to
me) that you are considered just an "interesting but not useful" prophet
ranting in the wilderness.

G

George Backus, President
Policy Assessment Corporation
14604 West 62nd Place
Arvada, Colorado, USA 80004-3621
Bus: +1-303-467-3566
Fax: +1-303-467-3576
George_Backus@ENERGY2020.com
C Thomas Higgins
Junior Member
Posts: 6
Joined: Fri Mar 29, 2002 3:39 am

Forecasting Accuracy

Post by C Thomas Higgins »

The following is a required disclosure for a certified public accountant
when reporting on the presentation of a forecasted or projected financial
statement.

"there will usually be differences between the projected/forecasted and
actual results, because events and circumstances frequently do not occur as
expected, and those differences may be material."

The accounting literature makes a distinction between forecasts and
projections. A forecast is the presentation of financial information using
managements best guess about future events. A projection is the
presentation of financial information under one or more hypothetical
assumptions (e.g. including a possible but not yet obtained bank loan).

The above disclosure amounts to the accounting profession saying that you
cannot rely on forecasts to predict the future accurately. Yet the
financial community continues to produce and rely on forecasts. Why? The
SBA requires candidates seeking loans under their programs to prepare
forecasts and submit them with most applications. Silly? Perhaps not.
Through my observation of financial forecasting as a social phenomenon I
have concluded that there are at least two good reasons to prepare
forecasts. Neither of which is linked specifically to the accuracy of the
prediction.

First the process of preparing the forecast provides an intimacy with the
financial information that cant be obtained otherwise. Second the
imposition of the formal forecasting structure affects the outcomes of
actual events.

Consequently my interest in SD is for the rich framework it provides in
understanding relationships between financial and social behavior. And for
its ability to connect (be intimate with) financial and non financial
factors). If the result is better accuracy in prediction - thats OK but
dont hold your breath. I believe the real benefit is to involve the players
(all participants in the system) in the process. Then the more
sophisticated the tool the more the intrinsic the benefit of well designed
models.

I dont hold much hope for high predictability on specific data points in
social economic structures. Other than those that are obvious. Such as
rent will be the same for the term of the lease. Thats like saying next
summer will be warmer than next winter. The value of SD seems to me to be
in doing it more than in what it does.

C. Thomas Higgins, CPA
loscann@telebyte.net
Locked