In response to John Noble;
Our group, for right of for wrong, has used SD extensively for pin-point
forecasting in the energy industry for 20 years. We spend a great deal of
time going head to head with econometric forecasts and their entrenched
staffs. So far we have always "won" against "classical" econometric efforts
for mid and long-term analyses. ("Classical" is a critical distinction that
I will clarify later). In our case, mid-term starts three years out. The
errors for a few of our models, over a ten year period, (sit down at this
point) have typically been in the tenth (singular) of a percent absolute
error range. By "ten-years" I mean, "What was the error in the forecast
after you wait ten years and look back on how you did?" In most cases we
cannot check that far back because "historical memories" get fuzzy.
Nonetheless, we can show that the mid-term-plus results tend to be an order
of magnitude better than the alternative models (engineering, input-output,
optimization -- the worst, accounting, and econometrics.)
We find there are no right reasons for this and many WRONG ones. But first
I must digress. When we took reasonably complex models and let the
parameters vary over their range to obtain the best fit, we did not do
better than the econometric models for that very reason that we were
treating the model as a "huge regression equation." The problem was, of
course, that it was a huge, non-linear, multi-dimensional "regression
equation" and only very sophisticated methods that "rewrite" the SD model
in "standard regression forms" even begin to do the job validly. And to
say "validly" denies truth and to say "adequate" would imply some heroic,
if not foolish, judgment. If the forecasts of the model fell within the
values of history, these parameter fits did well -- just like any
semi-reasonable, simple, econometric equation would. If the model was
going to go into "new territory," our experience was that the "parameter
fitting" process gave "explainable" results easily refuted by experts.
The trick was to take the "conventionally" parameterized model and (please
sit down again) perform "extremum tests" from SD 101. This showed us what
caused the model to really deviate from "reasonable" results. Now "what
are reasonable results?," you should be saying as you stand up again. They
are the results from other non-SD models. These results are "reasonable"
because they are often scrutinized by others and modified AS NEEDED to give
a consensus agreement on what will be. Rev. J. Sterman showed many years
ago that the "official" forecasts were just exponential smooths of the
"more correct" marginal forecasts (called adaptive expectations in the
other thread going on in the SD server.)
If our forecast was different, we could ask "why?" If the other forecasters
had a GOOD reason for there judgments, as they often did, it meant we had
missed a structure in our model important to the new conditions, but not to
the old. If they had a bad (usually paradoxical) reasoning, we could look
in our model to see if we were equally duped. (We often were by "current
wisdom" and needed to throw out those structures and rethink our
understanding.
And now a diversion to the diversion. The duping came from assuming we
"knew" the system and did not really need to do historical validation to
verify our very blind and false assumptions. George Richardson has spoken
to this point previously on the SD server, but still many seem to believe
in personal omnipotence. I agree there are new "things" in the future, but
I can only use what I understand from the past, even if by analogy, to
grasp them. More importantly, if I cant understand the past, I surely
cannot claim to understand the future. The future came from levels
accumulated from the past. The processes that made those accumulation dont
just disappear. And those levels are usually "the problem." The forces --
structures-- that made the levels as they are, usually act to preserve
them. I, for one, have not been able to determine how to change the
results of the past unless I understand how that past made the present.
Mark twain said, "The easiest person to fool is oneself." Historical data
is the only unbiased, though still inaccurate, critic that I can trust to
keep me in touch with reality But I digress...
When we saw dumb behavior modes during the extremum tests, it simply meant
we missed important feedback structure or miss-specified others. The more
we had a one-to-one correspondence among the structure, the experts
stories, and history (even if just analogous history in another field), the
better the model held its own. We then could make the simple parameter
estimations on the parts of the model where we could validly isolate that
structures were we felt that we understood the input, output and the
feedback adequately plus understood the limitations of the always
simplified structure and its by-design limited impact on the rest of the
model. These "regressions" made the econometricians happy but it was the
new feedback structures that counted.
Now the real answer to why the model made "good forecasts" is that once you
have a system that counter-responds to any changes no mater how dramatic,
you have a system that probably reflects a quite unique system solution.
Since the world exists, it usually finds these structural solutions. As
our forecasts showed, for example, the actual demand growth, the authors
of the "failing" forecasts finally concluded that it was because they
missed the feedback in price and the economy. Andy Ford and John Sterman
have written extensively about this. The boundary of the problem (the
system) in terms of what feedback had to be in the model to handle the
"hypothesized" future extreme conditions was found to be the simple key to
the not-so-modest accuracy that the SD model enjoyed. (I still have to sit
down when I realize how accurate we have been through no real fault of our
own.)
I know you are getting bored, but the story has a twist or two. First, here
is the simple twist is on the short-term forecast. To capture all the
short-term dynamics would require immense effort, a super computer, and, we
find, literally hundreds of equally believable (or unbelievable)
explanations of what happened. We cannot specify the system, we cant
understand the important levels, and we cant find a unique answer. Bummer.
But these phenomena usually only have transient affects. The major forces
dominate - think of gravity and pouring water into a container -- I
understand the asymptotic solution - the early splashy transient is
chaotic. Thus, an econometric equation with an asymptotic value of 0.0
always fits the data, indicates important leading and lagging indicators
and makes the short-term forecasts "on-the money. Note that these equations
have typically three or fewer terms. They look like SD auxiliary equation
that form a multiplier on the main "result" equation. They stink, they
give some understanding of what you dont know, and they work. (And they
are probably better that the assumed non-monotonic table functions some
SDers love to inflict on client models.)
Now to the last points. When we tested our energy and macro models against
the comparable "new" econometric models at the Department of Applied
Economics of Cambridge University (UK), SD lost. The "new" econometrics (as
opposed to classical econometrics) ALWAYS did better! (Again.. what a
bummer..) The most important question we learn in SD indoctrination comes
up -- WHY? The "new" econometrics is called co-integration. (See the works
of C.W.J. Granger to get started on this.) Simply put, co-integration
looks for level-rate relationships among variables by taking differences
and putting them together in all possible combinations. The differences are
the "derivatives" of all degrees. (Temporary pardons to the purist
"difference equation" crowd Ive gone on too long already and being
correct wont help this discussion.)
SDers often like to say that a 3rd order system can produce all the
behaviors the world has to offer or that it takes a hundred parameter
econometric equation to match the abilities of a well formulated 3rd order
SD model. Well, these co-integration equation take advantage of all the
"derivatives" and the resulting equations can forecast economic recessions
in a county in a industry to the point you see no light between the model
and reality. And if you saw light between the curves, you would believe the
model before the data (for strong statistical reasons.) The models created
by co-integration look very much of like our models, but are proudly touted
as a-theoretical. The are no human biases, no preconceptions or hidden
assumptions that "blight" all other models especially SD models. The
cointegration statistical techniques are almost immorally powerful. They
have the level and rate concept. They exhibit great "extremum test"
capabilities (as if econometricians care), they handle very complex systems
and they are ungodly accurate. Its like econometricians headed in the SD
direction and overshot the target.
But when I try to understand how the system of co-integrated equations
predicated, say the recession, perfectly I find that it is because delta-x
differenced three times multiplied by delta-y differenced once multiplied
by delta-x second differenced flipped signs. And I say "Of course, why did
I think of that?" Cointegration does lead to helping understand many
"whys" and I think should be part of the SD education but, to me, its
validity lies in its statistical power to filter relationships and not in
its impeccable forecasting ability. The forecast has no real "why?" built
into it. When an good SD model provides a forecast, its limitations are
understood and a "bad forecast can be used to make a good decision.
Which now brings us to the actual purpose of this tirade. (Only two more
paragraphs. Trust me. Im a System Dynamicist.) When we have made these
wonderfully accurate forecasts NO ONE has EVER believed them before the
future occurred. Even if we make a half dozen outlandish predictions that
occur as stated, there is NEVER any belief that the next prediction is
remotely valid. In over one hundred projects in the last 20 years not one
of our clients has ever really believed a forecast that did not verify the
past expectation. None, zero, zip. From a null hypothesis perspective, the
statistics are getting a tad significant on this issue. I want to claim
here that the adaptive expectations perspective that J. Sterman discusses
is an immutable law of human nature. If the forecast does not conform to a
modest adjustment to historical expectations, it is beyond credibility.
Period. (I hope everybody guessed we would cover every topic in the
universe here. After all, SD says everything is connected to everything
else via feedback)
Now again back to SD 101. Why is this? Well, why did someone pay good money
to have the @#$&^ forecast to begin with? The damn SD 101 again gives the
answer: There was a problem not a system, an assumed reference mode. They
feared the unspeakable fact that they might be wrong and there would be big
costly troubles if it were so. Thus, an accurate forecast is of no
intrinsic value. What is of value is the unasked question. How does an
organization deal with the unknown? What policies or strategies does it
need to have in-place or available? I hope we all know that the forecast is
just part of the system. The system affect the forecasts. A forecast
outside of the rest of the system is useless. When we included analyses
that expanded the boundary (remember the sub-tirade above?) and simulated
what one would do if the forecast were wrong, that got points. Big points.
The company moved. The model help them understand how to respond to the
unknown future. It described the control (feedback) system for them. It
made them safe and ensured that those terrible losses from a bad forecast
could be mitigated. (We use HYPERSENS for this but Vensim has similar
capabilities.) So the bottom line is: It not how good your forecast is, it
is knowing how bad the forecast can be and what to do if it is. SD is not
useful for providing good forecasts as much as it is useful for providing
understandable bad forecasts.
Okay, I once again get off the soap box and end this trite discussion.
Three paragraphs?. Oh yes, I lied. I made up the number two for that
forecast. I didnt need historical data. I knew better. I am a system
dynamicist.
George
From: "George Backus" <
gbackus@boulder.earthnet.net>
