SD vs. Econometrics

["Pelham Barton"]

Jon Noble asked for the "best prediction for a complex system, say a
commodity price for the next 6 months". What an SD model can do is
incorporate the effect of your actions, based on your prediction, on
what that price will be. Does best prediction mean "most likely
result if I just sit back and watch" or "prediction that will
encourage me to behave in the most useful way". For more on this, see
(for example) Chapter 9 of "Management System Dynamics", by R.G.
Coyle (Wiley, 1977).

Pelham Barton
University of Birmingham, UK
p.m.barton@bham.ac.uk

[Bill Harris]

George,

Thanks for your _very_ interesting posting! Besides some seemingly
impressive data, you included some important philosophical reminders and
(for me) new (or newly packaged) insights.

> errors for a few of our models, over a ten year period, (sit down at this
> point) have typically been in the tenth (singular) of a percent absolute
> error range. By "ten-years" I mean, "What was the error in the forecast

Okay, you got my attention.

> Now the real answer to why the model made "good forecasts" is that once you
> have a system that counter-responds to any changes no mater how dramatic,
^^^^^^^^^^^^^^^^
Can you explain what you mean here a bit more? Do you mean that it
responds to counter(act) the change --- to preserve its status quo? Or do
you mean that it responds in some reasonable fashion (as in none of the
integrations saturate OR none of the levels --- or rates --- go outside
their designed ranges) to imposed change (forces)?

> But these phenomena usually only have transient affects. The major forces
> dominate - think of gravity and pouring water into a container -- I
> understand the asymptotic solution - the early splashy transient is
> chaotic. Thus, an econometric equation with an asymptotic value of 0.0
> always fits the data, indicates important leading and lagging indicators
> and makes the short-term forecasts "on-the money. Note that these equations
> have typically three or fewer terms. They look like SD auxiliary equation
> that form a multiplier on the main "result" equation. They stink, they
> give some understanding of what you dont know, and they work. (And they
> are probably better that the assumed non-monotonic table functions some
> SDers love to inflict on client models.)

Can you explain that a bit more? It sounds like you are fitting short term
behavior with some econometric (read "statistical correlations") equations
which are used to modify rates. You say they have an asymptotic value of
0.0, which implies to me that they are additive corrections (so that they
make no difference in the longer term), but then you say, "... that form a
multiplier ...". Is it really that some of these are additive effects and
some are multiplicative, and you dont care, or is the asymptotic value of
0.0 important?

Also, are the econometric equations really just some econometric or
statistical fitting of the data, or are they derived from causal effects
(as in SD) with the time constants so short compared to the main model that
they can be best represented as constants, not integrations?

> the comparable "new" econometric models at the Department of Applied
> Economics of Cambridge University (UK), SD lost. The "new" econometrics (as
> opposed to classical econometrics) ALWAYS did better! (Again.. what a

Bummer again. Now theres something else I have to read.

(Seriously, thanks.)

Again, any time you want to tie up the SD list with "trite" "diversions"
like this,

... be my guest!

Bill
--
Bill Harris Hewlett-Packard Co.
R&D Engineering Processes Lake Stevens Division
domain: billh@lsid.hp.com M/S 330
phone: (425) 335-2200 8600 Soper Hill Road
fax: (425) 335-2828 Everett, WA 98205-1298

[jnoble1@mmm.com]

I*ve been watching with interest the discussion about behaviour understanding
and point prediction using system dynamics. While all the comments have been
favorable towards SD for pattern understanding, a few have been made to the
effect that SD is the best method for point prediction as well for dynamic
systems. I personally don*t have a great deal of experience with forecasting,
but do understand the benefits of using causal models over extrapolative
methods for future predictions when time, data, and resources allow.

My question for the list boils down to this; if time and resources were not
constaints and I wanted to have the best prediction for a complex system, say a
commodity price for the next 6 months, would statistical methods (such as
econometric models) or SD be the best approach? In it*s structure, the SD
model would seem to give a more *operational* view of system components based
on knowledge of the system than a set of statistically-derived regression
equations. Also, with current parameter optimization capabilities in many
packages, the effort to *tune* such SD models is reduced. Clearly, there
are many situations where predicting the future based on historical data does
not make sense, but when it does, is SD the best choice?

Jon Noble
3M Mfg. Planning Systems
jnoble1@mmm.com

[Bruce Hannon]

Jon Noble, the best way I have found to think of this question is to regard
the dynamic model as a giant regression equation, that contains more than a
little specific knowledge about the process involved. Then manipulate the
parameters in the model within their known ranges until the best fit with
known data occurs. This latter step is basically one used in what I call
statistical modeling. I regard the above approach as superior because the
"regression equation" is much truer.
I use a combination of STELLA and MADONNA to do this, but there are many
techniques.


Bruce Hannon, Jubilee Professor
Liberal Arts and Sciences
Geog/NCSA
220 Davenport Hall, MC 150
University of Illinois
Urbana, IL 61801
reply to: b-hannon@uiuc.edu
http://www.gis.uiuc.edu
217 333-0348 office
217 244 1785 fax

["George Backus"]

In response to John Noble;

Our group, for right of for wrong, has used SD extensively for pin-point
forecasting in the energy industry for 20 years. We spend a great deal of
time going head to head with econometric forecasts and their entrenched
staffs. So far we have always "won" against "classical" econometric efforts
for mid and long-term analyses. ("Classical" is a critical distinction that
I will clarify later). In our case, mid-term starts three years out. The
errors for a few of our models, over a ten year period, (sit down at this
point) have typically been in the tenth (singular) of a percent absolute
error range. By "ten-years" I mean, "What was the error in the forecast
after you wait ten years and look back on how you did?" In most cases we
cannot check that far back because "historical memories" get fuzzy.
Nonetheless, we can show that the mid-term-plus results tend to be an order
of magnitude better than the alternative models (engineering, input-output,
optimization -- the worst, accounting, and econometrics.)

We find there are no right reasons for this and many WRONG ones. But first
I must digress. When we took reasonably complex models and let the
parameters vary over their range to obtain the best fit, we did not do
better than the econometric models for that very reason that we were
treating the model as a "huge regression equation." The problem was, of
course, that it was a huge, non-linear, multi-dimensional "regression
equation" and only very sophisticated methods that "rewrite" the SD model
in "standard regression forms" even begin to do the job validly. And to
say "validly" denies truth and to say "adequate" would imply some heroic,
if not foolish, judgment. If the forecasts of the model fell within the
values of history, these parameter fits did well -- just like any
semi-reasonable, simple, econometric equation would. If the model was
going to go into "new territory," our experience was that the "parameter
fitting" process gave "explainable" results easily refuted by experts.

The trick was to take the "conventionally" parameterized model and (please
sit down again) perform "extremum tests" from SD 101. This showed us what
caused the model to really deviate from "reasonable" results. Now "what
are reasonable results?," you should be saying as you stand up again. They
are the results from other non-SD models. These results are "reasonable"
because they are often scrutinized by others and modified AS NEEDED to give
a consensus agreement on what will be. Rev. J. Sterman showed many years
ago that the "official" forecasts were just exponential smooths of the
"more correct" marginal forecasts (called adaptive expectations in the
other thread going on in the SD server.)

If our forecast was different, we could ask "why?" If the other forecasters
had a GOOD reason for there judgments, as they often did, it meant we had
missed a structure in our model important to the new conditions, but not to
the old. If they had a bad (usually paradoxical) reasoning, we could look
in our model to see if we were equally duped. (We often were by "current
wisdom" and needed to throw out those structures and rethink our
understanding.

And now a diversion to the diversion. The duping came from assuming we
"knew" the system and did not really need to do historical validation to
verify our very blind and false assumptions. George Richardson has spoken
to this point previously on the SD server, but still many seem to believe
in personal omnipotence. I agree there are new "things" in the future, but
I can only use what I understand from the past, even if by analogy, to
grasp them. More importantly, if I cant understand the past, I surely
cannot claim to understand the future. The future came from levels
accumulated from the past. The processes that made those accumulation dont
just disappear. And those levels are usually "the problem." The forces --
structures-- that made the levels as they are, usually act to preserve
them. I, for one, have not been able to determine how to change the
results of the past unless I understand how that past made the present.
Mark twain said, "The easiest person to fool is oneself." Historical data
is the only unbiased, though still inaccurate, critic that I can trust to
keep me in touch with reality But I digress...

When we saw dumb behavior modes during the extremum tests, it simply meant
we missed important feedback structure or miss-specified others. The more
we had a one-to-one correspondence among the structure, the experts
stories, and history (even if just analogous history in another field), the
better the model held its own. We then could make the simple parameter
estimations on the parts of the model where we could validly isolate that
structures were we felt that we understood the input, output and the
feedback adequately plus understood the limitations of the always
simplified structure and its by-design limited impact on the rest of the
model. These "regressions" made the econometricians happy but it was the
new feedback structures that counted.

Now the real answer to why the model made "good forecasts" is that once you
have a system that counter-responds to any changes no mater how dramatic,
you have a system that probably reflects a quite unique system solution.
Since the world exists, it usually finds these structural solutions. As
our forecasts showed, for example, the actual demand growth, the authors
of the "failing" forecasts finally concluded that it was because they
missed the feedback in price and the economy. Andy Ford and John Sterman
have written extensively about this. The boundary of the problem (the
system) in terms of what feedback had to be in the model to handle the
"hypothesized" future extreme conditions was found to be the simple key to
the not-so-modest accuracy that the SD model enjoyed. (I still have to sit
down when I realize how accurate we have been through no real fault of our
own.)

I know you are getting bored, but the story has a twist or two. First, here
is the simple twist is on the short-term forecast. To capture all the
short-term dynamics would require immense effort, a super computer, and, we
find, literally hundreds of equally believable (or unbelievable)
explanations of what happened. We cannot specify the system, we cant
understand the important levels, and we cant find a unique answer. Bummer.
But these phenomena usually only have transient affects. The major forces
dominate - think of gravity and pouring water into a container -- I
understand the asymptotic solution - the early splashy transient is
chaotic. Thus, an econometric equation with an asymptotic value of 0.0
always fits the data, indicates important leading and lagging indicators
and makes the short-term forecasts "on-the money. Note that these equations
have typically three or fewer terms. They look like SD auxiliary equation
that form a multiplier on the main "result" equation. They stink, they
give some understanding of what you dont know, and they work. (And they
are probably better that the assumed non-monotonic table functions some
SDers love to inflict on client models.)

Now to the last points. When we tested our energy and macro models against
the comparable "new" econometric models at the Department of Applied
Economics of Cambridge University (UK), SD lost. The "new" econometrics (as
opposed to classical econometrics) ALWAYS did better! (Again.. what a
bummer..) The most important question we learn in SD indoctrination comes
up -- WHY? The "new" econometrics is called co-integration. (See the works
of C.W.J. Granger to get started on this.) Simply put, co-integration
looks for level-rate relationships among variables by taking differences
and putting them together in all possible combinations. The differences are
the "derivatives" of all degrees. (Temporary pardons to the purist
"difference equation" crowd Ive gone on too long already and being
correct wont help this discussion.)

SDers often like to say that a 3rd order system can produce all the
behaviors the world has to offer or that it takes a hundred parameter
econometric equation to match the abilities of a well formulated 3rd order
SD model. Well, these co-integration equation take advantage of all the
"derivatives" and the resulting equations can forecast economic recessions
in a county in a industry to the point you see no light between the model
and reality. And if you saw light between the curves, you would believe the
model before the data (for strong statistical reasons.) The models created
by co-integration look very much of like our models, but are proudly touted
as a-theoretical. The are no human biases, no preconceptions or hidden
assumptions that "blight" all other models especially SD models. The
cointegration statistical techniques are almost immorally powerful. They
have the level and rate concept. They exhibit great "extremum test"
capabilities (as if econometricians care), they handle very complex systems
and they are ungodly accurate. Its like econometricians headed in the SD
direction and overshot the target.

But when I try to understand how the system of co-integrated equations
predicated, say the recession, perfectly I find that it is because delta-x
differenced three times multiplied by delta-y differenced once multiplied
by delta-x second differenced flipped signs. And I say "Of course, why did
I think of that?" Cointegration does lead to helping understand many
"whys" and I think should be part of the SD education but, to me, its
validity lies in its statistical power to filter relationships and not in
its impeccable forecasting ability. The forecast has no real "why?" built
into it. When an good SD model provides a forecast, its limitations are
understood and a "bad forecast can be used to make a good decision.

Which now brings us to the actual purpose of this tirade. (Only two more
paragraphs. Trust me. Im a System Dynamicist.) When we have made these
wonderfully accurate forecasts NO ONE has EVER believed them before the
future occurred. Even if we make a half dozen outlandish predictions that
occur as stated, there is NEVER any belief that the next prediction is
remotely valid. In over one hundred projects in the last 20 years not one
of our clients has ever really believed a forecast that did not verify the
past expectation. None, zero, zip. From a null hypothesis perspective, the
statistics are getting a tad significant on this issue. I want to claim
here that the adaptive expectations perspective that J. Sterman discusses
is an immutable law of human nature. If the forecast does not conform to a
modest adjustment to historical expectations, it is beyond credibility.
Period. (I hope everybody guessed we would cover every topic in the
universe here. After all, SD says everything is connected to everything
else via feedback)

Now again back to SD 101. Why is this? Well, why did someone pay good money
to have the @#$&^ forecast to begin with? The damn SD 101 again gives the
answer: There was a problem not a system, an assumed reference mode. They
feared the unspeakable fact that they might be wrong and there would be big
costly troubles if it were so. Thus, an accurate forecast is of no
intrinsic value. What is of value is the unasked question. How does an
organization deal with the unknown? What policies or strategies does it
need to have in-place or available? I hope we all know that the forecast is
just part of the system. The system affect the forecasts. A forecast
outside of the rest of the system is useless. When we included analyses
that expanded the boundary (remember the sub-tirade above?) and simulated
what one would do if the forecast were wrong, that got points. Big points.
The company moved. The model help them understand how to respond to the
unknown future. It described the control (feedback) system for them. It
made them safe and ensured that those terrible losses from a bad forecast
could be mitigated. (We use HYPERSENS for this but Vensim has similar
capabilities.) So the bottom line is: It not how good your forecast is, it
is knowing how bad the forecast can be and what to do if it is. SD is not
useful for providing good forecasts as much as it is useful for providing
understandable bad forecasts.

Okay, I once again get off the soap box and end this trite discussion.
Three paragraphs?. Oh yes, I lied. I made up the number two for that
forecast. I didnt need historical data. I knew better. I am a system
dynamicist.

George
From: "George Backus" <gbackus@boulder.earthnet.net>

["George Backus"]

In reply to Bill Harris:

> Now the real answer to why the model made "good forecasts" is that once you
> have a system that counter-responds to any changes no mater how dramatic,
> ^^^^^^^^^^^^^^^^
> Can you explain what you mean here a bit more? Do you mean that it

By this I mean one of two things, possibly (usually) combined. The system
either attempts (not necessarily succeeding) to maintain a status quo by
hitting a secondary mechanism, or its response moves it to a new operating
condition or mode that is also stable behavior (a limit cycle counts as
much as "stable" as does a slowly changing trend). Saying the fact in
reverse is easier. The model does not go to infinity or 0 (or negative,
depending on the level definition). There must be (my biased view) a
feedback loop jumping in to correct any large or rapid excursion outside of
operating conditions. I use something like 20% per DT as a measure of
excess change. If the model moves that fast I am clearly missing a
mechanism (or misunderstanding the system) because I only modeled
mechanisms that had time constants significantly longer than the DT.

Also the world exists. If I do something physically realizable, I expect
the world to continue to exist and make a physically realizable response.
This approach is heuristic but its basis is well founded in control theory
(Nyquist criteria) and SD (the DT problem).

> Can you explain that a bit more? It sounds like you are fitting short term
> behavior with some econometric (read "statistical correlations") equations

I miss-stated the idea here. The econometric equation has a "zero" long
term effect. It long term value is 1.0 -- a multiplier. There are some old
discussion on adders versus multipliers in the SD literature. Maybe another
of the "old ones" will go into this. The adders would generally give
inconsistent logic

A simple example of a transient equation might be a budget response of the
form (NB/AB)^E where NB is the new budget, AB is the old budget (smoothed
budget - Koyck lagged or discrete lagged in econometrics) , and E an
elasticity. In the "long-term" NB/AB=1.0 and (NB/AB)^E =1.0. Note that the
regression would be logarithmic so the independent variable value from the
regression is 0.0 but the e^0 used in the model is 1.0 (This is my weak
excuse saying 0.0 in the original test.) Thus, the "1.0" value is
important. I multiply the econometric results on to the value the "actual:
SD model delivers.

I like to rationalize that the equations have causality and try to write
them as such, but I lie to myself too. It is true (I hope) that the short
term response is just too fast for me to catch and has to many interactions
for me to understand. It is really just fitting the data with some SD
remorse to throw out variables that could not have a causal relationship.
I should note that co-integration techniques do tell me whether these
short-term econometrics have a long-term impact. Thus the "transient"
assumption can be tested and thereby keep the main SD model as legitimate
(and pure of heart) as possible.

----------------------
I think I need to doubly clarify the clarification I made to Bill Harris:
(Is this called progress?)

First, to keep life overly simple, I used the concept of "DT" to define
"too fast" of a model response. Many of the modern SD languages do not
limit themselves to the Euler integration method and its explicit "DT"
(delta- time , or time interval for calculation updates). The correct but
complicated answer is that I have explicit time constants in the model and
implicit ones possibly in table functions. (I try to make sure I explicitly
recognize the table function impacts). If the system can be put in a
"realizable" situation where the gain between an "input" and a system
"output" reflects a time constant greater than any I have recognized, I
argue that I have missed something important in the model structure -- if I
am making a forecast. This may be true for other uses of SD but I am not
in a position to make that claim.

This brings in point two. I dont know the above as fact. It is a design
philosophy I use based on the notion that : The system really exits. If
the system really could undergo the conditions leading up to the "too fast
excursion," it probably did so at least once in its history. But the system
still exists. Therefore, the system must have contained a mechanism to let
it survive -- to counter respond. Amen.

I need that indicated feedback loop in the model if my focus is to assure
that the model has predictive capability --- as opposed to showing that
there is something wrong in the system with a purpose of designing (a
possibly redundant or conflicting) means to control the phenomena. I, to
my limited knowledge, cannot design a test or experiment to show that the
logic of the previous paragraph is true. I can always falsely make up a
mechanism to prove to myself that the mechanism exists. I cannot (yet) find
a way to keep from fooling myself. I can only use ""evolutionary logic."
If the system exists and has probably experienced the precipitating
condition I assumed, then the system most probably contains a mechanism to
perpetuate its survival. Please note that this is only true for an
aggregate system where I can argue that some of a sufficiently large
population has the "gene" structure needed for survival. If I am modeling
a discrete system, say a single company, then I cannot assume that it
experienced the assumed conditions or that it would have survived. This
later situation justifies the "standard" practice in SD of not worrying
about the issue presented here.

I would try to make this more unclear but I better quite while I am only
falling behind at a rate consistent with pre-assumed time constants...

George Backus
From: "George Backus" <gbackus@boulder.earthnet.net>
Policy Assessment Corporation
14604 West 62nd Place
Arvada, Colorado USA 80004-3621
Bus: +1-303-467-3566
Fax: +1-303-467-3576