John Gunkler raises some interesting questions. Reading "Industrial=20
Dynamics" a few years ago changed how I think about models, but=20
unfortunately it didnt change the views of the kinds of clients I=20
work for! Forrester and others have described vividly what they mean=20
by validity, but these days it isnt enough to talk about comparing=20
the SD model with mental models - at least not in my field=20
(transport). There is a whole range of alternative metodologies=20
available, many of them very sophisticated, not all of them as useful=20
as claimed.
=20
I do think that one of the problems with SD is that it can appear=20
tendentious. You see it sometimes on this mailing list, when people=20
say they want to build a model to prove that x, y and z. Throw in=20
enough variables and you almost certainly will. The idea of modelling=
=20
parsimony always seems to be very useful, to me, but I dont think=20
its stressed enough in SD thinking.
=20
John Swanson
From: j.swanson@sdg.co.uk
Validity of Models
-
"John W. Gunkler"
- Junior Member
- Posts: 10
- Joined: Fri Mar 29, 2002 3:39 am
Validity of Models
I am continually impressed, as I read and reread Jay Forresters seminal
works in our field, by how frequently he was able to anticipate developments
and issues forty years before they arose. To not only create a new field
but to bolster it against its self-destruction is the work of genius.
One issue he took on more than 25 years ago was the issue of model validity.
In oft-quoted phrases he described (in Principles of Systems) what a model
was and wasnt good for and how it was to be judged -- namely, against "the
mental image or other abstract model which would be used instead." Such an
admonition serves us well today, in much the same way it served him then --
by taking the legs out from under critics who might want to ignore the
insights of system dynamics models on the basis that they arent "valid"
because they fail, in some detail, to capture the whole reality of a
situation.
I believe that we are still fighting this barrier to acceptance today and
that Forresters answer is still the best approach.
But Im worried, just among us, that we take his advice too far. Once we
leave the context of ill-informed critics, do we pay enough attention to the
validity of our models. Just saying to a client, "Oh, this model is not
supposed to represent the real world, its just supposed to help you think
about it" is a bit disingenuous, dont you think? Its a lot like what
focus group facilitators do when they admonish their clients, "Now, dont
read too much into what one group of eight people say." -- then allow the
clients to sit behind one-way mirrors and watch those eight people. And
what do the clients do? Of course they go away and form conclusions about
"the market" based on what they heard and saw! In a world where real data
is scarce, human beings use whatever real data they have (and usually
overextrapolate from it.) So, I ask my colleagues, what does a client do
with the models you help them build? Do they treat them as some "aid to
thinking" -- or do they begin to believe that the model is reality? Do they
begin to trust that the consequences of decisions they see in simulation
will be the consequences they see in the real world?
And, to be honest, at some level isnt this exactly what we tell (and sell)
them (with caveats and admonishments about due diligence and with a sense of
trying things out rather than betting the company)?
I want to take this several steps further but will do so in another post
after seeing reactions to this one. I want to make the case that we need to
go beyond a standard of "replicating historical behavior modes" as a test of
model validity. I have seen prominent members of this field explicitly deny
that their models are supposed to be predictive. I interpret this as
following Forresters advice and, in the appropriate context (ill-informed
critics) have already said I agree with it. But, just among ourselves,
shouldnt our standards be moving into predictive tests of model validity?
Shouldnt we (at least sometimes) move past "usefulness" and into "validity"
standards? I would like to see this argument developed in future posts.
John
From: "John W. Gunkler" <jgunkler@sprintmail.com>
works in our field, by how frequently he was able to anticipate developments
and issues forty years before they arose. To not only create a new field
but to bolster it against its self-destruction is the work of genius.
One issue he took on more than 25 years ago was the issue of model validity.
In oft-quoted phrases he described (in Principles of Systems) what a model
was and wasnt good for and how it was to be judged -- namely, against "the
mental image or other abstract model which would be used instead." Such an
admonition serves us well today, in much the same way it served him then --
by taking the legs out from under critics who might want to ignore the
insights of system dynamics models on the basis that they arent "valid"
because they fail, in some detail, to capture the whole reality of a
situation.
I believe that we are still fighting this barrier to acceptance today and
that Forresters answer is still the best approach.
But Im worried, just among us, that we take his advice too far. Once we
leave the context of ill-informed critics, do we pay enough attention to the
validity of our models. Just saying to a client, "Oh, this model is not
supposed to represent the real world, its just supposed to help you think
about it" is a bit disingenuous, dont you think? Its a lot like what
focus group facilitators do when they admonish their clients, "Now, dont
read too much into what one group of eight people say." -- then allow the
clients to sit behind one-way mirrors and watch those eight people. And
what do the clients do? Of course they go away and form conclusions about
"the market" based on what they heard and saw! In a world where real data
is scarce, human beings use whatever real data they have (and usually
overextrapolate from it.) So, I ask my colleagues, what does a client do
with the models you help them build? Do they treat them as some "aid to
thinking" -- or do they begin to believe that the model is reality? Do they
begin to trust that the consequences of decisions they see in simulation
will be the consequences they see in the real world?
And, to be honest, at some level isnt this exactly what we tell (and sell)
them (with caveats and admonishments about due diligence and with a sense of
trying things out rather than betting the company)?
I want to take this several steps further but will do so in another post
after seeing reactions to this one. I want to make the case that we need to
go beyond a standard of "replicating historical behavior modes" as a test of
model validity. I have seen prominent members of this field explicitly deny
that their models are supposed to be predictive. I interpret this as
following Forresters advice and, in the appropriate context (ill-informed
critics) have already said I agree with it. But, just among ourselves,
shouldnt our standards be moving into predictive tests of model validity?
Shouldnt we (at least sometimes) move past "usefulness" and into "validity"
standards? I would like to see this argument developed in future posts.
John
From: "John W. Gunkler" <jgunkler@sprintmail.com>
Validity of Models
This is a response to John Gunklers thoughtful posting
(1) I predict that the word "prediction" is going to cause problems in
this discussion. Some people (like me) think of it as meaning "point
prediction", others (like David Peterson) think of it as meaning any
statement about what the future might hold. Obviously, you dont intend
to start a discussion about the meaning of words.
(2) Much of the controversy around validity comes from differences in
what people hope to learn about the future. At one extreme are people
who hope to learn the approximate future values of real-world time
series. At the other extreme are people who hope to learn how feedback
processes in the real-world will influence the future.
(3) The use of models at the two extremes differ in terms of
completeness and parameter accuracy. People hoping for close
approximations to future time series need models that contain all of the
important feedback processes, all important exogenous inputs, and as
much of the unimportant feedback and exogeneity as their budget will
permit. Parameters need to be numerically accurate. In contrast,
people hoping to understand how a particular feedback process (or two or
three) are likely to influence the future only need to represent a few
analogous feedback processes. Parameter estimates often can be much
looser (e.g. "hiringTime" is positive).
Regards,
Jim Hines
LeapTec and MIT
From: Jim Hines <jimhines@interserv.com>
(1) I predict that the word "prediction" is going to cause problems in
this discussion. Some people (like me) think of it as meaning "point
prediction", others (like David Peterson) think of it as meaning any
statement about what the future might hold. Obviously, you dont intend
to start a discussion about the meaning of words.
(2) Much of the controversy around validity comes from differences in
what people hope to learn about the future. At one extreme are people
who hope to learn the approximate future values of real-world time
series. At the other extreme are people who hope to learn how feedback
processes in the real-world will influence the future.
(3) The use of models at the two extremes differ in terms of
completeness and parameter accuracy. People hoping for close
approximations to future time series need models that contain all of the
important feedback processes, all important exogenous inputs, and as
much of the unimportant feedback and exogeneity as their budget will
permit. Parameters need to be numerically accurate. In contrast,
people hoping to understand how a particular feedback process (or two or
three) are likely to influence the future only need to represent a few
analogous feedback processes. Parameter estimates often can be much
looser (e.g. "hiringTime" is positive).
Regards,
Jim Hines
LeapTec and MIT
From: Jim Hines <jimhines@interserv.com>
-
Ed Gallaher
- Junior Member
- Posts: 5
- Joined: Fri Mar 29, 2002 3:39 am
Validity of Models
I tried for several years to obtain neuroscience grant funding which
included a promising and innovative (I thought) use of SD to interpret drug
diffusion throughout the brain after microinjections in mice. As expected,
the reviewers were not familiar with SD, there is little biomedical track
record (largely because they wont fund the research . . . ), and it was
not funded.
But I kept at it. In the final attempt, after re-reading JWFs description
of model validity (described above), I paraphrased and quoted liberally to
indicate as clearly as possible the strengths and weaknesses of the
modeling approach. This grant was funded! Although I had made iterative
improvements with each submission and probably wore them down, I believe
this conceptual information played a key role in convincing the reviewers
that modeling and simulation, carefully done, can contribute significantly
to our growing understanding of such processes.
Furthermore, JWF wrote a description of systems analysis which compared new
engineering design (e.g. new airplane; modeling; testing; refining design
to meet specs, etc.) with socio-economic modeling which uses feedback
concepts to help solve difficult societal problems. (I dont have the D-##
at my fingertips). Socio-economic problems cannot be experimented on very
reliably; next year we may have a different president, interest rates,
balance of trade, public opinion, etc. etc.
>From my perspective JWF omitted another important viewpoint (but of course
this viewpoint is much more relevant to me as a biomedical researcher).
That is, I can model an existing, but poorly understood system, reliably
and repeatedly in the laboratory. Therefore I can develop a growing
understanding about the structure and function of the system, confirmed
with reliable data.
I can build a model of a homeostatic system (e.g. thermoregulation), and
then compare its performance with a genetically constant lab mouse.
Furthermore, I can control environmental conditions very closely. Will the
model act just like the mouse? Very doubtful, because our mental models of
biological processes remain fuzzy.
BUT: I can revise the model (based on rational SD concepts and rigorous
attention to experimental details) and repeat the experiment -almost
exactly- in another mouse next month, or next year, or in another
laboratory. -This cannot be accomplished with socio-economic models!-
Therefore, the application of SD to biomedical systems offers a tremendous
opportunity for new knowledge. In contrast to modeling physical systems
which we already understand (i.e. for teaching purposes; pendulums, cooling
cup of coffee), we are modeling a reasonably stable, repeatable system
about which we have a lot to learn. I suggest that this approach will
indeed lead to predictions about mouse thermoregulation which would
otherwise have been difficult or impossible.
We have recently been informed in these discussions (again) about several
biological models developed by SD students (diabetes, menstrual cycle), and
I applaud this work. But I am really waiting (and pushing) for the time
when these models are developed by the biomedical researchers themselves,
within their own laboratories. These models, and the modeling process,
should be an integral part of the weekly lab meeting, experimental design,
interpretation, and teaching. It should become difficult to obtain a
research grant that does -not- utilize modeling, rather than the other way
around.
Ed Gallaher, Ph.D.
From: Ed Gallaher <gallaher@teleport.com>
Assoc. Prof. Behavioral Neuroscience and Physiology-Pharmacology
Oregon Health Sciences University
Portland, OR
included a promising and innovative (I thought) use of SD to interpret drug
diffusion throughout the brain after microinjections in mice. As expected,
the reviewers were not familiar with SD, there is little biomedical track
record (largely because they wont fund the research . . . ), and it was
not funded.
But I kept at it. In the final attempt, after re-reading JWFs description
of model validity (described above), I paraphrased and quoted liberally to
indicate as clearly as possible the strengths and weaknesses of the
modeling approach. This grant was funded! Although I had made iterative
improvements with each submission and probably wore them down, I believe
this conceptual information played a key role in convincing the reviewers
that modeling and simulation, carefully done, can contribute significantly
to our growing understanding of such processes.
Furthermore, JWF wrote a description of systems analysis which compared new
engineering design (e.g. new airplane; modeling; testing; refining design
to meet specs, etc.) with socio-economic modeling which uses feedback
concepts to help solve difficult societal problems. (I dont have the D-##
at my fingertips). Socio-economic problems cannot be experimented on very
reliably; next year we may have a different president, interest rates,
balance of trade, public opinion, etc. etc.
>From my perspective JWF omitted another important viewpoint (but of course
this viewpoint is much more relevant to me as a biomedical researcher).
That is, I can model an existing, but poorly understood system, reliably
and repeatedly in the laboratory. Therefore I can develop a growing
understanding about the structure and function of the system, confirmed
with reliable data.
I can build a model of a homeostatic system (e.g. thermoregulation), and
then compare its performance with a genetically constant lab mouse.
Furthermore, I can control environmental conditions very closely. Will the
model act just like the mouse? Very doubtful, because our mental models of
biological processes remain fuzzy.
BUT: I can revise the model (based on rational SD concepts and rigorous
attention to experimental details) and repeat the experiment -almost
exactly- in another mouse next month, or next year, or in another
laboratory. -This cannot be accomplished with socio-economic models!-
Therefore, the application of SD to biomedical systems offers a tremendous
opportunity for new knowledge. In contrast to modeling physical systems
which we already understand (i.e. for teaching purposes; pendulums, cooling
cup of coffee), we are modeling a reasonably stable, repeatable system
about which we have a lot to learn. I suggest that this approach will
indeed lead to predictions about mouse thermoregulation which would
otherwise have been difficult or impossible.
We have recently been informed in these discussions (again) about several
biological models developed by SD students (diabetes, menstrual cycle), and
I applaud this work. But I am really waiting (and pushing) for the time
when these models are developed by the biomedical researchers themselves,
within their own laboratories. These models, and the modeling process,
should be an integral part of the weekly lab meeting, experimental design,
interpretation, and teaching. It should become difficult to obtain a
research grant that does -not- utilize modeling, rather than the other way
around.
Ed Gallaher, Ph.D.
From: Ed Gallaher <gallaher@teleport.com>
Assoc. Prof. Behavioral Neuroscience and Physiology-Pharmacology
Oregon Health Sciences University
Portland, OR
-
Tom Fiddaman
- Senior Member
- Posts: 55
- Joined: Fri Mar 29, 2002 3:39 am
Validity of Models
>We have recently been informed in these discussions (again) about several
>biological models developed by SD students (diabetes, menstrual cycle), and
>I applaud this work. But I am really waiting (and pushing) for the time
>when these models are developed by the biomedical researchers themselves,
>within their own laboratories. ...
Two useful examples of models with which biomedical researchers were
involved are included in my model library at:
http://home.earthlink.net/~tomfid/models/models.html
Short citations for each follow.
The second model (of oscillations in rat kidneys) is particularly
interesting. As I understand the story (second- or third-hand), the
researchers initially wrote a journal article describing the empirical data
on kidney pressure oscillations. This was rejected by journals because no
one believed the data. The researchers then constructed a model that
explained the data, and the subsequent article was accepted.
There is also a link on my bookmarks page,
http://home.earthlink.net/~tomfid/sdbookmarks.html
to a medical modeling package called SAAM, which appears to have a
substantial community. I havent actually used it, but it appears to be
targeted at modeling diffusion and transport processes. The state variables
in a model are mainly concentrations within various body or cell
compartments, with generic tools provided for representing reaction and
transport processes within/among compartments.
Since differential equations are the same for everybody, Id be interested
to hear what differences in style separate SAAM or other biomedical models
from typical SD models, and what methods or habits SD might contribute to
the work.
- Tom
Ultradian Oscillations of Insulin and Glucose
Replicated by Hank Taylor from Jeppe Sturis, Kenneth S. Polonsky, Erik
Mosekilde, and Eve van Cauter. Computer
Model for Mechanisms Underlying Ultradian Oscillations of Insulin and
Glucose. Am. J. Physiol. 260 (Endocrinol.
Metab. 23): E801-E809, 1991.
Chaos in Rat Kidneys
Replicated by Tom Fiddaman from Jensen, K.S., Mosekilde, Erik, and
Holstein-Rathlou, N. Self-Sustained Oscillations
and Chaotic Behavior in Kidney Pressure Regulation. In I. Prigogine
and M. Sanglier, eds., Laws of Nature and
Human Conduct. Brussels: Taskforce of Research Information and Study
on Science. See also a related abstract.
Includes a useful example of the Vensim FIND ZERO function (a
simultaneous equation solver), phase plots, and
Poincare maps and sections. See header of .mdl file for details
concerning use.
****************************************************
Thomas Fiddaman, Ph.D.
Ventana Systems http://www.vensim.com
34025 Mann Road Tel (360) 793-0903
Sultan, WA 98294 Fax (360) 793-2911
Tom@Vensim.com http://home.earthlink.net/~tomfid/
****************************************************
>biological models developed by SD students (diabetes, menstrual cycle), and
>I applaud this work. But I am really waiting (and pushing) for the time
>when these models are developed by the biomedical researchers themselves,
>within their own laboratories. ...
Two useful examples of models with which biomedical researchers were
involved are included in my model library at:
http://home.earthlink.net/~tomfid/models/models.html
Short citations for each follow.
The second model (of oscillations in rat kidneys) is particularly
interesting. As I understand the story (second- or third-hand), the
researchers initially wrote a journal article describing the empirical data
on kidney pressure oscillations. This was rejected by journals because no
one believed the data. The researchers then constructed a model that
explained the data, and the subsequent article was accepted.
There is also a link on my bookmarks page,
http://home.earthlink.net/~tomfid/sdbookmarks.html
to a medical modeling package called SAAM, which appears to have a
substantial community. I havent actually used it, but it appears to be
targeted at modeling diffusion and transport processes. The state variables
in a model are mainly concentrations within various body or cell
compartments, with generic tools provided for representing reaction and
transport processes within/among compartments.
Since differential equations are the same for everybody, Id be interested
to hear what differences in style separate SAAM or other biomedical models
from typical SD models, and what methods or habits SD might contribute to
the work.
- Tom
Ultradian Oscillations of Insulin and Glucose
Replicated by Hank Taylor from Jeppe Sturis, Kenneth S. Polonsky, Erik
Mosekilde, and Eve van Cauter. Computer
Model for Mechanisms Underlying Ultradian Oscillations of Insulin and
Glucose. Am. J. Physiol. 260 (Endocrinol.
Metab. 23): E801-E809, 1991.
Chaos in Rat Kidneys
Replicated by Tom Fiddaman from Jensen, K.S., Mosekilde, Erik, and
Holstein-Rathlou, N. Self-Sustained Oscillations
and Chaotic Behavior in Kidney Pressure Regulation. In I. Prigogine
and M. Sanglier, eds., Laws of Nature and
Human Conduct. Brussels: Taskforce of Research Information and Study
on Science. See also a related abstract.
Includes a useful example of the Vensim FIND ZERO function (a
simultaneous equation solver), phase plots, and
Poincare maps and sections. See header of .mdl file for details
concerning use.
****************************************************
Thomas Fiddaman, Ph.D.
Ventana Systems http://www.vensim.com
34025 Mann Road Tel (360) 793-0903
Sultan, WA 98294 Fax (360) 793-2911
Tom@Vensim.com http://home.earthlink.net/~tomfid/
****************************************************