Ventana software support forum

Tom,

I think your ideas of a informal (my word, not yours) code of ethics is a
good one. However,

> - thou shalt perform parameter change experiments one at a time

Doesnt the field of design of experiments teach that factorial designs
may be significantly more efficient than changing one parameter at a time
without adding any significant risk of bad conclusions? As I recall, they
have tests to tell you when you incur risks by factorial or fractional
factorial designs; the potential gains can be significant.

This concern about the specifics of one of your proposals really supports
your later statement:

> The hard part for me is defining codes for intervention. How do we know a

It might be nice to start out with a "SD modelers FAQ". One could start
creating a list of such issues as you mentioned that are at the level of
someone who has already read a significant amount of the literature and is
doing modeling, as opposed to the level of what is SD?. Many FAQs seem
to evolve into something fairly useful after somewhat of an informal peer
review process.

BTW, you speak of professional societies. One of my emotional reactions
against the accreditation proposal was that it sounded like you first had
to join the SD Society and then you had to pass a test that they created.
Im not trying to offend anyone, but that did make it sound like a way to
make the SD Society artificially important.

> Perhaps the SD Society could create a set of questions prospective clients
> might ask, along the lines of Sterman, J. D. (1988). A Skeptics Guide to

That sounds like a good idea. Such efforts by professional groups not only
can help but can also lead to others joining as they see something valuable
being generated.

Just my $0.02 worth.

Bill
--
Bill Harris Hewlett-Packard Co.
R&D Engineering Processes Lake Stevens Division
domain: billh@lsid.hp.com M/S 330
phone: (425) 335-2200 8600 Soper Hill Road
fax: (425) 335-2828 Everett, WA 98205-1298

Bill,

Thanks for pointing out an issue I should have anticipated.

>> - thou shalt perform parameter change experiments one at a time
>
>Doesnt the field of design of experiments teach that factorial designs
>may be significantly more efficient than changing one parameter at a time
>without adding any significant risk of bad conclusions? As I recall, they
>have tests to tell you when you incur risks by factorial or fractional
>factorial designs; the potential gains can be significant.

Your point is well taken, and I didnt mean to suggest that we should
exclude this kind of experimentation, or any kind of multivariate
sensitivity analysis, as long as its informed by an understanding of
scientific method. I actually had in mind a particular case in the SD
literature in which an excellent practitioner made 2 changes simultaneously
in a model, and attributed the outcome to the wrong one.

I like the idea of creating an SD FAQ, perhaps as a precursor to something
more. It seems to me that there are two purposes to this: to help new
practitioners get up to speed quickly and effectively, and to help
prospective customers choose quality practitioners. I think its easy to
serve the first goal, as new practitioners are likely to be seeking
information about the field. It sounds tougher to reach and inform
prospective customers - any thoughts on how (or whether) to do this?

- Tom

****************************************************
Thomas Fiddaman, Ph.D.
Ventana Systems http://www.vensim.com
34025 Mann Road Tel (360) 793-0903
Sultan, WA 98294 Fax (360) 793-2911
Tom@Vensim.com http://home.earthlink.net/~tomfid/
****************************************************

Tom,

> scientific method. I actually had in mind a particular case in the SD
> literature in which an excellent practitioner made 2 changes simultaneously
> in a model, and attributed the outcome to the wrong one.

This reminds me of an article I read on proofs in mathematics which pointed
out that they are as much a social as a logical affair. That is, even the
best of people striving to produce good proofs dont _really_ know theyve
gotten it right until a number of their peers have reviewed the work and
agreed (or modified) it.

> I like the idea of creating an SD FAQ, perhaps as a precursor to something
> more. It seems to me that there are two purposes to this: to help new
> practitioners get up to speed quickly and effectively, and to help
> prospective customers choose quality practitioners. I think its easy to
> serve the first goal, as new practitioners are likely to be seeking
> information about the field. It sounds tougher to reach and inform
> prospective customers - any thoughts on how (or whether) to do this?

I guess I was thinking about the former. While Im not a bona fide expert,
Id even consider (seriously) helping with the effort. I see the utility
of a FAQ on the level of "Whats a level?", but Im not as interested in
that; theres so much good literature one can find relatively easily, both
books and articles. Im more interested in the level of information that
may border on the "tricks of the trade" for those who are consulting
practitioners in the field, along the lines of what you mentioned as
guidelines. It could also include (definitely not in priority order)

information on when to use CLDs and when to use stocks and flows (so as
not to start another war, it could mention the various serious approaches
to the topic, acknowledging multiple philosophies);

integration methods, dt and how to determine whether your model exhibits
integration problems;

conformance of models with reality (how to test; how to determine if
youre close enough; how close can one really get; how to add features to
a model to improve its accuracy);

common "idioms" of SD models (much like the molecules, I guess);

standard (pragmatic) approaches to model building and to SD interventions
in organizations (starting with focusing on the problem, not the model);

some simple control theory concepts.

As for the latter purpose, what if we started simply with a list of
recommended questions for prospective clients to ask prospective
practitioners? Id be inclined to keep it simple and start with real-world
related questions. That is, Id be most inclined to include a question
that relates to a behavior or knowledge someone found really correlated to
success in an intervention or modeling activity or to failure (as opposed
to imagined success or failure). Thats not to say there arent "good
practice" items one would really like to include, but such lists can become
long laundry lists of attributes. Sort of like TQC, focusing on what has
really happened (gone wrong or gone right) might give this list a focus and
some credibility.

Comments?

Bill
From: Bill Harris <billh@lsid.hp.com>

PS: Out of curiosity, I note that we dont often talk about control theory
concepts in those specific terms, even though SD did come out of control
theory. Ive seen an MIT dissertation from a person whos name always
escapes me (Graham, I think, or was it Greene) which seemed to do in a
quantitative sense what Senges archetypes did in a qualitative sense: give
one some rules of thumb to use in place of detailed simulations. It was
based on a study of simulation models, and it gave guidance into what kind
of dynamic behavior one would see as a function of the delays in the model,
as well as how one would change a model (and thus system) to try to modify
dynamic behavior in specific ways. It seemed handy, and I copied and
occasionally use the list of rules of thumb it contained, but I never see
reference to it or to related developments. (Unfortunately, I cant find
the few pages I copied from it right now to give a better reference.)

--
Bill Harris Hewlett-Packard Co.
R&D Engineering Processes Lake Stevens Division
domain: billh@lsid.hp.com M/S 330
phone: (425) 335-2200 8600 Soper Hill Road
fax: (425) 335-2828 Everett, WA 98205-1298