The "usefulness" of SD

This forum contains all archives from the SD Mailing list (go to http://www.systemdynamics.org/forum/ for more information). This is here as a read-only resource, please post any SD related questions to the SD Discussion forum.
Locked
Colm Toolan subscriptions toolan
Junior Member
Posts: 9
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by Colm Toolan subscriptions toolan »

Posted by ""Colm Toolan"" <subscriptions@toolan.de>

Query: Are there any studies that document SD and its application in multiple, diverse situations from a ""usefulness"" point of view? By ""useful"" I mean Return on Investment for commercial applications and proven, measurable change for the better in environmental/cultural/not-for-profit scenarios.

I ask the question because - after about three years as a (relatively) passive member of this list - I find the ongoing (and sometime repeated) discussions of ""SD is this, but not that..."", ""SD is better than xyz, because ..."" somewhat tedious.

Some members seem to be wishing for a major breakthrough in SD acceptance (reading between the lines in the ""SD and other approaches"" thread) and others (http://justinlyonandsimulation.blogspot.com/) see 2006 as ""The Tipping Point for Simulation Science"".

I have to ask ""what are we doing to help?""

John Sterman's story of Winston J Ledet's (Snr) use and application of SD to ""Play the Maintenance Game"" (Business Dynamics p66 et seq) is a perfect example of what I mean. Here was someone taking a theoretical modelling approach to understand a very real problem and to develop appropriate solutions. But he then went on to translate the insights into a hands-on experience that shop floor labour could understand and implement. He developed a board game! And the whole process resulted in measurable Return on Investment!

I'd be prepared to take a position that if we only could (or would) document the results of our SD efforts in terms of tangible results - RoI, quantifiable before and after metrics - then the discussions with the CEOs, strategists, policy makers etc. regarding the pros and cons of the methodology would quickly disappear. If the SD community were to concentrate more on demonstrating the usefulness of the methodology than on the nuances of its application, we might even stop being our own worst enemy!

OK - my forte is business development and getting project goals achieved and I've no formal qualifications in SD. But, believe me, it doesn't matter what stocks and flows are in our system, if we can't explain to our customers/sponsors/supporters that there'll be more value/volume/profit when we've completed our work!

Regards,

Colm Toolan, Business Architect
Germany
Posted by ""Colm Toolan"" <subscriptions@toolan.de>
posting date Tue, 27 Dec 2005 18:12:00 +0100
nickols att.net (nickols att.net
Newbie
Posts: 1
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by nickols att.net (nickols att.net »

Posted by nickols@att.net (nickols@att.net)
Regarding Colm Toolan's query about the ""usefulness"" of SD...


>> Some members seem to be wishing for a major breakthrough in SD
>> acceptance (reading between the lines in the ""SD and other
>> approaches"" thread)


I've seen similar discussions on at least three other lists (Training, HR and OD). Groups of professionals, competent in their practice area, seem to have in common a longing for greater acceptance of their practice area, especially on the part of CEOs. Thus, trainers press to show the ROI of training, OD practitioners wonder why they no longer have a seat at the table, and HR folks are continuously angling to get such a seat. I've come to call this general pattern the ""Rodney Dangerfield syndrome"" (referring to Rodney's famous comment: ""I don't get no respect."").

In all cases, or so it seems to me, the underlying issue - namely, a perceived lack of acceptance (and, of course, the corresponding influence that goes with it) - ties not to an inability to produce proven business results of value - but to a failure to perceive the role of a particular practice and practitioners. Trainers think they can establish their worth if they can show the ROI of training. HR folks think they can get a seat at the table if they can show the business impact of HR. Ditto for OD consultants. And now, or so it seems, the same is true of SD.

It ain't gonna happen, people. Why? Because no tool, no matter how expertly wielded, can buy access to the halls of power. A distinction is frequently drawn between management and leadership. Press a little farther and you encounter governance. Here, another distinction can be made: between governing and ruling. CEOs, in case you haven't noticed, are members of the ruling class. Your technologies and your methodologies are, to them, nothing more than tools of situational relevance and usefulness. So are the practitioners who use those tools.

It is entirely conceivable that some trainer, or OD consultant, or HR specialist or SD practitioner will indeed be invited to sit at the table - but not owing to the tool or even to the results produced. That invitations will come because the person being invited to sit at the table has demonstrated loyalty and fealty in addition to some technical competence and, above all, a considerable amount of political and business acumen.

""Selling"" your technology won't buy you a ticket to the big game; neither will demonstrating its bottom-line value or impact. It'll earn you a good living and even establilsh a ""rep"" in certain quarters but it won't get you through the door to the corner office or the boardroom. That's a close-knit group and the game they're playing is anything but rational- analytic in nature.

So, unless you're prepared to play politics - with all that implies - my advice would be to stay focused on making SD an even better tool than it already is and increasing your mastery of it. There's always plenty of call for good tools and people who know how to use them (and the pay is pretty good, too).

--
Regards,

Fred Nickols, CPT
Senior Consultant
Distance Consulting
Posted by nickols@att.net (nickols@att.net)
posting date Wed, 28 Dec 2005 17:46:57 +0000
Bill Braun bbraun hlthsys.com
Member
Posts: 29
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by Bill Braun bbraun hlthsys.com »

Posted by Bill Braun <bbraun@hlthsys.com>

>>I find the ongoing (and sometime repeated)
>>discussions of ""SD is this, but not that..."", ""SD is better than xyz,
>>because ..."" somewhat tedious.


To revisit a topic of conversation is not the same as having the same conversation all over again.

It seems to me that people are working through their thinking. And, Toolan may have some thoughts worth talking about. But to dismiss past efforts to clarify how we understand SD and the future possibilities that it holds flies in the face of what it means to be a community of learners.

Toolan stakes out the position that ""it doesn't matter what stocks and flows are in our system, if we can't explain to our customers/sponsors/supporters that there'll be more value/volume/profit when we've completed our work!""

The purpose of the list (for me) is not singular in nature; ROI is not the only conversation to be had. If the list's purpose does becomes singular it will have diminshed itself to the detriment of all.

Bill Braun
Posted by Bill Braun <bbraun@hlthsys.com>
posting date Wed, 28 Dec 2005 14:04:22 -0500
sarah sheard sheardsheard yahoo.
Newbie
Posts: 1
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by sarah sheard sheardsheard yahoo. »

Posted by sarah sheard <sheardsheard@yahoo.com>
What? Are you suggesting that this community might
adapt and evolve? as the individual agents within
it discover various different ways in which they might
meet their own needs by participating differently?
Could it be that this community has characteristics a
complex adaptive system?

Sarah Sheard (not usually quite so obnoxious)
Posted by sarah sheard <sheardsheard@yahoo.com>
posting date Thu, 29 Dec 2005 19:47:57 -0800 (PST)
Jean-Jacques Laublé jean-jacques
Senior Member
Posts: 68
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by Jean-Jacques Laublé jean-jacques »

Posted by =?iso-8859-1?Q?Jean-Jacques_Laubl=E9?= <jean-jacques.lauble@wanadoo.fr> Hi Fred

The problem of acceptance discussed in this thread has to my opinion nothing to do with trying to be accepted at the 'table'. To be accepted at the table you need as you say a 'personality' that has nothing to do with SD and the people whom SDers try to convince are not the C.E.O. but the vast community of people interested in the solving of complex problems using structured methodology.

I am myself a C.E.O. and I know plenty of others and they have never heard of SD nor of any other methodologies D.E.S. or A.B.M. or even widely used methods like common statistics. The tool they use commonly is the rule of three. If you want to sit at their table you must understand their preoccupations, discuss with them at their own level of understanding using their language and not be boring. You must learn to sell yourself and not a method. Raspoutine did not need any rationality but still partly governed Russia during some time.

You write:
<my advice would be to stay focused on making SD an even better tool than it <already is and increasing your mastery of it. This is what we all try to do and I do not think that we are trying to sit at any table as it depends on each specific personalities. Regards. Jean-Jacques Laublé Allocar Strasbourg France Posted by =?iso-8859-1?Q?Jean-Jacques_Laubl=E9?= <jean-jacques.lauble@wanadoo.fr> posting date Fri, 30 Dec 2005 16:42:47 +0100
Colm Toolan subscriptions toolan
Junior Member
Posts: 9
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by Colm Toolan subscriptions toolan »

Posted by ""Colm Toolan"" <subscriptions@toolan.de>

I'd like to acknowledge the responses so far and address some possible misunderstandings.


Posted by nickols@att.net (nickols@att.net):
>It is entirely conceivable that some trainer, or OD consultant, or HR
>specialist or SD practitioner will indeed be invited to sit at the
>[Directors'] table - but not owing to the tool or even to the results
>produced. That invitations will come because the person being invited
>to sit at the table has demonstrated loyalty and fealty in addition to
>some technical competence and, above all, a considerable amount of
>political and business acumen.

I'd agree that SD alone or proof of its value alone is not sufficient. My argument is that it can't do any harm - no matter what other credentials one might have - to be able to prove the usefulness of the SD approach by reference to well-documented case studies.

Posted by Bill Braun <bbraun@hlthsys.com>:
>To revisit a topic of conversation is not the same as having the same
>conversation all over again.

Agreed.

>But to dismiss past efforts to clarify how we understand SD and the
>future possibilities that it holds flies in the face of what it means
>to be a community of learners.

Acknowledged. My use of the word ""tedious"" was a subjective statement and in no way intended to question the learning aspects of the group. My query was intended to access the body of knowledge and opinion relating to the experiences of applying SD in specific circumstances with documented results.

I would submit that any community of learners must accept that individual members might be interested in different aspects of the same subject area. The practitioners will attach more importance to the ""how?"", while the passive observers, the recommenders and the gatekeepers are more interested in the ""why?""


>The purpose of the list (for me) is not singular in nature; ROI is not
>the only conversation to be had. If the list's purpose does becomes
>singular it will have diminshed itself to the detriment of all.

I agree wholeheartedly with those sentiments. Which is why - after 2 to 3 years watching this list - I raised the query I did. As far as I am aware, there has not in that time, been a single, detailed discussion about ROI and documentation regarding the ""usefulness"" of SD. I don't ask for ROI-singularity, but it would be nice to get some access to members' experiences in this area.

Posted by sarah sheard <sheardsheard@yahoo.com>:
>What? Are you suggesting that this community might adapt and evolve?
>as the individual agents within it discover various different ways in
>which they might meet their own needs by participating differently?

:-)
I don't see my query particularly as a different type of participation. If
*this* list can not address the issue of usefulness, past successes etc. then who can?

By the way, despite the detailed responses, the original question still remains unanswered: Are there documented Case Studies?

Regards,

Colm Toolan, Business Architect
Posted by ""Colm Toolan"" <subscriptions@toolan.de>
posting date Sat, 31 Dec 2005 15:08:33 +0100
Colm Toolan subscriptions toolan
Junior Member
Posts: 9
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by Colm Toolan subscriptions toolan »

Posted by ""Colm Toolan"" <subscriptions@toolan.de>
Jean-Jacques wrote:
>There have been many discussions about the usefulness of SD before and
>effectively no documentations on real applications. You can find some
>interesting applications on the Association paper, but documentation
>about the usefulness of SD are scarce. [Discussion about RoI]


Firstly, let me clarify that my issue is NOT exclusively about RoI. That is merely a measure of usefulness used in some circles to measure the value of some types of things and activities. In other contexts - where SD is arguably much more useful and appropriate - RoI is certainly not an easy to use measure (environmetal, health, development issues etc.)

But why is there ""effectively no documentation about the usefulness of SD""?

Surely, if I decide to analyse some problem, pick an appropriate method, and conduct the analysis or build the model then I can document the results?

In (at least) some cases, this documentation would include metrics of success (revenue/profit increased by x%; emissions/traffic/occurrences reduced by y% - depending on the nature of the problem).
I would submit that in every single case, documentation is possible that describes the appropriateness of the model to the problem in hand, the ease (or otherwise) of use in the specific case, the insights gained and the policy/strategy recommendations made.
If the process does not result in some form of success, I would expect the documentation to outline and to explain the reasons for this ""failure"" - perhaps the method was inappropriate, the domain knowledge was not available/correct, maybe the recommendations were simply never implemented.

The use of SD is ALWAYS an investment decision - the resources (time, people, money) could always be used in an alternative fashion. Every decision-maker - especially the not-for-profit organisations! - must be able to demonstrate that their choice of problem solving approach represents the best use of the available resources.

I simply cannot believe that no one out there has had to justify that the SD approach represents a good choice and then gone on to document that justification within specific projects/applications!

Colm Toolan, Business Architect
Posted by ""Colm Toolan"" <subscriptions@toolan.de>
posting date Thu, 5 Jan 2006 14:36:01 +0100
Jean-Jacques Laublé jean-jacques
Senior Member
Posts: 68
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by Jean-Jacques Laublé jean-jacques »

Posted by Jean-Jacques Laublé <jean-jacques.lauble@wanadoo.fr> Hi Colm I recognize that there are not many documents about the usefulness of SD. But is it really specific to SD? Generally people do not like to document their results because it would make them depressive. Generally too because they are motivated by the future and not by the past, unless they are accountants.

A second reason of scarce documentation about results is that as I wrote it before people will not propagate ideas that give them a competitive advantage and do not want to make known that they were mystified by some original ideas. I recognize too that documentation about results would be very instructive as long as it is read! In many circumstances you have to learn the hard way by trial and error. See the excellent book from Sterman: business dynamic. Everything I learned in 4 years is in it. But the problem is that nobody told me what would better suit my case, and it is only today that I can reread the book and interpret it differently relatively to my past experiences. I think that the best solution to learn about SD and the usefulness of the method, would be to work some months with a very experienced modeller that would teach you that there is no definitive answer to your question but only depending on each specific case. That would economize years of struggling in the dark. Regards. Jean-Jacques Laublé Allocar Strasbourg France. Posted by Jean-Jacques Laublé <jean-jacques.lauble@wanadoo.fr> posting date Fri, 6 Jan 2006 17:20:47 +0100
wakeland pdx.edu
Junior Member
Posts: 3
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by wakeland pdx.edu »

Posted by wakeland@pdx.edu
Are insights useful? Of course they are. Is there an accepted way to quantify how much an insight is worth? Not to my knowledge.

Two of the best (among many) demonstrations of how building SD models can lead to useful insights are G. Richardson's ""Insightful Little Models"" presentation and N. Repenning's ""Firefighting in New Product Development"" article. In these cases, very simple models reveal deep and useful insights. However, these marvelous little models did just pop into existence. Many larger, and probably less useful, models were constructed first. The little models came later. So it is really the **process** of applying SD that is most valuable, not the final model (although an insightful little model sure helps to communicate the insight).

A number of readers of the SD list have insisted that ""good"" explanations of the usefulness of SD are scarce. Since the ""goodness"" of an explanation is judged by the reader, it could be that the many publications describing useful SD applications resonate only with the true believers and that other readers fail to see the usefulness. Or, it could be that people have failed to ""discover"" the papers that would really speak to them. In either case, it seems like a call to action for the SD community:

1) When documenting the results of SD applications make sure that the usefulness of the method is clearly explained...in non-SD terms

2) We need to make it easy for interested parties to find publications that do a good job of explaining the usefulness of SD in a wide variety of applications

Wayne Wakeland, Ph.D.
Associate Professor of Systems Science
Portland State Univ.
Posted by wakeland@pdx.edu
posting date Fri, 06 Jan 2006 10:53:18 -0800
E.Rouwette E.Rouwette fm.ru.nl
Junior Member
Posts: 2
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by E.Rouwette E.Rouwette fm.ru.nl »

Posted by ""E.Rouwette"" <E.Rouwette@fm.ru.nl>
Dear Colm and Jean-Jacques,

There is evidence on the usefulness of SD and related approaches. Colm writes
that 'if I decide to analyse some problem, pick an appropriate method, and
conduct the analysis or build the model then I can document the results?' and
this is indeed what has been done in several conference papers, journal
publications and books. Most of these have the form of in-depth case
descriptions and usually list outcomes such as insights learned, conclusions
that were implemented and results of implementation. I know of only very few
examples of cases where return on investment or monetary value is actually
calculated. An example that comes to mind is the case on Ingalls Shipbuilding
(Cooper, 1980 and section 2.3 of Sterman, 2000) which lists the monetary value
of the modeling effort but not the investment. I think an important reason for
this is that data on or evidence for results are progressively more difficult to
obtain when going from insights to implementation to results. In the last years
several reviews of case reports on modeling have been published. I listed a few
publications below which I think might be relevant to your question.

Andersen DF, Richardson GP, Vennix JAM. 1997. Group model building: adding more
science to the craft. /System Dynamics Review/ 13(2): 187 – 203. This paper
discusses the basics of research into modeling effectiveness. Rouwette, EAJA, Vennix, JAM, Van Mullekom T. Group model building effectiveness:
a review of assessment studies. /System Dynamics Review/, Vol. 18, No. 1. Spring
2002: 5-45. Focuses on group model building cases and analyzes 107 published
reports.
Sterman J. 2000. /Business dynamics/. Boston: McGraw-Hill. chapter 2 for three
studies on applications.

Publications in two other fields might be relevant too. On the topic of
usefulness or effectiveness of soft Operational Research methods (or problem
structuring systems, although these do not completely overlap) reviews are
beginning to appear, see for example:
Mingers J, Rosenhead J, 2004. Problem structuring methods in action. /European
Journal of Operational Research *152*: 530 – 554.

In the electronic meeting systems effectiveness studies are more common and
usually are conducted in laboratory settings. This makes it easier to determine
which causal factors are at work, but more difficult to determine the external
validity of results.
Dennis AR, Wixom BH, Berg RJ van den, 2001. Understanding fit and appropriation
effects in group support systems via meta-analysis. /Management Information
Systems Quarterly/, vol. 25, 167-183.
Fjermestad J, Hiltz SR, 1999. An assessment of group support systems
experimental research: methodology and results. /Journal of Management
Information Systems/, vol. 15: 7-149.
Fjermestad J, Hiltz SR, 2001. A descriptive evaluation of group support systems
case and field studies. /Journal of Management Information Systems./ vol. 17. Pervan G, Lewis LF, Bajwa DS, 2004. Adoption and use of electronic meeting
systems in large Australian and New Zealand organizations. /Group Decision and
Negotiation/ 13(5): 403-414.

Hope this helps.

Best regards,


Etiënne Rouwette

Radboud University Nijmegen
The Netherlands
Posted by ""E.Rouwette"" <E.Rouwette@fm.ru.nl>
posting date Fri, 06 Jan 2006 14:31:48 +0100
Johnson Scott T Scott.Johnson2 b
Junior Member
Posts: 2
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by Johnson Scott T Scott.Johnson2 b »

Posted by ""Johnson, Scott T"" <Scott.Johnson2@bp.com>
I really like Wayne's suggestions, especially ....

> 1) When documenting the results of SD applications make sure that the
> usefulness of the method is clearly explained...in non-SD terms""

A excellent example is ....

""Modelling growth strategy in a biotechnology startup firm"", Morecroft J D W, Lane D C, Viita P S, in Richardson G P ed., Modelling for management, Dartmouth Press, 1996

Following is a specific illustration of the ongoing challenge of getting organizations - one individual at a time - to even consider alternative problem solving approaches ....

George Backus and I have spent the last 12 months trying to get the attention of a BP team working on an important business issue. In the first 10 months we had to rely on email and a few phone calls (unfortunately not including George as he is an outsider) to explain why we thought our offer was worth considering. Eventually we knew when to stop pushing as it became apparent the team was convinced they were already on the correct modeling path (non SD, non agent-based), using smart folks from a respected university. The only concession was a promise to meet George in person in the near future.

Eventually a BP team representative showed up to meet with George in his lab for one hour. George learned that this BP individual mostly does not like big computer models as he doesn't trust them and thinks talking a problem through or using the back of an envelope will always work. George and I are tasked to produce a white paper showing some simple compelling situations where the feedback and delays make it so the human mind gets the wrong answer and then show a more disaggregated (SD,
agent-based) model where the local (emergent) phenomena dominates the problem space.

Not sure where this will end up but we are pleased to at least have a slightly open door and a working rapport.

Scott
Posted by ""Johnson, Scott T"" <Scott.Johnson2@bp.com>
posting date Sat, 7 Jan 2006 11:06:53 -0600
Jean-Jacques Laublé jean-jacques
Senior Member
Posts: 68
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by Jean-Jacques Laublé jean-jacques »

Posted by Jean-Jacques Laublé <jean-jacques.lauble@wanadoo.fr> Hi Etienne This thread is never going to end! I reread the original question from Colm. I reproduce the last phrases of his first post. <But, believe me, it doesn't matter what <stocks and flows are in our system, if we can't explain to our <customers/sponsors/supporters that there'll be more value/volume/profitwhen <we've completed our work!

There is a first question: when Colm writes about usefulness, does he mean the average usefulness through the years or does he mean the documentation about how the SD models where built or is it a bit both? Or saying it in another way: is the question about usefulness of SD, the ability of people in general to use SD usefully or about the insufficient use of SD in general? Colm seems to think that SD is like something you buy from a shop hoping that the value of the object is worth its cost. Excuse me Colm, I am exaggerating. First the profit of the modelling process is continuous during the modelling process and afterwards, and the work is never really completed. This is why SD is difficult to sell too.

Depending on the modelling process you can learn a lot or nothing. This is why describing the usefulness of a modelling process is difficult. If I had to sell a model to somebody I would first propose to build a very simplistic model at first, not too expensive, and work with the model and the client. And I mean a simplistic model has one stock and no more then three feed back loops and preferably less. And finding that first simple model is not so easy. I have started three weeks ago a model and I have now a simplistic model that I feel will be a good starting point for the next steps. But this first starting point has necessitated 9 first models that I have regularly simplified until I got to that simple to understand and to analyse model. And I expect to work many weeks (eventually months) on it until I will go to the next step. The first stock is the one that best represents the situation of our problem evaluating through the time. I would then propose to use that simple model for some times until comes the time when a need for something more elaborate comes. At that time I would study what would be the next step, adding a new stock or refining the rates of the first stock, weighing the advantages and drawbacks of all the options of refinement relative to the original objective, the ability to validate the hypothesis, the easiness to find data, the complexity it will add to the first model, the number of feed back loops it will generate (the less the
better) etc. etc.. Having chosen the next step I will be better prepared to build the second generation model etc. If this process is well shared with the client, the usefulness is an accumulation of experiences that is difficult to measure. We can learn a lot in the first step, a lot with the second and the third and nothing with the fourth if we went to quickly to it. Regards. Jean-Jacques Laublé Allocar Strasbourg France. Posted by Jean-Jacques Laublé <jean-jacques.lauble@wanadoo.fr> posting date Sat, 7 Jan 2006 17:02:03 +0100
Jack Homer jhomer comcast.net
Junior Member
Posts: 5
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by Jack Homer jhomer comcast.net »

Posted by ""Jack Homer"" <jhomer@comcast.net>
I'd like to respond to this question of how one can demonstrate the value of SD models, from the perspective of someone who makes his living doing SD for both commercial and government clients. Let's first make the assumption that the client likes the results of the modeling and actually succeeds in making the appropriate changes based on the lessons learned from the modeling. (Quite often such implementation does not occur, through no fault of the SD study, but for reasons of organizational resistance.) There should then be no problem in documenting what happens subsequent to
implementation: this is the future that actually comes to pass. What is not possible to document is what would have occurred if there had been no implementation, that is, the ""status quo future"": ""What would have happened if we had not implemented the changes?"" One can make assumptions about that status quo, and indeed, the model itself should already have simulated one version of that status quo future, but still these remain unprovable assumptions. And therefore, skeptics can always claim, ""These improvements would have occurred anyway, even without the changes in policy.""

Why is this any different from other cases of investment where an expected ROI can be computed? It's because SD models are done to address problems where everything seems to be changing and interacting, and one can not easily tease out impacts, say, in the form of a spreadsheet. SD models are typically done when things are already in flux and the status quo is not so easy to predict. It's not, ""How much more tax revenue will our community make by allowing in this new mall?"", where one could safely assume a status quo future than looks just like the past. Instead, it's more like, ""How could the employment that this mall provides not only bring us tax revenue in the short term, but also keep people from moving out of this unstable community, and thereby prevent a loss of tax revenue in the longer term?"" In such an environment, there may be fundamental disagreement about what the status quo future of the community is, and therefore disagreement about what the ROI actually is.

Thus, the measure of an SD model's success is ultimately a subjective one, and involves the decision-maker saying, ""I believe such-and-such would have happened if we had not implemented those changes."" In other words, it requires a testimonial, which, I think, is about the best one can expect.

Another tricky issue, one I will address only briefly here, is the problem of skeptics claiming after the fact, ""Those changes would have been made anyway, even without the model."" This is a common situation, because effective SD models change people's minds as the modeling project moves along, to the point that folks in the organization often can't remember what they believed before the modeling study began. The only way I know of to address this problem is to get a ""stake in the ground"", asking the client before the study begins, what they would do if they had to act today. I have tried this with clients, and the answer is often, ""I can tell you what I would do, but I don't think I could get such changes approved in my organization without a stronger argument and more evidence backing me up. That's why you're here, to help me make that argument!"" I think it's true we could make more of an effort at the beginning of projects to get that stake in the ground from the organization (from all key decision-makers and not only from the lead client), and that this would be a helpful way to gauge the impact of a study on decisions made subsequent to the SD study.

Jack Homer
Posted by ""Jack Homer"" <jhomer@comcast.net>
posting date Sat, 7 Jan 2006 09:12:57 -0500
the invigilator the_invigilator
Junior Member
Posts: 3
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by the invigilator the_invigilator »

Posted by ""the invigilator"" <the_invigilator@hotmail.com> Thanks to everyone for a very interesting thread.

I would like to throw another idea into the pot.

Is there any focus on accessibility of SD computer models
to those untrained in the art - making it more intuitive
in other words. For instance, a model that incorporates
explanatory hotlinks for explanations and real-life
examples (with a right mouse click), rather than only
mathematical formulas.

Perhaps also a layered approach that allows one to 'drill down'
into the different levels of abstraction of the model (like the
'internet maps' that you can zoom in). Thus, the casual examiner
could get a rough idea of the logic at a higher level, and not
be alienated by a high level of complexity. Then, those interested
in the detail can zoom in to that a single 'global' stock is made
up of another SD model entirely - one that operates independently
within that stock and from which its relationships with the higher
level variables are derived.

This might also allow other types of modellers - such as agent- based and econometric - to work with us through informing these
'sub (or sub sub) models'. They might be based on the model of small company, or typical 20 year old consumer, who then
participate in the workings of the overall model.

Apologies if this is an old idea.

Invig.
Posted by ""the invigilator"" <the_invigilator@hotmail.com> posting date Mon, 9 Jan 2006 16:04:14 +1100
Jean-Jacques Laublé jean-jacques
Senior Member
Posts: 68
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by Jean-Jacques Laublé jean-jacques »

Posted by Jean-Jacques Laublé <jean-jacques.lauble@wanadoo.fr> Hi Scott

About the PB representative liking simple back of the envelop solution I would ask him to show me how he would resolve the problem or any other problem using his method. I whould then transform his simple demonstration into a very simplistic SD model giving the same result. If his solution is logic this will be possible.I would then make the simplistic model very slightly evoluate and show him that for a little investment, the model adds a little more value and produces a little better solution.

Regards.
Jean-Jacques Laublé Allocar
Strasbourg France
Posted by Jean-Jacques Laublé <jean-jacques.lauble@wanadoo.fr> posting date Sun, 8 Jan 2006 17:31:38 +0100
Joel Rahn jrahn sympatico.ca
Junior Member
Posts: 19
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by Joel Rahn jrahn sympatico.ca »

Posted by Joel Rahn <jrahn@sympatico.ca>
Vensim (and probably other SD modelling systems) can do this by
hot-linking Views (pages with text, diagrams, output) in a model
or, with considerably more effort, by building a VenApp to present
and run the model.

Posted by Joel Rahn <jrahn@sympatico.ca>
posting date Mon, 09 Jan 2006 10:10:50 -0500
Colm Toolan subscriptions toolan
Junior Member
Posts: 9
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by Colm Toolan subscriptions toolan »

Posted by ""Colm Toolan"" <subscriptions@toolan.de>
Thanks to all who have responded and particularly those who pointed at specific documentation.

[ No more postings on this thread please ]

Posted by Wayne Wakeland:
>In either case, it seems like a call to action for the SD community:
>1) When documenting the results of SD applications make sure that the
>usefulness of the method is clearly explained...in non-SD terms
>2) We need to make it easy for interested parties to find publications
>that do a good job of explaining the usefulness of SD in a wide variety
>of applications

Wayne stresses ""non-SD terms"" and "" the usefulness of SD in a wide variety of applications"". Like myself, he is suggesting that we need to go beyond ""preaching to the converted"" - we need to give the potential sponsors the material they need to consider using SD. Assuming that they are not now aware of the benefits then we need to demonstrate what these benefits are, in language that they understand, addressing issues that concern them.

Posted by ""Johnson, Scott T"":
> George and I are tasked to produce a white paper showing some simple
>compelling situations where the feedback and delays make it so the
>human mind gets the wrong answer and then show a more disaggregated
>(SD,agent-based) model where the local (emergent) phenomena dominates
>the problem space.

Is this White Paper (or extracts) something that could be made available? Seems like it could form the basis for some useful insights to persuade the uninitiated.


Posted by Jean-Jacques Laublé:
> Colm seems to think that SD is like something you buy from a shop
>hoping that the value of the object is worth its cost. Excuse me Colm,
>I am exaggerating. First the profit of the modelling process is
>continuous during the modelling process and afterwards, and the work is
>never really completed. This is why SD is difficult to sell too.

Jean-Jaques, I've no problem with your exaggeration - maybe it's just what we need to focus on what it is I'm looking for. For lots of people SD
***is*** something you buy in a shop: It's one way of utilising scarce resources to help address an issue or a problem. I need someway to compare it with other approaches that I could spend my resources on! Telling me that ""the work is never really completed"" doesn't wash as an argument!


> If this process is well shared with the client, the usefulness is an
> accumulation of experiences that is difficult to measure.

I accept that there is value in the experience and in the insights gained in the process. However, I feel that this is not sufficient to persuade someone who is not fully aware of the quality and depth of these insights to allocate the necessary resources. Even if we cannot produce concrete, desired results in every specific case, we still need the body of documentation that illustrates the value of SD over other methods. People after all are convinced of the generic merits of IT projects and of the associated, hoped-for value, notwithstanding the fact that The Standish Group claims that 84% of projects do not meet the original goals in full or in part.

Posted by Jack Homer:
> The only way I know of to address this problem is to get a ""stake in
>the ground"", asking the client before the study begins, what they would
>do if they had to act today.

I see Jack's argument and agree with the points raised. However, I'm attempting to address the issues that occur earlier in the process than the situation quoted above. How do we demonstrate the value of the SD approach
***before*** the project begins?

It's obvious we are talking something very nebulous here - the usefulness, the value are sometimes hard to identify, are often unexpected (the Jean-Jacques ""experience"") and are always difficult to measure. It reminds me a little of the way one has to explain and justify one's CV in a job interview!

Maybe that's the approach we should take - ""I did that at that time, because .. The results were .... In hindsight, I see .... But overall the project was successful because of ...."" Obviously, if metrics (RoI, increases in x, decreases in y) are available they can be included, but even without the numbers it should be possible that we did something usefull!


Colm Toolan, Business Architect
Posted by ""Colm Toolan"" <subscriptions@toolan.de>
posting date Mon, 9 Jan 2006 21:10:57 +0100
Tobias Lorenz space56 freenet.de
Junior Member
Posts: 10
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by Tobias Lorenz space56 freenet.de »

Posted by ""Tobias Lorenz"" <space56@freenet.de>
Dear Invigilator,

We had a similar idea when writing the paper ""comprehensibility
as a discriminaton criterion for agent-based modelling and system
dynamics: an empirical approach"" for the SD conference in Boston.
It is available online via the following link:
http://www.systemdynamics.org/conf2005/ ... SSI175.pdf

The idea behind the experiment described there was quite simple.
Describe the same problem in an SD-manner and an AB-manner and then
check empirically which approach shows the better results. Although
i'd say that the results are not representative yet for an overall
client group, they at least make way for a less ideological and more
empirical discussion...

Yours
Tobias Lorenz
Posted by ""Tobias Lorenz"" <space56@freenet.de>
posting date Mon, 9 Jan 2006 13:14:04 +0100
John Gunkler jgunkler sprintmail
Member
Posts: 30
Joined: Fri Mar 29, 2002 3:39 am

The ""usefulness"" of SD

Post by John Gunkler jgunkler sprintmail »

Posted by ""John Gunkler"" <jgunkler@sprintmail.com>
Jack Homer makes an excellent point, as he usually does, when he talks about putting a ""stake in the ground."" I have found this method useful in lots of non-SD situations, as well -- for the same reasons Jack mentions: namely, that people forget how things were at the beginning of a project.

That's why I often start projects with an investigation into the status quo that includes projections of financial and operational results ""if we continue to do what we've been doing."" The popular Lean and Six Sigma methodologies for process improvement both start with this step -- in Lean enterprise we call it capturing the ""current state."" And this includes filling ""data boxes"" with information describing the current processes -- lead time, cycle time, number of defects produced, machine uptime, process cycle efficiency, etc.

I believe that it's fair to compare what is subsequently done with the ""current state"" when the project began -- but, as Jack says, some people want to compare what is subsequently done with what ""would have been done without your project."" This can get ridiculous. The only way to deal with it, as Jack suggests, is to force people at the very beginning of the project to agree to document what they think they would do (before you start working with them.) The advantage of an SD model is that you can then simulate their policy changes and, to the extent that the clients accept the model's validity, compare their simulated results with actual results obtained.


John
Posted by ""John Gunkler"" <jgunkler@sprintmail.com>
posting date Mon, 9 Jan 2006 09:49:33 -0600
Locked