Empirical Evaluations of Computer Simulations

[wtignor@erols.com (Warren Tignor]

I am working on a study that may be of interest to you. I am interested in
any comments from the readership. I appreciate the help to date that I
have received from several participants of this mailing list. If anyone
would like to participate with the analysis of some business cases, please
let me know. Thank you.

Abstract

THE ROLE OF INFORMATION TECHNOLOGY
IN A LEARNING ORGANIZATION:

Case Study and Causal Diagramming

by

Warren Willard Tignor

Introduction

Competition requires businesses to leverage capabilities at every
opportunity. Businesses use information technology as a strategic asset to
improve their competitive positions. In the context of this study,
information technology includes the technologies of computers and
telecommunications (hardware and software), (Sprague & McNurlin, 1993).

Case study provides a low technology method to learn about business
capabilities. Case study has produced insights that help businesses
leverage or avoid situations similar to those documented in cases.
Presently, a typical way that decision makers are trained to prepare for
business careers is learning through the case study of hypothetical or
actual business experiences. The case study process involves reading the
case text to determine the facts of the case, the issue or problem, the
business decision, and lastly, some reasoning as to whether the decision is
supported by the factual results of the case.

With the advent of computerized information systems, new tools are
available to help businesses use information technology as a competitive,
strategic asset. One tool is Systems Dynamics. According to Wolstenholme
(1990), Systems Dynamics is a rigorous method for qualitative description,
exploitation and analysis of complex systems in terms of their processes,
information, organizational boundaries and strategies. System Dynamics
facilitates quantitative simulation modeling and analysis for the design of
the system structure and control.

Within System Dynamics, causal diagramming offers a convenient way to
represent the structure and behavior of systems composed of interacting
feedback loops. Causal diagrams identify the principal feedback loops
without distinguishing between the nature of the interconnected variables.
These diagrams play two important roles in Systems Dynamics:

1. They serve as preliminary sketches of causal hypothesis during
model development.

2. They can simplify the illustration of the model.

In both roles, causal diagrams allow the analyst to quickly communicate the
structural assumptions underlying a model.

The learning organization effective use of case study and causal diagrams
will require consistent examination of the whole business system, rather
than just trying to fix isolated problems. According to Dodgson (1993), a
learning organization is a firm that purposefully constructs structure and
strategies to enhance and maximize organizational learning.

The "fifth discipline" uses this conceptual framework of examination of the
whole and tools of systems thinking to clarify fully problem patterns to
understand how to change them most effectively, Senge (1990). System
Dynamics readily lends itself to learning organization environments.
=46orrester (1991) uses Systems Dynamics as the theory, method, and
philosophy needed to analyze the behavior of systems in not only business,
but also in environmental change, politics, economic behavior, medicine,
engineering, and other fields using simulation technology.

Statement Of The Problem

Businesses that are "learning organizations" will capitalize upon
techniques and tools that improve competitiveness. A learning organization
may use case study and causal diagramming as well as other approaches to
inquire into the systemic consequences of their plans or actions.

Businesses need to find better and faster ways to adapt in order to compete
in todays high technology and fast paced environment. Learning
organizations provide an environment that encourages finding better and
faster ways to adapt in todays high technology and fast paced world by:

1. Detecting and correcting errors.
2. Improving actions through knowledge, and
3. Developing the broad skills of their work-force.


This study addresses the role information technology plays in a learning
organization. In general, a learning organization takes advantage of
current and past knowledge to further the goals of the business.
Information technology permeates business organizations today as the trend
to automate continues to accelerate from the 1960s to the present. In many
instances, information technology helps businesses do things quicker,
better and cheaper; in most cases this results in better profitability.

Case study represents a vast source of past business knowledge available
for future application. Causal diagrams will help the business analyst
identify the major influencing factors of a case study and the feedback
mechanism that impacts the case results. This study presents the
hypothesis that the collaborative effect of case study and information
technology using causal diagramming in a learning organization will improve
the potential of businesses to adapt to new competitive situations.

Brief Description Of The Research Method And Design

A qualitative comparison of case studies and causal diagrams will be
performed to illustrate the collaborative role of information technology in
a learning organization to test the study hypothesis. The qualitative
comparison will show how case study and causal diagrams will improve the
opportunity for learning.

The methodology proposed consists of the following procedures:

1. Determine the attributes of a learning organization as defined and
referenced in the research material. This is the work of Argyris and Sch=F6=
n
(1978), Fiol and Lyles (1985), Dodgson (1993), Kofman and Senge (1995),
Nevis et al. (1995), and Senge (1990). Once a list of the attributes of a
learning organization for each author is established, an analysis of common
and unique attributes will be constructed. The importance of this
procedure is to establish a basis for determining what is a "learning
organization" based on a set of shared attributes. A set of common
attributes and unique attributes are needed in order to support the next
step of the methodology, Case Study and System Dynamics.

2. Review the case study material to identify the attributes of a case
study to: a. determine whether Case Study complements the needs of a
Learning Organization and b. identify the format of a case study in order
to data gather case study results in a uniform manner and to have the
results consistently address the same questions, e.g., the facts, the
issue.

3. Review the System Dynamics material to identify the attributes of
causal diagrams to: a. determine whether causal diagramming complements
the needs of a Learning Organization and b. identify the format of a
causal diagram in order to data gather causal diagramming results in a
uniform manner and to have the results consistently address the same
questions, e.g., the structure, the feedback.

The steps above will show the generic attributes of a learning organization
as well as those for a case study and causal diagramming. At the top
level, this part of the methodology will establish whether case study and
System Dynamics do address different parts of a set of learning
organization attributes. Additionally, the procedure will establish
whether there are attributes of case study and causal diagramming that
overlap and others that are unique. Lastly, an initial basis will exist to
support the study hypothesis that case study and causal diagramming support
a learning organization in a collaborative manner.

Once the above steps are completed, the methodology will be extended to
review specific business cases using the case study approach and then
review the same cases using System Dynamics in the form of causal diagrams
for comparison. The formats established in the initial stages of the
methodology will be reused during this stage to keep a consistent set of
data results. For individual cases, the Socratic method will be used to
determine the case facts, the business issue, the decision or results of
the case, and the reasoning behind decision or results as established by
the format. Using causal diagrams as defined by for System Dynamics, the
methodology will determine the major influencing factors of the case, based
on variables and levels, particularly the feedback loops involved. The
influencing factors will be identified as positive or negative influences.
Lastly, the entire causal diagram will be identified as either a positive
or negative influence diagram.

After completing the case and causal diagram analysis, the results will be
compared to the learning organization attributes to determine the
collaborative manner in which the two methods support learning or not.

Major Findings And Their Significance

TBD

Conclusions

TBD

References

Argyris, C & Sch=F6n, D.A. (1978) . Organizational learning: A theory of
action perspective. Reading, MA: Addison-Wesley.

Dodgson, M. (1993) . Organizational learning: A review of some
literature. Organization Studies, 14/3, 375-394.

=46iol, C.M., & Lyles, M.A. (1985) . Organizational learning. Academy of
Management Review, 10/4, 803-813.

=46orrester, J. W. (1991) . Systems dynamics and the lessons of 35 years.
In Kenyon B. De Greene (Ed.), Systemic basis of policy making in the 1990s
(pp. 5-34). [http://tfnet.ils.unc.edu/~gotwals/stell ... essons.txt].

Kofman, F., & Senge, P. (1995) . Communities of commitment: The heart of
the learning organization. In Sarita Chawla & John Renesch (Eds.),
Learning Organizations (pp. 15-43). Productivity Press: Portland, OR.

Nevis,.C., DiBella, J, & Gould, M. (1995) . Understanding organizations
as learning systems. Sloan Management Review, Winter, 73-85.

Senge, P. M. (1990) . The fifth discipline: the art & practice of the
learning organization. New York: Doubleday/Currency.

Sprague, H. & McNurlin, C. (1993) . Information systems management in
practice. New Jersey: Prentice Hall.

Wolstenholme, E. F. (1990) . System enquiry: A system dynamic approach.
Chicester: Wiley & Sons.



Warren Tignor
wtignor@erols.com
TRW
410-290-0500 work
410-290-0515 fax

[Nathu@aol.com]

I am currently working on my Doctoral Dissertation where I am attempting to
empirically evaluate a computer simulation using ithink as a teaching tool to
a group of managers. I am thinking of having them draw a mnetal model using
causal loops pre and post the simulation intervention with a two week follow
up survey on whether they have or plan to use the concepts learned in their
various jobs.

I would VERY MUCH appreciate any leads you might have in this regard
(references) and/or any suggestions you might have. John Sterman suggested I
contact you and mentioned that you are editing an SD Review Special Issue on
a related topic.

Any information/suggestions you could give me would be MOST WELCOME!!!

THANK YOU!!!!

Nathan H. Taylor
Nathu@aol.com
109 E. Murray Avenue
Durham, NC 27704
(919) 220-5425

[Guenther Ossimitz]

Hello,

I did some similar research work in a project called
"Entwicklung vernetzten Denkens" ("development of
systemic thinking") among about 120 students of grade 8 - 11
(age 15-19) in some Austrian upper secondary schools.

The students got about 20 hours of teaching about
system dynamics modelling and computer simulation
using Powersim by their teachers
(in mathematics and informatics courses).

Prior and after the teaching the students were tested
by a neutral person (the paper-and-pencil oriented test
was not known to the teachers). Some students were also
interviewed about their written test-answers.

The test was constructed to give some clues about the
state and the developement of systemic thinking skill
of the students.
A raw version of the evaluation of the test is already
finished; I would appreciate to discuss some results
with you.

Best wishes

Guenther Ossimitz


---------------------------
Dr. Guenther Ossimitz
University of Klagenfurt
A-9020 Universitaetsstr. 65
ossimitz@bigfoot.com
http://www.uni-klu.ac.at/users/gossimit/main.htm

[E.Rouwette@MAW.KUN.NL]

In reply to Nathan Taylors mail about evaluation:

I am a PhD. student working with Jac Vennix in Nijmegen, the Netherlands. My
research is also about the effects of model-building on individuals. In his
dissertation Jac compared the effects of SD and traditional classroom
lectures on interest, knowledge and quality of policy theory (assumptions
underlying the policy of interest). I know of a number of other studies done
in the Netherlands, mostly case studies, and some of them using qualitative
modelling only. I have listed several below.


1. Akkermans, H.A. 1995. Modelling With Managers. Participative Business
Modelling for Effective Strategic Decision-making. University of Eindhoven.
2. Bronkhorst, E.M. 1995. Modelling the Dutch Dental Health Care System: a
Comprehensive System Dynamics Approach. University of Nijmegen.
3. Scheper, W.J. 1991. Group Decision Support Systems. An Inquiry Into
Theoretical and Philosophical Issues. University of Utrecht.
4. Vennix, J.A.M., Scheper, W. J., and R. Willems. 1993. Group Model
Building: What Does The Client Think Of It? In Proceedings System Dynamics
1993, Ed. E. Zepeda and J.A.D. Machuca.
5. Vennix, J.A.M. 1990. Mental Models and Computer Models. Design and
Evaluation Of A Computer-Based Learning Environment For Policy Making.
University of Nijmegen.
6. Vennix, J.A.M. 1995. Building Consensus in Strategic Decision Making:
Insights From the Process of Group Model-Building. In Group Decision and
Negotiation. Vol. 4 no. 3.
7. Vennix, J.A.M. 1996. Group Model-Building: Facilitating Team Learning
Using System Dynamics. Chichester: John Wiley & Sons.
8. Vennix, J.A.M., H.A. Akkermans, and E.A.J.A. Rouwette. 1996. Group
Model-Building to Facilitate Organizational Change: An Exploratory Study.
System Dynamics Review. Vol. 12 No. 1: 39-58.
9. Verburgh, L.D. 1994. Participative Policy Modelling Applied to the Health
Care Insurance Industry. University of Nijmegen.

In trying to make a contribution to all of this I focus on the connection
between making a model in collaboration with managers of a client
organization and behavior of participants. (So perhaps I should clarify some
concepts: Id like to do research into the effects of Group Model-Building on
behavior of participants, behavior in the sense of putting the
recommandations of the session into practice).

In my opinion the existing theories pose a link between modelling and
learning (and consensus in some cases), but two things remain unclear: (a)
what exactly produces the effects in model-building? The simulation results,
causal diagrams, discussions or the facilitator? Only few studies that I know
of have tried to open the black box, and in these only reactions of
subjects were used ("how much do you think the diagrams contributed to the
effect of the sessions?").

(b) What is the link between learning and action? Most of the above studies
concentrate on changes in mental models, but one can ask the question what
use change in a mental model is if nothing shows in behavior. The next
question is what in a mental model influences behavior. I am aware that the
connection between learning and action is probably what half of psychology is
about, but some advances can be made. In no. 7 of the above, for example, we
used a well-tested model from social psychology that predicts action from
beliefs and evaluations. In my opinion (and I hope to test it in my
research!) attitudes and emotion or evaluation need to be taken into account
for changing behavior, as well as cognition. The concept mental model seems
to highlight only the cognitive part.

I am very interested in any comments, and I hope Nathans mail started a
discussion about this subject.

Thank you,

Etienne Rouwette
E.Rouwette@MAW.KUN.NL