Im afraid that Alexander Leus has somewhat missed the point I was trying to
express which is the degree of uncertainty introduced into a model by
multiplying factors which are inherently uncertain. Put simply, if we do
not know the real values of parameters we do not know if the model output
is right. If we dont know whether or not the model is producing the
right answers we do not know how much confidence we can have in the
conclusions we draw from the output and hence in the recommendations we
might make, whether those are made to a fee-paying client or to a body of
scholarship. My A, B, C example, where A, B and C are dynamic variables, was
only to show how small amounts of parametric uncertainty, well within the
poor data (or subject matter expertise as Leus call it) can produce a very
large degree of variability in the output, my example was more than a factor
of 3. No amount of fiddling with speculative probability distributions
undermines this inherent difficulty.
Let me pose some research questions:
SAMPLING Even with the Leus approach, how many model runs would be needed to
give a distribution of model outputs? How do you measure output? (A, B and
C are usually defined by some sort of curve so we might need a huge number
of runs to sample between an upper and a lower curve). Even if A, B and C
are constants (in which case they would not be multiplied), there are 2^3
(i.e.
combinations and therefore 8 runs, over and above the base case, areneeded. If there are 10 constants, 2^10 runs are needed and I dont even
want to think about how many that is. Even with a distribution of output,
measured somehow, it is no help to say that there is a small probability
that this could be the right answer and another small probability that
something quite different is the right conclusion and a large probability
that it could be something which is neither of those.
FORMULATION Going back to three variables, how do we know that V=A*B*C is a
reasonable model, as opposed to a convenient way of throwing in an equation?
If V, A, B and C are NOT dimensionless then dimensional analysis will
resolve the issue (after > 30 years in SD, dimensional analysis is my most
basic tool, but I am depressed by the people I meet, some with PhDs in SD,
who have no idea what DA is). If, however, C , for instance, is
dimensionless (and I am even more depressed by the people who have no idea
at all of when something can legitimately be dimensionless) then there are
no strong dimensional grounds for distinguishing between A*B/C, A*B*C or
even A*B^C. If two are dimensionless, the problem is more acute and if all
of V, A, B and C are dimensionless the problem is unsolvable except by
assumption. (and that isnt good practice!).
INDEPENDENCE How do we know that A, B and C are independent? My book goes
into this issue and shows some ways of treating cases of multiplier
dependency.
Maybe with some thought we could pose a series of research issues (or rules
of good practice) though most of those rules are in several well-established
texts.
Geoff
From: "geoff coyle" <geoff.coyle@btinternet.com>