Using Statistics in Dynamics Models
Posted: Tue Feb 03, 2004 5:35 pm
Hi Jim
The 1,2,3,4 is what I effectively use with some modifications and it does
not annoy me at all. This is why models are wrong. And I am not sure that it
horrifies a statistician if you explain him that you have not the pretension
to certify that the A hypothesis is true and you use the 1,2,3,4 method with
the following precisions.
I think that most modellers are obliged to use the 1,2,3,4 method, having
nothing better.
To cope with the all models are wrong sentence, I do three things: I will
not use the word truth, I will carefully chose the B assumption and the
other assumptions specified in the 4.
That means that any B assumption is not automatically adapted, and only some
implications of A may be considered as true.
So what does a real statistician do when faced with a situation in
which he knows from first principles that the model is wrong?
I can reply to this question with some preliminary clarifications.
It is necessary to define the word true or wrong.
For me these words do not represent well the reality of the problems I am
trying to solve.
For the word true I would prefer adapted to the purpose.
For the word wrong not adapted to the purpose.
As nothing is perfectly adapted to any purpose I will use adapted to a
certain degree to the purpose. So it the degree is 0 it is not adapted at
all.
Then the problem I am faced with is not a wrong model but a model which I
want to verify that it is sufficiently adapted to a certain purpose. It is my
decision to fix the level of adaptation and the purpose.
To proceed further on I will take a concrete example, which I am presently
working on.
Suppose I want to reconsider the way I have been pricing a type of car in my
company a short term rent a truck, vans and cars company.
The purpose of the model is to increase the margin of the type of car for
the next 24 months.
It is my responsibility to fix the minimum percentage of increase. To resume
the purpose is to find a better method of pricing then the one currently
used.
If I have a finished model, I will for instance test different pricing
strategy and choose the one that has the best margin.
To be confident in the model, I will first verify that if I put the same
price applied for instance the last year, it will produce the real margin
experienced over the same period.
This is of course not the final purpose. I do not want to use that model to
prove that what happened could really happen. But I decide that if my model
is able to reproduce to a certain level the results of the past, it will
already be a first point. And I can assure you that it is not
evident, due to the number of different prices depending on the time in the
year, the kilometres per day, the duration of the renting, the kind of
customer, etc..
To verify this I can use any method I decide, for instance the classical
maximum likelihood
implemented in Vensim as I have a set of data, the real and the calculated
margin over a 12 months period.
It is too my responsibility and my experience to decide the level of
likelihood.
Of course I can verify other outcome from the model, but I can decide that
being concerned with the margin over a 24 months period, this test is
enough. This is where the purpose is important. I do not need a true model,
but a model that is adapted to a certain purpose.
If the level of adaptation is not high enough, then I have to correct my
model until it reaches the correct level of adaptation.
Once I have verified that the model can reproduce approximately the past, I
can decide that the test is sufficient enough to be used with different
prices than the one applied last year, and
observe the behaviour of the margin. I know that my model may be bugged,
because I tested the model with past data, and nothing proves that the model
is still adapted to the purpose with different prices. Suppose I test an
increase of price of 20%, my model may not describe the reaction of the
customers if I am currently pricing 20% more then the competition.
It is then up to me to decide if the first purpose is enough or not. If not
I will have to define a second purpose closer to the original one.
That kind of problem is automatic if you test a new policy. You have never
the real data that correspond to all the possible policies. It is then
necessary to believe that the policy calculated will produce sufficient
results.
I have in my case, a great chance. I can reproduce an experiment like in the
scientific word.
I can then be more confident on my model, by producing the data that
correspond to the new policy, to see if the reality is conform to the
prediction of the model.
I will have tested the model with old data and with the new data produced by
the effective application of the policy. This new testing using new data,
will of course be more adapted to the purpose which is to modify prices.
To resume, there is always a moment when some belief eventually irrational
is necessary to take a decision. The problem is to make the irrational part
as small as possible. But I can assure you that compared to the process I
used years ago (I prefer not to talk about it) this new method is much more
adapted to the purpose then the previous one, the only thing I care about.
And I have not written anything about truth.
Regards.
J.J. Laublé, Allocar rent a car company
From: =?iso-8859-1?Q?Jean-Jacques_Laubl=E9?= <JEAN-JACQUES.LAUBLE@WANADOO.FR>
Strasbourg France
The 1,2,3,4 is what I effectively use with some modifications and it does
not annoy me at all. This is why models are wrong. And I am not sure that it
horrifies a statistician if you explain him that you have not the pretension
to certify that the A hypothesis is true and you use the 1,2,3,4 method with
the following precisions.
I think that most modellers are obliged to use the 1,2,3,4 method, having
nothing better.
To cope with the all models are wrong sentence, I do three things: I will
not use the word truth, I will carefully chose the B assumption and the
other assumptions specified in the 4.
That means that any B assumption is not automatically adapted, and only some
implications of A may be considered as true.
So what does a real statistician do when faced with a situation in
which he knows from first principles that the model is wrong?
I can reply to this question with some preliminary clarifications.
It is necessary to define the word true or wrong.
For me these words do not represent well the reality of the problems I am
trying to solve.
For the word true I would prefer adapted to the purpose.
For the word wrong not adapted to the purpose.
As nothing is perfectly adapted to any purpose I will use adapted to a
certain degree to the purpose. So it the degree is 0 it is not adapted at
all.
Then the problem I am faced with is not a wrong model but a model which I
want to verify that it is sufficiently adapted to a certain purpose. It is my
decision to fix the level of adaptation and the purpose.
To proceed further on I will take a concrete example, which I am presently
working on.
Suppose I want to reconsider the way I have been pricing a type of car in my
company a short term rent a truck, vans and cars company.
The purpose of the model is to increase the margin of the type of car for
the next 24 months.
It is my responsibility to fix the minimum percentage of increase. To resume
the purpose is to find a better method of pricing then the one currently
used.
If I have a finished model, I will for instance test different pricing
strategy and choose the one that has the best margin.
To be confident in the model, I will first verify that if I put the same
price applied for instance the last year, it will produce the real margin
experienced over the same period.
This is of course not the final purpose. I do not want to use that model to
prove that what happened could really happen. But I decide that if my model
is able to reproduce to a certain level the results of the past, it will
already be a first point. And I can assure you that it is not
evident, due to the number of different prices depending on the time in the
year, the kilometres per day, the duration of the renting, the kind of
customer, etc..
To verify this I can use any method I decide, for instance the classical
maximum likelihood
implemented in Vensim as I have a set of data, the real and the calculated
margin over a 12 months period.
It is too my responsibility and my experience to decide the level of
likelihood.
Of course I can verify other outcome from the model, but I can decide that
being concerned with the margin over a 24 months period, this test is
enough. This is where the purpose is important. I do not need a true model,
but a model that is adapted to a certain purpose.
If the level of adaptation is not high enough, then I have to correct my
model until it reaches the correct level of adaptation.
Once I have verified that the model can reproduce approximately the past, I
can decide that the test is sufficient enough to be used with different
prices than the one applied last year, and
observe the behaviour of the margin. I know that my model may be bugged,
because I tested the model with past data, and nothing proves that the model
is still adapted to the purpose with different prices. Suppose I test an
increase of price of 20%, my model may not describe the reaction of the
customers if I am currently pricing 20% more then the competition.
It is then up to me to decide if the first purpose is enough or not. If not
I will have to define a second purpose closer to the original one.
That kind of problem is automatic if you test a new policy. You have never
the real data that correspond to all the possible policies. It is then
necessary to believe that the policy calculated will produce sufficient
results.
I have in my case, a great chance. I can reproduce an experiment like in the
scientific word.
I can then be more confident on my model, by producing the data that
correspond to the new policy, to see if the reality is conform to the
prediction of the model.
I will have tested the model with old data and with the new data produced by
the effective application of the policy. This new testing using new data,
will of course be more adapted to the purpose which is to modify prices.
To resume, there is always a moment when some belief eventually irrational
is necessary to take a decision. The problem is to make the irrational part
as small as possible. But I can assure you that compared to the process I
used years ago (I prefer not to talk about it) this new method is much more
adapted to the purpose then the previous one, the only thing I care about.
And I have not written anything about truth.
Regards.
J.J. Laublé, Allocar rent a car company
From: =?iso-8859-1?Q?Jean-Jacques_Laubl=E9?= <JEAN-JACQUES.LAUBLE@WANADOO.FR>
Strasbourg France