Help with model calibration

Use this forum to post Vensim related questions.
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

The last simulation values are in the .out file. I'll take a look at the rest.

Here's my .out:

Code: Select all

:COMSYS After 19509169 simulations
:COMSYS Best payoff is 14853.3
:COMSYS User terminated multiple search session
:OPTIMIZER=Powell
:SENSITIVITY=Off
:MULTIPLE_START=RRandom
:RANDOM_NUMBER=Default
:OUTPUT_LEVEL=On
:TRACE=Off
:MAX_ITERATIONS=1000
:RESTART_MAX=0
:PASS_LIMIT=2
:FRACTIONAL_TOLERANCE=0.0003
:TOLERANCE_MULTIPLIER=21
:ABSOLUTE_TOLERANCE=1
:SCALE_ABSOLUTE=1
:VECTOR_POINTS=25
:MCINITMETHOD=0
:MCPAYOFFTYPE=0
:MCRECORD=0
:MCSCHEDULE=0
:MCLIMIT=0
:MCBURNIN=0
:MCNCHAINS=2
:MCOUTLIER=0.05
:MCGAMMA=1
:MCEPSILON=0.01
:MCDELTA=0.0001
:MCJUMP=0.05
:MCUPDATEPAIRS=2
:MCXOVER=0.2
:MCTEMP=1
:MCFTEMP=1
:MCCOOLING=1000
10 <= baseline memories[Driver1] = 115.874  <= 315
10 <= baseline memories[Driver2] = 162.782  <= 315
10 <= baseline memories[Driver3] = 150.436  <= 315
10 <= baseline memories[Driver4] = 97.4185  <= 315
10 <= baseline memories[Driver5] = 10  <= 315
10 <= baseline memories[Driver6] = 51.9877  <= 315
10 <= baseline memories[Driver7] = 128.865  <= 315
10 <= baseline memories[Driver9] = 10  <= 315
10 <= baseline memories[Driver11] = 32.0282  <= 315
10 <= baseline memories[Driver13] = 171.574  <= 315
10 <= baseline memories[Driver14] = 73.7726  <= 315
10 <= baseline memories[Driver15] = 89.5762  <= 315
10 <= baseline memories[Driver16] = 11.477  <= 315
10 <= baseline memories[Driver17] = 172.32  <= 315
10 <= baseline memories[Driver18] = 45.0006  <= 315
10 <= baseline memories[Driver19] = 106.077  <= 315
10 <= baseline memories[Driver21] = 65.1144  <= 315
10 <= baseline memories[Driver23] = 10.484  <= 315
10 <= baseline memories[Driver24] = 72.9756  <= 315
10 <= baseline memories[Driver25] = 145.071  <= 315
10 <= baseline memories[Driver27] = 168.835  <= 315
10 <= baseline memories[Driver28] = 10  <= 315
10 <= baseline memories[Driver29] = 80.1412  <= 315
10 <= baseline memories[Driver30] = 122.208  <= 315
10 <= baseline memories[Driver31] = 49.9086  <= 315
10 <= baseline memories[Driver32] = 172.015  <= 315
10 <= baseline memories[Driver33] = 124.297  <= 315
10 <= baseline memories[Driver34] = 10  <= 315
10 <= baseline memories[Driver35] = 10  <= 315
10 <= baseline memories[Driver36] = 26.4813  <= 315
10 <= baseline memories[Driver37] = 11.27  <= 315
10 <= baseline memories[Driver42] = 190.135  <= 315
10 <= baseline memories[Driver43] = 73.826  <= 315
10 <= baseline memories[Driver44] = 180.835  <= 315
10 <= baseline memories[Driver45] = 104.835  <= 315
10 <= baseline memories[Driver51] = 14.027  <= 315
10 <= baseline memories[Driver52] = 76.7154  <= 315
10 <= baseline memories[Driver53] = 36.3467  <= 315
10 <= baseline memories[Driver54] = 104.517  <= 315
10 <= baseline memories[Driver55] = 38.19  <= 315
10 <= baseline memories[Driver56] = 123.513  <= 315
10 <= baseline memories[Driver57] = 10  <= 315
10 <= baseline memories[Driver58] = 104.569  <= 315
10 <= baseline memories[Driver60] = 112.869  <= 315
10 <= baseline memories[Driver208] = 126.681  <= 315
10 <= baseline memories[Driver212] = 82.2941  <= 315
10 <= baseline memories[Driver246] = 88.25  <= 315
200 <= baseline mileage[Driver1] = 202.732  <= 9330
200 <= baseline mileage[Driver2] = 1034.98  <= 9330
200 <= baseline mileage[Driver3] = 8921.56  <= 9330
200 <= baseline mileage[Driver4] = 3855.69  <= 9330
200 <= baseline mileage[Driver5] = 1890.98  <= 9330
200 <= baseline mileage[Driver6] = 4575.54  <= 9330
200 <= baseline mileage[Driver7] = 8456.8  <= 9330
200 <= baseline mileage[Driver9] = 8736.06  <= 9330
200 <= baseline mileage[Driver11] = 9330  <= 9330
200 <= baseline mileage[Driver13] = 200  <= 9330
200 <= baseline mileage[Driver14] = 3447.38  <= 9330
200 <= baseline mileage[Driver15] = 3541.39  <= 9330
200 <= baseline mileage[Driver16] = 2108.11  <= 9330
200 <= baseline mileage[Driver17] = 5696.27  <= 9330
200 <= baseline mileage[Driver18] = 5665.03  <= 9330
200 <= baseline mileage[Driver19] = 5161.84  <= 9330
200 <= baseline mileage[Driver21] = 2901.56  <= 9330
200 <= baseline mileage[Driver23] = 305.661  <= 9330
200 <= baseline mileage[Driver24] = 9284.73  <= 9330
200 <= baseline mileage[Driver25] = 3562.77  <= 9330
200 <= baseline mileage[Driver27] = 3786.87  <= 9330
200 <= baseline mileage[Driver28] = 9330  <= 9330
200 <= baseline mileage[Driver29] = 8709.63  <= 9330
200 <= baseline mileage[Driver30] = 1247.1  <= 9330
200 <= baseline mileage[Driver31] = 2190.45  <= 9330
200 <= baseline mileage[Driver32] = 6079.56  <= 9330
200 <= baseline mileage[Driver33] = 5964.06  <= 9330
200 <= baseline mileage[Driver34] = 249.339  <= 9330
200 <= baseline mileage[Driver35] = 7963.68  <= 9330
200 <= baseline mileage[Driver36] = 719.812  <= 9330
200 <= baseline mileage[Driver37] = 8689.67  <= 9330
200 <= baseline mileage[Driver42] = 5863.98  <= 9330
200 <= baseline mileage[Driver43] = 9330  <= 9330
200 <= baseline mileage[Driver44] = 2011.24  <= 9330
200 <= baseline mileage[Driver45] = 5764.23  <= 9330
200 <= baseline mileage[Driver51] = 9274.42  <= 9330
200 <= baseline mileage[Driver52] = 3105.41  <= 9330
200 <= baseline mileage[Driver53] = 6721.54  <= 9330
200 <= baseline mileage[Driver54] = 4354.22  <= 9330
200 <= baseline mileage[Driver55] = 206.22  <= 9330
200 <= baseline mileage[Driver56] = 4542.22  <= 9330
200 <= baseline mileage[Driver57] = 201.621  <= 9330
200 <= baseline mileage[Driver58] = 6007.5  <= 9330
200 <= baseline mileage[Driver60] = 9226.56  <= 9330
200 <= baseline mileage[Driver208] = 200  <= 9330
200 <= baseline mileage[Driver212] = 5982.4  <= 9330
200 <= baseline mileage[Driver246] = 4132.41  <= 9330
15 <= driving need[Driver1] = 1256.85  <= 1500
15 <= driving need[Driver2] = 1500  <= 1500
15 <= driving need[Driver3] = 1496.47  <= 1500
15 <= driving need[Driver4] = 1500  <= 1500
15 <= driving need[Driver5] = 262.732  <= 1500
15 <= driving need[Driver6] = 588.06  <= 1500
15 <= driving need[Driver7] = 1500  <= 1500
15 <= driving need[Driver9] = 599.572  <= 1500
15 <= driving need[Driver11] = 584.667  <= 1500
15 <= driving need[Driver13] = 1500  <= 1500
15 <= driving need[Driver14] = 1500  <= 1500
15 <= driving need[Driver15] = 928.103  <= 1500
15 <= driving need[Driver16] = 455.195  <= 1500
15 <= driving need[Driver17] = 1500  <= 1500
15 <= driving need[Driver18] = 799.619  <= 1500
15 <= driving need[Driver19] = 1500  <= 1500
15 <= driving need[Driver21] = 1124.88  <= 1500
15 <= driving need[Driver23] = 696.898  <= 1500
15 <= driving need[Driver24] = 1337.84  <= 1500
15 <= driving need[Driver25] = 1212.87  <= 1500
15 <= driving need[Driver27] = 1500  <= 1500
15 <= driving need[Driver28] = 658.159  <= 1500
15 <= driving need[Driver29] = 1380.22  <= 1500
15 <= driving need[Driver30] = 1496.48  <= 1500
15 <= driving need[Driver31] = 789.27  <= 1500
15 <= driving need[Driver32] = 1500  <= 1500
15 <= driving need[Driver33] = 1500  <= 1500
15 <= driving need[Driver34] = 901.373  <= 1500
15 <= driving need[Driver35] = 524.767  <= 1500
15 <= driving need[Driver36] = 417.325  <= 1500
15 <= driving need[Driver37] = 365.473  <= 1500
15 <= driving need[Driver42] = 1499.82  <= 1500
15 <= driving need[Driver43] = 850.876  <= 1500
15 <= driving need[Driver44] = 1482.68  <= 1500
15 <= driving need[Driver45] = 1080.39  <= 1500
15 <= driving need[Driver51] = 395.97  <= 1500
15 <= driving need[Driver52] = 760.984  <= 1500
15 <= driving need[Driver53] = 1500  <= 1500
15 <= driving need[Driver54] = 1500  <= 1500
15 <= driving need[Driver55] = 523.21  <= 1500
15 <= driving need[Driver56] = 1500  <= 1500
15 <= driving need[Driver57] = 828.543  <= 1500
15 <= driving need[Driver58] = 1500  <= 1500
15 <= driving need[Driver60] = 1499.72  <= 1500
15 <= driving need[Driver208] = 1500  <= 1500
15 <= driving need[Driver212] = 1016.69  <= 1500
15 <= driving need[Driver246] = 868.081  <= 1500
1e-005 <= initial event rate[Driver1] = 0.111853  <= 0.3
1e-005 <= initial event rate[Driver2] = 0.0136537  <= 0.3
1e-005 <= initial event rate[Driver3] = 0.00576947  <= 0.3
1e-005 <= initial event rate[Driver4] = 0.0307075  <= 0.3
1e-005 <= initial event rate[Driver5] = 0.00800071  <= 0.3
1e-005 <= initial event rate[Driver6] = 0.0654256  <= 0.3
1e-005 <= initial event rate[Driver7] = 0.0199648  <= 0.3
1e-005 <= initial event rate[Driver9] = 0.0514226  <= 0.3
1e-005 <= initial event rate[Driver11] = 1e-005  <= 0.3
1e-005 <= initial event rate[Driver13] = 0.3  <= 0.3
1e-005 <= initial event rate[Driver14] = 0.257073  <= 0.3
1e-005 <= initial event rate[Driver15] = 0.00129547  <= 0.3
1e-005 <= initial event rate[Driver16] = 0.133978  <= 0.3
1e-005 <= initial event rate[Driver17] = 0.202199  <= 0.3
1e-005 <= initial event rate[Driver18] = 0.0123813  <= 0.3
1e-005 <= initial event rate[Driver19] = 0.0392037  <= 0.3
1e-005 <= initial event rate[Driver21] = 0.00270743  <= 0.3
1e-005 <= initial event rate[Driver23] = 0.296444  <= 0.3
1e-005 <= initial event rate[Driver24] = 0.0183645  <= 0.3
1e-005 <= initial event rate[Driver25] = 0.0104826  <= 0.3
1e-005 <= initial event rate[Driver27] = 0.0361611  <= 0.3
1e-005 <= initial event rate[Driver28] = 1e-005  <= 0.3
1e-005 <= initial event rate[Driver29] = 0.00395405  <= 0.3
1e-005 <= initial event rate[Driver30] = 0.00189994  <= 0.3
1e-005 <= initial event rate[Driver31] = 0.179248  <= 0.3
1e-005 <= initial event rate[Driver32] = 0.0175246  <= 0.3
1e-005 <= initial event rate[Driver33] = 0.0411129  <= 0.3
1e-005 <= initial event rate[Driver34] = 0.159283  <= 0.3
1e-005 <= initial event rate[Driver35] = 0.0915423  <= 0.3
1e-005 <= initial event rate[Driver36] = 0.00509304  <= 0.3
1e-005 <= initial event rate[Driver37] = 0.00468767  <= 0.3
1e-005 <= initial event rate[Driver42] = 0.298366  <= 0.3
1e-005 <= initial event rate[Driver43] = 1e-005  <= 0.3
1e-005 <= initial event rate[Driver44] = 0.168415  <= 0.3
1e-005 <= initial event rate[Driver45] = 0.29803  <= 0.3
1e-005 <= initial event rate[Driver51] = 0.0102097  <= 0.3
1e-005 <= initial event rate[Driver52] = 0.00296696  <= 0.3
1e-005 <= initial event rate[Driver53] = 0.00115278  <= 0.3
1e-005 <= initial event rate[Driver54] = 0.166162  <= 0.3
1e-005 <= initial event rate[Driver55] = 0.0313833  <= 0.3
1e-005 <= initial event rate[Driver56] = 0.00189994  <= 0.3
1e-005 <= initial event rate[Driver57] = 0.142458  <= 0.3
1e-005 <= initial event rate[Driver58] = 0.00348168  <= 0.3
1e-005 <= initial event rate[Driver60] = 0.0427219  <= 0.3
1e-005 <= initial event rate[Driver208] = 0.3  <= 0.3
1e-005 <= initial event rate[Driver212] = 0.0272722  <= 0.3
1e-005 <= initial event rate[Driver246] = 0.13646  <= 0.3
-0.0008 <= effect of cumulative miles driven on event per mile = -0.000208534  <= -1e-006
-0.8 <= effect of recent memories on event per mile = -0.0002  <= -0.0002
0.01 <= fraction of events known = 0.0503015  <= 0.2
1 <= retention time = 7.67886  <= 12
-8 <= effect of recent memories on monthly driving = -7.70627  <= -0.05
My immediate concern is that effect of recent memories on event per mile is at the .0002 boundary.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Hi Tomfid,
I changed the effect of recent memories on event per mile to .00002 (instead of .0002) for the upper boundary and ran the simulation overnight. The concern remains the same, ie the effect of recent memories on event per mile is at the .00002 boundary. Do you have any ideas about how to get around this problem?
Thanks
Attachments
doc5.docx
(13.98 KiB) Downloaded 265 times
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

I guess what that says is that recent memories either have no effect, or (more likely) that the time constant of the effect is much less than 1 month, so you can't see it in the data.

You might try letting it range even to positive values - perhaps there's a kind of "serial offender" effect.

I don't think you need overnight runs - I found the solution after a few million runs to be the same as the solution after a few hundred thousand.

Another experiment worth trying would be to compute the errors/month on the basis of simulated errors/mile and data miles/month - i.e., estimate only one thing, not two.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks Tomfid.
I don't know if I might get some insights from you. But here is what the statistical analysis of the data shows:
1) I used past cumulative miles driven and past errors (errors in the last month) to predict current errors. The results showed that these factors are significant predictors of current events. Specifically, past cumulative miles driven have a negative effect on errors/mile and past errors have a positive effect on errors/mile.
2) There was a clinical trial in which drivers were constantly given feedback about their errors by alerting them on the errors and sharing the errors data with them. The study showed that providing feedback results in a significant decline in errors.
One of my goals was to predict the results of that clinical trial.
Note: Not all errors are known to the drivers. Some errors are unnoticed (errors were detected by a recording device). I think that was the basis of the clinical trial.

My explanation for the positive effect of past errors on current errors was a faster decline in memory of events since few errors are known. That could be why providing feedback on errors resulted in faster decline in errors according to the clinical trial.I guess providing feedback might result in build-up of memory.

I know you have an extensive experience in system modeling. So, do you have any suggestions about how I can restructure this model and be able to simulate the results of that clinical trial? That is the main goal here.

Thank you so much.
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

If #2 is true, then the effect of past errors should be positive, right? You have
-0.8 <= effect of recent memories on event per mile = -0.0002 <= -0.0002
which is negative. I'd try a range of -.8 to +.8 for both parameters to see what happens in the unconstrained case.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Should the effect of recent memories be positive (ie increase error/mile)? Shouldn’t memories decrease errors/mile?

If I have to draw the system structure directly from the insights gained from the statistical analysis, it would be like the structure shown in the attached file. The structure shows that recent errors have a positive effect on event/mile and cumulative miles driven have a negative effect on mile/mile. The challenge with this structure is that I have no idea about how to simulation the policy of providing feedback to drivers about errors.

I built the structure that we have been working on with the hope of testing the policy of providing feedback to drivers about errors. The main assumption for this structure is that every error should result in a memory for a continuous error improvement. But because some errors are unnoticed (thus, there is a fraction of errors known), there are not enough memories of errors, and this leads to an increase in error/mile. This structure would allow to test the policy of feedback. To test the policy, I would simply increase the fraction of errors known.

What do you think? Any thoughts?
Attachments
Doc6.docx
(15.33 KiB) Downloaded 266 times
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

OK - I think I misunderstood the situation earlier. To play back what (I think) you said:

1 - Initially, there's a positive effect of errors on future errors/mile.
2 - a treatment increases driver awareness of errors, and the error rate decreases

One possible interpretation is that #2 reduces the positive coefficient in #1, but errors could go down whether the final value is negative or remains positive.

Another possibility is that there are two different mechanisms at work. #1 has a positive coefficient, and #2 is negative, perhaps with a different time constant.
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

In any case, the thing that's missing from the model is the treatment itself. If it happened within the dataset, it needs to be in there somewhere. A simple approach would be to provide a step function in the error coefficient or fraction remembered.

However, if the signs differ, I think you'd have to have two different mechanisms.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks for the thoughts.
Actually, there are 2 groups of drivers. One group received the treatment and the second group did not get the treatment (control group).
The data I loaded in the software is the control data. I'm not using the treatment data.
Statistic analysis of the control data shows that amount of driving (cumulative miles driven) and past errors are significant predictors of current errors (amount of driving has a negative coefficient and past errors have a positive coefficient). The main critique I got from the model I uploaded yesterday (doc6) is to explain the mechanism by which past errors can increase future errors. Do drivers view excitement in past errors to increase their future errors? So, it was unbelievable, and difficult to explain. So I came up with the hypothesis that every error should result in a memory, but many errors are unnoticed and lost to memory. So, I would expect memory to decrease errors. Thus, a faster decline in the stock of memory will increase errors. That was the basis of the model we have been working on.
For example, if we go with the idea that a fraction of errors is known, we can increase the fraction to 100% and see if the results of that policy with match up with the results of the treatment group we did not use.
That's pretty much the big picture.

Do you have more suggestions/ideas? Maybe the idea of a fraction of errors known is not a good idea.
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

I think the "fraction known" idea makes a lot of sense. Also, I would expect errors to have a negative effect on future errors (i.e. people would learn). So, I think the real puzzle is why there appears to be a positive effect of errors on future errors in the control group. Both the dynamic model and whatever statistical approach you used seem to suggest that.
LAUJJL
Senior Member
Posts: 1421
Joined: Fri May 23, 2003 10:09 am
Vensim version: DSS

Re: Help with model calibration

Post by LAUJJL »

Hi Leon

4 remarks concerning your last model joined teen-driver-model.

1. Why one unit error and why not zero?

2. Adding subscripts has no utility at least at this level as each subscript result is independent from the others. It adds a complexity without any utility. It is always possible to add subscripts later on if
you need it.

3. I do not see how the cumulative miles driven can influence the events per mile. The events per month are already influenced by the recent memories of event in two ways, by influencing the monthly driving (how much they drive) and the events per mile (the way people drive). Adding the cumulative stock adds a third way of recent memories influencing the event per mile. Then recent memories have now three ways of influencing the events per month. I would try just forgetting this third influence and erasing the cumulative miles driven.

4. If you still want to keep the cumulative miles driven effect why not smooth it like you smoothed the recent memories of event. You are working with two stocks, one smoothed and the other not. Why?

Regards.

JJ
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Hi JJ,
Thanks so much for your input. I like your idea of removing cumulative miles driven and working with one stock. I will experiment that idea and let you know what happens.

Tomfid,
How should I set the zero unit error JJ is suggesting?
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

You just need to set the units for "weight".
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Hi Tomfid,
For some reason, I got an error saying "weight" is not used in the model. Despite the error, I ran the model and got a lot of floating errors.

Hi JJ,
I made some changes to the model based on your helpful suggestions. Feel free to comment.

Thanks so much.
Attachments
Driver6.zip
(1.99 MiB) Downloaded 253 times
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

"not used" is a warning rather than an error. You can suppress it by checking the "supplementary" box in the equation editor. (Or, ignore it.)
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Here's a revised .voc for your v5 model. Two changes: effect of recent memories on event per mile is permitted to be positive, and fraction of events known is not estimated. The event fraction can't be determined independent of the effect of memories. In any case, after a long optimization, the effect of memories is still negative, which makes sense to me.

I don't know what method you used in the statistical estimate, but I don't trust the positive effect of events on errors as a causal interpretation. It may just be some kind of serial correlation of errors effect. In any case, there aren't enough time points to be confident either way.
driver.voc
(900 Bytes) Downloaded 251 times
LAUJJL
Senior Member
Posts: 1421
Joined: Fri May 23, 2003 10:09 am
Vensim version: DSS

Re: Help with model calibration

Post by LAUJJL »

Hi Leon

I have experienced with the time that it is not easy and sometimes misleading to comment other works. I have no idea of your problem, of your environment, your constraints etc..

When modeling one must take into account all these factors.

I work on my own problems and my ideas about what should be a model may be very different from someone else. So it is up to you to use or discard my suggestions.

I personally would take away all the subscripts that can be added later on for calibration for instance. See the model joined where I have taken away the subscripts.

There are 7 parameters plus 7 variables depending on them. I do not see what is the weight for.

I would first put minimum and maximum on all the parameters and build a view with sliders for all the parameters and graphs for the 7 variables.

I would then start with one parameter and chose a set of average values for all the other parameters and using the synthesim vary that parameter between its minimum and maximum and try to understand how the variables behave when doing this. After having well understood that. I would change the set of the 6 other parameters values and see if my appreciation of the behaviours change and if so why. After having experimented with different set of the 6 parameter values I would go to another parameter and repeat the process. This is of course very time consuming and cumbersome but may be very instructive. Once you have done that with all the parameters you can do the same but choosing different pair of parameters that you may vary together. Of course this is very systematic but you can be inventive and chose your own strategy of analysis.

The purpose is to understand your model and verify that it correctly represents the reality or at least at a reasonable precision.

Once you have done that you will be more able to calibrate whatever you judge useful.

But I think it has no utility calibrating a model that is wrong whatever the quality of your data.

One thing after the other.

Regards.

JJ
Attachments
Teen_Driver_Model_simplified.mdl
(3.97 KiB) Downloaded 235 times
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

I'd leave the subscripts in, because they permit quick switching among the various drivers' data. You only have to look at one trace at a time.

But I agree that hand calibration and experimentation is extremely useful for insight into the structure.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

tomfid wrote:Here's a revised .voc for your v5 model. Two changes: effect of recent memories on event per mile is permitted to be positive, and fraction of events known is not estimated. The event fraction can't be determined independent of the effect of memories. In any case, after a long optimization, the effect of memories is still negative, which makes sense to me.

I don't know what method you used in the statistical estimate, but I don't trust the positive effect of events on errors as a causal interpretation. It may just be some kind of serial correlation of errors effect. In any case, there aren't enough time points to be confident either way.

driver.voc
Thanks so much for your help.
Do you mean I should remove "fraction of events known" from parameters to be estimated? I was not able to open the attached file
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Correct.

You should be able to download the file into your model directory (rename it to avoid overwriting the original). You can open it with any text editor (like most Vensim files).
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks so much Tomfid.
I've removed the fraction of event known from unknown parameters and set it to 20%. I think the estimates make sense to me now. The individual trends also look good.
I have just few floating errors. What could that be due to? Do you think the parameter estimates would be the same regardless of the the fraction of event known used?
Again, thank you so much for your time and help.
Attachments
Driver7.zip
(608.29 KiB) Downloaded 206 times
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

LAUJJL wrote:Hi Leon

I have experienced with the time that it is not easy and sometimes misleading to comment other works. I have no idea of your problem, of your environment, your constraints etc..

When modeling one must take into account all these factors.

I work on my own problems and my ideas about what should be a model may be very different from someone else. So it is up to you to use or discard my suggestions.

I personally would take away all the subscripts that can be added later on for calibration for instance. See the model joined where I have taken away the subscripts.

There are 7 parameters plus 7 variables depending on them. I do not see what is the weight for.

I would first put minimum and maximum on all the parameters and build a view with sliders for all the parameters and graphs for the 7 variables.

I would then start with one parameter and chose a set of average values for all the other parameters and using the synthesim vary that parameter between its minimum and maximum and try to understand how the variables behave when doing this. After having well understood that. I would change the set of the 6 other parameters values and see if my appreciation of the behaviours change and if so why. After having experimented with different set of the 6 parameter values I would go to another parameter and repeat the process. This is of course very time consuming and cumbersome but may be very instructive. Once you have done that with all the parameters you can do the same but choosing different pair of parameters that you may vary together. Of course this is very systematic but you can be inventive and chose your own strategy of analysis.

The purpose is to understand your model and verify that it correctly represents the reality or at least at a reasonable precision.

Once you have done that you will be more able to calibrate whatever you judge useful.

But I think it has no utility calibrating a model that is wrong whatever the quality of your data.

One thing after the other.

Regards.

JJ

Hi JJ,
Thank you for sharing your ideas.
What I learned from the data is that the event/mile has a S-shape decline. There is a first increase in the event/mile over 2 months followed by a slow decline in event/mile. So, it is likely there are 2 forces influencing event/mile. We hypothesize that one force is "recent memories of events" and the second force is "cumulative miles driven". A decrease in the first force (recent memories) leads to an increase in event/mile. The second force (cumulative miles) quickly overtakes the decline in the first force (recent memories); this results in an overall decline in event/mile. I would say that the effect of the decrease of the first force (recent memories) is stronger at the beginning and the second force (cumulative miles) is weak at the beginning.
Another insight is that events seem to decrease monthly driving.

The weight is set to standardize the driving need because there are substantial differences in the distribution of driving need.

Please, feel free to share your thoughts/ideas. For example, could you defend why it would be better to smooth cumulative miles driven and how?

Thanks
LAUJJL
Senior Member
Posts: 1421
Joined: Fri May 23, 2003 10:09 am
Vensim version: DSS

Re: Help with model calibration

Post by LAUJJL »

Hi Leon.

I have made deep changes to the model.

I have added again the cumulative miles driven as I had not understood its long term effect on the driving quality and of course there is no reason to smooth it.

I have first replaced memories by forgotten events which is the same and avoids the cumbersome changing events into memories.

I have then used normalised lookup to clearly show the different effects. See normalized lookups in the vensim guides.

I have too added a loop that was not in your model, that describes the fact that past events or past driven quality affects the present driving quality a kind of learning process.

The model needs to be run to eventually change the lookups or different parameters.

Regards.

JJ
Attachments
Teen_Driver_Model_simplified_bis.mdl
(8.1 KiB) Downloaded 231 times
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

To identify the floating point error, you need to find the parameters that cause it to occur (should be in the output or .log file) and run the model with those to see what the issue is.

A few FP errors out of a million simulations might not be an issue. Most likely it's an overflow in exp() or something like that. You don't know until you investigate.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

LAUJJL wrote:Hi Leon.

I have made deep changes to the model.

I have added again the cumulative miles driven as I had not understood its long term effect on the driving quality and of course there is no reason to smooth it.

I have first replaced memories by forgotten events which is the same and avoids the cumbersome changing events into memories.

I have then used normalised lookup to clearly show the different effects. See normalized lookups in the vensim guides.

I have too added a loop that was not in your model, that describes the fact that past events or past driven quality affects the present driving quality a kind of learning process.

The model needs to be run to eventually change the lookups or different parameters.

Regards.

JJ

Hi JJ,
Thanks so much for the time and efforts you have put into helping out.
The very first model had a variable labelled “confidence” in the causal pathway between recent memories and monthly driving. It also had a variable labelled “caution” between recent memories and event/mile. So, I thought about these intermediates variables. One of the critiques I got from the very first model was on these variables. One critique said: “Will I accept your model without accepting your concept of caution”. I ended up removing them and kept variables everybody would agree on. We could add the stock “event per mile smoothed” you are suggesting if it would be easier to explain and justify. Otherwise, it would still be better to stick to the last version of the model because that one is much more convincing.
What is the purpose of the stock “event per mile smoothed”? How is that stock independent of the 2 stocks already in the model?
I like your labels “events remembered” and “time to forget”; I can use them instead of memories.
While I have you here, I would like to ask if you have any ideas about how I could incorporate to the model a variable labeled “behavioral factors”. You can think of behaviors factors as cell phone use, distraction, traffic violations, etc. I was thinking about adding behavioral factors as an external factor, which would directly affect event/mile. I don’t know if it can find a place inside the internal structure of the model. Please, feel free to share your ideas.
Please find attached the S-shape decline I mentioned earlier as well as the trends in behavioral factors. You can see from both my data and another study published that the typical trend in event rate is of a first increase followed by a decrease.

Again, thank you so much
Attachments
Graph.docx
(261.73 KiB) Downloaded 196 times
Post Reply