Help with model calibration

Use this forum to post Vensim related questions.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I think you are correct. How can I apply subscripts to a subset of variables in the model? I tried to directly to remove the subscripts from few variables, but it did not work.
Thanks
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Three options:

1. Remove the subscript from the variable in the equation editor, and accept the offer to change uses after you hit OK.

2. Remove the subscript from the variable in the equation editor, and remove it from uses of the variable manually.

3. Select the variable and use Edit>Set Subscripts.
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

If you get an error, Vensim can't necessarily determine the cause. For example:

x = 1
y[a] = x[a]

Which equation has the wrong dimension for x? Hard to tell, but Vensim generally assumes that the variable definition is correct, even though the use could be what the user actually intended.. But the actual error is never more than one causal link away from the reported error.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks so much Tomfid.
You made the right suggestion, there is only 1 floating error now. Any ideas about how I can get around it? I've uploaded the current stage of the calibration
I'm so appreciative of your help
Attachments
Driver2.zip
(158.6 KiB) Downloaded 305 times
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

I think the error happens because you don't have upper bounds on the parameters. Therefore it's searching 0 ... infinity.
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Here's an attempt at confidence bounds on the results, using MCMC. The full output's too big to upload, but you can rerun it with this setup.
Driver3.zip
(416.63 KiB) Downloaded 307 times
The basic procedure is:
- run a powell optimization (already done here)
- load powell.out as a changes file to restart from the optimal solution
- run mcmc, using the driver-mcmc.voc control file

The control file uses mostly default settings, except that I set limits on run length and set a higher initial temperature to get some initial dispersion of points.

The result will be in a file called <runname>_MCMC_sample.tab. This is a tab delimited file containing a sample of points for the parameters. This represents the joint distribution of all the parameter values. I haven't really looked these over, but the results seem OK individually. The only surprise is that the effect of cum miles is ~0 (maybe it's positive in reality - i.e. a learning effect rather than a fatigue effect?).

It looks like the payoff is dominated by a few items:

Code: Select all

events per month[Driver208]/1
events per month[Driver23]/1
events per month[Driver13]/1
events per month[Driver34]/1
events per month[Driver14]/1
The payoff report isn't fully populated - this may be a bug.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thank you very much Tomfid.
You are correct in your observation. The drivers 208, 23, 13, 34, 14 are risky drivers; they made a lot of driving errors.
I think we should expect the effect of cum. miles on event rate (ie. events/mile) to be negative. Statistically, I was able to document that effect.
A brief idea about the data. New drivers (no driving experience before) were enrolled and a video recording system was installed in their vehicles to track their driving mistakes over 5 months as well as their mileage. So, it is unrealistic to have cumulative miles driven to increase driving mistakes. We should expect a decrease
I run the mcmc, I'm not sure I did it right, but attached is what I got.
Again, thanks so much for your very helpful suggestions, I really appreciate.
Attachments
powell.docx
(12.28 KiB) Downloaded 292 times
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Did you unzip and run it in a fresh directory? Your output is completely different. In particular, the negative event data warnings are concerning - that suggests that something is corrupted.

Unfortunately, I'm leaving on vacation for 2 weeks and won't be able to do anything further with this until I return.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Yes I unzipped the file. I loaded the model, clicked on "optimize", then I saw the payoff definition window; I hit next button, I saw the optimization control file window with MCMC, then I hit on next button, saw the data source window, then hit "Finish" button. That's what I did, I don't know if that what I should do
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Hi Tomfid, hope you had a great vacation!

I was wondering if you give me some direction on the MCMC procedure.
Thanks so much!
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

What are you struggling with?
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Try running it from the simulation control dialog. The setup should look like this:
driverSetup.png
driverSetup.png (7.07 KiB) Viewed 8501 times
and the result should have no errors.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks so much.
I think I was not doing the right thing. Please, find attached the steps I went through.
How did you get the set up you sent me?

I also found out that the estimated parameters largely depend on the random model estimates assigned to the parameters. For example, if I assign a value of 2 to retention time, the estimate for retention time derived from the calibration would be close to 2. Likewise, if I assign 4 to retention time, the estimate for retention time would be close to 4. So, I believe something is wrong because the estimated parameters should be the same regardless of the initial assigned parameters. Do you have any ideas why this might be happening? I know the estimates are not yet stable
Attachments
powel2.docx
(78.75 KiB) Downloaded 276 times
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I think I found the control dialog you used. But I still don't get the same results as you
Attachments
doc.docx
(40.82 KiB) Downloaded 284 times
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Hi Tomfid,
Do you have any suggestions for the issues I'm encountering?
Thanks so much
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

leon wrote:I also found out that the estimated parameters largely depend on the random model estimates assigned to the parameters. For example, if I assign a value of 2 to retention time, the estimate for retention time derived from the calibration would be close to 2. Likewise, if I assign 4 to retention time, the estimate for retention time would be close to 4. So, I believe something is wrong because the estimated parameters should be the same regardless of the initial assigned parameters. Do you have any ideas why this might be happening? I know the estimates are not yet stable
What happens if you make the initial estimate for retention time something like 8? Does the final estimate stay around 8 or go to 4
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

One thing I notice is that you have the lower bound for the initial event rate set to 0. This isn't really plausible, as it implies that a driver could never make a mistake. I'd set it to a small positive value, like 1e-6.
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

The estimates can vary with the initial values if the model has multiple optima. The best thing to do in that case is to turn on multiple start and let it run overnight.
https://www.vensim.com/documentation/multiple_start.htm
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

I think your fraction of events known is a redundant parameter. It's confounded with effect of recent memories on event per mile - if one is 10x bigger and the other 10x smaller, the effect is the same. Right?
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

I think you could make more improvements to bounds. For example,
- the time constants have to have a lower bound of at least the TIME STEP; otherwise the model is unstable.
- the baseline memories etc. should be bounded above by some reasonable limit - for example, the maximum monthly events in the data is < 200, so 200 * fraction known * retention time is probably a reasonable upper bound for memories.

For drivers with no events, you probably can't say much about their individual parameters.
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Here's an update that implements some of my comments, and includes a fresh copy of the MCMC sample.

I notice one more thing. The event fading time comes out quite long - longer than the model horizon. Is that plausible? It may simply be generating a trend. Is it really justifiable to have two separate event memory processes with very different time constants?

I think it would help to spend some time going through the drivers one by one and looking at the calibration results.
Driver3b.zip
(1.09 MiB) Downloaded 264 times
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

tomfid wrote:I think your fraction of events known is a redundant parameter. It's confounded with effect of recent memories on event per mile - if one is 10x bigger and the other 10x smaller, the effect is the same. Right?
I think you raised an important question here. There are different types of events (driving errors): minor errors (not known to the driver and detected by a video recording device), near-collision and collision. The last 2 types of errors are assumed to be known by the driver. So I created the parameter “fraction of events know” to account for the fact that some events are lost to memory. One of the interventions I’m hoping to test is sharing the error data with the driver so that they are aware of all their errors. So, do you think it makes sense to have such a parameter (fraction of events known)?
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

tomfid wrote:Here's an update that implements some of my comments, and includes a fresh copy of the MCMC sample.

I notice one more thing. The event fading time comes out quite long - longer than the model horizon. Is that plausible? It may simply be generating a trend. Is it really justifiable to have two separate event memory processes with very different time constants?

I think it would help to spend some time going through the drivers one by one and looking at the calibration results.

Driver3b.zip
I went through the drivers and I think the fit looks good.
I don’t know what the MCMC sample is used for. But I looked up the powel.log for the parameter estimates (I did not run the model, I directly looked up your results). The longer event fading time may be correct. One thing I noticed: the initial values for effect of memories and effect of miles driven did not change after calibration. I’m wondering if the event fading time will change if we modify these initial values.
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

leon wrote:I’m wondering if the event fading time will change if we modify these initial values.
The best thing to do is try it and see if it does.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I honestly don't know how Tomfid executed the MCMC procedure. Could you please lay out the different steps to do this? After I load the model into in VENSIM, what should I do next?
Post Reply