Help with model calibration

Use this forum to post Vensim related questions.
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

If you run with the driver-mcmc optimization control file in the last model archive I uploaded, it does MCMC. However, I think you need to address some of the items I mentioned in my last few posts.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks.
I would like to directly use the last model you uploaded moving forward. I unzipped the file, but don't event know how to run it. After I unzipped the file, I clicked on "Teen_Driver_Model" to upload the model into VENSIM. What should I do from that step? I want to change the initial values for effect of memory and effect of miles driven and see if the estimated parameters derived from calibration will change. Could you please provide more guidance on it? I went through the MCMC documentation, but I don't think I will be able to execute this procedure, it seems very complex.
You also recommended to turn on multiple start and let it run overnight. If you think it is important, I would like to do that.
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

I want to change the initial values for effect of memory and effect of miles driven and see if the estimated parameters derived from calibration will change.
You also recommended to turn on multiple start and let it run overnight. If you think it is important, I would like to do that.
These really only make sense for regular optimization, not MCMC. I would start with the second item, which is just a setting change for the Powell optimizer - see the Help page I linked.

But before doing that, I would strongly recommend spending some time on hand calibration to gain insight, considering whether you need two memory stocks, and setting upper and lower bounds on all the parameters in the .voc.

MCMC is implemented by an alternative optimization control file. I supplied two:
- driver.voc is your original, with small modifications by me (I think I set a nonzero lower bound on the event rate, for example) - this is using the Powell optimizer.
- driver-mcmc.voc - this implements MCMC (to run it, just use the Optimize button as you would normally).

The endpoint of MCMC is a sample of parameters in <runname>_mcmc_sample.tab (I provided an example). However, the results may not make sense until you address some of the other issues.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thank you so much, I appreciate.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Hi Tomfid,
I did what you recommended. Few questions
1) What multiple start should I use ? I used Random, it never stopped and I had to stop it. It had some floating errors
2) You mentioned 2 memory stocks. Does it mean 2 stocks with different memory retention times? How to aggregate 2 stocks in Vensim?
Thanks a lot
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

1) I prefer RRandom, but the difference is not normally important.

2) You already have 2 memory stocks as I recall. I was thinking that you might consider eliminating one of them - i.e. have a similar memory stock driving both effects.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thank you so much for your time. I think it is a good idea, less of parameters.
The model is not capturing monthly driving at all. I don't know if it's a weight problem. Please, let me know what you think
Attachments
Driver4.zip
(1.78 MiB) Downloaded 198 times
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

How to refresh the directory of VENSIM? My model may have been corrupted and can't run anymore
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Hi Tomfid,
Will you be willing to look at the model for me? The fit was good when each either monthly driving or events/months was calibrated. But the fit was not good when I attempted to calibrate both variables. Any suggestions?
Thanks so much
Attachments
Driver4.zip
(71.11 KiB) Downloaded 188 times
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Please, use this file. Disregard the previous file
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Please, use this file. Disregard the previous file, I forgot to include a parameter in the previous file
Attachments
Driver4.zip
(76.95 KiB) Downloaded 193 times
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Hi Tomfid,
Could you please give me some more directions? I have attached the current model. I also attached a mean profile of errors/mile by gender. Do you think that the retention time should depend on gender? I don't know it that it's influencing the error shape. Also what do you think about the weight of 0.05 used for monthly driving?
Thanks so much for your time.
Attachments
Graph.docx
(67.07 KiB) Downloaded 190 times
Driver4.zip
(688.8 KiB) Downloaded 193 times
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

I like this simplified version. I'll take a look.
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

WRT the weight on driving, I think you want a variable weight. Looking at the standard deviations of the drivers, they're anywhere from 10 to almost 300, or .05 to .4 normalized. I'd recommend a weight expressed as a fraction of driving rather than a single absolute value.

If you think of the weight as the inverse of the measurement error, it should be closer to .01, but with a lot of variation.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks so much Tomfid.
Do you mean I should use 1/driving need at the place of weight in the optimization file? I tried that, but it was declined by the software.
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

You can't put expressions in .vpd etc. Create a model variable to contain the weight.

Using a constant weight for each driver is probably good enough. But you could go further and compute the weight point by point. That would require a change in the data approach, however. Currently you have the data variable names match the model variable, which is really convenient. The switch would be to use separate data variable names. Not sure it's worth the trouble.

Yet another option might be to use the log transform in the payoff, so that each data point has the same fractional SD, but that's problematic if there are any 0s in the data.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks so much Tomfid.
1) Could you please take a lot at the attached document? Did I incorporate the variable "weight" as you were suggesting?
2) You mentioned Log transformation. So, if I select transform=Log for the variable "monthly driving" in the vpd, will that work as well?
Attachments
doc4.docx
(15.3 KiB) Downloaded 149 times
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

1) looks reasonable, but what's the equation? If you assume that measurement error = 20% of driving need, then weight = 1/(.2*driving need).
2) using the log transform with weight = .2 would give measurement error = 20% of driving data, but any data points with 0 driving will be skipped, which may be undesirable.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Is there any easier way to assign a subscript to the new variable added to the model?
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

You can select it and use Edit>Set Subscripts.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks Tomfid.
The fit is not good, especially for monthly driving. Do you have any ideas about what I could do to improve it?
Attachments
Driver5.zip
(588.08 KiB) Downloaded 142 times
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

I actually wouldn't expect the fit to driving to be good. I would expect random life demands to greatly outweigh memories of recent errors in the decision to drive. So, it may just be that large errors are expected, and don't indicate anything "poor."
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Sorry I attached the wrong file. The fit is not good at all. Could you please take a look at it?
Attachments
Driver5.zip
(80.89 KiB) Downloaded 129 times
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Hi Tomfid,
I let the simulation run overnight by turning on multiple optima RRandom. I set weight= 1/(0.1*driving need[DriverID]). I've tried to attach the file here, but it said "error, file too large". So I emailed you the files.
My concerns are:
1) if you look at the average trend of the sample, you can see an increase in event/mile over the first 2 months followed by a slow decline. From the simulation, no driver seems to exhibit a trend similar to the sample average trend in event/mile. This makes me wonder if there is something I'm missing. I'm not expecting every driver to exhibit the average trend, but some should have a similar trend as the average trend.
2) Are the model estimates from the final simulation (.log)? The last simulation seems to pick the upper bounds of some of the parameters, for eg. retention time=12. What do you think about that?
3) Could you please see if you can try the calibration with mcmc?

Thanks so much for your time. Any suggestions would be greatly appreciated.
Post Reply