Help with model calibration

Use this forum to post Vensim related questions.
Post Reply
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Vensim has to open the data.vdf to read the data. Some other process may have locked the file. Restarting Vensim will probably clear the error.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

First of all, thank you a lot. The weight did really improve the model fit for both measures per driver. I have some floating errors, also the model did not capture at all 1 driver. I don't know if I still need to refine the weight or if there is something I need to do. Also some of the initial conditions estimated from the model do not seem realistic at all. For example, I would not expect a new driver to have driven 3,000 miles before their formal driver license.
Attachments
calibration.docx
(15.65 KiB) Downloaded 272 times
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

I think the first thing to pay attention to is these warnings:

Code: Select all

WARNING:     Non-positive Poisson event rate for calibration payoff element events per month[Driver56]/0.25 - skipping.    
Either the data or the model prediction is for a negative error rate, which is obviously not possible.
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

BTW the weight for Poisson elements in the payoff will normally be 1, because the distribution assumption already takes care of the scaling (making the mean = variance assumption).
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I'm still getting the same warning after changing the weight to 1.

Let me try this and see what happens. Instead of calibrating 2 measures (errors and monthly driving) per driver, I think I will calculate error/mile and calibrate that variable.
I don't know if you will be willing to reset this model for me. I'm trying to start over, but it's not allowing me.
I got the attached when I clicked on "simulate", I don't know why it's not running
Attachments
Doc9.docx
(32.78 KiB) Downloaded 291 times
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

sorry I forgot to attach the model. I'm unable to reset it to start the calibration over
Attachments
submodel1.mdl
(6.34 KiB) Downloaded 288 times
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

This isn't a model issue. The data file could be missing or corrupted, or the operating system could be locking it for some other process.

The file it's attempting to load is designated in the Data Source field of the Advanced tab of the Simulation Control dialog.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

And what should I do? Please, advise. I've restarted VENSIM several times and I'm not even able to do a simple run.
Thanks
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

leon wrote:And what should I do? Please, advise. I've restarted VENSIM several times and I'm not even able to do a simple run.
Thanks
Tom has already told you,
"The file it's attempting to load is designated in the Data Source field of the Advanced tab of the Simulation Control dialog."
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

The file has probably been deleted, renamed, corrupted or locked by another process. You need to fix that - either repoint the Data Source field at the correct file (if it still exists) or restore the file from a backup (if it's damaged). Failing that, you can reimport the data or perhaps re-download the file that was posted here. But you should always have a backup.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks for all your help. The file was probably corrupted. I deleted the file and downloaded a file posted here.
I was trying to calibrate errors/mile since calibrating one variable provided me a good fit before. But I didn't get good results.
1) Do you think I still need a poisson distribution when calibrating errors/mile? Or will a normal distribution better?
2) Do I need a weight other than 1 since errors/mile is very very small?
3) Do you think calibrating one measure (errors/mile) is better than calibrating 2 measures (errors per month and monthly driving)
Thanks so much for the time and efforts you have been putting to assist me
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

1) The distribution will not be Normal - it's Poisson/Normal. I have no idea what that yields, but it's probably asymptotically normalish, so I'd use that.
2) For a single measure, the weight doesn't matter for the central estimate (it only affects confidence bounds), unless you think the measurement error varies from point to point or driver to driver.
3) I think 2 measures is better. If calibrating with errors/mile gets you a better fit of errors & miles separately too, there's something strange going on.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

What are your thoughts about sample size for my work? Since I'm trying to calibrate per driver, should I be concerned about sample size?
What are the metrics to make the model believable?
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

I think the only way to determine if the sample size is adequate is to check your confidence bounds on parameters. I don't know offhand what the expected values are for the Poisson distribution. Probably the simplest thing to do would be to use MCMC. https://www.vensim.com/documentation/mcmc_sa.htm
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thank you so much for your help Tom!
I actually have more drivers that I can add. So, if I fit the model to all the drivers' data, what do you suggest to validate the model, especially since the model is fitted to a set of drivers?. Should I compute the R-statistics for each driver and average across the drivers?
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

If you take a look at the payoff report that gets generated when you optimize, you already have a lot of stats. It's in <runname>.rep (tab delimited).
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Another thing worth looking at is the parameter distributions. Presumably things like "driving need" are personal to each driver. But you'd hope that "retention time," "effect of cumulative miles" etc. would be fairly consistent across drivers. If those parameters are all over the place, that's a red flag.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks a lot. I will take a look at those parameters across the drivers.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks Tomfid.
If it's possible, would you be willing to take a look at the model and make me some simple suggestions? There is a lot of floating errors and I'm getting frustrated with it. The constants (retention time, event fading time, etc.) do not seem similar across the drivers and I'm worrying about it. I have loaded enough drivers in the model now
I appreciate your time and efforts you have putting into guiding me on this work. I've attached the model
Attachments
Teen_Driver_Model.mdl
(6.77 KiB) Downloaded 236 times
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Can you zip up the whole project, with the data and optimization files, and upload that?
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks. I've attached the file.
I have the data by gender, but I thought it would not make any difference to calibrate by gender group. Unless you think otherwise.
Thanks so much for your help, I'm so appreciative.
Attachments
Driver.zip
(101.34 KiB) Downloaded 232 times
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Hi Tomfid, I just wanted to follow-up with you, I hope you did not forget about me.
Thanks for all your help
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Sorry, my travel schedule has been too busy.

I think there's a risk here of overfitting. It appears that several of the parameter vectors are bimodally distributed, with a lot of extreme values. You essentially have a unique parameterization of the model for each driver. I think that's a valuable experiment, but I'd want to try versions with fewer parameters.

Driving need, initial event rate, baseline driving and baseline memories probably should remain subscripted by [DriverID]. For the others (especially the 3 effect...s and retention), I'd try estimating a single value (remove the subscript) and see how well you can do.

The fits will degrade (they look pretty good now), but I think the reliability of the estimates will be substantially improved.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks so much.
I removed the subscripts from the 3 effects, retention time, fading time. I got an error, please find attached what I got.
Thanks
Attachments
Doc1.docx
(47.21 KiB) Downloaded 225 times
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

leon wrote:Thanks so much.
I removed the subscripts from the 3 effects, retention time, fading time. I got an error, please find attached what I got.
Thanks
The error is displayed in your screen shot. In your optimization control file, "EFFECT OF CUMULATIVE MILES DRIVEN ON EVENT PER MILE" is not subscripted, but it is in the model. They need to match. You need to review your optimization files.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Post Reply