Help with model calibration
Re: Help with model calibration
Vensim has to open the data.vdf to read the data. Some other process may have locked the file. Restarting Vensim will probably clear the error.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
First of all, thank you a lot. The weight did really improve the model fit for both measures per driver. I have some floating errors, also the model did not capture at all 1 driver. I don't know if I still need to refine the weight or if there is something I need to do. Also some of the initial conditions estimated from the model do not seem realistic at all. For example, I would not expect a new driver to have driven 3,000 miles before their formal driver license.
- Attachments
-
- calibration.docx
- (15.65 KiB) Downloaded 279 times
Re: Help with model calibration
I think the first thing to pay attention to is these warnings:
Either the data or the model prediction is for a negative error rate, which is obviously not possible.
Code: Select all
WARNING: Non-positive Poisson event rate for calibration payoff element events per month[Driver56]/0.25 - skipping.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
BTW the weight for Poisson elements in the payoff will normally be 1, because the distribution assumption already takes care of the scaling (making the mean = variance assumption).
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
I'm still getting the same warning after changing the weight to 1.
Let me try this and see what happens. Instead of calibrating 2 measures (errors and monthly driving) per driver, I think I will calculate error/mile and calibrate that variable.
I don't know if you will be willing to reset this model for me. I'm trying to start over, but it's not allowing me.
I got the attached when I clicked on "simulate", I don't know why it's not running
Let me try this and see what happens. Instead of calibrating 2 measures (errors and monthly driving) per driver, I think I will calculate error/mile and calibrate that variable.
I don't know if you will be willing to reset this model for me. I'm trying to start over, but it's not allowing me.
I got the attached when I clicked on "simulate", I don't know why it's not running
- Attachments
-
- Doc9.docx
- (32.78 KiB) Downloaded 301 times
Re: Help with model calibration
sorry I forgot to attach the model. I'm unable to reset it to start the calibration over
- Attachments
-
- submodel1.mdl
- (6.34 KiB) Downloaded 298 times
Re: Help with model calibration
This isn't a model issue. The data file could be missing or corrupted, or the operating system could be locking it for some other process.
The file it's attempting to load is designated in the Data Source field of the Advanced tab of the Simulation Control dialog.
The file it's attempting to load is designated in the Data Source field of the Advanced tab of the Simulation Control dialog.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
And what should I do? Please, advise. I've restarted VENSIM several times and I'm not even able to do a simple run.
Thanks
Thanks
-
- Super Administrator
- Posts: 4620
- Joined: Wed Mar 05, 2003 3:10 am
Re: Help with model calibration
Tom has already told you,leon wrote:And what should I do? Please, advise. I've restarted VENSIM several times and I'm not even able to do a simple run.
Thanks
"The file it's attempting to load is designated in the Data Source field of the Advanced tab of the Simulation Control dialog."
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Re: Help with model calibration
The file has probably been deleted, renamed, corrupted or locked by another process. You need to fix that - either repoint the Data Source field at the correct file (if it still exists) or restore the file from a backup (if it's damaged). Failing that, you can reimport the data or perhaps re-download the file that was posted here. But you should always have a backup.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Thanks for all your help. The file was probably corrupted. I deleted the file and downloaded a file posted here.
I was trying to calibrate errors/mile since calibrating one variable provided me a good fit before. But I didn't get good results.
1) Do you think I still need a poisson distribution when calibrating errors/mile? Or will a normal distribution better?
2) Do I need a weight other than 1 since errors/mile is very very small?
3) Do you think calibrating one measure (errors/mile) is better than calibrating 2 measures (errors per month and monthly driving)
Thanks so much for the time and efforts you have been putting to assist me
I was trying to calibrate errors/mile since calibrating one variable provided me a good fit before. But I didn't get good results.
1) Do you think I still need a poisson distribution when calibrating errors/mile? Or will a normal distribution better?
2) Do I need a weight other than 1 since errors/mile is very very small?
3) Do you think calibrating one measure (errors/mile) is better than calibrating 2 measures (errors per month and monthly driving)
Thanks so much for the time and efforts you have been putting to assist me
Re: Help with model calibration
1) The distribution will not be Normal - it's Poisson/Normal. I have no idea what that yields, but it's probably asymptotically normalish, so I'd use that.
2) For a single measure, the weight doesn't matter for the central estimate (it only affects confidence bounds), unless you think the measurement error varies from point to point or driver to driver.
3) I think 2 measures is better. If calibrating with errors/mile gets you a better fit of errors & miles separately too, there's something strange going on.
2) For a single measure, the weight doesn't matter for the central estimate (it only affects confidence bounds), unless you think the measurement error varies from point to point or driver to driver.
3) I think 2 measures is better. If calibrating with errors/mile gets you a better fit of errors & miles separately too, there's something strange going on.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
What are your thoughts about sample size for my work? Since I'm trying to calibrate per driver, should I be concerned about sample size?
What are the metrics to make the model believable?
What are the metrics to make the model believable?
Re: Help with model calibration
I think the only way to determine if the sample size is adequate is to check your confidence bounds on parameters. I don't know offhand what the expected values are for the Poisson distribution. Probably the simplest thing to do would be to use MCMC. https://www.vensim.com/documentation/mcmc_sa.htm
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Thank you so much for your help Tom!
I actually have more drivers that I can add. So, if I fit the model to all the drivers' data, what do you suggest to validate the model, especially since the model is fitted to a set of drivers?. Should I compute the R-statistics for each driver and average across the drivers?
I actually have more drivers that I can add. So, if I fit the model to all the drivers' data, what do you suggest to validate the model, especially since the model is fitted to a set of drivers?. Should I compute the R-statistics for each driver and average across the drivers?
Re: Help with model calibration
If you take a look at the payoff report that gets generated when you optimize, you already have a lot of stats. It's in <runname>.rep (tab delimited).
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Another thing worth looking at is the parameter distributions. Presumably things like "driving need" are personal to each driver. But you'd hope that "retention time," "effect of cumulative miles" etc. would be fairly consistent across drivers. If those parameters are all over the place, that's a red flag.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Thanks a lot. I will take a look at those parameters across the drivers.
Re: Help with model calibration
Thanks Tomfid.
If it's possible, would you be willing to take a look at the model and make me some simple suggestions? There is a lot of floating errors and I'm getting frustrated with it. The constants (retention time, event fading time, etc.) do not seem similar across the drivers and I'm worrying about it. I have loaded enough drivers in the model now
I appreciate your time and efforts you have putting into guiding me on this work. I've attached the model
If it's possible, would you be willing to take a look at the model and make me some simple suggestions? There is a lot of floating errors and I'm getting frustrated with it. The constants (retention time, event fading time, etc.) do not seem similar across the drivers and I'm worrying about it. I have loaded enough drivers in the model now
I appreciate your time and efforts you have putting into guiding me on this work. I've attached the model
- Attachments
-
- Teen_Driver_Model.mdl
- (6.77 KiB) Downloaded 246 times
Re: Help with model calibration
Can you zip up the whole project, with the data and optimization files, and upload that?
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Thanks. I've attached the file.
I have the data by gender, but I thought it would not make any difference to calibrate by gender group. Unless you think otherwise.
Thanks so much for your help, I'm so appreciative.
I have the data by gender, but I thought it would not make any difference to calibrate by gender group. Unless you think otherwise.
Thanks so much for your help, I'm so appreciative.
- Attachments
-
- Driver.zip
- (101.34 KiB) Downloaded 242 times
Re: Help with model calibration
Hi Tomfid, I just wanted to follow-up with you, I hope you did not forget about me.
Thanks for all your help
Thanks for all your help
Re: Help with model calibration
Sorry, my travel schedule has been too busy.
I think there's a risk here of overfitting. It appears that several of the parameter vectors are bimodally distributed, with a lot of extreme values. You essentially have a unique parameterization of the model for each driver. I think that's a valuable experiment, but I'd want to try versions with fewer parameters.
Driving need, initial event rate, baseline driving and baseline memories probably should remain subscripted by [DriverID]. For the others (especially the 3 effect...s and retention), I'd try estimating a single value (remove the subscript) and see how well you can do.
The fits will degrade (they look pretty good now), but I think the reliability of the estimates will be substantially improved.
I think there's a risk here of overfitting. It appears that several of the parameter vectors are bimodally distributed, with a lot of extreme values. You essentially have a unique parameterization of the model for each driver. I think that's a valuable experiment, but I'd want to try versions with fewer parameters.
Driving need, initial event rate, baseline driving and baseline memories probably should remain subscripted by [DriverID]. For the others (especially the 3 effect...s and retention), I'd try estimating a single value (remove the subscript) and see how well you can do.
The fits will degrade (they look pretty good now), but I think the reliability of the estimates will be substantially improved.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Thanks so much.
I removed the subscripts from the 3 effects, retention time, fading time. I got an error, please find attached what I got.
Thanks
I removed the subscripts from the 3 effects, retention time, fading time. I got an error, please find attached what I got.
Thanks
- Attachments
-
- Doc1.docx
- (47.21 KiB) Downloaded 234 times
-
- Super Administrator
- Posts: 4620
- Joined: Wed Mar 05, 2003 3:10 am
Re: Help with model calibration
The error is displayed in your screen shot. In your optimization control file, "EFFECT OF CUMULATIVE MILES DRIVEN ON EVENT PER MILE" is not subscripted, but it is in the model. They need to match. You need to review your optimization files.leon wrote:Thanks so much.
I removed the subscripts from the 3 effects, retention time, fading time. I got an error, please find attached what I got.
Thanks
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559