Help with model calibration
Re: Help with model calibration
I think you are correct. How can I apply subscripts to a subset of variables in the model? I tried to directly to remove the subscripts from few variables, but it did not work.
Thanks
Thanks
Re: Help with model calibration
Three options:
1. Remove the subscript from the variable in the equation editor, and accept the offer to change uses after you hit OK.
2. Remove the subscript from the variable in the equation editor, and remove it from uses of the variable manually.
3. Select the variable and use Edit>Set Subscripts.
1. Remove the subscript from the variable in the equation editor, and accept the offer to change uses after you hit OK.
2. Remove the subscript from the variable in the equation editor, and remove it from uses of the variable manually.
3. Select the variable and use Edit>Set Subscripts.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
If you get an error, Vensim can't necessarily determine the cause. For example:
x = 1
y[a] = x[a]
Which equation has the wrong dimension for x? Hard to tell, but Vensim generally assumes that the variable definition is correct, even though the use could be what the user actually intended.. But the actual error is never more than one causal link away from the reported error.
x = 1
y[a] = x[a]
Which equation has the wrong dimension for x? Hard to tell, but Vensim generally assumes that the variable definition is correct, even though the use could be what the user actually intended.. But the actual error is never more than one causal link away from the reported error.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Thanks so much Tomfid.
You made the right suggestion, there is only 1 floating error now. Any ideas about how I can get around it? I've uploaded the current stage of the calibration
I'm so appreciative of your help
You made the right suggestion, there is only 1 floating error now. Any ideas about how I can get around it? I've uploaded the current stage of the calibration
I'm so appreciative of your help
- Attachments
-
- Driver2.zip
- (158.6 KiB) Downloaded 305 times
Re: Help with model calibration
I think the error happens because you don't have upper bounds on the parameters. Therefore it's searching 0 ... infinity.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Here's an attempt at confidence bounds on the results, using MCMC. The full output's too big to upload, but you can rerun it with this setup.
The basic procedure is:
- run a powell optimization (already done here)
- load powell.out as a changes file to restart from the optimal solution
- run mcmc, using the driver-mcmc.voc control file
The control file uses mostly default settings, except that I set limits on run length and set a higher initial temperature to get some initial dispersion of points.
The result will be in a file called <runname>_MCMC_sample.tab. This is a tab delimited file containing a sample of points for the parameters. This represents the joint distribution of all the parameter values. I haven't really looked these over, but the results seem OK individually. The only surprise is that the effect of cum miles is ~0 (maybe it's positive in reality - i.e. a learning effect rather than a fatigue effect?).
It looks like the payoff is dominated by a few items:
The payoff report isn't fully populated - this may be a bug.
The basic procedure is:
- run a powell optimization (already done here)
- load powell.out as a changes file to restart from the optimal solution
- run mcmc, using the driver-mcmc.voc control file
The control file uses mostly default settings, except that I set limits on run length and set a higher initial temperature to get some initial dispersion of points.
The result will be in a file called <runname>_MCMC_sample.tab. This is a tab delimited file containing a sample of points for the parameters. This represents the joint distribution of all the parameter values. I haven't really looked these over, but the results seem OK individually. The only surprise is that the effect of cum miles is ~0 (maybe it's positive in reality - i.e. a learning effect rather than a fatigue effect?).
It looks like the payoff is dominated by a few items:
Code: Select all
events per month[Driver208]/1
events per month[Driver23]/1
events per month[Driver13]/1
events per month[Driver34]/1
events per month[Driver14]/1
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Thank you very much Tomfid.
You are correct in your observation. The drivers 208, 23, 13, 34, 14 are risky drivers; they made a lot of driving errors.
I think we should expect the effect of cum. miles on event rate (ie. events/mile) to be negative. Statistically, I was able to document that effect.
A brief idea about the data. New drivers (no driving experience before) were enrolled and a video recording system was installed in their vehicles to track their driving mistakes over 5 months as well as their mileage. So, it is unrealistic to have cumulative miles driven to increase driving mistakes. We should expect a decrease
I run the mcmc, I'm not sure I did it right, but attached is what I got.
Again, thanks so much for your very helpful suggestions, I really appreciate.
You are correct in your observation. The drivers 208, 23, 13, 34, 14 are risky drivers; they made a lot of driving errors.
I think we should expect the effect of cum. miles on event rate (ie. events/mile) to be negative. Statistically, I was able to document that effect.
A brief idea about the data. New drivers (no driving experience before) were enrolled and a video recording system was installed in their vehicles to track their driving mistakes over 5 months as well as their mileage. So, it is unrealistic to have cumulative miles driven to increase driving mistakes. We should expect a decrease
I run the mcmc, I'm not sure I did it right, but attached is what I got.
Again, thanks so much for your very helpful suggestions, I really appreciate.
- Attachments
-
- powell.docx
- (12.28 KiB) Downloaded 292 times
Re: Help with model calibration
Did you unzip and run it in a fresh directory? Your output is completely different. In particular, the negative event data warnings are concerning - that suggests that something is corrupted.
Unfortunately, I'm leaving on vacation for 2 weeks and won't be able to do anything further with this until I return.
Unfortunately, I'm leaving on vacation for 2 weeks and won't be able to do anything further with this until I return.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Yes I unzipped the file. I loaded the model, clicked on "optimize", then I saw the payoff definition window; I hit next button, I saw the optimization control file window with MCMC, then I hit on next button, saw the data source window, then hit "Finish" button. That's what I did, I don't know if that what I should do
Re: Help with model calibration
Hi Tomfid, hope you had a great vacation!
I was wondering if you give me some direction on the MCMC procedure.
Thanks so much!
I was wondering if you give me some direction on the MCMC procedure.
Thanks so much!
-
- Super Administrator
- Posts: 4573
- Joined: Wed Mar 05, 2003 3:10 am
Re: Help with model calibration
What are you struggling with?
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Re: Help with model calibration
Try running it from the simulation control dialog. The setup should look like this:
and the result should have no errors./*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Thanks so much.
I think I was not doing the right thing. Please, find attached the steps I went through.
How did you get the set up you sent me?
I also found out that the estimated parameters largely depend on the random model estimates assigned to the parameters. For example, if I assign a value of 2 to retention time, the estimate for retention time derived from the calibration would be close to 2. Likewise, if I assign 4 to retention time, the estimate for retention time would be close to 4. So, I believe something is wrong because the estimated parameters should be the same regardless of the initial assigned parameters. Do you have any ideas why this might be happening? I know the estimates are not yet stable
I think I was not doing the right thing. Please, find attached the steps I went through.
How did you get the set up you sent me?
I also found out that the estimated parameters largely depend on the random model estimates assigned to the parameters. For example, if I assign a value of 2 to retention time, the estimate for retention time derived from the calibration would be close to 2. Likewise, if I assign 4 to retention time, the estimate for retention time would be close to 4. So, I believe something is wrong because the estimated parameters should be the same regardless of the initial assigned parameters. Do you have any ideas why this might be happening? I know the estimates are not yet stable
- Attachments
-
- powel2.docx
- (78.75 KiB) Downloaded 276 times
Re: Help with model calibration
I think I found the control dialog you used. But I still don't get the same results as you
- Attachments
-
- doc.docx
- (40.82 KiB) Downloaded 284 times
Re: Help with model calibration
Hi Tomfid,
Do you have any suggestions for the issues I'm encountering?
Thanks so much
Do you have any suggestions for the issues I'm encountering?
Thanks so much
-
- Super Administrator
- Posts: 4573
- Joined: Wed Mar 05, 2003 3:10 am
Re: Help with model calibration
What happens if you make the initial estimate for retention time something like 8? Does the final estimate stay around 8 or go to 4leon wrote:I also found out that the estimated parameters largely depend on the random model estimates assigned to the parameters. For example, if I assign a value of 2 to retention time, the estimate for retention time derived from the calibration would be close to 2. Likewise, if I assign 4 to retention time, the estimate for retention time would be close to 4. So, I believe something is wrong because the estimated parameters should be the same regardless of the initial assigned parameters. Do you have any ideas why this might be happening? I know the estimates are not yet stable
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Re: Help with model calibration
One thing I notice is that you have the lower bound for the initial event rate set to 0. This isn't really plausible, as it implies that a driver could never make a mistake. I'd set it to a small positive value, like 1e-6.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
The estimates can vary with the initial values if the model has multiple optima. The best thing to do in that case is to turn on multiple start and let it run overnight.
https://www.vensim.com/documentation/multiple_start.htm
https://www.vensim.com/documentation/multiple_start.htm
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
I think your fraction of events known is a redundant parameter. It's confounded with effect of recent memories on event per mile - if one is 10x bigger and the other 10x smaller, the effect is the same. Right?
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
I think you could make more improvements to bounds. For example,
- the time constants have to have a lower bound of at least the TIME STEP; otherwise the model is unstable.
- the baseline memories etc. should be bounded above by some reasonable limit - for example, the maximum monthly events in the data is < 200, so 200 * fraction known * retention time is probably a reasonable upper bound for memories.
For drivers with no events, you probably can't say much about their individual parameters.
- the time constants have to have a lower bound of at least the TIME STEP; otherwise the model is unstable.
- the baseline memories etc. should be bounded above by some reasonable limit - for example, the maximum monthly events in the data is < 200, so 200 * fraction known * retention time is probably a reasonable upper bound for memories.
For drivers with no events, you probably can't say much about their individual parameters.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Here's an update that implements some of my comments, and includes a fresh copy of the MCMC sample.
I notice one more thing. The event fading time comes out quite long - longer than the model horizon. Is that plausible? It may simply be generating a trend. Is it really justifiable to have two separate event memory processes with very different time constants?
I think it would help to spend some time going through the drivers one by one and looking at the calibration results.
I notice one more thing. The event fading time comes out quite long - longer than the model horizon. Is that plausible? It may simply be generating a trend. Is it really justifiable to have two separate event memory processes with very different time constants?
I think it would help to spend some time going through the drivers one by one and looking at the calibration results.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
I think you raised an important question here. There are different types of events (driving errors): minor errors (not known to the driver and detected by a video recording device), near-collision and collision. The last 2 types of errors are assumed to be known by the driver. So I created the parameter “fraction of events know” to account for the fact that some events are lost to memory. One of the interventions I’m hoping to test is sharing the error data with the driver so that they are aware of all their errors. So, do you think it makes sense to have such a parameter (fraction of events known)?tomfid wrote:I think your fraction of events known is a redundant parameter. It's confounded with effect of recent memories on event per mile - if one is 10x bigger and the other 10x smaller, the effect is the same. Right?
Re: Help with model calibration
I went through the drivers and I think the fit looks good.tomfid wrote:Here's an update that implements some of my comments, and includes a fresh copy of the MCMC sample.
I notice one more thing. The event fading time comes out quite long - longer than the model horizon. Is that plausible? It may simply be generating a trend. Is it really justifiable to have two separate event memory processes with very different time constants?
I think it would help to spend some time going through the drivers one by one and looking at the calibration results.
Driver3b.zip
I don’t know what the MCMC sample is used for. But I looked up the powel.log for the parameter estimates (I did not run the model, I directly looked up your results). The longer event fading time may be correct. One thing I noticed: the initial values for effect of memories and effect of miles driven did not change after calibration. I’m wondering if the event fading time will change if we modify these initial values.
-
- Super Administrator
- Posts: 4573
- Joined: Wed Mar 05, 2003 3:10 am
Re: Help with model calibration
The best thing to do is try it and see if it does.leon wrote:I’m wondering if the event fading time will change if we modify these initial values.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Re: Help with model calibration
I honestly don't know how Tomfid executed the MCMC procedure. Could you please lay out the different steps to do this? After I load the model into in VENSIM, what should I do next?