Help with model calibration
Re: Help with model calibration
It would seem that age is a major factor as well.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Thank you so much Tomfid.tomfid wrote:To identify the floating point error, you need to find the parameters that cause it to occur (should be in the output or .log file) and run the model with those to see what the issue is.
A few FP errors out of a million simulations might not be an issue. Most likely it's an overflow in exp() or something like that. You don't know until you investigate.
This work would not have been possible without your help. I changed the lower bound of retention time to 1 (it was 0.1 before) and I got no floating point. I'm also satisfied with the estimate of cumulative miles because it falls between the confidence interval of the estimate I got using statistical methods.
1) Where should I see the metrics of the model?
2) Do the estimated parameters have confidence intervals?
3) What is the goal of the sensitivity analysis?
Re: Help with model calibration
I agree age is an important factor. I believe the initial event rate is a surrogate of maturity level (which is a function of age). The data loaded in the model were collected from 16 years old drivers.tomfid wrote:It would seem that age is a major factor as well.
Do you think I should add maturity level into the model and make it affect initial event rate?
Thanks a lot
Re: Help with model calibration
I think age and the experimental treatment should both be in there if they vary within the dataset. If all the drivers are 16, and all in the control group, there's no particular advantage of adding the parameters.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Drivers in both groups (control and experimental) were 16 years old.
Control data: natural driving.
For the experimental data:
The treatment provided feedback in two ways. First, a small light visible to the teen driver flashed to alert the driver whenever a driving error has been made. Second, parents received a weekly report card that summarizes the teen’s driving behavior and video clips of driving errors and behaviors. Parents had clear, contextual feedback (video and audio) of each error and reviewed the clips with their teens to discuss a wide variety of driving skills and behaviors. So, both known and unknown errors were shared with drivers.
What do you think if I make the variable "feedback intervention" influence "fraction of events know"? This means that "feedback intervention" will take on the value of fraction of events know under natural conditions if there is no intervention, then it will take on the value of 100% if there is treatment.
It would be interesting to test behavioral interventions. This would mean we need to add behavioral factors (eg. traffic violations, cell phone use) to the model. Do you have any ideas about how these factors can be included in the internal structure of the model? Otherwise, what we have now is already fine.
Thanks
Control data: natural driving.
For the experimental data:
The treatment provided feedback in two ways. First, a small light visible to the teen driver flashed to alert the driver whenever a driving error has been made. Second, parents received a weekly report card that summarizes the teen’s driving behavior and video clips of driving errors and behaviors. Parents had clear, contextual feedback (video and audio) of each error and reviewed the clips with their teens to discuss a wide variety of driving skills and behaviors. So, both known and unknown errors were shared with drivers.
What do you think if I make the variable "feedback intervention" influence "fraction of events know"? This means that "feedback intervention" will take on the value of fraction of events know under natural conditions if there is no intervention, then it will take on the value of 100% if there is treatment.
It would be interesting to test behavioral interventions. This would mean we need to add behavioral factors (eg. traffic violations, cell phone use) to the model. Do you have any ideas about how these factors can be included in the internal structure of the model? Otherwise, what we have now is already fine.
Thanks
Re: Help with model calibration
Hi Leon.
I prefer to stop my remarks because they may be inappropriate.
You have a continuous deterministic model and seem to have discreet data. This is why you use a small time step to be able to introduce your discreet data. I have no experience of that.
I have too a lot of exogenous discreet data (demand from prospects) and their treatment is highly stochastic (the client will buy or not buy depending on the price offer) the price offer depending on the availability that is changing depending on the acceptance of the prospect). This means that the whole availability system is changing at each demand. It means that each demand has an influence on the other next demands. In this situation it is useless to transform the discreet demand into a continuous one. I build then discreet stochastic models with a lot of random values generator to be able to simulate a discreet reality.
But in your case, using a continuous model may work, but having no experience with it, I cannot make any advise about it.
In my last model, I was smoothing events to capture the past driving capacity that will automatically influence the present driving practice even if it is influenced too by other factors.
To work correctly my model should have a greater time step, to allow the flows to better represent the reality. With longer time steps, you have more events that can be considered continuous when you aggregate them.
Regards.
JJ
I prefer to stop my remarks because they may be inappropriate.
You have a continuous deterministic model and seem to have discreet data. This is why you use a small time step to be able to introduce your discreet data. I have no experience of that.
I have too a lot of exogenous discreet data (demand from prospects) and their treatment is highly stochastic (the client will buy or not buy depending on the price offer) the price offer depending on the availability that is changing depending on the acceptance of the prospect). This means that the whole availability system is changing at each demand. It means that each demand has an influence on the other next demands. In this situation it is useless to transform the discreet demand into a continuous one. I build then discreet stochastic models with a lot of random values generator to be able to simulate a discreet reality.
But in your case, using a continuous model may work, but having no experience with it, I cannot make any advise about it.
In my last model, I was smoothing events to capture the past driving capacity that will automatically influence the present driving practice even if it is influenced too by other factors.
To work correctly my model should have a greater time step, to allow the flows to better represent the reality. With longer time steps, you have more events that can be considered continuous when you aggregate them.
Regards.
JJ
Re: Help with model calibration
Hi Tomfid,
I have few questions
1) Does the calibration give some performance metrics (eg. R-squarre for instance)
2)What should be the next step after the calibration? Sensitivity analysis or policy optimization?
3) Is there any way I can change the color of the graph?
Thanks so much
I have few questions
1) Does the calibration give some performance metrics (eg. R-squarre for instance)
2)What should be the next step after the calibration? Sensitivity analysis or policy optimization?
3) Is there any way I can change the color of the graph?
Thanks so much
Re: Help with model calibration
Hi Tomfid,
I'm trying to find the optimal number of months the drivers should receive the treatment (intervention). Is it something VENSIM can do?
Thanks so much
I'm trying to find the optimal number of months the drivers should receive the treatment (intervention). Is it something VENSIM can do?
Thanks so much
Re: Help with model calibration
Hi,
I'm sorry for any mistakes I may have made.
Is there a link for policy analysis in the Help?
Your help is greatly appreciated.
Thanks
I'm sorry for any mistakes I may have made.
Is there a link for policy analysis in the Help?
Your help is greatly appreciated.
Thanks
Re: Help with model calibration
For a policy optimization, you need to express the tradeoffs involved:
- what does the policy do? (Presumably changes the fraction of events known?) I think you want to have the full sample in the model, with and without treatment, so you can estimate the differences.
- what does the policy cost? (Cost of time, money spent implementing driver feedback.)
- what's the benefit? (Value of an avoided error.)
Then the payoff is (value-cost) and the control variable is the extent of treatment (0-100%, or time/$ budget for effort).
If the situation isn't very nonlinear, the answer may be boring - 0 or 100% effort.
- what does the policy do? (Presumably changes the fraction of events known?) I think you want to have the full sample in the model, with and without treatment, so you can estimate the differences.
- what does the policy cost? (Cost of time, money spent implementing driver feedback.)
- what's the benefit? (Value of an avoided error.)
Then the payoff is (value-cost) and the control variable is the extent of treatment (0-100%, or time/$ budget for effort).
If the situation isn't very nonlinear, the answer may be boring - 0 or 100% effort.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Thanks so much for your help.
I'm sorry, but I'm struggling to apply the concept you explained.
The cost (from the information I got) is baseline fee+a monthly fee.
What do you think of the way I applied the concept?
I'm sorry, but I'm struggling to apply the concept you explained.
The cost (from the information I got) is baseline fee+a monthly fee.
What do you think of the way I applied the concept?
- Attachments
-
- Doc1.docx
- (13.01 KiB) Downloaded 287 times
Re: Help with model calibration
I think you want to set it up so that you have an indicator of whether the intervention is active, maybe a 0-1 switch, with the effect (fraction known) and cost dependent on that.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
It seems like I understood now.
Feedback intervention=indicator*(fraction known + cost)
The payoff is the difference in errors/month when the indicator switches between 0 and 1.
Will it be possible to determine the optimal duration of the intervention?
Feedback intervention=indicator*(fraction known + cost)
The payoff is the difference in errors/month when the indicator switches between 0 and 1.
Will it be possible to determine the optimal duration of the intervention?
- Attachments
-
- Doc1.docx
- (13.12 KiB) Downloaded 282 times
Re: Help with model calibration
No. Think about the units of "fraction known + cost" - it doesn't work.
Roughly,
payoff = -cost
cost = cost of errors + cost of intervention
cost of errors = cost per error * errors = f(intervention via fraction known)
intervention = {0 or 1 switch}
fraction known = base fraction + intervention*effect of intervention
cost of intervention = intervention*unit cost of intervention
Roughly,
payoff = -cost
cost = cost of errors + cost of intervention
cost of errors = cost per error * errors = f(intervention via fraction known)
intervention = {0 or 1 switch}
fraction known = base fraction + intervention*effect of intervention
cost of intervention = intervention*unit cost of intervention
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Does it diagram reflect what you explained?
- Attachments
-
- Doc1.docx
- (13.97 KiB) Downloaded 274 times
Re: Help with model calibration
Almost. The only difference is that I would expect events to have a cost (i.e. risk) whether known or not.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Is this correct?
Also, what are we trying to determine from this diagram? Fraction?
Also, what are we trying to determine from this diagram? Fraction?
- Attachments
-
- Doc1.docx
- (13.83 KiB) Downloaded 261 times
Re: Help with model calibration
The link from fraction of events known to cost of events is not needed. The fraction influences the number of events (via a longer causal path).
The point here is to construct a measure of cost to optimize (minimize). It has to capture the tradeoff between the cost of the problem (events) and cost of the solution (intervention).
The point here is to construct a measure of cost to optimize (minimize). It has to capture the tradeoff between the cost of the problem (events) and cost of the solution (intervention).
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
I've added the diagram to the model.
I guess the new variables should not have subscripts.
What do you think?
I guess the new variables should not have subscripts.
What do you think?
- Attachments
-
- Doc1.docx
- (17.92 KiB) Downloaded 243 times
Re: Help with model calibration
Hi,
I'm sorry, but I have 2 Vensim questions:
1) I'm trying to export model output into excel. Specifically, I want to export the model predictions for events per month and monthly driving into excel. I was doing by hands, but it's taking time. Is there a way to expect model predictions from Vensim into excel?
2) I'm also trying to load the data for the treatment group into the model. How should I specify which subject belongs to the control or the treatment?
Thanks so much for all your help, I'm so appreciative.
I'm sorry, but I have 2 Vensim questions:
1) I'm trying to export model output into excel. Specifically, I want to export the model predictions for events per month and monthly driving into excel. I was doing by hands, but it's taking time. Is there a way to expect model predictions from Vensim into excel?
2) I'm also trying to load the data for the treatment group into the model. How should I specify which subject belongs to the control or the treatment?
Thanks so much for all your help, I'm so appreciative.
-
- Super Administrator
- Posts: 4605
- Joined: Wed Mar 05, 2003 3:10 am
Re: Help with model calibration
Do you mean an automated way? This is possible only with Vensim DSS (your profile has Pro as your version of Vensim).leon wrote:1) I'm trying to export model output into excel. Specifically, I want to export the model predictions for events per month and monthly driving into excel. I was doing by hands, but it's taking time. Is there a way to expect model predictions from Vensim into excel?
Can you upload an example? It's impossible to help/assist with this description.leon wrote:2) I'm also trying to load the data for the treatment group into the model. How should I specify which subject belongs to the control or the treatment?
Thanks so much for all your help, I'm so appreciative.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Re: Help with model calibration
Thanks so much.
I've attached an example of the data.
Exporting into Excel:
After I ran the model, I hit on table on the left corner, it displayed the prediction time series. I want to export that table into excel. Or if I can copy the table, that would be fine too. But software won't let me copy it.
I've attached an example of the data.
Exporting into Excel:
After I ran the model, I hit on table on the left corner, it displayed the prediction time series. I want to export that table into excel. Or if I can copy the table, that would be fine too. But software won't let me copy it.
- Attachments
-
- Data outline.xlsx
- (8.69 KiB) Downloaded 209 times
-
- Super Administrator
- Posts: 4605
- Joined: Wed Mar 05, 2003 3:10 am
Re: Help with model calibration
Why can't you copy it? Hit the clipboard button in the table window (top left) and it will copy the contents to the clipboard.leon wrote:Exporting into Excel:
After I ran the model, I hit on table on the left corner, it displayed the prediction time series. I want to export that table into excel. Or if I can copy the table, that would be fine too. But software won't let me copy it.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
-
- Super Administrator
- Posts: 4605
- Joined: Wed Mar 05, 2003 3:10 am
Re: Help with model calibration
What is the problem importing the data? It looks like it should import without issue. What errors are you getting?leon wrote:Thanks so much.
I've attached an example of the data.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Re: Help with model calibration
I was asking how to format the data. Remember we used subscripts for driver ID. There is a new variable treatment and drivers are categorized by treatment group. How to format the data taking into account treatment group?