Help with model calibration

Use this forum to post Vensim related questions.
tomfid
Administrator
Posts: 2014
Joined: Wed May 24, 2006 4:54 am
Location: Montana, USA
Contact:

Re: Help with model calibration

Postby tomfid » Fri Aug 04, 2017 10:00 pm

It would seem that age is a major factor as well.
/*
Advice to posters (it really helps us to help you)
viewtopic.php?f=2&t=4391

Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics

*/

leon
Senior Member
Posts: 181
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Postby leon » Fri Aug 04, 2017 10:03 pm

tomfid wrote:To identify the floating point error, you need to find the parameters that cause it to occur (should be in the output or .log file) and run the model with those to see what the issue is.

A few FP errors out of a million simulations might not be an issue. Most likely it's an overflow in exp() or something like that. You don't know until you investigate.


Thank you so much Tomfid.
This work would not have been possible without your help. I changed the lower bound of retention time to 1 (it was 0.1 before) and I got no floating point. I'm also satisfied with the estimate of cumulative miles because it falls between the confidence interval of the estimate I got using statistical methods.

1) Where should I see the metrics of the model?
2) Do the estimated parameters have confidence intervals?
3) What is the goal of the sensitivity analysis?

leon
Senior Member
Posts: 181
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Postby leon » Fri Aug 04, 2017 10:14 pm

tomfid wrote:It would seem that age is a major factor as well.


I agree age is an important factor. I believe the initial event rate is a surrogate of maturity level (which is a function of age). The data loaded in the model were collected from 16 years old drivers.
Do you think I should add maturity level into the model and make it affect initial event rate?

Thanks a lot

tomfid
Administrator
Posts: 2014
Joined: Wed May 24, 2006 4:54 am
Location: Montana, USA
Contact:

Re: Help with model calibration

Postby tomfid » Fri Aug 04, 2017 11:24 pm

I think age and the experimental treatment should both be in there if they vary within the dataset. If all the drivers are 16, and all in the control group, there's no particular advantage of adding the parameters.
/*
Advice to posters (it really helps us to help you)
viewtopic.php?f=2&t=4391

Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics

*/

leon
Senior Member
Posts: 181
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Postby leon » Sat Aug 05, 2017 5:11 am

Drivers in both groups (control and experimental) were 16 years old.

Control data: natural driving.

For the experimental data:
The treatment provided feedback in two ways. First, a small light visible to the teen driver flashed to alert the driver whenever a driving error has been made. Second, parents received a weekly report card that summarizes the teen’s driving behavior and video clips of driving errors and behaviors. Parents had clear, contextual feedback (video and audio) of each error and reviewed the clips with their teens to discuss a wide variety of driving skills and behaviors. So, both known and unknown errors were shared with drivers.

What do you think if I make the variable "feedback intervention" influence "fraction of events know"? This means that "feedback intervention" will take on the value of fraction of events know under natural conditions if there is no intervention, then it will take on the value of 100% if there is treatment.

It would be interesting to test behavioral interventions. This would mean we need to add behavioral factors (eg. traffic violations, cell phone use) to the model. Do you have any ideas about how these factors can be included in the internal structure of the model? Otherwise, what we have now is already fine.

Thanks

LAUJJL
Senior Member
Posts: 1155
Joined: Fri May 23, 2003 10:09 am
Location: STRASBOURG FRANCE

Re: Help with model calibration

Postby LAUJJL » Sat Aug 05, 2017 4:51 pm

Hi Leon.

I prefer to stop my remarks because they may be inappropriate.

You have a continuous deterministic model and seem to have discreet data. This is why you use a small time step to be able to introduce your discreet data. I have no experience of that.

I have too a lot of exogenous discreet data (demand from prospects) and their treatment is highly stochastic (the client will buy or not buy depending on the price offer) the price offer depending on the availability that is changing depending on the acceptance of the prospect). This means that the whole availability system is changing at each demand. It means that each demand has an influence on the other next demands. In this situation it is useless to transform the discreet demand into a continuous one. I build then discreet stochastic models with a lot of random values generator to be able to simulate a discreet reality.
But in your case, using a continuous model may work, but having no experience with it, I cannot make any advise about it.

In my last model, I was smoothing events to capture the past driving capacity that will automatically influence the present driving practice even if it is influenced too by other factors.

To work correctly my model should have a greater time step, to allow the flows to better represent the reality. With longer time steps, you have more events that can be considered continuous when you aggregate them.

Regards.

JJ

leon
Senior Member
Posts: 181
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Postby leon » Tue Aug 08, 2017 3:50 pm

Hi Tomfid,
I have few questions
1) Does the calibration give some performance metrics (eg. R-squarre for instance)
2)What should be the next step after the calibration? Sensitivity analysis or policy optimization?
3) Is there any way I can change the color of the graph?

Thanks so much

leon
Senior Member
Posts: 181
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Postby leon » Wed Aug 16, 2017 3:12 pm

Hi Tomfid,
I'm trying to find the optimal number of months the drivers should receive the treatment (intervention). Is it something VENSIM can do?
Thanks so much

leon
Senior Member
Posts: 181
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Postby leon » Mon Aug 28, 2017 1:05 pm

Hi,
I'm sorry for any mistakes I may have made.
Is there a link for policy analysis in the Help?
Your help is greatly appreciated.
Thanks

tomfid
Administrator
Posts: 2014
Joined: Wed May 24, 2006 4:54 am
Location: Montana, USA
Contact:

Re: Help with model calibration

Postby tomfid » Mon Aug 28, 2017 2:26 pm

For a policy optimization, you need to express the tradeoffs involved:

- what does the policy do? (Presumably changes the fraction of events known?) I think you want to have the full sample in the model, with and without treatment, so you can estimate the differences.
- what does the policy cost? (Cost of time, money spent implementing driver feedback.)
- what's the benefit? (Value of an avoided error.)

Then the payoff is (value-cost) and the control variable is the extent of treatment (0-100%, or time/$ budget for effort).

If the situation isn't very nonlinear, the answer may be boring - 0 or 100% effort.
/*
Advice to posters (it really helps us to help you)
viewtopic.php?f=2&t=4391

Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics

*/

leon
Senior Member
Posts: 181
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Postby leon » Wed Sep 13, 2017 1:11 pm

Thanks so much for your help.
I'm sorry, but I'm struggling to apply the concept you explained.
The cost (from the information I got) is baseline fee+a monthly fee.
What do you think of the way I applied the concept?
Attachments
Doc1.docx
(13.01 KiB) Downloaded 5 times

tomfid
Administrator
Posts: 2014
Joined: Wed May 24, 2006 4:54 am
Location: Montana, USA
Contact:

Re: Help with model calibration

Postby tomfid » Wed Sep 13, 2017 1:37 pm

I think you want to set it up so that you have an indicator of whether the intervention is active, maybe a 0-1 switch, with the effect (fraction known) and cost dependent on that.
/*
Advice to posters (it really helps us to help you)
viewtopic.php?f=2&t=4391

Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics

*/

leon
Senior Member
Posts: 181
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Postby leon » Wed Sep 13, 2017 3:19 pm

It seems like I understood now.
Feedback intervention=indicator*(fraction known + cost)
The payoff is the difference in errors/month when the indicator switches between 0 and 1.
Will it be possible to determine the optimal duration of the intervention?
Attachments
Doc1.docx
(13.12 KiB) Downloaded 5 times

tomfid
Administrator
Posts: 2014
Joined: Wed May 24, 2006 4:54 am
Location: Montana, USA
Contact:

Re: Help with model calibration

Postby tomfid » Wed Sep 13, 2017 3:27 pm

No. Think about the units of "fraction known + cost" - it doesn't work.

Roughly,

payoff = -cost
cost = cost of errors + cost of intervention
cost of errors = cost per error * errors = f(intervention via fraction known)
intervention = {0 or 1 switch}
fraction known = base fraction + intervention*effect of intervention
cost of intervention = intervention*unit cost of intervention
/*
Advice to posters (it really helps us to help you)
viewtopic.php?f=2&t=4391

Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics

*/

leon
Senior Member
Posts: 181
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Postby leon » Wed Sep 13, 2017 6:43 pm

Does it diagram reflect what you explained?
Attachments
Doc1.docx
(13.97 KiB) Downloaded 4 times

tomfid
Administrator
Posts: 2014
Joined: Wed May 24, 2006 4:54 am
Location: Montana, USA
Contact:

Re: Help with model calibration

Postby tomfid » Wed Sep 13, 2017 7:55 pm

Almost. The only difference is that I would expect events to have a cost (i.e. risk) whether known or not.
/*
Advice to posters (it really helps us to help you)
viewtopic.php?f=2&t=4391

Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics

*/

leon
Senior Member
Posts: 181
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Postby leon » Wed Sep 13, 2017 8:41 pm

Is this correct?
Also, what are we trying to determine from this diagram? Fraction?
Attachments
Doc1.docx
(13.83 KiB) Downloaded 5 times

tomfid
Administrator
Posts: 2014
Joined: Wed May 24, 2006 4:54 am
Location: Montana, USA
Contact:

Re: Help with model calibration

Postby tomfid » Wed Sep 13, 2017 10:41 pm

The link from fraction of events known to cost of events is not needed. The fraction influences the number of events (via a longer causal path).

The point here is to construct a measure of cost to optimize (minimize). It has to capture the tradeoff between the cost of the problem (events) and cost of the solution (intervention).
/*
Advice to posters (it really helps us to help you)
viewtopic.php?f=2&t=4391

Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics

*/

leon
Senior Member
Posts: 181
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Postby leon » Thu Sep 14, 2017 12:28 am

I've added the diagram to the model.
I guess the new variables should not have subscripts.
What do you think?
Attachments
Doc1.docx
(17.92 KiB) Downloaded 5 times

leon
Senior Member
Posts: 181
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Postby leon » Mon Sep 18, 2017 1:35 pm

Hi,
I'm sorry, but I have 2 Vensim questions:
1) I'm trying to export model output into excel. Specifically, I want to export the model predictions for events per month and monthly driving into excel. I was doing by hands, but it's taking time. Is there a way to expect model predictions from Vensim into excel?
2) I'm also trying to load the data for the treatment group into the model. How should I specify which subject belongs to the control or the treatment?
Thanks so much for all your help, I'm so appreciative.

Administrator
Super Administrator
Posts: 2829
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Postby Administrator » Tue Sep 19, 2017 8:21 am

leon wrote:1) I'm trying to export model output into excel. Specifically, I want to export the model predictions for events per month and monthly driving into excel. I was doing by hands, but it's taking time. Is there a way to expect model predictions from Vensim into excel?

Do you mean an automated way? This is possible only with Vensim DSS (your profile has Pro as your version of Vensim).

leon wrote:2) I'm also trying to load the data for the treatment group into the model. How should I specify which subject belongs to the control or the treatment?
Thanks so much for all your help, I'm so appreciative.

Can you upload an example? It's impossible to help/assist with this description.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/viewtopic.php?f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559

leon
Senior Member
Posts: 181
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Postby leon » Tue Sep 19, 2017 5:04 pm

Thanks so much.
I've attached an example of the data.

Exporting into Excel:
After I ran the model, I hit on table on the left corner, it displayed the prediction time series. I want to export that table into excel. Or if I can copy the table, that would be fine too. But software won't let me copy it.
Attachments
Data outline.xlsx
(8.69 KiB) Downloaded 5 times

Administrator
Super Administrator
Posts: 2829
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Postby Administrator » Tue Sep 19, 2017 5:19 pm

leon wrote:Exporting into Excel:
After I ran the model, I hit on table on the left corner, it displayed the prediction time series. I want to export that table into excel. Or if I can copy the table, that would be fine too. But software won't let me copy it.

Why can't you copy it? Hit the clipboard button in the table window (top left) and it will copy the contents to the clipboard.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/viewtopic.php?f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559

Administrator
Super Administrator
Posts: 2829
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Postby Administrator » Tue Sep 19, 2017 5:20 pm

leon wrote:Thanks so much.
I've attached an example of the data.

What is the problem importing the data? It looks like it should import without issue. What errors are you getting?
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/viewtopic.php?f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559

leon
Senior Member
Posts: 181
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Postby leon » Tue Sep 19, 2017 6:08 pm

I was asking how to format the data. Remember we used subscripts for driver ID. There is a new variable treatment and drivers are categorized by treatment group. How to format the data taking into account treatment group?


Return to “Vensim”

Who is online

Users browsing this forum: No registered users and 3 guests