Help with model calibration

Use this forum to post Vensim related questions.
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

It would seem that age is a major factor as well.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

tomfid wrote:To identify the floating point error, you need to find the parameters that cause it to occur (should be in the output or .log file) and run the model with those to see what the issue is.

A few FP errors out of a million simulations might not be an issue. Most likely it's an overflow in exp() or something like that. You don't know until you investigate.
Thank you so much Tomfid.
This work would not have been possible without your help. I changed the lower bound of retention time to 1 (it was 0.1 before) and I got no floating point. I'm also satisfied with the estimate of cumulative miles because it falls between the confidence interval of the estimate I got using statistical methods.

1) Where should I see the metrics of the model?
2) Do the estimated parameters have confidence intervals?
3) What is the goal of the sensitivity analysis?
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

tomfid wrote:It would seem that age is a major factor as well.
I agree age is an important factor. I believe the initial event rate is a surrogate of maturity level (which is a function of age). The data loaded in the model were collected from 16 years old drivers.
Do you think I should add maturity level into the model and make it affect initial event rate?

Thanks a lot
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

I think age and the experimental treatment should both be in there if they vary within the dataset. If all the drivers are 16, and all in the control group, there's no particular advantage of adding the parameters.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Drivers in both groups (control and experimental) were 16 years old.

Control data: natural driving.

For the experimental data:
The treatment provided feedback in two ways. First, a small light visible to the teen driver flashed to alert the driver whenever a driving error has been made. Second, parents received a weekly report card that summarizes the teen’s driving behavior and video clips of driving errors and behaviors. Parents had clear, contextual feedback (video and audio) of each error and reviewed the clips with their teens to discuss a wide variety of driving skills and behaviors. So, both known and unknown errors were shared with drivers.

What do you think if I make the variable "feedback intervention" influence "fraction of events know"? This means that "feedback intervention" will take on the value of fraction of events know under natural conditions if there is no intervention, then it will take on the value of 100% if there is treatment.

It would be interesting to test behavioral interventions. This would mean we need to add behavioral factors (eg. traffic violations, cell phone use) to the model. Do you have any ideas about how these factors can be included in the internal structure of the model? Otherwise, what we have now is already fine.

Thanks
LAUJJL
Senior Member
Posts: 1421
Joined: Fri May 23, 2003 10:09 am
Vensim version: DSS

Re: Help with model calibration

Post by LAUJJL »

Hi Leon.

I prefer to stop my remarks because they may be inappropriate.

You have a continuous deterministic model and seem to have discreet data. This is why you use a small time step to be able to introduce your discreet data. I have no experience of that.

I have too a lot of exogenous discreet data (demand from prospects) and their treatment is highly stochastic (the client will buy or not buy depending on the price offer) the price offer depending on the availability that is changing depending on the acceptance of the prospect). This means that the whole availability system is changing at each demand. It means that each demand has an influence on the other next demands. In this situation it is useless to transform the discreet demand into a continuous one. I build then discreet stochastic models with a lot of random values generator to be able to simulate a discreet reality.
But in your case, using a continuous model may work, but having no experience with it, I cannot make any advise about it.

In my last model, I was smoothing events to capture the past driving capacity that will automatically influence the present driving practice even if it is influenced too by other factors.

To work correctly my model should have a greater time step, to allow the flows to better represent the reality. With longer time steps, you have more events that can be considered continuous when you aggregate them.

Regards.

JJ
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Hi Tomfid,
I have few questions
1) Does the calibration give some performance metrics (eg. R-squarre for instance)
2)What should be the next step after the calibration? Sensitivity analysis or policy optimization?
3) Is there any way I can change the color of the graph?

Thanks so much
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Hi Tomfid,
I'm trying to find the optimal number of months the drivers should receive the treatment (intervention). Is it something VENSIM can do?
Thanks so much
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Hi,
I'm sorry for any mistakes I may have made.
Is there a link for policy analysis in the Help?
Your help is greatly appreciated.
Thanks
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

For a policy optimization, you need to express the tradeoffs involved:

- what does the policy do? (Presumably changes the fraction of events known?) I think you want to have the full sample in the model, with and without treatment, so you can estimate the differences.
- what does the policy cost? (Cost of time, money spent implementing driver feedback.)
- what's the benefit? (Value of an avoided error.)

Then the payoff is (value-cost) and the control variable is the extent of treatment (0-100%, or time/$ budget for effort).

If the situation isn't very nonlinear, the answer may be boring - 0 or 100% effort.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks so much for your help.
I'm sorry, but I'm struggling to apply the concept you explained.
The cost (from the information I got) is baseline fee+a monthly fee.
What do you think of the way I applied the concept?
Attachments
Doc1.docx
(13.01 KiB) Downloaded 235 times
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

I think you want to set it up so that you have an indicator of whether the intervention is active, maybe a 0-1 switch, with the effect (fraction known) and cost dependent on that.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

It seems like I understood now.
Feedback intervention=indicator*(fraction known + cost)
The payoff is the difference in errors/month when the indicator switches between 0 and 1.
Will it be possible to determine the optimal duration of the intervention?
Attachments
Doc1.docx
(13.12 KiB) Downloaded 230 times
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

No. Think about the units of "fraction known + cost" - it doesn't work.

Roughly,

payoff = -cost
cost = cost of errors + cost of intervention
cost of errors = cost per error * errors = f(intervention via fraction known)
intervention = {0 or 1 switch}
fraction known = base fraction + intervention*effect of intervention
cost of intervention = intervention*unit cost of intervention
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Does it diagram reflect what you explained?
Attachments
Doc1.docx
(13.97 KiB) Downloaded 221 times
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Almost. The only difference is that I would expect events to have a cost (i.e. risk) whether known or not.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Is this correct?
Also, what are we trying to determine from this diagram? Fraction?
Attachments
Doc1.docx
(13.83 KiB) Downloaded 208 times
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

The link from fraction of events known to cost of events is not needed. The fraction influences the number of events (via a longer causal path).

The point here is to construct a measure of cost to optimize (minimize). It has to capture the tradeoff between the cost of the problem (events) and cost of the solution (intervention).
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I've added the diagram to the model.
I guess the new variables should not have subscripts.
What do you think?
Attachments
Doc1.docx
(17.92 KiB) Downloaded 192 times
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Hi,
I'm sorry, but I have 2 Vensim questions:
1) I'm trying to export model output into excel. Specifically, I want to export the model predictions for events per month and monthly driving into excel. I was doing by hands, but it's taking time. Is there a way to expect model predictions from Vensim into excel?
2) I'm also trying to load the data for the treatment group into the model. How should I specify which subject belongs to the control or the treatment?
Thanks so much for all your help, I'm so appreciative.
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

leon wrote:1) I'm trying to export model output into excel. Specifically, I want to export the model predictions for events per month and monthly driving into excel. I was doing by hands, but it's taking time. Is there a way to expect model predictions from Vensim into excel?
Do you mean an automated way? This is possible only with Vensim DSS (your profile has Pro as your version of Vensim).
leon wrote:2) I'm also trying to load the data for the treatment group into the model. How should I specify which subject belongs to the control or the treatment?
Thanks so much for all your help, I'm so appreciative.
Can you upload an example? It's impossible to help/assist with this description.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks so much.
I've attached an example of the data.

Exporting into Excel:
After I ran the model, I hit on table on the left corner, it displayed the prediction time series. I want to export that table into excel. Or if I can copy the table, that would be fine too. But software won't let me copy it.
Attachments
Data outline.xlsx
(8.69 KiB) Downloaded 157 times
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

leon wrote:Exporting into Excel:
After I ran the model, I hit on table on the left corner, it displayed the prediction time series. I want to export that table into excel. Or if I can copy the table, that would be fine too. But software won't let me copy it.
Why can't you copy it? Hit the clipboard button in the table window (top left) and it will copy the contents to the clipboard.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

leon wrote:Thanks so much.
I've attached an example of the data.
What is the problem importing the data? It looks like it should import without issue. What errors are you getting?
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I was asking how to format the data. Remember we used subscripts for driver ID. There is a new variable treatment and drivers are categorized by treatment group. How to format the data taking into account treatment group?
Post Reply