Help with model calibration

Use this forum to post Vensim related questions.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Do you know why I'm getting this error?
Attachments
Optimization.zip
(34.49 KiB) Downloaded 239 times
Doc_error.docx
(123.35 KiB) Downloaded 230 times
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

You've told the optimizer to look at "monthly driving" and not "monthly driving[driverid]".
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks so much, I fixed.

One more question. Do you know what the variable cost mean in the model? I'm trying to have a better sense of it. What would be the interpretation of that variable. I'm sorry I'm not getting that part of the model. Tom gave me the idea
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

Isn't it the actual cost (in $ I assume, you haven't put units in your model) of what you are doing?

I'd suggest putting units in your model before optimisation. This is the most basic of checks on a model.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
mike
Senior Member
Posts: 56
Joined: Fri May 21, 2004 12:53 am

Re: Help with model calibration

Post by mike »

My God,
you Vensim help guys have the patience of a Saint.
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

leon wrote:One more question. Do you know what the variable cost mean in the model? I'm trying to have a better sense of it. What would be the interpretation of that variable. I'm sorry I'm not getting that part of the model. Tom gave me the idea
If that's not clear, I think I would reframe with a question: What question do you hope to answer with the optimization?
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Dear Tomfid,
As you know I'm new to the field of SD. It has been a good learning process.
The effect of the treatment is not of a great interest for me because I can easily estimate that effect using statistical models.
My thought process was to increase the fraction of events known to 100% and compare the simulated to the historical treatment data. I was thinking about using statistical methods for the comparison.

You suggested an idea that I was very interested to learn, however I'm not grasping the concept.
The model now has a cost of events and a cost of treatment. The total cost is the sum of these 2 costs. I don't even know what that total cost means. If I understand what you previously explained, it seems like we want to know by much we can increase the fraction base of events known to have a minimum value for total cost.

My second confusion is about the cost of intervention. This cost is independent of events, it only depends on time. In fact, it is 140$+(50$/month)
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Looking back at the history,
I'm trying to find the optimal number of months the drivers should receive the treatment (intervention). Is it something VENSIM can do?
To do that, you need to express the tradeoffs involved. Otherwise the optimal treatment is all or nothing.

That's the point of the cost metric: to convert the tradeoff into some common currency. Then you can compare the benefit of additional treatment (in value of avoided errors) to the cost (presumably in $ spent delivering the treatment).

Varying the fraction of events known also seems like an interesting experiment, but you don't need optimization for that.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Oh seems like that's where I was confused. So, the cost of intervention is not the actual cost of delivering the intervention? It's just an indicator. Same thing for cost of event?
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

It's more than "just an indicator" because you'll rely on it to make a decision.

I think only you can answer the question however. How you frame it really depends on the situation. If the intervention works to reduce errors, what would stop you from using it 100% of the time?
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I think using the intervention 100% would not be an issue because the actual cost of the intervention is a function of time, it does not depend on how many errors were reported as part of the feedback intervention. The real question is when is it reasonable to stop providing the intervention?. In other words, is providing driver feedback for 2 or 3 months is enough?
My confusion is mostly about the cost metric. Can I use a random value for unit cost of the intervention and a random value for cost per event and still get the answer to my question?
The actual cost of the intervention is a flat fee+a monthly fee.
Thanks so much for your input
LAUJJL
Senior Member
Posts: 1421
Joined: Fri May 23, 2003 10:09 am
Vensim version: DSS

Re: Help with model calibration

Post by LAUJJL »

Hi Leon

As your deadline is Saturday, it is time to accept the fact that by this time you will not have the model finished and to act in consequence.

You are not an expert in SD. Then nobody expects a prefessional work from you

At your place, I would change this negative situation into a positive one.

I would explain to your professors that a model is never finished and has always weaknesses, which is true, and that what is important is to know these weaknesses.

I would then take some time to make your model as tidy as possible. For instance a model without unit errors is already a proof of some intelligent work.

I would then list all the problems that this model generates by itself, plus all the problems that it has not solved and take the time to explain them.

This will take already some time and will prove that you are conscious of the difficulties that are in fact real (the problem itself is rather complex) and the limitations and eventual errors of the model.

A good idea, if you have some time left, is to build some reality check, even if they detect some errors, if you have some experience to build them. Otherwise forget it. Using reality checks needs some experience. The least you can build, with or without reality checks, some mass balance, which is already something.

Good luck for your presentation.

Regards.

JJ
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

tomfid wrote:It's more than "just an indicator" because you'll rely on it to make a decision.

I think only you can answer the question however. How you frame it really depends on the situation. If the intervention works to reduce errors, what would stop you from using it 100% of the time?
Maybe I'm not posing my question well. As you know, some drivers have a really low error rate over time and some have moderate and some have a high rate. I believe that some drivers do not need the intervention and those who need the intervention probably do not need it for the same duration. For example, some may need it for 2 months while some might need it for 4 or 5 or more months. That's what I would like the model to be able to help us with. Like you explained, the cost metric might help for it.
The equation: unit cost of intervention*intervention is not expressed as a function of time. My question is what would you look at to tell how long a particular driver might need the intervention.
Also, how would you express the unit cost of the intervention? The actual cost=140$+50$/month
Let's assume the intervention is used 100%

I'm very sorry for abusing of your time.
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

unit cost of intervention*intervention is a function of time if you make the intervention switch a function of time (or anything else).

What you want is a decision rule for intervention control. For example, you could switch the intervention on for all drivers, then switch it off when their error rate falls below some threshold.

Economically, what you want to do is stop the intervention when the discounted cumulative flow of value from avoided errors falls below the intervention cost. It might be possible to work this out analytically, since the model is simple, though I'm not sure it's worth the trouble.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Sorry, but I'm struggling to specify the decision rule for the intervention control. I tried to use "if then else" statement, but apparently it did not work.
Attachments
Docm.docx
(64.37 KiB) Downloaded 214 times
Optimization.zip
(379.7 KiB) Downloaded 206 times
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

Can you give a little more information instead of telling us "it didn't work"?

What didn't work? What error messages did you get (if any)?
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

It says " the following inputs were not expected (but are used): cost of events, cost of treatment. Do you want to update the input list and modify the diagram"
For decision rule on treatment, I specified:
Treatment=if then else (cost of events<cost of treatment, 0, 1)

I'm not sure this way of specifying the decision rule is correct because it's asking me to modify the diagram
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

This just means you didn't draw the arrows before typing in the equation.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Perfect! Thank you.

One more question. Can you share a help link I can follow to determine 95% confidence interval around parameters estimated from model calibration?
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

One more question. Can you share a help link I can follow to determine 95% confidence interval around parameters estimated from model calibration?
You'll need to export the dataset and calculate it yourself using something like Excel.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

The calibration results do not show standard errors around the estimates. Or where to find those standard errors?
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks Tomnfid.

Is it possible to obtain summary statistics from the current version of Vensim? See attached what I found online.

Thanks
Attachments
Summary statistics.docx
(37.45 KiB) Downloaded 157 times
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Sorry, one more question.
I set the sensitivity to payoff value during the calibration (optimization) and I got the attached results. Are those confidence intervals around the parameter estimates 95%CI?
Thanks
Attachments
Confidence intervals.docx
(15.65 KiB) Downloaded 148 times
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

leon wrote:Thanks Tomnfid.

Is it possible to obtain summary statistics from the current version of Vensim? See attached what I found online.

Thanks
Do you mean the stats tool? It's all in the help system,
http://www.vensim.com/documentation/ind ... s_tool.htm
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Post Reply