Help with model calibration
Re: Help with model calibration
Do you know why I'm getting this error?
- Attachments
-
- Optimization.zip
- (34.49 KiB) Downloaded 264 times
-
- Doc_error.docx
- (123.35 KiB) Downloaded 260 times
-
- Super Administrator
- Posts: 4620
- Joined: Wed Mar 05, 2003 3:10 am
Re: Help with model calibration
You've told the optimizer to look at "monthly driving" and not "monthly driving[driverid]".
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Re: Help with model calibration
Thanks so much, I fixed.
One more question. Do you know what the variable cost mean in the model? I'm trying to have a better sense of it. What would be the interpretation of that variable. I'm sorry I'm not getting that part of the model. Tom gave me the idea
One more question. Do you know what the variable cost mean in the model? I'm trying to have a better sense of it. What would be the interpretation of that variable. I'm sorry I'm not getting that part of the model. Tom gave me the idea
-
- Super Administrator
- Posts: 4620
- Joined: Wed Mar 05, 2003 3:10 am
Re: Help with model calibration
Isn't it the actual cost (in $ I assume, you haven't put units in your model) of what you are doing?
I'd suggest putting units in your model before optimisation. This is the most basic of checks on a model.
I'd suggest putting units in your model before optimisation. This is the most basic of checks on a model.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Re: Help with model calibration
My God,
you Vensim help guys have the patience of a Saint.
you Vensim help guys have the patience of a Saint.
Re: Help with model calibration
If that's not clear, I think I would reframe with a question: What question do you hope to answer with the optimization?leon wrote:One more question. Do you know what the variable cost mean in the model? I'm trying to have a better sense of it. What would be the interpretation of that variable. I'm sorry I'm not getting that part of the model. Tom gave me the idea
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Dear Tomfid,
As you know I'm new to the field of SD. It has been a good learning process.
The effect of the treatment is not of a great interest for me because I can easily estimate that effect using statistical models.
My thought process was to increase the fraction of events known to 100% and compare the simulated to the historical treatment data. I was thinking about using statistical methods for the comparison.
You suggested an idea that I was very interested to learn, however I'm not grasping the concept.
The model now has a cost of events and a cost of treatment. The total cost is the sum of these 2 costs. I don't even know what that total cost means. If I understand what you previously explained, it seems like we want to know by much we can increase the fraction base of events known to have a minimum value for total cost.
My second confusion is about the cost of intervention. This cost is independent of events, it only depends on time. In fact, it is 140$+(50$/month)
As you know I'm new to the field of SD. It has been a good learning process.
The effect of the treatment is not of a great interest for me because I can easily estimate that effect using statistical models.
My thought process was to increase the fraction of events known to 100% and compare the simulated to the historical treatment data. I was thinking about using statistical methods for the comparison.
You suggested an idea that I was very interested to learn, however I'm not grasping the concept.
The model now has a cost of events and a cost of treatment. The total cost is the sum of these 2 costs. I don't even know what that total cost means. If I understand what you previously explained, it seems like we want to know by much we can increase the fraction base of events known to have a minimum value for total cost.
My second confusion is about the cost of intervention. This cost is independent of events, it only depends on time. In fact, it is 140$+(50$/month)
Re: Help with model calibration
Looking back at the history,
That's the point of the cost metric: to convert the tradeoff into some common currency. Then you can compare the benefit of additional treatment (in value of avoided errors) to the cost (presumably in $ spent delivering the treatment).
Varying the fraction of events known also seems like an interesting experiment, but you don't need optimization for that.
To do that, you need to express the tradeoffs involved. Otherwise the optimal treatment is all or nothing.I'm trying to find the optimal number of months the drivers should receive the treatment (intervention). Is it something VENSIM can do?
That's the point of the cost metric: to convert the tradeoff into some common currency. Then you can compare the benefit of additional treatment (in value of avoided errors) to the cost (presumably in $ spent delivering the treatment).
Varying the fraction of events known also seems like an interesting experiment, but you don't need optimization for that.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Oh seems like that's where I was confused. So, the cost of intervention is not the actual cost of delivering the intervention? It's just an indicator. Same thing for cost of event?
Re: Help with model calibration
It's more than "just an indicator" because you'll rely on it to make a decision.
I think only you can answer the question however. How you frame it really depends on the situation. If the intervention works to reduce errors, what would stop you from using it 100% of the time?
I think only you can answer the question however. How you frame it really depends on the situation. If the intervention works to reduce errors, what would stop you from using it 100% of the time?
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
I think using the intervention 100% would not be an issue because the actual cost of the intervention is a function of time, it does not depend on how many errors were reported as part of the feedback intervention. The real question is when is it reasonable to stop providing the intervention?. In other words, is providing driver feedback for 2 or 3 months is enough?
My confusion is mostly about the cost metric. Can I use a random value for unit cost of the intervention and a random value for cost per event and still get the answer to my question?
The actual cost of the intervention is a flat fee+a monthly fee.
Thanks so much for your input
My confusion is mostly about the cost metric. Can I use a random value for unit cost of the intervention and a random value for cost per event and still get the answer to my question?
The actual cost of the intervention is a flat fee+a monthly fee.
Thanks so much for your input
Re: Help with model calibration
Hi Leon
As your deadline is Saturday, it is time to accept the fact that by this time you will not have the model finished and to act in consequence.
You are not an expert in SD. Then nobody expects a prefessional work from you
At your place, I would change this negative situation into a positive one.
I would explain to your professors that a model is never finished and has always weaknesses, which is true, and that what is important is to know these weaknesses.
I would then take some time to make your model as tidy as possible. For instance a model without unit errors is already a proof of some intelligent work.
I would then list all the problems that this model generates by itself, plus all the problems that it has not solved and take the time to explain them.
This will take already some time and will prove that you are conscious of the difficulties that are in fact real (the problem itself is rather complex) and the limitations and eventual errors of the model.
A good idea, if you have some time left, is to build some reality check, even if they detect some errors, if you have some experience to build them. Otherwise forget it. Using reality checks needs some experience. The least you can build, with or without reality checks, some mass balance, which is already something.
Good luck for your presentation.
Regards.
JJ
As your deadline is Saturday, it is time to accept the fact that by this time you will not have the model finished and to act in consequence.
You are not an expert in SD. Then nobody expects a prefessional work from you
At your place, I would change this negative situation into a positive one.
I would explain to your professors that a model is never finished and has always weaknesses, which is true, and that what is important is to know these weaknesses.
I would then take some time to make your model as tidy as possible. For instance a model without unit errors is already a proof of some intelligent work.
I would then list all the problems that this model generates by itself, plus all the problems that it has not solved and take the time to explain them.
This will take already some time and will prove that you are conscious of the difficulties that are in fact real (the problem itself is rather complex) and the limitations and eventual errors of the model.
A good idea, if you have some time left, is to build some reality check, even if they detect some errors, if you have some experience to build them. Otherwise forget it. Using reality checks needs some experience. The least you can build, with or without reality checks, some mass balance, which is already something.
Good luck for your presentation.
Regards.
JJ
Re: Help with model calibration
Maybe I'm not posing my question well. As you know, some drivers have a really low error rate over time and some have moderate and some have a high rate. I believe that some drivers do not need the intervention and those who need the intervention probably do not need it for the same duration. For example, some may need it for 2 months while some might need it for 4 or 5 or more months. That's what I would like the model to be able to help us with. Like you explained, the cost metric might help for it.tomfid wrote:It's more than "just an indicator" because you'll rely on it to make a decision.
I think only you can answer the question however. How you frame it really depends on the situation. If the intervention works to reduce errors, what would stop you from using it 100% of the time?
The equation: unit cost of intervention*intervention is not expressed as a function of time. My question is what would you look at to tell how long a particular driver might need the intervention.
Also, how would you express the unit cost of the intervention? The actual cost=140$+50$/month
Let's assume the intervention is used 100%
I'm very sorry for abusing of your time.
Re: Help with model calibration
unit cost of intervention*intervention is a function of time if you make the intervention switch a function of time (or anything else).
What you want is a decision rule for intervention control. For example, you could switch the intervention on for all drivers, then switch it off when their error rate falls below some threshold.
Economically, what you want to do is stop the intervention when the discounted cumulative flow of value from avoided errors falls below the intervention cost. It might be possible to work this out analytically, since the model is simple, though I'm not sure it's worth the trouble.
What you want is a decision rule for intervention control. For example, you could switch the intervention on for all drivers, then switch it off when their error rate falls below some threshold.
Economically, what you want to do is stop the intervention when the discounted cumulative flow of value from avoided errors falls below the intervention cost. It might be possible to work this out analytically, since the model is simple, though I'm not sure it's worth the trouble.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Sorry, but I'm struggling to specify the decision rule for the intervention control. I tried to use "if then else" statement, but apparently it did not work.
- Attachments
-
- Docm.docx
- (64.37 KiB) Downloaded 242 times
-
- Optimization.zip
- (379.7 KiB) Downloaded 237 times
-
- Super Administrator
- Posts: 4620
- Joined: Wed Mar 05, 2003 3:10 am
Re: Help with model calibration
Can you give a little more information instead of telling us "it didn't work"?
What didn't work? What error messages did you get (if any)?
What didn't work? What error messages did you get (if any)?
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Re: Help with model calibration
It says " the following inputs were not expected (but are used): cost of events, cost of treatment. Do you want to update the input list and modify the diagram"
For decision rule on treatment, I specified:
Treatment=if then else (cost of events<cost of treatment, 0, 1)
I'm not sure this way of specifying the decision rule is correct because it's asking me to modify the diagram
For decision rule on treatment, I specified:
Treatment=if then else (cost of events<cost of treatment, 0, 1)
I'm not sure this way of specifying the decision rule is correct because it's asking me to modify the diagram
-
- Super Administrator
- Posts: 4620
- Joined: Wed Mar 05, 2003 3:10 am
Re: Help with model calibration
This just means you didn't draw the arrows before typing in the equation.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Re: Help with model calibration
Perfect! Thank you.
One more question. Can you share a help link I can follow to determine 95% confidence interval around parameters estimated from model calibration?
One more question. Can you share a help link I can follow to determine 95% confidence interval around parameters estimated from model calibration?
-
- Super Administrator
- Posts: 4620
- Joined: Wed Mar 05, 2003 3:10 am
Re: Help with model calibration
You'll need to export the dataset and calculate it yourself using something like Excel.One more question. Can you share a help link I can follow to determine 95% confidence interval around parameters estimated from model calibration?
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Re: Help with model calibration
The calibration results do not show standard errors around the estimates. Or where to find those standard errors?
Re: Help with model calibration
There are two options:
https://www.vensim.com/documentation/pa ... tivity.htm
https://www.vensim.com/documentation/mcmc_sa.htm
https://www.vensim.com/documentation/pa ... tivity.htm
https://www.vensim.com/documentation/mcmc_sa.htm
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Help with model calibration
Thanks Tomnfid.
Is it possible to obtain summary statistics from the current version of Vensim? See attached what I found online.
Thanks
Is it possible to obtain summary statistics from the current version of Vensim? See attached what I found online.
Thanks
- Attachments
-
- Summary statistics.docx
- (37.45 KiB) Downloaded 187 times
Re: Help with model calibration
Sorry, one more question.
I set the sensitivity to payoff value during the calibration (optimization) and I got the attached results. Are those confidence intervals around the parameter estimates 95%CI?
Thanks
I set the sensitivity to payoff value during the calibration (optimization) and I got the attached results. Are those confidence intervals around the parameter estimates 95%CI?
Thanks
- Attachments
-
- Confidence intervals.docx
- (15.65 KiB) Downloaded 178 times
-
- Super Administrator
- Posts: 4620
- Joined: Wed Mar 05, 2003 3:10 am
Re: Help with model calibration
Do you mean the stats tool? It's all in the help system,leon wrote:Thanks Tomnfid.
Is it possible to obtain summary statistics from the current version of Vensim? See attached what I found online.
Thanks
http://www.vensim.com/documentation/ind ... s_tool.htm
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Units are important!
http://www.bbc.co.uk/news/magazine-27509559