Help with model calibration

Use this forum to post Vensim related questions.
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

leon wrote:Sorry, one more question.
I set the sensitivity to payoff value during the calibration (optimization) and I got the attached results. Are those confidence intervals around the parameter estimates 95%CI?
Thanks
Does this help?
http://www.vensim.com/documentation/ind ... tivity.htm

It describes what is in the output files.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

The stat tools are showing means, median, std etc.; but I'm looking for model performance metrics (eg. R-squared). Please, see the file (summary statistics) I attached to you.
Thanks
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Sorry, but which file has the payoff report? I went through the different outputs generated from the model calibration and I did not see the payoff report file.
Thanks
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

You have to turn it on via the checkbox in the Advanced tab of the simulation control dialog. Then it'll be a .rep file.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I've have been trying to make sense of the payoff report over the last several days, but I'm not getting it.
Do I need to do some computation to get the R-square or did I check the wrong box? Please help
Thanks
Attachments
payoff report.zip
(26.14 KiB) Downloaded 223 times
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

What version are you running? There's a bug in 7.0 that causes this to happen, I think.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I'm using the version 7. Is there any other ways around?
Thanks
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

It's fixed in 7.1, and also wasn't in 6.4 I think.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I thought I was using the last version. The 7 version has been downloaded recently. So I need to download the 7.1?
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I've upgraded to the new version.
Could you please help me with the interpretation?
I don't know if I need to do some computation to get the R-Square for each subscript.
What do RMSE, Um, Us, Uc, MAE, MAPE, MAEoM mean?
Attachments
payoff report.zip
(6.3 KiB) Downloaded 224 times
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

The reported items are documented here:
http://vensim.com/documentation/extende ... orting.htm
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

The r^2 is reported for every subscript that's included in the payoff.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks so much!
Is it approach to average each statistics across all subscripts to get a summary statistics?
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Depends on the stat. Generally, I'd say no, but reporting a range of MAPE values (for example) seems like a pretty reasonable thing to do.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thank you. Also thanks for the Sterman's paper, it is very helpful.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Are there any cutoffs for the interpretation of the metrics? What can be considered high for MAPE?
I have a lot of undefined (NA) for the variable "events/month"
Attachments
performance metric2.xlsx
(17.56 KiB) Downloaded 227 times
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Dear,
I applied my simulated driver feedback intervention to the model. I simply increased the fraction of events known to 100% for each driver and recorded the simulated intervention data. I compared the simulated data to the actual feedback intervention. I used a statistical model for the comparison and adjusted by the fact that the 2 groups of drivers have different baseline event rates. Please, see attached the mean profiles over time for both the simulated and actual intervention. I was asked to offset this in the model. Is this possible? I don't know if I clearly posed the question
Attachments
Doc1.docx
(115.45 KiB) Downloaded 171 times
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

It seems like if I offset this in the model, the 2 mean profiles will overlap. I guess
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Not sure what you mean by "offset" - delay?
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I don't know if this is feasible in Vensim. I don't know if I used the right term for "offset"
Basically let's say I have 2 drivers A and B with initial event rates of 0.1/mile and 0.5/mile respectively. Driver A received the actual treatment starting at Month 2 and I simulated the treatment on Driver B using my model (I simply increased the fraction of events known to 100% starting at Month 2 and recorded the simulated intervention data). Then my research question is: do the actual treatment and simulated treatment agree?
Statistically for the comparison, I need to account for the fact that both drivers had different initial event rates to begin with. So, I will throw my initial event rate into my statistical model.
I was asked why I adjusted for the initial event rate. Isn't that initial event rate in the dynamic model? So I'm confused
Is there anyway to account for this in Vensim?
I'm still not sure if you got the issue
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Something is definitely amiss if the simulated and real means differ by this much.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

You are correct. But you would agree with me that the trajectory of the actual treatment response on a driver with an initial event rate of 0.5/mile would not be the same as the trajectory of the actual treatment response on a driver with an initial event rate of 0.1/mile. That's why in statistics, the initial event rates should be accounted for if I want to compare the treatment response for these 2 drivers
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Another way to think about this is:
The actual treatment was applied to a group of "good" drivers. They had a very low initial event rate.
My simulation was applied to a group of "bad" drivers. They had a high initial event event rate. I would not expect the mean profiles of these 2 groups of drivers to overlap because of the differential initial event rates. But I would expect them to have the same shape. That's why I adjusted for the initial event rates using statistical approach.

What do you think Tom?
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Understood. I think the short answer is "yes" - should be possible to account for this in Vensim. I'm traveling tomorrow but will answer as I can.
Post Reply