Ventana software support forum

leon wrote:Sorry, one more question.
I set the sensitivity to payoff value during the calibration (optimization) and I got the attached results. Are those confidence intervals around the parameter estimates 95%CI?
Thanks

Does this help?
http://www.vensim.com/documentation/ind ... tivity.htm

It describes what is in the output files.

The stat tools are showing means, median, std etc.; but I'm looking for model performance metrics (eg. R-squared). Please, see the file (summary statistics) I attached to you.
Thanks

It's in the Payoff Report: https://www.vensim.com/documentation/ex ... orting.htm

Sorry, but which file has the payoff report? I went through the different outputs generated from the model calibration and I did not see the payoff report file.
Thanks

You have to turn it on via the checkbox in the Advanced tab of the simulation control dialog. Then it'll be a .rep file.

I've have been trying to make sense of the payoff report over the last several days, but I'm not getting it.
Do I need to do some computation to get the R-square or did I check the wrong box? Please help
Thanks

What version are you running? There's a bug in 7.0 that causes this to happen, I think.

I'm using the version 7. Is there any other ways around?
Thanks

It's fixed in 7.1, and also wasn't in 6.4 I think.

I thought I was using the last version. The 7 version has been downloaded recently. So I need to download the 7.1?

I've upgraded to the new version.
Could you please help me with the interpretation?
I don't know if I need to do some computation to get the R-Square for each subscript.
What do RMSE, Um, Us, Uc, MAE, MAPE, MAEoM mean?

The reported items are documented here:
http://vensim.com/documentation/extende ... orting.htm

The r^2 is reported for every subscript that's included in the payoff.

Thanks so much!
Is it approach to average each statistics across all subscripts to get a summary statistics?

Depends on the stat. Generally, I'd say no, but reporting a range of MAPE values (for example) seems like a pretty reasonable thing to do.

Thank you. Also thanks for the Sterman's paper, it is very helpful.

Are there any cutoffs for the interpretation of the metrics? What can be considered high for MAPE?
I have a lot of undefined (NA) for the variable "events/month"

Dear,
I applied my simulated driver feedback intervention to the model. I simply increased the fraction of events known to 100% for each driver and recorded the simulated intervention data. I compared the simulated data to the actual feedback intervention. I used a statistical model for the comparison and adjusted by the fact that the 2 groups of drivers have different baseline event rates. Please, see attached the mean profiles over time for both the simulated and actual intervention. I was asked to offset this in the model. Is this possible? I don't know if I clearly posed the question

It seems like if I offset this in the model, the 2 mean profiles will overlap. I guess

Not sure what you mean by "offset" - delay?

I don't know if this is feasible in Vensim. I don't know if I used the right term for "offset"
Basically let's say I have 2 drivers A and B with initial event rates of 0.1/mile and 0.5/mile respectively. Driver A received the actual treatment starting at Month 2 and I simulated the treatment on Driver B using my model (I simply increased the fraction of events known to 100% starting at Month 2 and recorded the simulated intervention data). Then my research question is: do the actual treatment and simulated treatment agree?
Statistically for the comparison, I need to account for the fact that both drivers had different initial event rates to begin with. So, I will throw my initial event rate into my statistical model.
I was asked why I adjusted for the initial event rate. Isn't that initial event rate in the dynamic model? So I'm confused
Is there anyway to account for this in Vensim?
I'm still not sure if you got the issue

Something is definitely amiss if the simulated and real means differ by this much.

You are correct. But you would agree with me that the trajectory of the actual treatment response on a driver with an initial event rate of 0.5/mile would not be the same as the trajectory of the actual treatment response on a driver with an initial event rate of 0.1/mile. That's why in statistics, the initial event rates should be accounted for if I want to compare the treatment response for these 2 drivers

Another way to think about this is:
The actual treatment was applied to a group of "good" drivers. They had a very low initial event rate.
My simulation was applied to a group of "bad" drivers. They had a high initial event event rate. I would not expect the mean profiles of these 2 groups of drivers to overlap because of the differential initial event rates. But I would expect them to have the same shape. That's why I adjusted for the initial event rates using statistical approach.

What do you think Tom?

Understood. I think the short answer is "yes" - should be possible to account for this in Vensim. I'm traveling tomorrow but will answer as I can.