Ventana software support forum

I just found in Dogan, Bootstrapping for Confidence Interval Estimation, in: SDR 23 (2007) 4 that the way Vensim uses the likelihood ratio method to come up with confidence intervals for parameter estimates in calibration is not compatible with standard econometrics texts (Dogan references Greene, 2003). Would someone have further references or a more detailled description in what ways there are differences?

Kind regards,

Guido

I was just looking at that paper the other day, and I'm not convinced that it's right. I have done a variety of estimation tests on simple problems (e.g. finding the mean and confidence bounds on a normal distribution) and everything was fine. I'll take another look at the paper and report back.

I just checked some old correspondence, and the issue is not what I was thinking of. Gokhan is correct that there is a difference; the origin is reporting conventions rather than a technical error. Here's an attempt at an explanation.

Suppose you have a model with parameters a that you're fitting to data. For simplicity, assume that i has dimension 2 ( i : i1, i2 ). You optimize to find a minimizing your calibration payoff. Now you want to know what the payoff surface looks like around the optimum. One way to summarize that is to calculate confidence bounds on a[i1] & a[i2].

Vensim does this by slicing the payoff surface along the parameter axes. So, to calculate bounds on a[i1], fix a[i2] at the optimum and vary a[i1] until the payoff changes to a cutoff value (from the Chi-squared distribution). If the parameters have independent effects on the payoff, in which case the surface looks like a round bowl or an ellipse with axes parallel to the parameter axes, this works fine. However, in the case where a[i1] & a[i2] interact, such that the payoff surface is an off-axis ellipse (e.g., a long, narrow bowl oriented at 45 degrees to the axes), it doesn't work so well. By slicing along the parameter axes, you miss the long (major) axis of the ellipse, along which a[i1] & a[i2] can covary a lot without changing the payoff much.

There are two ways around this. One is to estimate the local curvature around the optimum and find eigenvalues to estimate the orientation of the ellipse, then search along the principal axes of the ellipse rather than along the parameter axes. Another is to find the optimal a[i2] at every test point for a[i1], and vice versa. Either way, you can explore the long dimension of off-axis troughs in the payoff surface.

The first method, used in Vensim, is giving you only conditional confidence bounds, which may be narrower than the unconditional bounds, sometimes by a lot.

The second method is evidently what you typically get from regression packages. If you're doing estimation via linear algebra, it's cheap to compute, so this makes sense. However, it would potentially be computationally prohibitive for more general nonlinear models that are slow to execute. Assuming that the eigenvalue approach would be defeated by nonlinearity or discontinuity of the payoff surface, you'd have to do p*v*n(p-1) model runs, where p = number of parameters, v = number of executions to slice along a given parameter, and n(p-1) is the number of executions required to suboptimize all but one parameter.

A forthcoming beta feature should provide a better way to handle this.

Tom

Tom,

thank you very much for the research and your explanation.

Kind regards,

Guido

Thank you for your explanation,
Would you tell me where I can find the confidence interval in the output of model?
I am a beginner and I can't find a way to calculate it.
Does Vensim provide confidence interval for parameters that are being calibrated or I should follow the Dogan paper?
Thanks for your time

I'd start with https://www.vensim.com/documentation/usr18.htm

A lot has changed since Dogan's paper, including the introduction of MCMC, which is the preferred way to do confidence bounds in many cases.
https://www.vensim.com/documentation/mcmc_sa.htm

Thank you so much

I calculated the 95% Confidence Intervals according to (https://www.vensim.com/documentation/in ... ?usr18.htm).
But when I look at the results, there is a sentence that makes me worry. It says that: * Means a bound was reached, i.e. payoff not at criterion

And for most of the variables, the confidence interval is the same as the intervals that I have set in the optimization. Like these ones:
0.1 *<= Initial Infected Females[P15] = 20 <= 20 *
0.1 *<= Initial Infected Females[P18] = 0.174616 <= 9.83206

Now, there are three questions:
1-Does it mean that I should increase the intervals for variables and optimize the model and then calculate the 95% CI again?

2- or does it mean that these variables that their CI is the same as their intervals in optimization are not sensitive and the changes in their value is not affecting the model significantly?

3-Or it doesn't matter
I appreciate your feedback as always.
Thank you,

The * means that the bounds on the variable in your control file are tighter than the CI.

There could be three reasons for this:
- the parameter only matters in conjunction with others (the principal axis problem raised in the Dogan paper)
- the data don't supply much information about the parameter
- the payoff isn't weighted properly (specifically, if the weights are too low, it'll take large variations to change the payoff by the ChiSq criteria)

If you have a long vector of "initial infected" parameters, I think it's likely that the data don't say much about them individually.

One option would be to change your parameterization, so that you have a "total initial infected" combined with subscripted variables that determine the share of the total allocated to each gender, cohort etc.

Thank you so much. it was very helpful