payoff sensitivity

Use this forum to post Vensim related questions.
Post Reply
fabioR
Junior Member
Posts: 14
Joined: Thu Feb 22, 2018 1:09 pm
Vensim version: DSS

payoff sensitivity

Post by fabioR »

Hello!

I have a doubt regarding the MCMC option in Vensim DSS. I would like to evaluate the confidence bounds of all the parameters that I calibrated in my model. In the Help (https://www.vensim.com/documentation/in ... ptions.htm), it is written "If the payoff is a properly weighted sum of squares, it should be distributed Χ2 (Chi squared) with one degree of freedom".

I have two questions about it:

1 - Does "properly weighted sum of squares" means setting the Payoff with Transform=None and Distribution=Normal, with a proper weight?
2 - How can I check if my Payoff are "distributed Χ2 (Chi squared) with one degree of freedom"? Is there a statistical demonstration somewhere?

Thank you,

Fabio
fabioR
Junior Member
Posts: 14
Joined: Thu Feb 22, 2018 1:09 pm
Vensim version: DSS

Re: payoff sensitivity

Post by fabioR »

fabioR wrote:Hello!

I have a doubt regarding the MCMC option in Vensim DSS. I would like to evaluate the confidence bounds of all the parameters that I calibrated in my model. In the Help (https://www.vensim.com/documentation/in ... ptions.htm), it is written "If the payoff is a properly weighted sum of squares, it should be distributed Χ2 (Chi squared) with one degree of freedom".

I have two questions about it:

1 - Does "properly weighted sum of squares" means setting the Payoff with Transform=None and Distribution=Normal, with a proper weight?
2 - How can I check if my Payoff are "distributed Χ2 (Chi squared) with one degree of freedom"? Is there a statistical demonstration somewhere?

Thank you,

Fabio
Is there someone who can help me? Thanks
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: payoff sensitivity

Post by tomfid »

1 - correct, as long as you're not estimating the weights. If you're estimating the weights, you need to use Distribution=Gaussian (or there are a few others that work). The Gaussian is the same, except that the likelihood includes an extra term for the weight. You can use the log transform, as long as your errors and weight can be interpreted as fractions of the data rather than absolute values.

2 - having the payoff distributed X^2 is a consequence of the payoff being a sum of weighted squared error terms. If the weighting is right, the expectation for each point is ~N(0,1)^2, so the payoff is a sum of squares. So, you can test whether the contribution of each data series to the payoff is roughly equal to the number of points (i.e. mean contribution per point = 1). In most cases, you can't really do more than that, because you'll only have one dataset.
fabioR
Junior Member
Posts: 14
Joined: Thu Feb 22, 2018 1:09 pm
Vensim version: DSS

Re: payoff sensitivity

Post by fabioR »

tomfid wrote:1 - correct, as long as you're not estimating the weights. If you're estimating the weights, you need to use Distribution=Gaussian (or there are a few others that work). The Gaussian is the same, except that the likelihood includes an extra term for the weight. You can use the log transform, as long as your errors and weight can be interpreted as fractions of the data rather than absolute values.

2 - having the payoff distributed X^2 is a consequence of the payoff being a sum of weighted squared error terms. If the weighting is right, the expectation for each point is ~N(0,1)^2, so the payoff is a sum of squares. So, you can test whether the contribution of each data series to the payoff is roughly equal to the number of points (i.e. mean contribution per point = 1). In most cases, you can't really do more than that, because you'll only have one dataset.
Thank you Tom. Regarding the two answers:

1. The only problem I have is that I calibrated my model with 4 different time series simultaneously (viz. in an African village: (1) number of electrical connections of households (2) number of electrical connections of business activities, (3) monthly electricity demand of households and (4) monthly electricity demand of business activities), so the Payoff is composed by these 4 controbutions. These time series have different order of magnitude in the values, and therefore I do not know how to calculate the "weight" (in case of Normal distribution) or the ST.DEV (in case of Gaussian distribution). Indeed, I have no errors in the measurements.

2. When you say "the expectation for each point is ~N(0,1)" you mean that the weighted residulas are distribuited as ~N(0,1) point by point?

Thanks
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: payoff sensitivity

Post by tomfid »

1. If you use Normal, the weight should be 1/stdDev, and for Gaussian the weight is just stdDev (the latter makes the interpretation of stdDev more consistent across distributions - for Robust, it's a scale parameter representing the MAE, for example).

"I have no errors in the measurements." - this is more or less impossible. Data always contains noise and systematic problems. In physical science, you might have a pretty good idea about the noise in a measurement, but in socioeconomics, that's rare, so you have to estimate the noise. What you're really assuming when you calibrate is that the data was generated by an instance of the model, plus measurement error. Then you're using optimization to identify the set of parameters that maximizes the likelihood of getting that data. (What you still don't really know is how likely the model is, given the data, particularly because the real world process that generated the data is not the same as the model.)

There are several ways to estimate how big that error might be. The key thing that all of these accomplish is to scale the errors (and therefore the data) to account for the differences in order of magnitude.

a. If the data series is stationary, the standard deviation of the data is a reasonable upper limit. If it's not stationary, but can be detrended in some way, the SD around the trend may also be a reasonable guess.
b. Guess. You can just assume that the errors are some constant fraction, e.g., 5% of the data value. This works especially well if the data grows exponentially and therefore has dramatic differences in scale.
c. Iterate. From some initial guess, you can look at the payoff and see whether the mean error is ~1, and adjust accordingly.
d. Formally estimate. If you use Gaussian (or Robust or some of the others) you can make the error scale parameter part of the .voc.
In practice, the weights typically only have to be approximately correct to get reasonable results. You'll normally have other, bigger problems (systematic and dynamic errors, with the latter requiring Kalman filtering). I usually start with "b" above.

2. Correct.
fabioR
Junior Member
Posts: 14
Joined: Thu Feb 22, 2018 1:09 pm
Vensim version: DSS

Re: payoff sensitivity

Post by fabioR »

tomfid wrote:
2. Correct.

Thank you Tom, this will really help to find a proper way to set the weight in the payoff.

I have just an observation regarding the chi-square distribution. Since, by definition, the chi-square with k degrees of freedom is the distribution of a sum of the squares of k independent standard normal random variables, I still do not understand why in the Vensim Help it is written that "Payoff are distributed Χ2 (Chi squared) with one degree of freedom". Indeed, supposing that terms in the payoff are normal (viz. the residuals are ~N(0,1) point by point as you explained to me) and independent (which is not trivial!), why just one degree of freedom if the payoff is the sum of many (>1) weighted squared terms??

Thanks!
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: payoff sensitivity

Post by tomfid »

It's not the distribution of the sum of squares itself, but of the deviance function, i.e. the deviation in likelihood as you move away from the best parameters.
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: payoff sensitivity

Post by tomfid »

If you use MCMC, you avoid the question of determining the proper X^2 cutoff entirely. But then you should use the Gaussian option in order to have the likelihood properly scaled (Normal is the sum of squares, whereas Gaussian uses the sum/2, corresponding with the actual likelihood for the distribution).
fabioR
Junior Member
Posts: 14
Joined: Thu Feb 22, 2018 1:09 pm
Vensim version: DSS

Re: payoff sensitivity

Post by fabioR »

tomfid wrote:If you use MCMC, you avoid the question of determining the proper X^2 cutoff entirely. But then you should use the Gaussian option in order to have the likelihood properly scaled (Normal is the sum of squares, whereas Gaussian uses the sum/2, corresponding with the actual likelihood for the distribution).
Yes, I had already tried, but as soon as I run the MCMC with "Log likelihood" as Payoff type, Vensim shuts down ("Vensim family of products has stopped working"). If I specify just "Likelihood" as Payoff type, the .tab output file is meaningless since it reports just the name of the variables. Could be the issue related to the high number of variables to calibrate (viz. 110)?
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: payoff sensitivity

Post by tomfid »

I think this is a known bug that will be fixed in the next release. If you can share the model & supporting files (post here or email), that would help us to ensure it's fixed. Also, try turning off the payoff report.

Whether 110 is a lot depends on the structure of the problem, but I'd say that generally, it is. That's probably not the cause, though - it's not a lot of memory; it's a lot of computation.
fabioR
Junior Member
Posts: 14
Joined: Thu Feb 22, 2018 1:09 pm
Vensim version: DSS

Re: payoff sensitivity

Post by fabioR »

tomfid wrote:I think this is a known bug that will be fixed in the next release. If you can share the model & supporting files (post here or email), that would help us to ensure it's fixed. Also, try turning off the payoff report.

Whether 110 is a lot depends on the structure of the problem, but I'd say that generally, it is. That's probably not the cause, though - it's not a lot of memory; it's a lot of computation.
Thank you Tom, I prefer via email. To which address?

/Fabio
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: payoff sensitivity

Post by tomfid »

Vensim at vensim.com works.
fabioR
Junior Member
Posts: 14
Joined: Thu Feb 22, 2018 1:09 pm
Vensim version: DSS

Re: payoff sensitivity

Post by fabioR »

tomfid wrote:Vensim at vensim.com works.
Hi Tom, now it works! I just changed the MCINITMETHOD. If i use the hybrid method, it now works perfectly.

I have one question about how to interpret the results: I do, for examples, 20'000 MCMC simulations; then, in the output myrun_MCMC_sample.tab I have all the accepted points listed, viz. this .tab file contains Y columns (as the calibration parameters) and X=20'000 +1 rows containing the names of the Y parameters and below their values in all the accepted points.

In your opinion, could I say that a good proxy of the confidence bounds of each calibration parameter is the [MIN-MAX] of each column?

Thank you
WayneZhou2009
Senior Member
Posts: 105
Joined: Wed Oct 25, 2017 3:52 pm
Vensim version: PRO

Re: payoff sensitivity

Post by WayneZhou2009 »

Hi Tom,

I have been trying to understand why the payoff likelihood follows a Chi squared distribution with 1 degree of freedom. I found several posts including this one and some related to calibration, where you offered very useful insights and clarifications.

I just wanted to ask some further questions to make sure my understanding is correct.

(1) The payoff likelihood function, on it own, follows a Chi-squared distribution X2, with k degree of freedom, where k is the number of model parameters to be estimated. For example, theta may contain 3 parameters. Is this correct?

(2) When conducting confidence interval estimation, if likelihood ratio method is used, the confidence intervals of parameters are estimated as:

payoff log-likelihood (optimal parameter values, i.e. theta cap) - payoff log-likelihood (alternative parameter values, i.e. theta star) = the critical value of X2 distribution with 1 degree of freedom (for example, 3.84, for 95% confidence interval)

(3) Now, the above X2 distribution has only 1 degree of freedom because each time Vensim conducts calculation for only 1 parameter with the other parameters being fixed at their optimal values (the theta cap). Is this understanding correct?

Thank you so much for your time.
tomfid wrote: Thu May 03, 2018 2:15 pm It's not the distribution of the sum of squares itself, but of the deviance function, i.e. the deviation in likelihood as you move away from the best parameters.
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: payoff sensitivity

Post by tomfid »

The formatting is screwy, but this is the math: https://maths.lancs.ac.uk/grabows1/web/ ... 15.S1.html
Post Reply