payoff sensitivity
payoff sensitivity
Hello!
I have a doubt regarding the MCMC option in Vensim DSS. I would like to evaluate the confidence bounds of all the parameters that I calibrated in my model. In the Help (https://www.vensim.com/documentation/in ... ptions.htm), it is written "If the payoff is a properly weighted sum of squares, it should be distributed Χ2 (Chi squared) with one degree of freedom".
I have two questions about it:
1 - Does "properly weighted sum of squares" means setting the Payoff with Transform=None and Distribution=Normal, with a proper weight?
2 - How can I check if my Payoff are "distributed Χ2 (Chi squared) with one degree of freedom"? Is there a statistical demonstration somewhere?
Thank you,
Fabio
I have a doubt regarding the MCMC option in Vensim DSS. I would like to evaluate the confidence bounds of all the parameters that I calibrated in my model. In the Help (https://www.vensim.com/documentation/in ... ptions.htm), it is written "If the payoff is a properly weighted sum of squares, it should be distributed Χ2 (Chi squared) with one degree of freedom".
I have two questions about it:
1 - Does "properly weighted sum of squares" means setting the Payoff with Transform=None and Distribution=Normal, with a proper weight?
2 - How can I check if my Payoff are "distributed Χ2 (Chi squared) with one degree of freedom"? Is there a statistical demonstration somewhere?
Thank you,
Fabio
Re: payoff sensitivity
Is there someone who can help me? ThanksfabioR wrote:Hello!
I have a doubt regarding the MCMC option in Vensim DSS. I would like to evaluate the confidence bounds of all the parameters that I calibrated in my model. In the Help (https://www.vensim.com/documentation/in ... ptions.htm), it is written "If the payoff is a properly weighted sum of squares, it should be distributed Χ2 (Chi squared) with one degree of freedom".
I have two questions about it:
1 - Does "properly weighted sum of squares" means setting the Payoff with Transform=None and Distribution=Normal, with a proper weight?
2 - How can I check if my Payoff are "distributed Χ2 (Chi squared) with one degree of freedom"? Is there a statistical demonstration somewhere?
Thank you,
Fabio
Re: payoff sensitivity
1 - correct, as long as you're not estimating the weights. If you're estimating the weights, you need to use Distribution=Gaussian (or there are a few others that work). The Gaussian is the same, except that the likelihood includes an extra term for the weight. You can use the log transform, as long as your errors and weight can be interpreted as fractions of the data rather than absolute values.
2 - having the payoff distributed X^2 is a consequence of the payoff being a sum of weighted squared error terms. If the weighting is right, the expectation for each point is ~N(0,1)^2, so the payoff is a sum of squares. So, you can test whether the contribution of each data series to the payoff is roughly equal to the number of points (i.e. mean contribution per point = 1). In most cases, you can't really do more than that, because you'll only have one dataset.
2 - having the payoff distributed X^2 is a consequence of the payoff being a sum of weighted squared error terms. If the weighting is right, the expectation for each point is ~N(0,1)^2, so the payoff is a sum of squares. So, you can test whether the contribution of each data series to the payoff is roughly equal to the number of points (i.e. mean contribution per point = 1). In most cases, you can't really do more than that, because you'll only have one dataset.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: payoff sensitivity
Thank you Tom. Regarding the two answers:tomfid wrote:1 - correct, as long as you're not estimating the weights. If you're estimating the weights, you need to use Distribution=Gaussian (or there are a few others that work). The Gaussian is the same, except that the likelihood includes an extra term for the weight. You can use the log transform, as long as your errors and weight can be interpreted as fractions of the data rather than absolute values.
2 - having the payoff distributed X^2 is a consequence of the payoff being a sum of weighted squared error terms. If the weighting is right, the expectation for each point is ~N(0,1)^2, so the payoff is a sum of squares. So, you can test whether the contribution of each data series to the payoff is roughly equal to the number of points (i.e. mean contribution per point = 1). In most cases, you can't really do more than that, because you'll only have one dataset.
1. The only problem I have is that I calibrated my model with 4 different time series simultaneously (viz. in an African village: (1) number of electrical connections of households (2) number of electrical connections of business activities, (3) monthly electricity demand of households and (4) monthly electricity demand of business activities), so the Payoff is composed by these 4 controbutions. These time series have different order of magnitude in the values, and therefore I do not know how to calculate the "weight" (in case of Normal distribution) or the ST.DEV (in case of Gaussian distribution). Indeed, I have no errors in the measurements.
2. When you say "the expectation for each point is ~N(0,1)" you mean that the weighted residulas are distribuited as ~N(0,1) point by point?
Thanks
Re: payoff sensitivity
1. If you use Normal, the weight should be 1/stdDev, and for Gaussian the weight is just stdDev (the latter makes the interpretation of stdDev more consistent across distributions - for Robust, it's a scale parameter representing the MAE, for example).
"I have no errors in the measurements." - this is more or less impossible. Data always contains noise and systematic problems. In physical science, you might have a pretty good idea about the noise in a measurement, but in socioeconomics, that's rare, so you have to estimate the noise. What you're really assuming when you calibrate is that the data was generated by an instance of the model, plus measurement error. Then you're using optimization to identify the set of parameters that maximizes the likelihood of getting that data. (What you still don't really know is how likely the model is, given the data, particularly because the real world process that generated the data is not the same as the model.)
There are several ways to estimate how big that error might be. The key thing that all of these accomplish is to scale the errors (and therefore the data) to account for the differences in order of magnitude.
a. If the data series is stationary, the standard deviation of the data is a reasonable upper limit. If it's not stationary, but can be detrended in some way, the SD around the trend may also be a reasonable guess.
b. Guess. You can just assume that the errors are some constant fraction, e.g., 5% of the data value. This works especially well if the data grows exponentially and therefore has dramatic differences in scale.
c. Iterate. From some initial guess, you can look at the payoff and see whether the mean error is ~1, and adjust accordingly.
d. Formally estimate. If you use Gaussian (or Robust or some of the others) you can make the error scale parameter part of the .voc.
In practice, the weights typically only have to be approximately correct to get reasonable results. You'll normally have other, bigger problems (systematic and dynamic errors, with the latter requiring Kalman filtering). I usually start with "b" above.
2. Correct.
"I have no errors in the measurements." - this is more or less impossible. Data always contains noise and systematic problems. In physical science, you might have a pretty good idea about the noise in a measurement, but in socioeconomics, that's rare, so you have to estimate the noise. What you're really assuming when you calibrate is that the data was generated by an instance of the model, plus measurement error. Then you're using optimization to identify the set of parameters that maximizes the likelihood of getting that data. (What you still don't really know is how likely the model is, given the data, particularly because the real world process that generated the data is not the same as the model.)
There are several ways to estimate how big that error might be. The key thing that all of these accomplish is to scale the errors (and therefore the data) to account for the differences in order of magnitude.
a. If the data series is stationary, the standard deviation of the data is a reasonable upper limit. If it's not stationary, but can be detrended in some way, the SD around the trend may also be a reasonable guess.
b. Guess. You can just assume that the errors are some constant fraction, e.g., 5% of the data value. This works especially well if the data grows exponentially and therefore has dramatic differences in scale.
c. Iterate. From some initial guess, you can look at the payoff and see whether the mean error is ~1, and adjust accordingly.
d. Formally estimate. If you use Gaussian (or Robust or some of the others) you can make the error scale parameter part of the .voc.
In practice, the weights typically only have to be approximately correct to get reasonable results. You'll normally have other, bigger problems (systematic and dynamic errors, with the latter requiring Kalman filtering). I usually start with "b" above.
2. Correct.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: payoff sensitivity
tomfid wrote:
2. Correct.
Thank you Tom, this will really help to find a proper way to set the weight in the payoff.
I have just an observation regarding the chi-square distribution. Since, by definition, the chi-square with k degrees of freedom is the distribution of a sum of the squares of k independent standard normal random variables, I still do not understand why in the Vensim Help it is written that "Payoff are distributed Χ2 (Chi squared) with one degree of freedom". Indeed, supposing that terms in the payoff are normal (viz. the residuals are ~N(0,1) point by point as you explained to me) and independent (which is not trivial!), why just one degree of freedom if the payoff is the sum of many (>1) weighted squared terms??
Thanks!
Re: payoff sensitivity
It's not the distribution of the sum of squares itself, but of the deviance function, i.e. the deviation in likelihood as you move away from the best parameters.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: payoff sensitivity
If you use MCMC, you avoid the question of determining the proper X^2 cutoff entirely. But then you should use the Gaussian option in order to have the likelihood properly scaled (Normal is the sum of squares, whereas Gaussian uses the sum/2, corresponding with the actual likelihood for the distribution).
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: payoff sensitivity
Yes, I had already tried, but as soon as I run the MCMC with "Log likelihood" as Payoff type, Vensim shuts down ("Vensim family of products has stopped working"). If I specify just "Likelihood" as Payoff type, the .tab output file is meaningless since it reports just the name of the variables. Could be the issue related to the high number of variables to calibrate (viz. 110)?tomfid wrote:If you use MCMC, you avoid the question of determining the proper X^2 cutoff entirely. But then you should use the Gaussian option in order to have the likelihood properly scaled (Normal is the sum of squares, whereas Gaussian uses the sum/2, corresponding with the actual likelihood for the distribution).
Re: payoff sensitivity
I think this is a known bug that will be fixed in the next release. If you can share the model & supporting files (post here or email), that would help us to ensure it's fixed. Also, try turning off the payoff report.
Whether 110 is a lot depends on the structure of the problem, but I'd say that generally, it is. That's probably not the cause, though - it's not a lot of memory; it's a lot of computation.
Whether 110 is a lot depends on the structure of the problem, but I'd say that generally, it is. That's probably not the cause, though - it's not a lot of memory; it's a lot of computation.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: payoff sensitivity
Thank you Tom, I prefer via email. To which address?tomfid wrote:I think this is a known bug that will be fixed in the next release. If you can share the model & supporting files (post here or email), that would help us to ensure it's fixed. Also, try turning off the payoff report.
Whether 110 is a lot depends on the structure of the problem, but I'd say that generally, it is. That's probably not the cause, though - it's not a lot of memory; it's a lot of computation.
/Fabio
Re: payoff sensitivity
Vensim at vensim.com works.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: payoff sensitivity
Hi Tom, now it works! I just changed the MCINITMETHOD. If i use the hybrid method, it now works perfectly.tomfid wrote:Vensim at vensim.com works.
I have one question about how to interpret the results: I do, for examples, 20'000 MCMC simulations; then, in the output myrun_MCMC_sample.tab I have all the accepted points listed, viz. this .tab file contains Y columns (as the calibration parameters) and X=20'000 +1 rows containing the names of the Y parameters and below their values in all the accepted points.
In your opinion, could I say that a good proxy of the confidence bounds of each calibration parameter is the [MIN-MAX] of each column?
Thank you
-
- Senior Member
- Posts: 105
- Joined: Wed Oct 25, 2017 3:52 pm
- Vensim version: PRO
Re: payoff sensitivity
Hi Tom,
I have been trying to understand why the payoff likelihood follows a Chi squared distribution with 1 degree of freedom. I found several posts including this one and some related to calibration, where you offered very useful insights and clarifications.
I just wanted to ask some further questions to make sure my understanding is correct.
(1) The payoff likelihood function, on it own, follows a Chi-squared distribution X2, with k degree of freedom, where k is the number of model parameters to be estimated. For example, theta may contain 3 parameters. Is this correct?
(2) When conducting confidence interval estimation, if likelihood ratio method is used, the confidence intervals of parameters are estimated as:
payoff log-likelihood (optimal parameter values, i.e. theta cap) - payoff log-likelihood (alternative parameter values, i.e. theta star) = the critical value of X2 distribution with 1 degree of freedom (for example, 3.84, for 95% confidence interval)
(3) Now, the above X2 distribution has only 1 degree of freedom because each time Vensim conducts calculation for only 1 parameter with the other parameters being fixed at their optimal values (the theta cap). Is this understanding correct?
Thank you so much for your time.
I have been trying to understand why the payoff likelihood follows a Chi squared distribution with 1 degree of freedom. I found several posts including this one and some related to calibration, where you offered very useful insights and clarifications.
I just wanted to ask some further questions to make sure my understanding is correct.
(1) The payoff likelihood function, on it own, follows a Chi-squared distribution X2, with k degree of freedom, where k is the number of model parameters to be estimated. For example, theta may contain 3 parameters. Is this correct?
(2) When conducting confidence interval estimation, if likelihood ratio method is used, the confidence intervals of parameters are estimated as:
payoff log-likelihood (optimal parameter values, i.e. theta cap) - payoff log-likelihood (alternative parameter values, i.e. theta star) = the critical value of X2 distribution with 1 degree of freedom (for example, 3.84, for 95% confidence interval)
(3) Now, the above X2 distribution has only 1 degree of freedom because each time Vensim conducts calculation for only 1 parameter with the other parameters being fixed at their optimal values (the theta cap). Is this understanding correct?
Thank you so much for your time.
Re: payoff sensitivity
The formatting is screwy, but this is the math: https://maths.lancs.ac.uk/grabows1/web/ ... 15.S1.html
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/