I am looking for a scientific way in Vensim to show that the obtained simulated results of my simulation model fit the historical data (reference mode).
I have two lines in a graph. The first one visualizes the historical data of a paramater between 1995 and 2005. The second line shows the obtained simulation results for that same parameter for the same period. However, there is by visual inspection no perfect match.
Can you tell me how to measure the fit between these two lines. I would like to tell my audience for example that the simulated data fit the historical data for 85% with a 5% significance level.
Hendrik Stouten
Fit between the simulated data and the reference mode
-
hendrikstouten
- Junior Member
- Posts: 9
- Joined: Mon Jun 25, 2007 9:13 am
data fitting
you probably mean variable and not parameter, as a parameter is not supposed to vary as a reference mode is.
A second idea is that if this a variable, fitting more or less one variable to real data, is good to eventually find some mistakes, if the data obviously do not fit, but if they fit more or less, the only thing that it can proove is a very weak confidence in the model, especialy if the model is big, with many parameters and you have only one variable verification.
Fitting data is good to find mistakes but does not proove anything unless there are a lot of variables being verified, and in that case, all variable do not generally fit all together, which even does not proove that the model is wrong.
And even if the participant see that the variable is fitting the real data, it would be better for them to understand why.
I personaly think that data fitting is black box demonstration and should not be essentially used for proove of validity.
To resume, I think that data fitting does not proove anything, but can trigger further exploration and questionning of the model or eventually the data that can be wrong too.
Regards.
JJ
A second idea is that if this a variable, fitting more or less one variable to real data, is good to eventually find some mistakes, if the data obviously do not fit, but if they fit more or less, the only thing that it can proove is a very weak confidence in the model, especialy if the model is big, with many parameters and you have only one variable verification.
Fitting data is good to find mistakes but does not proove anything unless there are a lot of variables being verified, and in that case, all variable do not generally fit all together, which even does not proove that the model is wrong.
And even if the participant see that the variable is fitting the real data, it would be better for them to understand why.
I personaly think that data fitting is black box demonstration and should not be essentially used for proove of validity.
To resume, I think that data fitting does not proove anything, but can trigger further exploration and questionning of the model or eventually the data that can be wrong too.
Regards.
JJ
-
bob@vensim.com
- Senior Member
- Posts: 1107
- Joined: Wed Mar 12, 2003 2:46 pm
One standard approach for the comparison of simulation and data is the R-square statistic. This is available in the stats tool - but because the means may be different it has a range of -infinity to 1 not 0 to 1 so be careful with it.
To go beyond that you need to get much more specific about the error generation model underlying the data. A standard approach for confidence bounds on parameters estimated using calibration to data is to generate confidence intervals by normalizing the payoff weights so that they are the reciprocal of the standard deviation of the prediction error the model makes when calibrated.
To go beyond that you need to get much more specific about the error generation model underlying the data. A standard approach for confidence bounds on parameters estimated using calibration to data is to generate confidence intervals by normalizing the payoff weights so that they are the reciprocal of the standard deviation of the prediction error the model makes when calibrated.
-
hendrikstouten
- Junior Member
- Posts: 9
- Joined: Mon Jun 25, 2007 9:13 am