Help with model calibration

Use this forum to post Vensim related questions.
Post Reply
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Help with model calibration

Post by leon »

Hi there. First of all, thanks so much for the help you have been providing me.
I'm trying to calibrate a model I'm developing as part of my dissertation research. I have a number of questions and I would like your help to go through the calibration process. I bought Vensim Pro. Is there anyway I could privately send you the model before I start asking my questions? Since this is related to my PhD, I don't want to post the model on the public Forum.
Thanks for all your help
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

Can you update your profile with the serial number for your Pro licence?

And feel free to post your questions on the forum, that would be the best thing to start with.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Administrator wrote:Can you update your profile with the serial number for your Pro licence?

And feel free to post your questions on the forum, that would be the best thing to start with.
I clicked on "Edit profile" and it says "Module not accessible". Is there any other way to update the profile? Or I can just post the serial number here
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

Ahh, sorry about that. If you email us with the info, we can update the profile for you.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Administrator wrote:Ahh, sorry about that. If you email us with the info, we can update the profile for you.
I just emailed through tony@ventanasystems.co.uk
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

leon wrote:
Administrator wrote:Can you update your profile with the serial number for your Pro licence?

And feel free to post your questions on the forum, that would be the best thing to start with.
I clicked on "Edit profile" and it says "Module not accessible". Is there any other way to update the profile? Or I can just post the serial number here

I'm trying to model the rate of driving errors over time among new adolescent drivers. The rate is defined as the number of errors per mile per week.
I have data on amount of weekly driving (in mile), errors per week, etc. over a period of 20 weeks. The rate is plotted as a function of week
My model has 2 variables affecting the rate: cumulative miles driven and report of recent errors.
If I were to express the rate using statistical modeling, I would write this: rate (errors/mile)=exp(bo+b1*cumulative_miles_driven+b2*report_recent_errors)
=initial_rate*exp(b1*cumulative_miles_driven+b2*report_recent_errors)
1) Is that equation as formulated consistent with the principle of a dynamic model?
2) As far as calibration goes: Should the calibration done using the aggregated data ie the sample average rate of errors over time?
3) the model has 3 stocks and the initial conditions ie the initial values for the stock variables are not known. Does SD offer any solutions regarding this issue?
4) What do you think of partial calibration ie calibration of the model piece by piece since we don't measure everything?

Thanks a lot for your help
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

leon wrote:1) Is that equation as formulated consistent with the principle of a dynamic model?
I'd assume so. Where did you get the equation?
leon wrote:2) As far as calibration goes: Should the calibration done using the aggregated data ie the sample average rate of errors over time?
I'd calibrate with the best data possible. If I need to calibrate, I always use real data if I can get it. So personally, I'd use the actual rate of errors if you have it.
leon wrote:3) the model has 3 stocks and the initial conditions ie the initial values for the stock variables are not known. Does SD offer any solutions regarding this issue?
Calibration is usually used to estimate unknown parameters. These initial values, those are the kinds of things you can estimate using calibration. Have you looked at the electric sample model that ships with Vensim?
leon wrote:4) What do you think of partial calibration ie calibration of the model piece by piece since we don't measure everything?
Partial calibration is fine as long as those parameters you are trying to estimate are not influenced by other areas of the model. This is a silly example, but does demonstrate what I'm referring to. If you are estimating the
number of male and female drivers, but only have error rates for drivers not separated by gender, there is no point calibrating the number of male drivers to this data.

Take a look at the electric model that ships with Vensim.
http://www.vensim.com/documentation/ind ... ?21295.htm
This sample takes you through importing data and calibration. From what you have described, it's exactly what you need to do.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Administrator wrote:
leon wrote:1) Is that equation as formulated consistent with the principle of a dynamic model?
I'd assume so. Where did you get the equation?
leon wrote:2) As far as calibration goes: Should the calibration done using the aggregated data ie the sample average rate of errors over time?
I'd calibrate with the best data possible. If I need to calibrate, I always use real data if I can get it. So personally, I'd use the actual rate of errors if you have it.
leon wrote:3) the model has 3 stocks and the initial conditions ie the initial values for the stock variables are not known. Does SD offer any solutions regarding this issue?
Calibration is usually used to estimate unknown parameters. These initial values, those are the kinds of things you can estimate using calibration. Have you looked at the electric sample model that ships with Vensim?
leon wrote:4) What do you think of partial calibration ie calibration of the model piece by piece since we don't measure everything?
Partial calibration is fine as long as those parameters you are trying to estimate are not influenced by other areas of the model. This is a silly example, but does demonstrate what I'm referring to. If you are estimating the
number of male and female drivers, but only have error rates for drivers not separated by gender, there is no point calibrating the number of male drivers to this data.

Take a look at the electric model that ships with Vensim.
http://www.vensim.com/documentation/ind ... ?21295.htm
This sample takes you through importing data and calibration. From what you have described, it's exactly what you need to do.

Thanks so much!

Let's say I have 10 drivers and the data are displayed like this
Driver ID Week errors miles rate
Driver1 1 5 150 5/150
Driver1 2 8 120 8/120
Driver1 3 9 115 9/115
. . . . .
. . . . .
. . . . .
Driver1 20 2 170 2/170

Driver2 1 1 100 1/100
Driver2 2 0 90 0/90
Driver2 3 3 95 3/95
. . . . .
. . . . .
. . . . .
Driver2 20 1 120 1/120

etc........

If I want to predict error/mile, the statistical model would be like this:
errors/mile=exp(b0+b1*variable1+b2*variable). Of course, the statistical analysis will account for repeated measures

My second question is this: since I have 10 individual data over time, what data would be used for the calibration? Their average? Median?
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

leon wrote:My second question is this: since I have 10 individual data over time, what data would be used for the calibration? Their average? Median?
I'd need to see more of the model to comment. If your model calculates the average errors, then calibrate to the average of the data.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Administrator wrote:
leon wrote:My second question is this: since I have 10 individual data over time, what data would be used for the calibration? Their average? Median?
I'd need to see more of the model to comment. If your model calculates the average errors, then calibrate to the average of the data.
I've emailed you the model. The model is built to simulate individual error rate over time.
Thanks for your help!
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

In general, there are two distinct situations:

- If you have a population of individuals, but you don't have input data for the particular causes of errors for each driver, you probably can't expect to replicate the time series error rate for any particular driver. Therefore comparing individual driver histories to data would be fruitless. In that case, you'd want to use Generalized Method of Means. See the chapter in https://www.amazon.com/Analytical-Metho ... 0262029499. The simplest version of that would be to compare the aggregate statistics (mean error rate or similar) for model and data.

- If your model does replicate the particulars of each driver's history, then you can compare those directly. If this works, it gives you a lot more information.

I guess there's a third, intermediate case, in which you might have enough information to generate individual driver's error statistics, but not the particular time series, in which case you'd compare the drivers' mean error rates to the data for the same.
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

One more thing to think about:

The error distribution is probably not Normal (Gaussian).
- If drivers make lots of errors, it's probably approximately Normal, in which case you could ignore this.
- If they make only a few in total, it's probably Poisson
- If you're looking at errors over time, it's probably Poisson or Bernouilli, depending on whether they can make more than one mistake per time step
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

tomfid wrote:One more thing to think about:

The error distribution is probably not Normal (Gaussian).
- If drivers make lots of errors, it's probably approximately Normal, in which case you could ignore this.
- If they make only a few in total, it's probably Poisson
- If you're looking at errors over time, it's probably Poisson or Bernouilli, depending on whether they can make more than one mistake per time step
You are correct, I used negative binomial distribution to statistically analyze the data.

Let's assume that the drivers have the same causes of errors. For sure, there would be individual (external) links to errors that the model would not capture. How to go about calibration in this situation? Individual statistics or group statistics?
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

tomfid wrote:In general, there are two distinct situations:

- If you have a population of individuals, but you don't have input data for the particular causes of errors for each driver, you probably can't expect to replicate the time series error rate for any particular driver. Therefore comparing individual driver histories to data would be fruitless. In that case, you'd want to use Generalized Method of Means. See the chapter in https://www.amazon.com/Analytical-Metho ... 0262029499. The simplest version of that would be to compare the aggregate statistics (mean error rate or similar) for model and data.

- If your model does replicate the particulars of each driver's history, then you can compare those directly. If this works, it gives you a lot more information.

I guess there's a third, intermediate case, in which you might have enough information to generate individual driver's error statistics, but not the particular time series, in which case you'd compare the drivers' mean error rates to the data for the same.
Also the model is built to replicate individual time series data depending on the initial conditions (driving demand and initial error rate). So what data should I calibrate the model on? Average statistics?
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

You might start with averages just to get a quick answer, but I'd lean toward individual stats.

For example, if the model reduces to something like the following:

error frequency = f ( state of driver ) ~ errors/sec

Then the probability of making an error in the next time step is P error = 1-exp(-error frequency*TIME STEP), or approximately error frequency*TIME STEP if the rate is low relative to the step.

If only one error can happen per time step, either due to the coding method or in practice because the error rate is low, this is essentially a series of Bernouilli trials. Presumably your data would look like
0 0 0 1 0 0 0 0 1 ...
(i.e. lots of zeroes, with a 1 wherever a mistake happens).

So your likelihood for a given driver [d] at a given time would be:
likelihood[d] = data error[d] * P err[d] + (1-data error[d])*P err[d]
Assuming P err is always >0, you'd take the log of this, sum it over drivers, and integrate it over time to get the total log likelihood, and maximize that.

The aggregate version would be to sum each driver's errors, resulting in a Poisson distribution, but in that case you're essentially ignoring any variation in f( state of driver ) over time.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I agree with you, individual stats would make more sense.
The binomial distribution seems appropriate, which I used to express error rate in the model.
error rate (error/mile) is a function of 2 variables (cumulative miles driven and event reporting).
I expressed error rate like this: error rate=exp(b0+b1*variable1+b2*variable2)=exp(b0)*exp(b1*variable1+b2*variable2)=initial_rate*exp(b1*variable1+b2*variable2)

I don't think I know how to execute in Vensim what you explained. Also how should the data layed out to import into Vensim?
Is there any documentation to guide me through this?

Thanks so much
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

leon wrote:Is there any documentation to guide me through this?
There is a whole chapter in the help system on data,
http://www.vensim.com/documentation/ind ... ?usr16.htm
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks so much!

I don't know how best I can use your help here. I have some background in quantitative methods, but I'm a very beginner in simulation. Your help would be very much appreciated!
You probably have got some idea about what I'm trying to achieve with my model. I've divided my sample of drivers into 2 groups. I want to use one group to get the model parameters. Then I will validate the model against the second group of drivers. Each driver's history in the validation group will be compared individually against the model simulation.
I went through the chapter and I have few questions: What does using data to drive a model mean? Does the software allow to load the data of the drivers? This is a group of drivers and I don't want to use their mean

Thanks a lot!
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

For data input, it looks like your data is already in Tidy format, so I'd use that - https://www.vensim.com/documentation/in ... dy2vdf.htm, or GET XLS DATA/GET DIRECT DATA if you want live spreadsheet links.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I will try, the software seems a bit complex. Do you provide technical support? What are the fees? I will talk to my advisor and see if they can sponsor me
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

leon wrote:What does using data to drive a model mean? Does the software allow to load the data of the drivers? This is a group of drivers and I don't want to use their mean
Have you looked at the sample I suggested yet? It takes you though a how to load data into a model and calibrate against it. It's exactly what you need to do.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

This conversation spurred me to add Poisson and Binomial distributions to the optimization payoff calculations, so that part will get easier as soon as we release 6.5. In the meantime, I'll post an example ASAP.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

tomfid wrote:This conversation spurred me to add Poisson and Binomial distributions to the optimization payoff calculations, so that part will get easier as soon as we release 6.5. In the meantime, I'll post an example ASAP.
I have not tried the calibration yet, I will message you when I get stuck.

So do you mean I won't be able to use binomial distributions for the optimization calculations?
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

For the moment, you have to calculate the binomial or Poisson likelihood explicitly in an equation. Adding it to the optimizer's builtin distributions is just a convenience.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

tomfid wrote:For the moment, you have to calculate the binomial or Poisson likelihood explicitly in an equation. Adding it to the optimizer's builtin distributions is just a convenience.
I have my data in excel xls format. How can I load the data into Vensim using the GET XLS DATA function? I don't even know where is the function in vensim

Thanks
Post Reply