Help with model calibration

[Administrator]

Open both models
select the structure from the first model you want to copy
Edit->Copy
switch to the model where you want to paste
edit->Paste.

[leon]

Thanks so much.
I don't know what I should expect to see after I paste the structure into my model. But attached is what I got. What should I do next?

[leon]

I don't see any options of connecting the MODEL and DATA variables

[tomfid]

It looks like you've copied and pasted your own model structure rather than pasting mine into your model.

Rather than copy/paste, I'd suggest just manually recreating my structure in your model.

[leon]

Thanks. I've created your structure. Please, find attached the model. What's the next step?

[Administrator]

1. Copy this structure into your model and connect the MODEL and DATA variables (for the data, the easiest thing is probably to rename it to match your existing data series).
2. Create a Policy payoff to maximize the payoff shown.
3. Optimize.
Driving3.mdl

[tomfid]

The next thing would be to replace the MODEL and DATA variables with your modeled and data events. The easiest way is often to use the Merge tool to drag the existing variables onto my templates, replacing them.

[tomfid]

If you don't have a data variable with a unique name, you'll need to create one (see Chapter 16, User Guide).

[leon]

I'm still not able to do this thing. You mentioned that a new version of VENSIM with poisson distribution built-in was going to be released 2 weeks ago. Is the new version out yet?
What do you mean by replacing MODEL and DATA variables with modeled and data events? I have a model variable called "events per month"; should I rename "DATA events" as "events per month" in your structure? And what about "MODEL expected events"? Where will that go?
I'm sorry, but I'm still not clear where we are going at this point

[Administrator]

I'm still not able to do this thing

What are you still struggling with?

You mentioned that a new version of VENSIM with poisson distribution built-in was going to be released 2 weeks ago. Is the new version out yet?

Not yet. I'm hoping to upload it today.

What do you mean by replacing MODEL and DATA variables with modeled and data events?

Tom suggested using his structure in your model. Paste it in, and replace the relevant parts in his structure with the parts from your model.

[tomfid]

Here's the big picture:

For any distribution, the likelihood (or log likelihood) is a function of the model and data values, plus some parameters. For the Normal distribution, it simplifies to:

((model-data)/std deviation)^2

which is why minimizing the sum of squared errors maximizes the likelihood, assuming Normal errors.

If the errors are exponential (double Laplace), it's ABS( (model-data)/scale ), and so on.

So, you just need to write an equation that calculates this value for the distribution you need, which is what my Poisson example is doing. Only my MODEL and DATA inputs need to be replaced by your model and data values. Then you use a policy payoff to maximize the likelihood variable.

Whether you use a builtin payoff element or write an explicit equation, the answer will be the same. You might try replicating the Normal version above, to verify that it works like your existing payoff.

[leon]

Thanks so much.
I know the theoretical concepts behind normal distribution, poisson distribution, binomial distribution, etc. My issue is the practical aspect of these concepts ie how to restructure my model in order to calibrate my model using poisson distribution.
If you are going to release the new version of Vensim with poisson/binomial distribution built-in today, I might just wait for the new version.

[leon]

Is the newer version of Vensim out?

[tomfid]

Probably tomorrow - you can sign in to the download page and see when it appears.

[leon]

Thanks. From your experience, do you think the Poisson/binomial distribution can improve the model fit? Or is it a problem with the model structure?
I appreciate your input

[tomfid]

I think it's unlikely to matter much. The issues are probably either structural, or it could be that error generation is just inherently noisy, so there isn't much information in the data.

[leon]

Thanks for your input. The thing is the model provides a better fit for drivers with high errors and a poor fit for those with low errors, especially at start-up. Don't you think this phenomenon could be due to the wrong distribution used? My guess the distribution would approximate a normal distribution in drivers with high events

[tomfid]

Could be. The trouble with infrequent events is that you need a long history in order to see much.

I guess the question is, what does the fit look like? Are there discernible trends that aren't captured?

[leon]

Thanks for your help.
I'm very hopeful that the newer version of VENSIM will be released tomorrow. Otherwise, you just have to help me again with the user-defined poisson distribution

[leon]

Thanks for releasing the VENSIM 7 version.
There is a good fit when one variable is calibrated; however when 2 variables are simultaneously calibrated, the fit does not look good. What do you suggest/recommend when that happens?
Thanks for all your help

[tomfid]

When you say 1 or 2 variables, do you mean 1 or 2 drivers, or 1 or 2 different measures per driver?

If the fit degrades when you add more drivers, then there may be a pooled parameter that needs to be distinct for each driver (or, it could be noise).

If the fit degrades when you consider multiple measures, there might be several causes:
- the measures need to be weighted to reflect different scales or measurement errors (e.g., one is order of magnitude 1e6 and the other is OM 1.0)
- the model can't explain both measures at once
- the measures somehow contain conflicting information (which might be a data issue)

[leon]

I mean 2 different measures per driver (driving errors and monthly driving). Probably I need different scales for the 2 variables. Should I use a scale of 1 for driving errors and 1,000,000 for monthly driving like you suggested?

[tomfid]

The scale parameter depends on what payoff type you're using. See
https://www.vensim.com/documentation/ex ... ayoffs.htm

If you're using the default "Normal" type, the weight should be the inverse of the standard error of the measurement, which may be unknown. Generally a decent guess is fine.

So, for driving, you could base your estimate on something like 10% of monthly driving, or just the standard deviation of monthly driving. You can refine these later - for example, by observing the residuals - or estimate them, but I wouldn't mess with that on a first attempt. Just as a wild guess, drivers probably average 1000 miles/month, so an sd of 10% would be 100 miles/month, for a weight of 1/100 = .01.

For errors, you could start with the fact that Poisson variance = mean, so if drivers average 16 mistakes per month, sd = sqrt(16) = 4, for a weight of 1/4 = .25.

[leon]

Also, how can I reset a simulation? I mean if I want to start over.
I wanted to do a simple run; I clicked on "simulate" and I got this attached message. I was not trying to open data.vdf
Thanks

[leon]

Thanks so much for the explanation!