Help with model calibration

[leon]

I can't thank you enough Tomfid, you have been very helpful.

1) I'm sure I understand how to use the file you have attached. So, do you mean I should use those estimates as the model parameters and re-run the calibration?
2) What do you think if I use the first month of driving (miles driven in the first month) as driving need instead of trying to figure out the driving need through calibration? Same thing for initial event rate. What if I compute the number of errors in the first month divided by the first month of driving and use that as the initial event rate instead of trying to determine the initial event rate?
3) Any link for how to use the Synthesim?

Again thank so much

[tomfid]

1) Just run the model with those parameters to see what the floating point error is. Then fix the structure so it can't happen. https://www.vensim.com/documentation/23320.htm

2) Seems like a bad idea. One month of driving will be a noisy measure. Better to use the mean, which you can get with INITIAL(GET DATA MEAN(...))

3) https://www.vensim.com/documentation/usr13.htm

[leon]

Thanks so much Tomfid

2 questions:

1) How can I force a parameter not to be zero in the optimization control file? I have attached a screenshot. For example for event fading time Vensim does not allow me to specify 0<event fading time.

2) This a modeling question. I went through the drivers one by one and I realized that the drivers that we were not picked up by the model have something in common. They have a decline in recent events from the beginning of the simulation. I've attached the simulation for 1 driver. This makes me wonder, if I should introduce a variable called "risk taking behavior" and make that variable depend on recent events. I think in the real world, it makes sense to think that if recent driving errors start to decline, people start to engage in risky behaviors (eg traffic violations). Then the variable "risk taking behavior" will affect "event per mile". I've attached the trajectory for traffic violations. So, my question is, how can I formulate the equation of "risk taking behavior" so that it can capture the trajectory of traffic violations?
It also very possible that risk taking behavior is actually affected by recent memories of events. So, I don't know what will work better

Thanks a lot, I appreciate your help

[tomfid]

1) set the minimum to a small value, like 0.001 < x. However, if your time step is 1 unit, the shortest sensible time constant for anything is 1 unit (or >2 units if you want a decent approximation of continuous time).

[leon]

Thanks so much!
Any ideas for my second question?

[Administrator]

if I should introduce a variable called "risk taking behavior" and make that variable depend on recent events. I think in the real world, it makes sense to think that if recent driving errors start to decline, people start to engage in risky behaviors (eg traffic violations). Then the variable "risk taking behavior" will affect "event per mile". So, my question is, how can I formulate the equation of "risk taking behavior" so that it can capture the trajectory of traffic violations?

Maybe use the smooth function? Or store the recent events in a level and drain it over 2 months or so?

[leon]

Thanks!

I think the issue is:
Using statistical modeling, I've expressed event per mile as: Event per mile=exp(β0+β1*mileage+β2*memories)=exp(β0)*exp(β1*mileage+β2*memories)
Then I called exp(β0) the initial event rate. From the equation above, I can see there is no way (initial event rate)*exp(β1*mileage+β2*memories) can be equal to zero. The only way (initial event rate)*exp(β1*mileage+β2*memories) can be equal to zero is when initial event rate=0. But it turns out from the statistical model that exp(β0) (that I called the initial event rate can never be equal to zero). The drivers that were not picked up by the SD model have zero event in their first months of driving (months 1 and 2). Is there any way to re-write this equation so that even if I assign zero to initial event rate, the SD model can still work well?

Your input would be much appreciated

[tomfid]

I think you want to think of exp(B0) as the _reference_event rate, not the _initial_event rate. Then make it a parameter to be optimized, not derived from the first month of driving.

[leon]

Thanks!
I actually added the_reference_event_rate (that I called initial event rate) to the optimization file. Is that reference rate different from the initial rate (baseline rate)? Is it not the rate when the other variables are held equal to zero? Why can't it be called initial event rate?
The model did not pick the drivers with zero event at the start of the time series data, and I wonder if the issue might not be from the exp(B0)

[tomfid]

You can call it whatever you want of course. The software doesn't care.

However, conceptually, the reason is just what you say - "Is it not the rate when the other variables are held equal to zero?" The initial error rate will be equal to exp(B0) only if all the other vars are initially zero too. I would not expect that to be the case in general. If mileage and memories are cumulative, then I guess this might be true.

However, it seems impossible that any driver would ever have an expected error rate of 0. In the real world, that's just common sense. In the statistical world, the log likelihood of the Poisson, Bernouilli and Binomial distributions is undefined for 0 probability or frequency.

[leon]

Thanks for the explanation!

I think I did something maybe wrong. I did the calibration with a distribution "Normal". Is there anyway I could specify binomial distribution? I actually used binomial distribution to conduct statistical analyses of the data.

Thanks

[tomfid]

Binomial makes sense if the number of errors a driver can make is limited to a finite set each month. That seems unlikely. I think you want Poisson. Both distributions will be built into v7 (any day now), but otherwise, see the original model I posted, illustrating the Poisson distribution.

[leon]

There was overdispersion when I conducted the statistical analysis of the data, which made me use binomial distribution in the analysis. But I can try the poisson distribution and see what happens. Could you please send the link of what you posted?

[tomfid]

It's linked to the 3rd post on the third page of this thread.

[leon]

The third post is the drivers zipped file. So the steps to specify the poisson distribution is in that file?

[tomfid]

Right. It's in the model.

Because there isn't a Poisson calibration payoff element in v6.4, you have to calculate a variable that represents the Poisson log likelihood. Then you use that as a policy payoff, and maximize it.

I wouldn't expect the difference between Normal and Poisson to be large - it will mainly affect confidence bounds.

[leon]

This is going to be another complication I guess. Remember the tidy data format did not work with Vensim Pro and you guys reformatted the data for me. So I did not load the data using the cmd command.
Assuming the data is already loaded in the software, could you please help with the different steps in simple terms to accomplish this?
Thanks a lot

[Administrator]

Just reformat it in the same way that I did. It's very easy.

[leon]

Tomfid said there is no Poisson calibration payoff in the current version of Vensim, and I have to calculate a variable that represents the Poisson log likelihood, then use that as a policy payoff, and maximize it. I was asking the different steps in simple terms to accomplish this

[leon]

Does that mean, I need to calculate exp(events per month) and replace that with events per month, then import the data into Vensim and redo the calibration?

[tomfid]

No. I'm not sure what exp(events) would mean.

In general, a likelihood (or log likelihood) is f(model,data). For the standard normal, f = (model-data)^2/2, which is why the payoff computes a sum of squares. The idea is the same for Poisson, just the functional form is different.

I'd suggest copying the structure out of my model and pasting it into yours. Then:
- replace "actual events" with your data
- replace "estimated event frequency" with your exp(b0 + b1*driving + b2*... )

You don't need to reformat your data, unless you don't have data for errors in your model with a name that is distinct from the simulated errors model variable. In that case, you just have to change the names in your data file, reimport, and add the required data variables to the model.

[leon]

I'm still not sure I understand how to do this. I honestly don't know what I'm doing by copying your structure into mine, so I'm confused here. I have attached both your model and mine. Could you please just set up the poisson distribution into my model for me so that I can directly use it? Thanks

[leon]

Could you please take a look at the model and help me with the poisson distribution? You could also lay out the different steps to apply the poisson distribution
Thanks

[tomfid]

1. Copy this structure into your model and connect the MODEL and DATA variables (for the data, the easiest thing is probably to rename it to match your existing data series).
2. Create a Policy payoff to maximize the payoff shown.
3. Optimize.

Driving3.mdl

(2.66 KiB) Downloaded 203 times

[leon]

Thanks for all your help.
Could you help me with the steps to copy the structure and paste into my model? I've tried to copy and paste, but it did not work