Help with model calibration

Use this forum to post Vensim related questions.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

How can this be done in Vensim?
My fear is my limited knowledge of Vensim. For the group I applied my simulated intervention to, I increased the fraction of events known to 100% starting from month=2 for each driver and recorded the simulated data into Excel for each driver. Then I used a statistical software for the comparison because it's much easier for me that way.
I didn't want to bother you for that.
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

I thought your initial event rate parameter had [driver] detail, so that the mean for every driver was centered?

Because the experimental design doesn't randomly assign the treatment, any conclusion about differences between the two groups is pretty speculative. One could always argue that the high-error treatment drivers are also fast learners, for example. So I think you have to lean on the time series behavior of the treatment pool for primary insight.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Yes the initial event rate was subscripted, I mean it was initial event rate[driver].
To be clear, I did not export the treatment group data into Vensim, so don't be thinking I had both sample data into Vensim. Forget about the treatment data for now.
I'm sure you would have done differently what I did.

After the model calibration using the control group data, I recorded the expected initial event rate [driver], expected driving need[driver], expected baseline stock values[driver], and all the parameter estimates. For the treatment simulation, I took each driver (one by one) and set their expected values and re-ran the simulation (driver by driver in the control group; it took me a lot of time) by changing the fraction of events to 100% starting at month=2. I recorded the data for each driver and used a statistical software to compare my simulated treatment data with the actual treatment data.

Hope you understand what I did. It's was a time consuming process, which I knew; but I did not want to bother you on that
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

I think this could be made quite easy in Vensim.

- Create a flag for each driver, indicating whether treatment was received (a constant) or when it was received (time series)
- Import the data for ALL drivers
- Rewrite "fraction perceived" as something like = IF THEN ELSE( treatment, 1, base fraction perceived)
- Estimate parameters with all drivers pooled
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I think I understood what you explained. There is something I'm trying to grasp here.
What do you mean by " Estimate parameters with all drivers pooled'? Calibration?
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Right.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

"fraction perceived" as something like = IF THEN ELSE( treatment, 1, base fraction perceived)

I agree that fraction perceived (for control group)=1 starting from time=2 (which means my simulated intervention is switched on) and it would be base fraction at time=1 (which is 0.07, the model assumption was 7% percent of events are known). This is clear to me

My confusing part of this is:
What would be the value of the fraction perceived for actual treatment group since this group already received the treatment? I'm not sure assigning 1 (starting from time=2) for fraction perceived for the treatment group makes sense because this group already received the treatment. Assigning 1 would mean that I'm applying my simulated treatment again to a group already treated. I think the fraction perceived should be zero for this group because I'm no longer applying my simulated intervention, which is increasing the fraction of events known.

I hope you get my point.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

tomfid wrote:I think this could be made quite easy in Vensim.

- Create a flag for each driver, indicating whether treatment was received (a constant) or when it was received (time series)
- Import the data for ALL drivers
- Rewrite "fraction perceived" as something like = IF THEN ELSE( treatment, 1, base fraction perceived)
- Estimate parameters with all drivers pooled


Dear Tomfid,
May I ask a technical question here?
Let me use an example to frame my question. Suppose that I have 2 drivers A and B. Driver A is a control (no treatment) and Driver B was treated (The treatment started at month=2). I have 5 data points (5 months of follow-up)
Here is the data structure of this example
Driver month Events Treatment
A 1 value 0
A 2 value 0
A 3 value 0
A 4 value 0
A 5 value 0
B 1 value 0
B 2 value 1
B 3 value 1
B 4 value 1
B 5 value 1
My calibrated model was based on Driver A data since it’s the control. I want to test whether the calibrated model can fit the data of Driver B. This means I will keep all the fixed parameters estimated using Driver A data (e.g. effect of cumulative miles driven, time to forget, etc.) and set treatment, driving need, initial event rate, and initial stock values as new parameters to estimate from Driver B data.
What do you think? I basically wanted the Driver B data to be an external validation process of the model
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Makes sense.

If the treatment was assigned on the basis of performance in month 1, you don't really know how much improvement is due to treatment vs. selection and regression to the mean, but presumably you can't easily acquire a new dataset.

I would be inclined to estimate the model on a subset of the data containing a mix of control and treatment drivers, and then test how it performs on a different subset. It would also be interesting to estimate over different subsets and compare the parameters that result.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks.
Let me try both ideas and see what happens.
Post Reply