Help with model calibration

Use this forum to post Vensim related questions.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I don't see any errors/warning other than "No values in run import - unloaded"
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Not sure what the problem is here. There was a bug in the initial implementation of tidy data, but I thought it was fixed in 6.4e. Maybe not - that would explain why this isn't working. I think you have two options: reformat the data in some other way, or wait for the next release (should be this week).
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

I think you've got your files confused. And I think you still need to read the help we keep posting, the original example I posted takes you through everything (including data import).

1. Download the "Drivers.zip" file that Tom uploaded.
2. Unzip the file.
3. Open Vensim.
4. Click "File->Open Model" and open "Driving1.mdl".
5. Click "File->Edit file" and select "import.cmd".
6. Click "File->Run Commands" and you should be able to import the data.

I'd really urge you to spend the time going through the tutorial first though.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

You're right. It's not a bug. I reverted to 6.4E and it works fine. I get:

Code: Select all

Writing 6 values for -events per month[Driver60]
Writing 6 values for -events per month[Driver57]
Writing 6 values for -events per month[Driver56]
Writing 6 values for -events per month[Driver54]
Writing 6 values for -events per month[Driver53]
Writing 6 values for -events per month[Driver52]
Writing 6 values for -events per month[Driver51]
Writing 6 values for -events per month[Driver50]
Writing 6 values for -events per month[Driver49]
Writing 6 values for -events per month[Driver48]
Writing 6 values for -monthly driving[Driver60]
Writing 6 values for -monthly driving[Driver57]
Writing 6 values for -monthly driving[Driver56]
Writing 6 values for -monthly driving[Driver54]
Writing 6 values for -monthly driving[Driver53]
Writing 6 values for -monthly driving[Driver52]
Writing 6 values for -monthly driving[Driver51]
Writing 6 values for -monthly driving[Driver50]
Writing 6 values for -monthly driving[Driver49]
Writing 6 values for -monthly driving[Driver48]
Writing 6 values for time base -Time
Loading data.vdf as data.  Origin is ..
data.csv converted to dataset data.vdf on Wed May 03 07:34:47 2017
So it has to be something else.
- running inside the .zip archive?
- files in a location without write privileges?
- wrong command?
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Administrator wrote:I think you've got your files confused. And I think you still need to read the help we keep posting, the original example I posted takes you through everything (including data import).

1. Download the "Drivers.zip" file that Tom uploaded.
2. Unzip the file.
3. Open Vensim.
4. Click "File->Open Model" and open "Driving1.mdl".
5. Click "File->Edit file" and select "import.cmd".
6. Click "File->Run Commands" and you should be able to import the data.

I'd really urge you to spend the time going through the tutorial first though.
I did exactly what you described, which is almost what I was doing. The only thing new here is the step 4, which I followed. I'm still getting the same thing "No values in the run import - unloaded". Are we using the same software?
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

tomfid wrote:You're right. It's not a bug. I reverted to 6.4E and it works fine. I get:

Code: Select all

Writing 6 values for -events per month[Driver60]
Writing 6 values for -events per month[Driver57]
Writing 6 values for -events per month[Driver56]
Writing 6 values for -events per month[Driver54]
Writing 6 values for -events per month[Driver53]
Writing 6 values for -events per month[Driver52]
Writing 6 values for -events per month[Driver51]
Writing 6 values for -events per month[Driver50]
Writing 6 values for -events per month[Driver49]
Writing 6 values for -events per month[Driver48]
Writing 6 values for -monthly driving[Driver60]
Writing 6 values for -monthly driving[Driver57]
Writing 6 values for -monthly driving[Driver56]
Writing 6 values for -monthly driving[Driver54]
Writing 6 values for -monthly driving[Driver53]
Writing 6 values for -monthly driving[Driver52]
Writing 6 values for -monthly driving[Driver51]
Writing 6 values for -monthly driving[Driver50]
Writing 6 values for -monthly driving[Driver49]
Writing 6 values for -monthly driving[Driver48]
Writing 6 values for time base -Time
Loading data.vdf as data.  Origin is ..
data.csv converted to dataset data.vdf on Wed May 03 07:34:47 2017
So it has to be something else.
- running inside the .zip archive? I unzipped the file
- files in a location without write privileges? I'm not sure I know location without write privilege. What does this mean?
- wrong command?
I just used your command as described/suggested
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

ok, I think I know what's going on.

When you click "File->Run commands", Are you getting a message saying "Do you want to treat this as input data"?

You are using Vensim Pro, we assumed you were using DSS (and command files are not supported in Pro). There is no easy way to load tidy format data into Vensim pro at the moment.

I'd urge you again to go through the tutorial I posted originally. This takes you through data import and calibration, you could have formatted your data into a similar way to how the tutorial asks for.

I've formatted the data in a way Vensim Pro can read.

1. Click Model->Import data
2. Select data.xlsx
3. Select the "VENSIM_FORMAT_driver_data" sheet.
4. In the next dialog, there is a setting (row 2), "Time axis name : ", following this, there is "Across" and "Down. Select "Down".
5. Click ok.
Attachments
data.xlsx
(18.45 KiB) Downloaded 182 times
VENSIM_FORMAT_driver_data.vdf
(4.46 KiB) Downloaded 185 times
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks so much, I appreciate. It works fine now the import.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I've tried to follow the electric example on this link http://www.vensim.com/documentation/ind ... ?21295.htm
But I guess changes have been made to the software and the steps described are not what I'm seeing.
When I Clicked on the Optimize button on the Toolbar (there is no optimization button on the toolbar as suggested by the tutorial), I didn't get The Optimization Wizard opens with the Payoff Definition. Rather I got what I've attached to this message. Maybe there is a step I missed out
Thanks for your help
Attachments
Doc4.docx
(79.15 KiB) Downloaded 177 times
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I figured it out. Sorry
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

What are the rules for setting weight during calibration? Should I use a weight of 1 for my data?
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I got this at the end of the calibration. Do you have any ideas about what went wrong?
Also, the initial values of the stocks are not really known and the software did not ask about them. I also wanted the stocks to be parameters to be estimated.
Thanks a lot, I appreciate your help
Attachments
Doc4.docx
(88.71 KiB) Downloaded 175 times
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

I'm not sure what you are doing. It looks like you've selected the data file as the payoff definition.

You really need to follow the example, it takes you through all the steps you need to go through.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I could be wrong, but I think I've exactly followed the different steps. I've attached all the steps I went through
Thanks so much for all your help
Attachments
Doc4.docx
(226.49 KiB) Downloaded 176 times
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: Help with model calibration

Post by Administrator »

Can you upload all the files you are using as it's impossible to help with just screenshots.

And I'd really suggest you do the Vensim example I keep telling you about.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

It appears that you're trying to load a dataset that doesn't exist. Check what you have specified in the Data Sources field of the Advanced tab of the Simulation Control dialog.
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

If you want the initial conditions of the stocks to be included, make them parameters:
stock = INTEG( flows, initial stock )
initial stock = 3.141
Then add the "initial ..." constant to the optimization control file.

Using a weight of 1 is probably a good start, as long as driver errors are similar in magnitude.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Thanks Tomfid for pointing out the problem, I'm so appreciative. I think you are correct. It worked fine, but with some floating errors.

I would like to address the initial values for the stocks, which are not known.
How do you specify that constant (initial condition) in the model? The control parameter file is not giving me the option of including the initial values of the stocks as parameters. Take a look at the parameter options I got when I clicked on "add constant"

One more question: there is also an auxiliary variable I made before, and now want to remove it. Any suggestions?

Thanks so much
Attachments
Doc5.docx
(42.09 KiB) Downloaded 136 times
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

You have to add the constant to the model first - edit your equation so that the stock initial conditions are named constants. Then you can use those constants in your control file.

You can delete a variable with the Delete tool in the toolbar, or in the Equation Editor, using the Delete Variable button.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

The equation for a stock was expressed as a function of flow and initial value, for example: cumulative miles driven=integ(monthly driving, 200) with 200 being the initial value for that stock. When I clicked on "add constant" in the optimization control file, a diag "optimization parameter" opened and has a list of variables. Stock names are not in the variable names, and I don't see a way to specify a stock initial value as a parameter.

Sorry, I wanted to ask how to delete a shadow variable. I used "time" as a shadow variable, and VENSIM said it cannot be deleted. I wanted to remove it
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

I guess I have to name the initial value a variable, and use an arrow to link the variable (initial condition) with the stock variable. What do you think?
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

Recall my example above:
stock = INTEG( flows, initial stock )
initial stock = 3.141
In the same way, rewrite your stock equation so that the 200 is a separate variable, "initial miles driven," with value 200 (or whatever). Then use that constant in your optimization setup.

To remove <time>, select it, hit the delete key, and choose "Remove from View."
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

What do you think?
I think you are correct.
leon
Senior Member
Posts: 192
Joined: Tue Feb 21, 2017 11:11 pm
Vensim version: PRO

Re: Help with model calibration

Post by leon »

Please, help me to improve this work. I'm so grateful for all the help you have been providing to me.
After calibration:
1) There were floating point errors
2) The results are not satisfactory as model results and actual data do not overlap very well. Issue with the weight?
3) Any inputs for improvements?

Also my thinking is to compute individual statistics and then average them to get the final statistics for the sample. Then I'm going to apply the model to a different set of drivers using the average model parameters

Please, find attached the model results. I've also attached the model
Attachments
Driving1.mdl
(4.06 KiB) Downloaded 140 times
Results.docx
(17.86 KiB) Downloaded 123 times
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Help with model calibration

Post by tomfid »

3) The model you attached is the same as my original concept, so I don't have any comment on that.

Just as a matter of good source control practice, I'd recommend keeping your whole project in a folder with a version number, and periodically copying the whole thing to a new folder with an incremented number. Then it's easy to .zip up everything for archiving or posting. We can't say much without a complete package. (This is good advice for everyone posting here btw.)

2) The weights are unlikely to be the problem, at least when the data all have similar scale, as here. More likely, the structure simply can't reproduce the desired behavior precisely, and/or there's a lot of noise in the error process.

A good experiment would be to calibrate one or more drivers by hand (in Synthesim). That'll give you some insight into what matters and how to do better.

1) Generally your model should not be causing floating point errors. Sometimes they're benign, like the division by zero you get from computing market shares if all companies have 0 sales, but often they represent a conceptual problem.

To test, replicate the conditions that caused the reported error. One way to do that is to paste the error report into a .cin file and load that at startup. Here's an example, from your report:
fp error.cin
(4.85 KiB) Downloaded 146 times
Post Reply