Page 1 of 1
TIME STEP and historical data
Posted: Sun Mar 29, 2020 11:29 am
by Simplifier
Since the historical data tat I am using to optimise and calibrate my model has been collected annually, should I define the TIME STEP annually too? Because the smaller TIME STEP that I use, the worse the optimisation payoff I obtain for the model.
Re: TIME STEP and historical data
Posted: Sun Mar 29, 2020 11:32 am
by Administrator
The time step you select should be 1/3 of the smallest delay in your model. So if your shortest delay is one year, your time step should be 0.25 year(s).
http://www.vensim.com/documentation/ref_time_step.htm
Re: TIME STEP and historical data
Posted: Sun Mar 29, 2020 4:08 pm
by tomfid
If shrinking the time step is changing the payoff, that's probably a sign that it's also changing the dynamics, and that in turn is changing the fit.
The choice of time step really shouldn't have anything to do with the data interval.
Re: TIME STEP and historical data
Posted: Sun Mar 29, 2020 4:09 pm
by tomfid
The best thing to do is create some custom graphs that compare the model to the data, and then do some hand calibration to see what's going on.
Re: TIME STEP and historical data
Posted: Mon Mar 30, 2020 7:58 am
by Simplifier
tomfid wrote: ↑Sun Mar 29, 2020 4:09 pm
The best thing to do is create some custom graphs that compare the model to the data, and then do some hand calibration to see what's going on.
I think I figured what the issue can be.
I'm using several RANDOM PINK NOISE functions in my model and it seems they are really sensitive to the change of TIME STEP. Note that I am optimising some of the elements that are used in any of the RANDOM PINK NOISE functions, i.e. mean, sd, and correlation time, but except seed. The outcome of these RANDOM PINK NOISE functions seems to eventually have made the value of the variables and stocks in my model fluctuate drastically with the the change of TIME STEP.
Have I observed and detected a right reason for it?? What's your recommendation to fix it?
Re: TIME STEP and historical data
Posted: Mon Mar 30, 2020 8:03 am
by Administrator
Have I observed and detected a right reason for it?? What's your recommendation to fix it?
As I've suggested, you should fix all the units errors in your model first. This will lead you to where all the errors are.
Re: TIME STEP and historical data
Posted: Mon Mar 30, 2020 10:40 am
by Simplifier
Administrator wrote: ↑Mon Mar 30, 2020 8:03 am
Have I observed and detected a right reason for it?? What's your recommendation to fix it?
As I've suggested, you should fix all the units errors in your model first. This will lead you to where all the errors are.
I have created a new small, simple model and I have used a pink noise in it. Again with the change of time step, the results and payoff function change.
Is it supposed to be working like this?
Re: TIME STEP and historical data
Posted: Mon Mar 30, 2020 10:44 am
by Administrator
If you upload the model, we can take a look. Make sure you upload all files required to run it.
Re: TIME STEP and historical data
Posted: Mon Mar 30, 2020 11:49 am
by Simplifier
Administrator wrote: ↑Mon Mar 30, 2020 10:44 am
If you upload the model, we can take a look. Make sure you upload all files required to run it.
Re: TIME STEP and historical data
Posted: Mon Mar 30, 2020 12:07 pm
by Administrator
I'd suggest putting in some realistic bounds on the things you are allowing the optimizer to modify, if you do this, the optimal solution is identical for time step = 0.5 and 0.03125.
Re: TIME STEP and historical data
Posted: Mon Mar 30, 2020 12:31 pm
by tomfid
Time step will have a big effect on any randomness - if you draw twice as many values, you get a completely different realization of the random streams.
To calibrate a model with random inputs, you really need to look at the performance over an ensemble of simulations, rather than a single run. This dramatically adds to the runtime of course.
Re: TIME STEP and historical data
Posted: Mon Mar 30, 2020 12:36 pm
by Simplifier
Administrator wrote: ↑Mon Mar 30, 2020 12:07 pm
I'd suggest putting in some realistic bounds on the things you are allowing the optimizer to modify, if you do this, the optimal solution is identical for time step = 0.5 and 0.03125.
I put some realistic bounds. The eventual results seem identical, though not entirely. Why does the outcome of pink noise change with the change of time step? I even disconnected the pink noise and it does change every time I change the time step.
Also my real model is way larger than this. How can i select the best time step and make sure that the selected time step is the one to go with?
Re: TIME STEP and historical data
Posted: Mon Mar 30, 2020 12:47 pm
by tomfid
If you think about a run with 4 time steps, the random draws might be
Time 1 2 3 4
Value 3 8 2 6
If you halve the time step, you get
Time 0.5 1 1.5 2 2.5 3 3.5 4
Value 3 8 2 6 1 7 7 9
The value 2 now occurs at time 1.5 instead of time 2, etc.
In a real model, it's more complex than that, because you may have multiple random variables drawing from the same stream of numbers.
If the behavior still changes when the pink noise is disconnected (e.g., standard deviation set to 0), then that must be a change in other behavior, related to stability or perhaps discrete events (PULSE etc.).
The easiest way to test is to cut time step in half successively until the behavior stops changing.
Re: TIME STEP and historical data
Posted: Mon Mar 30, 2020 12:58 pm
by Simplifier
tomfid wrote: ↑Mon Mar 30, 2020 12:47 pm
If you think about a run with 4 time steps, the random draws might be
Time 1 2 3 4
Value 3 8 2 6
If you halve the time step, you get
Time 0.5 1 1.5 2 2.5 3 3.5 4
Value 3 8 2 6 1 7 7 9
The value 2 now occurs at time 1.5 instead of time 2, etc.
In a real model, it's more complex than that, because you may have multiple random variables drawing from the same stream of numbers.
If the behavior still changes when the pink noise is disconnected (e.g., standard deviation set to 0), then that must be a change in other behavior, related to stability or perhaps discrete events (PULSE etc.).
The easiest way to test is to cut time step in half successively until the behavior stops changing.
Thanks Tom for clear explanation!!
I meant I disconnected the pink noise and watched whether the behaviour of the pink noise changes or not, which as a result it did. And obviously this proved your above explanation that any randomness changes with the change of time step. Its going to be a long overwhelming and frustrating optimisation process to work with all the pink noises and time step in my model i think.
Re: TIME STEP and historical data
Posted: Tue Mar 31, 2020 7:58 am
by Administrator
Simplifier wrote: ↑Mon Mar 30, 2020 12:58 pmIts going to be a long overwhelming and frustrating optimisation process to work with all the pink noises and time step in my model i think.
Your first action should be to correct all the unit errors in your model. These errors mean your model has bugs and is not producing the correct output at time steps other than 1.