System of differential equations-- estimating parameters

Use this forum to post Vensim related questions.
Post Reply
Simplifier
Senior Member
Posts: 54
Joined: Tue Sep 11, 2018 2:04 am
Vensim version: PRO

System of differential equations-- estimating parameters

Post by Simplifier »

I have a system of differential equations as followings:
Image

I have already collected the real world data (discrete data) in the form of time series for individual X, Y, and Z variables. Since I need to fit the above system to the real world data to estimate the parameters of ai, bi, ci, and di (i=1, 2, 3), I was wondering if it's possible to do it in the Vensim software.

Could you share your thoughts as experts in the Vensim with me?

Thanks very much in advance for your cooperation, and I look forward to your reply.
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: System of differential equations-- estimating parameters

Post by Administrator »

Have you looked at the calibration topic in the help system?
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
Simplifier
Senior Member
Posts: 54
Joined: Tue Sep 11, 2018 2:04 am
Vensim version: PRO

Re: System of differential equations-- estimating parameters

Post by Simplifier »

Administrator wrote: Thu Sep 13, 2018 12:51 pm Have you looked at the calibration topic in the help system?
Yes. heaps of times I've been looking. Also in the Vensim documentation. Mine is system of differential equations and especially with discrete historical data.

I gave up and decided to post as my clock is ticking.
Administrator
Super Administrator
Posts: 4573
Joined: Wed Mar 05, 2003 3:10 am

Re: System of differential equations-- estimating parameters

Post by Administrator »

Upload the model you have and a sample of the data and hopefully we can suggest something.
Advice to posters seeking help (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391

Units are important!
http://www.bbc.co.uk/news/magazine-27509559
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: System of differential equations-- estimating parameters

Post by tomfid »

Discrete doesn't matter - you just run your model with time step = 1, and exercise appropriate caution: http://metasd.com/2017/11/discrete-time-stinks/
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: System of differential equations-- estimating parameters

Post by tomfid »

Also, data's always discrete. You can calibrate a continuous model against discrete measurements, even if they're irregular in time.
Simplifier
Senior Member
Posts: 54
Joined: Tue Sep 11, 2018 2:04 am
Vensim version: PRO

Re: System of differential equations-- estimating parameters

Post by Simplifier »

Administrator wrote: Thu Sep 13, 2018 1:31 pm Upload the model you have and a sample of the data and hopefully we can suggest something.
Actually I performed it successfully, but the thing is that the estimations are really horrible (not even close). I dont know it's because there are twelve parameters in the system being estimated (large number of parameters), or because of the weakness of payoff function, which is - ((model-data))^2.

Could you advise me on bettering the estimations?
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: System of differential equations-- estimating parameters

Post by tomfid »

Twelve is not unreasonably large - this should work.

What do you mean by "not even close" - poor fit, or is this a synthetic data experiment with a known answer?
Simplifier
Senior Member
Posts: 54
Joined: Tue Sep 11, 2018 2:04 am
Vensim version: PRO

Re: System of differential equations-- estimating parameters

Post by Simplifier »

tomfid wrote: Mon Sep 17, 2018 12:57 pm Twelve is not unreasonably large - this should work.

What do you mean by "not even close" - poor fit, or is this a synthetic data experiment with a known answer?
I actually ran a data experiment for the following system of equations to see if the vensim calibration can reasonably estimate the twelve parameters of a1, b1, c1, d1, ..., a3, b3, c3, and d3. I already knew the values of the parameters, but I just wanted to observe the fitting capability of the vensim. It did have a very poor fit unfortunately.

d(x)/dt= a1x − b1x2 − c1xy + d1xz
d(y)/dt= a2y − b2y2 + c2yx − d2yz
d(z)/dt= a3z − b3z2 − c3zy + d3zx
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: System of differential equations-- estimating parameters

Post by tomfid »

As a check, do you get a better payoff if you run the model with the known true parameters?

Have you tried random multistart? Do you have bounds on your parameters in the control file?
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: System of differential equations-- estimating parameters

Post by tomfid »

Much easier to comment if you can post the model.
Simplifier
Senior Member
Posts: 54
Joined: Tue Sep 11, 2018 2:04 am
Vensim version: PRO

Re: System of differential equations-- estimating parameters

Post by Simplifier »

tomfid wrote: Mon Sep 17, 2018 4:14 pm Much easier to comment if you can post the model.
Attachments
Three (bridging) technologies(autosave1).mdl
(4.22 KiB) Downloaded 193 times
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: System of differential equations-- estimating parameters

Post by tomfid »

I think there are a multiple challenges here. There seem to be a lot of local optima. The usual approach to that would be to use random multistart to explore a lot of starting points, or to use MCMC as a simulated annealer. However, both of those approaches do poorly for me, because parameter bounds of [-1,1] result in huge numbers of floating point errors, and therefore low efficiency.

Better bounds or more robustness in the model might make one of these approaches practical.
Simplifier
Senior Member
Posts: 54
Joined: Tue Sep 11, 2018 2:04 am
Vensim version: PRO

Re: System of differential equations-- estimating parameters

Post by Simplifier »

tomfid wrote: Mon Sep 17, 2018 6:07 pm I think there are a multiple challenges here. There seem to be a lot of local optima. The usual approach to that would be to use random multistart to explore a lot of starting points, or to use MCMC as a simulated annealer. However, both of those approaches do poorly for me, because parameter bounds of [-1,1] result in huge numbers of floating point errors, and therefore low efficiency.

Better bounds or more robustness in the model might make one of these approaches practical.
Oh that's so sad, really sad, that I paid AUD800 for a software that wont be able to do what I need :x :x . Feel a whole waste of money :evil: . Wish the calibration was available on trial through PLE to make sure it can be useful here.

It's not fair. But thanks for taking your time to work on it
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: System of differential equations-- estimating parameters

Post by tomfid »

I don't think you've exhausted all the options yet! I think improving the robustness of the model should be a high priority. If multiple optima and floating point errors within the parameter bounds are the issue, other software's going to have the same kinds of problems.
Simplifier
Senior Member
Posts: 54
Joined: Tue Sep 11, 2018 2:04 am
Vensim version: PRO

Re: System of differential equations-- estimating parameters

Post by Simplifier »

tomfid wrote: Tue Sep 18, 2018 12:24 am I don't think you've exhausted all the options yet! I think improving the robustness of the model should be a high priority. If multiple optima and floating point errors within the parameter bounds are the issue, other software's going to have the same kinds of problems.
Thanks. How can I do the robustness or fix the floating point errors? I'm new to vensim, and I dont know what they mean in vensim.
LAUJJL
Senior Member
Posts: 1421
Joined: Fri May 23, 2003 10:09 am
Vensim version: DSS

Re: System of differential equations-- estimating parameters

Post by LAUJJL »

Hi Simplifier

You have little chance of ever finding a solution of your problem, using a powerful software that you apparently do not master because of too little practice. When one practices Vensim a bit one finds very quickly the utility of putting units in models, and then doing it becomes a second nature.
Not doing it proves that you have not seriously studied Vensim.

And then without knowledge of the software you expect that it will by some miracle solve your problem using its sophisticated features without having understood its basic principles!

Mastering a software like Vensim takes unfortunately years. Start studying the user guide, and once that done, simplify your problem. For example instead of working in three dimensions, work only with one dimension even if it necessitates using fantasy data. This will reduce the number of parameters from 12 to 4, etc…

Best regards.

JJ
LAUJJL
Senior Member
Posts: 1421
Joined: Fri May 23, 2003 10:09 am
Vensim version: DSS

Re: System of differential equations-- estimating parameters

Post by LAUJJL »

Hi Simplifier.

Another basic rule of SD that you do not respect: using constants with no significant name related to their meaning in the system that you pretend to describe.

How can I understand a model where the author uses a1,b1,b2 etc.. without knowing what these constants are about?

Best regards.

JJ
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: System of differential equations-- estimating parameters

Post by tomfid »

Robustness implies that any reasonable input or parameterization of the model should result in a reasonable output - i.e. bounded (not infinite), no negative physical quantities, etc.

Having an abstract model makes that rather hard to assess or enforce, because none of the parameters correspond with recognizable concepts. SD models are usually more operational, i.e. they describe the processes by which things happen, which makes it easier to parameterize them.

In this model, the only reality checks I can think of are:
a) technologies have limited growth rates in the short run
b) technologies have long run growth rates of 0 or at best a few % per year

Without units of measure or a real world interpretation for anything, it's hard to say more.

Consider:
growth rate 1 = a1*Technology 1-b1*Technology 1*Technology 1-c21*Technology 1*Technology 2-c31*Technology 1*Technology 3

I have no idea how to interpret this. Probably abs(growth rate) < 1, which implies abs(a1) < 1 in some conditions, but it's pretty hard to figure out what it requires in general for plausible parameters. Also, I'm not sure what "technology" means here - normally it's thought of as nondecreasing, except in the event of civilization collapse perhaps, but here it's clearly meant to go negative at times.
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: System of differential equations-- estimating parameters

Post by tomfid »

I'd guess that most of the floating point errors are simply cases in which there's explosive exponential growth. Using double precision helps with that, though only briefly at high rates. It also helps to use the log transform in the payoff, to reduce scale differences.

If I repeat the experiment with all parameters bounded 0 <= x <= 0.3, using random multistart, I get far fewer floating point errors. In 600k random multistart simulations, I got 238 endpoints. Many if not most appear to be distinct (not repeats), but they might not be true optima due to the curvature problem ( http://metasd.com/2018/05/optimization-banana-death/ ). I rather quickly get solutions that are very close to the true behavior, but with a few substantial parameter differences. None is exactly the true parameter set, but many have behavior that would be indistinguishable from the true case in the presence of noise.

Using the MCMC option for simulated annealing gets around the curvature problem, and I get into the correct basin (I think) within 200k simulations. A couple parameters are still wrong, but I think that's because they have no significant effect on the behavior, and therefore can't be identified.

So, with a little fooling around, Vensim can solve this problem. The fooling around is the price of abstraction, for the most part.
Attachments
wrong.cin
(129 Bytes) Downloaded 174 times
states.vpd
(60 Bytes) Downloaded 158 times
states-log.vpd
(63 Bytes) Downloaded 152 times
params-multi.voc
(674 Bytes) Downloaded 132 times
params-SA.voc
(678 Bytes) Downloaded 135 times
params-single.voc
(670 Bytes) Downloaded 132 times
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: System of differential equations-- estimating parameters

Post by tomfid »

Files attached above:
- wrong.cin = changes file for deliberately wrong starting point
- states.vpd = payoff comparing model state to data from a base run
- states-log.vpd = same, with log transform
- params-single.voc = a single Powell search
- params-multi.voc = RRandom multistart
- params-SA.voc = simulated annealing

Note that the last two will run until you stop them. I'm tempted to fire up a server and run this a billion times to see if random multistart finds the solution. I think the problem is that the true parameters have a very small basin in a very big space.
http://metasd.com/2014/03/in-rn-n-large ... ou-scream/

One other observation: I think the original rational function version of the model in the top post here is potentially more robust than the version that's currently implemented. As long as the parameters are positive, it ensures that growth is bounded for each of the stocks. Maximum growth rates are also bounded by the a# parameters. This seems like a more tractable situation.
Simplifier
Senior Member
Posts: 54
Joined: Tue Sep 11, 2018 2:04 am
Vensim version: PRO

Re: System of differential equations-- estimating parameters

Post by Simplifier »

tomfid wrote: Tue Sep 18, 2018 3:40 pm Files attached above:
- wrong.cin = changes file for deliberately wrong starting point
- states.vpd = payoff comparing model state to data from a base run
- states-log.vpd = same, with log transform
- params-single.voc = a single Powell search
- params-multi.voc = RRandom multistart
- params-SA.voc = simulated annealing

Note that the last two will run until you stop them. I'm tempted to fire up a server and run this a billion times to see if random multistart finds the solution. I think the problem is that the true parameters have a very small basin in a very big space.
http://metasd.com/2014/03/in-rn-n-large ... ou-scream/

One other observation: I think the original rational function version of the model in the top post here is potentially more robust than the version that's currently implemented. As long as the parameters are positive, it ensures that growth is bounded for each of the stocks. Maximum growth rates are also bounded by the a# parameters. This seems like a more tractable situation.
Wow... Really really thanks very much for your insightful and helpful comments. The comments are really highly sophisticated for me and it's hard for me to turn my heard around them at this moment as I'm a beginner in Vensim, but will get to them by studying more and more.

I just got an idea, but I still need to pick your brain. As you know there are three growth rates in my model and each growth rate does have four parameters. What if I simulate one growth rate at a time so the software would only need to estimate four rather than twelve parameters. Imagine this is the growth rate of Technology 1:

growth rate 1 = a1*Technology 1-b1*Technology 1*Technology 1-c21*Technology 1*Technology 2-c31*Technology 1*Technology 3

As you know I have the real data in terms of time series from 1990-2018 for Technology 1, Technology 2, and Technology 3. Now what if I dont define any equations for Technology 2 and 3 and ask the software to go straight to the corresponding real data and pick up their values for estimating the parameters used in the growth rate of Technology 1. I just dont know how I should define Technology 1 and Technology 2 or how I should ask the software to go and pick up their values. I'm a beginner and I dont know if it means lookup table, or ... .

Could you advise me again?
tomfid
Administrator
Posts: 3806
Joined: Wed May 24, 2006 4:54 am

Re: System of differential equations-- estimating parameters

Post by tomfid »

Generically, the idea of subdividing the problem into smaller estimation problems works. You lose feedback of course, which is a negative particularly if you're doing Kalman filtering.

In this case, I'm not sure it would help, because there seem to be several distinct parameterizations of the model that are very close to the data. So, whether 2 out of 3 techs are defined by data or endogenously might not change the ability to identify the parameters that seem to be ill-determined.

However, it's very easy to test this. You can change a variable - say, Tech 1, to a data variable (edit the equation and change the Type to Data). You'll notice that it's corresponding flow connection will disappear. Then it will be driven by the same (synthetic or real) dataset that you specify as the Data Source on the Advanced tab of the simulation control dialog.
Post Reply