Page 1 of 1

User-defined file setup in Sensitivity Simulation - for multiple parameters

Posted: Sun Jan 12, 2020 8:05 pm
by WayneZhou2009
Hi Tom, happy new year and I hope you had a great holiday season. I have a question about the "File" option in the Sensitivity Simulation Setup. I want to run Monte Carlo simulations of my model by randomly drawing samples from a distribution of Parameter A, a distribution of Parameter B, and a distribution of Parameter C.

Each distribution is not "standard" type of distribution available in Vensim. Rather, each distribution is the posterior distribution of the parameter concerned, previously produced separately from MCMC process in R. Each distribution is in a tab delimited text file. However, it appears that Vensim only allows one single file to be selected.

I do not think combining the 3 files into one file will solve the problem, because that would mean some sort of "fixed" combination of the 3 parameters. For example, if combined, the file may look like:

A B C
2.2 100 0.3
2.3 98 0.4
2.1 87 0.7
... ... ...

This will mean that every draw will correspond to a row giving a specific combination, such as the first row where A=2.2, B=100, and C=0.3. However, the draws shall allow multiple different combinations such as a combination of A=2.2, B=98 and C = 0.7, a combination of A=2.2, B=87 and C=0.3, etc.

How could I set this up in Vensim?

Thank you very much.

Re: User-defined file setup in Sensitivity Simulation - for multiple parameters

Posted: Mon Jan 13, 2020 12:28 am
by tomfid
Hi - Good question.

You are correct, that there isn't a direct way to resample the columns independently in Vensim. You'd have to use R or Python or Excel or something to do that.

However....

I think you don't want to resample in that way. If you do, you are in effect sampling each parameter's marginal distribution. But you lose the joint distribution of the parameters. If they're independent, fine, but most of the time they will be correlated, possibly in some non-obvious way. So, I would recommend keeping the rows of the sample intact, as the File option in Vensim does.

Re: User-defined file setup in Sensitivity Simulation - for multiple parameters

Posted: Mon Jan 13, 2020 2:25 pm
by WayneZhou2009
Hi Tom, many thanks for your helpful reply. I agree with you that the joint distribution should be more proper than marginal distribution if the parameters are correlated. But in my case, they are not correlated. Their posterior distributions were obtained from separate independent models. And there exist almost no theoretical or practical correlations between them. Hence I feel that it would be necessary to sample the columns independently.

So, suppose there are 10 samples in each of the 3 columns, that would mean a total of 10^3 = 1000 samples which cover all possible combinations of the 3 parameters. I think I can generate these 1000 samples in R and then put them into one single text file to be used by Vensim's Sensitivity Simulation. However, as I mentioned, each distribution is the posterior distribution of a parameter, and therefore its column is very long, having many samples, say, 30,000. That would give a unbearably huge number of combination possibilities...

I wonder whether there is any nicer way of doing this, in terms of data manipulation and computational efficiency?

Many thanks.
tomfid wrote: Mon Jan 13, 2020 12:28 am Hi - Good question.

You are correct, that there isn't a direct way to resample the columns independently in Vensim. You'd have to use R or Python or Excel or something to do that.

However....

I think you don't want to resample in that way. If you do, you are in effect sampling each parameter's marginal distribution. But you lose the joint distribution of the parameters. If they're independent, fine, but most of the time they will be correlated, possibly in some non-obvious way. So, I would recommend keeping the rows of the sample intact, as the File option in Vensim does.

Re: User-defined file setup in Sensitivity Simulation - for multiple parameters

Posted: Mon Jan 13, 2020 2:49 pm
by tomfid
I see. In that case, it does seem sensible to do some kind of combinatorial resampling from the individual columns. Unfortunately, the Vensim file sensitivity isn't set up to do this, but I'll take a look - it might be simple to implement.

Re: User-defined file setup in Sensitivity Simulation - for multiple parameters

Posted: Mon Jan 13, 2020 7:04 pm
by WayneZhou2009
Thanks a lot Tom for helping. I look forward to hearing from you.

Re: User-defined file setup in Sensitivity Simulation - for multiple parameters

Posted: Mon Jan 27, 2020 12:05 pm
by WayneZhou2009
Hi Tom, please may I follow up with you on this? Many thanks.
tomfid wrote: Mon Jan 13, 2020 2:49 pm I see. In that case, it does seem sensible to do some kind of combinatorial resampling from the individual columns. Unfortunately, the Vensim file sensitivity isn't set up to do this, but I'll take a look - it might be simple to implement.

Re: User-defined file setup in Sensitivity Simulation - for multiple parameters

Posted: Mon Feb 03, 2020 10:53 am
by RWTH_FCN
tomfid wrote: Mon Jan 13, 2020 2:49 pm I see. In that case, it does seem sensible to do some kind of combinatorial resampling from the individual columns. Unfortunately, the Vensim file sensitivity isn't set up to do this, but I'll take a look - it might be simple to implement.
I was also wondering how can we feed 4 scnearios simultaneously to a model. I have an energy system model in Vensim which I have 4 scenarios (4 different inputs) for it. How is it possible to do sensitivity analysis or stochastic modeling with these 4 scenarios?

Re: User-defined file setup in Sensitivity Simulation - for multiple parameters

Posted: Mon Feb 03, 2020 1:58 pm
by tomfid
When you say scenarios, do you mean that you have 4 files, each with a different set of variables, and you want to merge them, or ... ?

Re: User-defined file setup in Sensitivity Simulation - for multiple parameters

Posted: Mon Feb 03, 2020 3:46 pm
by RWTH_FCN
tomfid wrote: Mon Feb 03, 2020 1:58 pm When you say scenarios, do you mean that you have 4 files, each with a different set of variables, and you want to merge them, or ... ?
Yes, I have four input files ( four CIN files). Each of them includes many variables like fuel prices, installed capacities, co2 prices, transmission constraints, demand, costs assumptions and etc. But CO2 prices, fuel prices, demands and transmission capcacitis are different in different scenarios. I do not want to merge them, I was just wondering can Vensim read all of the simultaneously to do a kind of sensitivity analysis instead of monte carlo simulation for this large-scale model? Or is it possible to use stochastic optimization feature of Vensim with these four defined files instead of monte carlo simulation?

Re: User-defined file setup in Sensitivity Simulation - for multiple parameters

Posted: Tue Feb 04, 2020 8:42 am
by RWTH_FCN
In Stochastic Optimization page of Manual, I encountered this sentence:

"The stochastic optimization option facilitates optimization over an array of random or user-specified scenarios without first arraying the model. The total payoff is the sum of the payoffs of the individual runs in the sensitivity ensemble."

Can I ask you how is it possible to feed user-specified scenarios to the model to run it stochasticly?

Re: User-defined file setup in Sensitivity Simulation - for multiple parameters

Posted: Tue Feb 04, 2020 11:53 am
by tomfid
In your case, you'd have to transpose your cin file(s) into a sensitivity sample file.

This isn't as hard as it sounds. If you import a cin file into Excel, and use text to columns to break things at the =, you get a column of variable names and a column of values. Do this once per file, then merge so that column A is names, B is scenario 1, C is scenario 2 values, etc. Finally, use copy + paste special ... transpose so that you have the names in row 1 rather than column A. Then you can use this to drive the simulation.

Re: User-defined file setup in Sensitivity Simulation - for multiple parameters

Posted: Tue Feb 04, 2020 12:37 pm
by RWTH_FCN
tomfid wrote: Tue Feb 04, 2020 11:53 am In your case, you'd have to transpose your cin file(s) into a sensitivity sample file.

This isn't as hard as it sounds. If you import a cin file into Excel, and use text to columns to break things at the =, you get a column of variable names and a column of values. Do this once per file, then merge so that column A is names, B is scenario 1, C is scenario 2 values, etc. Finally, use copy + paste special ... transpose so that you have the names in row 1 rather than column A. Then you can use this to drive the simulation.
Thanks for the hint. Should I do it for all variables or is it fine to do it for variables which vary in different scenarios? Then, how vensim recognizes the scenarios?

Re: User-defined file setup in Sensitivity Simulation - for multiple parameters

Posted: Tue Feb 04, 2020 12:53 pm
by tomfid
I would just include the things that vary - the rest could be left in a default cin file. Then you include the sample in a sensitivity simulation via a vsc using the File method.

Re: User-defined file setup in Sensitivity Simulation - for multiple parameters

Posted: Tue Feb 04, 2020 1:27 pm
by RWTH_FCN
tomfid wrote: Tue Feb 04, 2020 12:53 pm I would just include the things that vary - the rest could be left in a default cin file. Then you include the sample in a sensitivity simulation via a vsc using the File method.
Sounds plausible. If model takes 2 hours for run, is it possible to do stochastic optimization as well or do you just suggest to do a sensitivity analysis?

Re: User-defined file setup in Sensitivity Simulation - for multiple parameters

Posted: Thu Feb 06, 2020 3:12 pm
by tomfid
I would guess that it probably won't get far. It should be possible to optimize over a small sample for a few control parameters.

We really need to get parallel simulation released - that's been held up by turbulence from the 64 bit changeover. However, there's a basic problem, which is that optimization doesn't parallelize very well. (Things like random multistart and MCMC do, but the efficiency of a single search doesn't scale with the number of cores.)