Lookup or time series for data input
Lookup or time series for data input
Hi,
I am having some issues trying to integrate data in Ventity. I have weather data in hourly values for a single year, i.e. 24*365 = 8760 data points, that I need to use in my model, repeatedly over several years, such that the value on, for instance, January 3rd, 4pm, will always be the same data point.
My initial idea was to do this via a lookup function specified via an excel sheet, using Modulo(Model.Time,8760) as input and the hourly values as output. This works to a certain extent, but since Ventity external data input files require the entire lookup function to be contained in a single cell, only about a fifth of the sequence fits (Excel has a character limit per cell of 32767).
The second idea was to input the data as Time Series data and use a Data Entity to access the values, however, there is no way to "repeat" the values once the first year is over, as far as I have been able to figure out.
Is there a functionality or workaround in Ventity that could solve my problem? I know that in Vensim it is possible to specify Lookup functions in Excel in a different format, where each row would be a new data point. This made it easy to access this type of longer data sets.
I am having some issues trying to integrate data in Ventity. I have weather data in hourly values for a single year, i.e. 24*365 = 8760 data points, that I need to use in my model, repeatedly over several years, such that the value on, for instance, January 3rd, 4pm, will always be the same data point.
My initial idea was to do this via a lookup function specified via an excel sheet, using Modulo(Model.Time,8760) as input and the hourly values as output. This works to a certain extent, but since Ventity external data input files require the entire lookup function to be contained in a single cell, only about a fifth of the sequence fits (Excel has a character limit per cell of 32767).
The second idea was to input the data as Time Series data and use a Data Entity to access the values, however, there is no way to "repeat" the values once the first year is over, as far as I have been able to figure out.
Is there a functionality or workaround in Ventity that could solve my problem? I know that in Vensim it is possible to specify Lookup functions in Excel in a different format, where each row would be a new data point. This made it easy to access this type of longer data sets.
Re: Lookup or time series for data input
Understood. I've put in a ticket for an improvement that addresses this.
In the meantime, what if you create a time series that repeats the lookup data N times?
In the meantime, what if you create a time series that repeats the lookup data N times?
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Lookup or time series for data input
I tried doing that but the excel files quickly became too large for Ventity to handle (at least on my computer), as my simulation runs 30 years, so I need nearly 300'000 data points for several variables. I'll try to find some parametric way to estimate the data points for now, but please keep me in the loop about any updates.
Re: Lookup or time series for data input
If you have a file that's big enough to choke the Ventity data input, I'd love to have a copy for testing.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Lookup or time series for data input
We're exploring options on this.
After some reflection, I'm wondering why you need to run hourly for 30 years. I'm assuming this is electric power, but no matter what the domain, running a tactical time step on a strategic horizon is tough. It implies that the long term model has to have a lot of short term detail that mostly gets in the way.
Is there another way to approach this problem?
After some reflection, I'm wondering why you need to run hourly for 30 years. I'm assuming this is electric power, but no matter what the domain, running a tactical time step on a strategic horizon is tough. It implies that the long term model has to have a lot of short term detail that mostly gets in the way.
Is there another way to approach this problem?
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Lookup or time series for data input
I use the hourly time step to evaluate shortterm effects like intraday price volatility and power shortages. Zooming out to daily/weekly/monthly/yearly time steps will make that much harder. The 30 year horizon is to see the longterm effects of policies on these effects.
I attach the document with the data for 1 year. I cannot upload the long one that breaks Ventity as it's too large for the forum, but it's basically this spreadsheet copied down 30 times.
I originally intended to use different historic yearly weather patterns and make the model randomly choose a historic year to follow each year. My work around for now is to use an average 24hour curve with a yearly pattern of daily averages, essentially cutting down the required data points to 365 instead of 8760, allowing the lookup solution to work. The down side for me here is that I have a repeating daily pattern and thus the hourly outliers are not as visible.
I attach the document with the data for 1 year. I cannot upload the long one that breaks Ventity as it's too large for the forum, but it's basically this spreadsheet copied down 30 times.
I originally intended to use different historic yearly weather patterns and make the model randomly choose a historic year to follow each year. My work around for now is to use an average 24hour curve with a yearly pattern of daily averages, essentially cutting down the required data points to 365 instead of 8760, allowing the lookup solution to work. The down side for me here is that I have a repeating daily pattern and thus the hourly outliers are not as visible.
 Attachments

 weather2short.xlsx
 (1.45 MiB) Downloaded 50 times
Re: Lookup or time series for data input
Thanks! We'll experiment.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Re: Lookup or time series for data input
A possible hack would be to edit the xml file containing the entity description with the lookup, and paste the data in there.
/*
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/
Advice to posters (it really helps us to help you)
http://www.ventanasystems.co.uk/forum/v ... f=2&t=4391
Blog: http://blog.metasd.com
Model library: http://models.metasd.com
Bookmarks: http://delicious.com/tomfid/SystemDynamics
*/