Page 1 of 1

Lookup or time series for data input

Posted: Tue Apr 17, 2018 2:52 pm
by paulvbaal
Hi,

I am having some issues trying to integrate data in Ventity. I have weather data in hourly values for a single year, i.e. 24*365 = 8760 data points, that I need to use in my model, repeatedly over several years, such that the value on, for instance, January 3rd, 4pm, will always be the same data point.

My initial idea was to do this via a lookup function specified via an excel sheet, using Modulo(Model.Time,8760) as input and the hourly values as output. This works to a certain extent, but since Ventity external data input files require the entire lookup function to be contained in a single cell, only about a fifth of the sequence fits (Excel has a character limit per cell of 32767).

The second idea was to input the data as Time Series data and use a Data Entity to access the values, however, there is no way to "repeat" the values once the first year is over, as far as I have been able to figure out.

Is there a functionality or workaround in Ventity that could solve my problem? I know that in Vensim it is possible to specify Lookup functions in Excel in a different format, where each row would be a new data point. This made it easy to access this type of longer data sets.

Re: Lookup or time series for data input

Posted: Tue Apr 17, 2018 3:16 pm
by tomfid
Understood. I've put in a ticket for an improvement that addresses this.

In the meantime, what if you create a time series that repeats the lookup data N times?

Re: Lookup or time series for data input

Posted: Wed Apr 18, 2018 12:47 pm
by paulvbaal
I tried doing that but the excel files quickly became too large for Ventity to handle (at least on my computer), as my simulation runs 30 years, so I need nearly 300'000 data points for several variables. I'll try to find some parametric way to estimate the data points for now, but please keep me in the loop about any updates.

Re: Lookup or time series for data input

Posted: Wed Apr 18, 2018 1:57 pm
by tomfid
If you have a file that's big enough to choke the Ventity data input, I'd love to have a copy for testing.

Re: Lookup or time series for data input

Posted: Thu Apr 19, 2018 2:20 pm
by tomfid
We're exploring options on this.

After some reflection, I'm wondering why you need to run hourly for 30 years. I'm assuming this is electric power, but no matter what the domain, running a tactical time step on a strategic horizon is tough. It implies that the long term model has to have a lot of short term detail that mostly gets in the way.

Is there another way to approach this problem?

Re: Lookup or time series for data input

Posted: Mon Apr 23, 2018 8:50 am
by paulvbaal
I use the hourly time step to evaluate short-term effects like intra-day price volatility and power shortages. Zooming out to daily/weekly/monthly/yearly time steps will make that much harder. The 30 year horizon is to see the long-term effects of policies on these effects.

I attach the document with the data for 1 year. I cannot upload the long one that breaks Ventity as it's too large for the forum, but it's basically this spreadsheet copied down 30 times.

I originally intended to use different historic yearly weather patterns and make the model randomly choose a historic year to follow each year. My work around for now is to use an average 24-hour curve with a yearly pattern of daily averages, essentially cutting down the required data points to 365 instead of 8760, allowing the lookup solution to work. The down side for me here is that I have a repeating daily pattern and thus the hourly outliers are not as visible.

Re: Lookup or time series for data input

Posted: Mon Apr 30, 2018 10:59 pm
by tomfid
Thanks! We'll experiment.

Re: Lookup or time series for data input

Posted: Mon Apr 30, 2018 11:01 pm
by tomfid
A possible hack would be to edit the xml file containing the entity description with the lookup, and paste the data in there.