Connecting with data

[sarahboyar@gmail.com]

Hi there,

I'd be grateful for links to past posts or direct advice.

1. I am working in Vensim Pro and have a model with 3 dimensions - Age, Operation, Health Trust. This comes to nearly 7000 specific dimensions in my model. Please don't comment on the appropriateness of these categories, it will eventually be a SD model theoretically, in terms of stocks and flows and feedbacks; just need the dimensions for client purposes as well.

2. I'm antiquated and working with linking my data via GET XLS DATA. It is crashing Vensim, in particular because I have 105 single-years of age. (Yes, I do sum up in subranges and can work with element, subranges, subscripts, and subscript mapping just fine. In fact, this is one of the reasons I've stuck with Vensim, finding R and Py to be a bit clunky in different ways, in a corporate context).

3. This GET XLS DATA is both for timeseries data, but it is actually important that I can choose a base year for different scenarios. The GET DATA AT TIME function crashes in particular when I tried to just get various base years out of the timeseries.

4. Even more humiliating, to import the data, I have 60 equations in one variable because I only know how to import this way (via the old .xls not even .xlsx)..... the first subscript is the 'master' and then i need to write new equations for each other pair of subscripts. Obviously I made 'age' the first subscript, or it would have been a lot more equations (age has the most elements).

5. Typically, I would run this model to import the data, and then save the .vdf file and create a new model for the real work. But I seem to be maxing out the potential of this method, and would love a cleaner way to import data as arrayed by my data scientist. CSV? How would it be formatted?

6. I do use .cin files for changing model constants and parameters. Just saying.

Any help much appreciated. Basically, I need to import a lot of data, and then it's a simple SD model, but with 7000 unique dimensions and (hopefully, eventually) calibrated to an imported timeseries so that I can initiate the model in equilibrium and then run scenarios where I disturb the equilibrium.

Thank you,
Sarah

[Administrator]

I don't really follow any of this.

But I'd suggest CSV as the import type. You can have something like

time,,,,0,1,2,3,4,5,6
data,sub1,sub2,sub3,10,20,30,40,50,60,70

[sarahboyar@gmail.com]

Thanks. So how does the import work? In an equation? Or via the sim control as a data source? In your example, is 'data' the variable name?

[sarahboyar@gmail.com]

Is there a sample model in the documentation for this?

[tomfid]

There's a chapter in the User Guide, https://www.vensim.com/documentation/21160.html

Last year's conference workshop is online https://vensim.com/conference/#using-data-in-vensim

[tomfid]

If the Age x Operation x Health Trust matrix has a lot of empty elements, Ventity might be a good alternative.

For example, a while ago I prototyped a model of consumer products, with detail like Market (country) x Flavor x Size. Not every flavor x size is offered in every country, so the full matrix has a lot of empty cells. In array-based languages you have to simulate 0s for those. In Ventity, you can just create a product entity, and define it with data only for the ad hoc combination of attributes that actually exists.

If it's possible to replicate the crash in #3 with a reduced example (fake data if necessary), we'd love to track that down. Using xls could be a cause, just because it gets less use than xlsx these days. The function should work for either format. You can use the '?filename' trick to point to the file easily.

[tomfid]

I think the issue you raise in #4 may be avoidable if you put cell addresses in an arrayed string function, but direct import from .csv or .tab is likely to be cleaner.

[tomfid]

In the example, 'data' is a variable name.

There are a lot of possible layouts, including Tidy an relational, but typically something like this is easiest:

2023-05-09_10-07-17.png (21.94 KiB) Viewed 1485 times

In the import dialog, you'll specify the "sub" columns as subscripts. The range names in the first row don't matter; only the element names below are used, and some can be blank for variables that don't have full detail.

[sarahboyar@gmail.com]

Thank you for all the advice.

The biggest learning was to learn to use data import: Model>Import dataset....

- I note that the subscripts in the variable probably need to be in the same order as the column layout in the csv/excel/whatever.
- Once imported, the data still needs to be set as a .vdfx (or .vdf) run in the simulation control (advanced tab >> data sources) in order that it can be used to drive other variables.