Can I take in input data paths from another input data file?

[jrissman]

I am curious if there is a way to get Vensim to read the paths (or at least the filenames) of CSV input data files from a different input data file, and then to use those filenames in a GET DIRECT DATA equation. This would allow for a user to change which input data files are read into my model by changing the contents of one input data file.

For example, suppose I have a model that calculates properties of the urban layout of cities. Suppose I've made a version for Arizona, and my file structure looks like this:

Code: Select all

CitySim.mdl
CityFileList.csv
Cities/Phoenix.csv
Cities/Tuscon.csv
Cities/Flagstaff.csv

Suppose the contents of CityFileList.csv are:

Code: Select all

Cities/Phoenix.csv,Cities/Tuscon.csv,Cities/Flagstaff.csv

Now in my model, CitySim.mdl, I want a variable to use GET DIRECT DATA to read from whichever files are specified in CityFileList.csv.

I imagine this would require two variables. For example, imagine there were a Vensim command called GET DIRECT STRINGS, and it reads the file paths inside CityFileList.csv in as text strings. Based on the GET DIRECT SUBSCRIPTS syntax, it would be something like:

Code: Select all

City Files[city] = GET DIRECT STRINGS('CityFileList.csv',',','A1','C1')

Then, imagine that the GET DIRECT DATA command could accept values from the City Files variable as paths to the files to read:

Code: Select all

City Properties[city1] = GET DIRECT DATA(City Files[city1],',','1','B2')
City Properties[city2] = GET DIRECT DATA(City Files[city2],',','1','B2')
City Properties[city3] = GET DIRECT DATA(City Files[city3],',','1','B2') 

Is there any way to accomplish reading from files whose paths and filenames are specified in input data, rather than hard-coded into the Vensim model?

[Administrator]

You should be able to use a string variable to hold the file names, these can be changed in a CIN file.

[tomfid]

I think you can already do this by (a) making the filename a string, and (b) updating the string via a change file prior to simulation.

So, your model would have:
filename :IS: 'default.xlsx'
data = GET DIRECT DATA( filename, 'tab', ...

And the .cin would contain
x=3.141
filename :IS: 'alternate.xlsx'

http://vensim.com/documentation/cin_files.htm

[jrissman]

Hi folks,

Your suggestion works when using the "Sim Control" button to select the .cin file prior to simulation, but it fails if I try to load the .cin file once I'm already in SyntheSim mode. (The change to the string variable is silently ignored when loading the .cin file once already in SyntheSim mode.) I've attached a simple test model where you can test this out. Is this a bug in Vensim?

Most people use SyntheSim. Few use the "Sim Control" button. So this would affect most users.

Another problem is that the filename :IS: 'alternate.xlsx' directive inside the .cin file is eliminated if I overwrite the .cin file from within SyntheSim mode, after adjusting some numerical variable settings, even if I had successfully loaded the filename :IS: 'alternate.xlsx' directive using "Sim Control" prior to entering SyntheSim mode. This means I can't make changes to any of the variables and save my changes as a .cin file without losing the path to the input data file.

Jeff

TestModel.zip

(858 Bytes) Downloaded 237 times

[tomfid]

Your suggestion works when using the "Sim Control" button to select the .cin file prior to simulation, but it fails if I try to load the .cin file once I'm already in SyntheSim mode. (The change to the string variable is silently ignored when loading the .cin file once already in SyntheSim mode.) I've attached a simple test model where you can test this out. Is this a bug in Vensim?

It's not a bug. It might be possible to change the behavior, though.

Another problem is that the filename :IS: 'alternate.xlsx' directive inside the .cin file is eliminated if I overwrite the .cin file from within SyntheSim mode, after adjusting some numerical variable settings, even if I had successfully loaded the filename :IS: 'alternate.xlsx' directive using "Sim Control" prior to entering SyntheSim mode. This means I can't make changes to any of the variables and save my changes as a .cin file without losing the path to the input data file.

This does sound like a bug. As a workaround, you could put the filename in one .cin in a hidden or locked location, and put the numbers in a different .cin.

Another option might be to subscript the data by city, and then use VECTOR ELM MAP to pick the active element needed.

[jrissman]

Thanks, Tom. The elements I care about are already subscripted, so I'll look into the workaround involving VECTOR ELM MAP. I assume the goal is to be able to save a constant value into the .cin file instead of a string, because we know that Vensim handles contants in .cin files successfully.

Is VECTOR ELM MAP the simplest function to select an element of a subscript based on its index position within the subscript, instead of its name? That is, to obtain the third element of a subscript (with zero-based indexing), I can't write something simple like:

Code: Select all

Chosen City = 2
Path to File of Chosen City = Paths to City Files[2]

I read the help documentation on VECTOR ELM MAP but haven't yet experimented to figure out how to use it to perform the operation shown above, that is, selecting an element from a vector based on a constant.

Also, is there a better way to load a bunch of strings into a subscripted string variable than giving each string its own equation? Something like GET DIRECT SUBSCRIPTS, except meant to act upon a file containing strings and to populate a string variable? (Aside from brevity, one advantage the GET DIRECT... equations have is that they neatly handle an arbitrary number of items. So if the number of strings in my input data file varies, I don't have to deal with a mismatch between the number of equations )

[jrissman]

In the post above, the first code block was supposed to read:

Code: Select all

Chosen City = 2
Path to File of Chosen City = Paths to City Files[Chosen City]

(I know this code doesn't work, it's just an example of simple syntax.)

[tomfid]

Typical application would be something like this:

City : Sodom, Gomorrah, WashingtonDC
Active city: 0 ~ this is an integer selector; arrays are 0 based in VECTOR ELM MAP, so 0 <= x < ELMCOUNT(city)
ConstVal[city] = 4,5,6
Active ConstVal = VECTOR ELM MAP( ConstVal[sodom], Active city )

It's a little trickier if the city isn't the last dimension. Then you have to calculate a more complex offset - I think the example in the help system covers that.

[jrissman]

Hi Tom,

I tried this approach. It works for selecting numerical values, but it fails when trying to select strings. Vensim gives the error:

Argument 1 to function VECTOR ELM MAP must be a normal variable.

In your example, your third line is:
ConstVal[city] = 4,5,6

But I need to select file paths, which are strings, not numbers. So I have a subscripted string variable, using multiple equations, one per string. Consider this example:

Active City=
0
~
~ |

city:
Phoenix, Tuscon, Flagstaff
~
~ |

Path to Data File for Active City=
VECTOR ELM MAP(Paths to City Data Files[Phoenix], Active City)
~
~ |

Paths to City Data Files[Phoenix]:IS:
'data/PhoenixData.csv' ~~|
Paths to City Data Files[Tuscon]:IS:
'data/TusconData.csv' ~~|
Paths to City Data Files[Flagstaff]:IS:
'data/FlagstaffData.csv'
~
~ |

This fails, with the error noted above. But when I assign numerical values rather than strings to each elements of Paths to City Data Files (and change the variable's type from String to Constant), it works.

So I still can't figure out how to use a numerical setting to select a particular string. Do you have any other ideas of functions that might successfully be able to select a string from a vector of strings? Or some other way to use a numerical value saved into a .cin file to select a file path from a set of options?

[LAUJJL]

Hi

Instead of selecting a string from a vector of strings, something that Vensim seems not to be able to do (the reason might come from the fact that string variables have different length and are then more difficult to spot) select the data that come from the strings.

Joined a packaged model (that can be opened with the vensim reader too) using the get vdf data that works like the get direct etc..

You can use the synthesim and vary the file from where come the data during the synthesim using the active file slider.

The file?.vdf have been generated with the string.mdl replacing the 'file?.vdf' by '' in the get vdf function.

You will notice that I could have used a vector elm map function to choose between the data, but I prefer to use a sum ( ) that is easier to understand, does not need complex calculations when there are many subscripts to calculate the offset and does not risk fetching data outside of the place where the variable is stored. I never use vector elm map.

Regards.

JJ

[jrissman]

Hello JJ,

Thank you so much for the suggestion and the sample model. It is an exceedingly clever approach, and it has enabled me to solve my problem. You are awesome.

For anyone else reading this thread in the future, although JJ's example pulls data from VDF files, his approach also works for other types of source data, such as data obtained via GET DIRECT DATA() from .csv files.

Jeff

[tomfid]

I think you've identified a capability gap, so I'll put this on the todo list.

[LAUJJL]

Hi

in fact there are two problems with strings in Vensim.

First, a string cannot never been equal to a formula. It can only be equal to a string constant surrounded with quotes. It is then impossible to change its value based for instance on another numeric value.

The second problem is that it seems impossible to change the value of a string during a synthesim session, except at the beginning before entering the synthesim at the simulation setup, clicking on the string variable or on the setup button.

The result is that you must do prior to the synthesim everything that you cannot do when in the synthesim.

And even if you can change a string during a synthesim, nothing guaranties that the software will download again anything via the get … because the name of the string representing the file has changed. Vensim eventually downloads the data at the simulation set up only.

See the string2 model, where it is easy to modify the file to fetch when in set up prior to the synthesim but impossible to do afterwards, even if you click on the C=1 button where you can change the name of the file, but it has no effect on the message nor on the data downloaded.

Regards.

JJ