Checking for NA input data with subscripts

[AurelienP]

Hi,

I am trying to migrate a model from version 6.3E to version 10 (yes I agree this was long overdue).
I encounter the following issue:
when testing an input data variable for :NA:, it sometimes seems not to work, if the variable is with multiple subscripts (but I want to use 1 specific element for at least one subscript).

With examples it is probably clearer:

Example 1:

Code: Select all

data input[countries,fuels] :INTERPOLATE: ~~|
variable[countries] = IF THEN ELSE(data input[countries,FUEL1] = :NA:,1,2) ~~|

I obtain 2 here, even if data input is :NA: for all countries and all fuels.
I also obtain 2 if there is some data, which makes sense.

Example 2:

Code: Select all

data input[countries] :INTERPOLATE: ~~|
variable[countries] = IF THEN ELSE(data input[countries] = :NA:,1,2) ~~|

Here it seems to work properly.

Any hint on why example 1 would not work?
It used to work as such on version 6.3E.

Thanks in advance!
Aurélien

[Administrator]

Can you upload the data file?

[tomfid]

If you want to check for :NA: in a data variable, you need to use the :RAW: interpolation method. :INTERPOLATE: only returns :NA: if the series has no data at all.

[AurelienP]

Thanks a lot for both your replies.

I ran a few additional tests and I believe I found the issue.

Changing from INTERPOLATE to RAW does not solve my issue (I actually tried that before).
Also, remember that the exact same model and cmd would run in version 6.3E without any problem.

But then I realised that I was using VDF format for my input file, and not VDFX, and for some reason the "NA" from the VDF file were interpreted as a "-nan" string in the VDFX format.
It took me a while to figure it out because:
- when displaying the Causes table, the data variable was not showing anything (i.e. not "-nan", so I thought it was OK)
- when converting the VDF into TAB or XLSX, the line was empty as expected
But then (and thanks to this I understood):
- When exporting the failed simulation output VDFX into XLS (with a reduced savelist), I noticed the "-nan" values instead of empty series.

So basically, I had to manually convert the VDF into VDFX by exporting it to TAB and then convert the TAB to VDFX.
Then everything worked as planned.
So I guess it is my mistake for trying to use a VDF format instead of VDFX with the new version! It was silly from my side here, I realize.

Side question: any better way to convert VDF to VDFX than goind through TAB format?
The VDF to TAB was quick, but then TAB to VDFX took around 1h to convert (VDF size = 500 Mo -> TAB size = 990 Mo -> VDFX size = 1.1Go).
From browsing the forum it seems that it is the way to go, although it is not ideal, but please let me know!

Thanks & best regards
Aurélien

[tomfid]

I think your instincts were good here.

Leaving the data in .vdf format should normally work. It's possible that there was a leak in 6.3 that allowed the NAN values to get into the vdf in the first place.

For doing a round-trip vdf/vdfx conversion, .dat format is often the easiest, because it doesn't require any format specs. However, in this case I'm not sure it would be faster.

The doubling of size is expected, because vdfx is double precision, whereas vdf is single. If the size is an annoyance, it might actually work to reimport the tab file to vdf using 6.3, assuming that my 'leak' idea is correct, and the tab has properly omitted NAN or NA entries.