Loading Data, Constants, Lookups from .csv files in WebAssembly

[jrissman]

Hello Ventana Team,

We at Energy Innovation are extremely interested in the new Vensim 8.1 feature that can convert a model to WebAssembly, to run in a browser. As you may recall, for the Energy Policy Simulator (EPS), we currently use a Linux multi-context version of Vensim running on a server to perform model runs when they are requested by a user. This means the user must send the parameters to the server, the server runs the model, and it sends the results data back to the user's browser. This is a slow process, due to network latency and also the time required for the server to perform a Vensim model run.

The possibility of converting the model to WebAssembly so that it is running locally in a user's web browser has the potential to dramatically speed up model performance, by eliminating latency and because modern browsers execute WebAssembly code very quickly (much more quickly than the server can complete a Vensim run). This means that interacting with a Vensim model online could be a much faster and more pleasant experience for users, while also allowing us to simplify our backend server architecture, which could save us money every month in Amazon Web Services hosting fees.

The EPS is a large model. It loads in constants, data, and lookups from hundreds of different .csv files, and it uses a wide variety of the various functions available in Vensim's programming language. I noticed in Vensim's documentation of the new WebAssembly converter feature that "All data and constants must be contained within the model. At present, Vensim WASM models cannot load external data or constants." That particular limitation would prevent the EPS from being converted to WebAssembly.

Emscripten provides a virtual file system that simulates the local file system, so that "code that uses traditional file APIs can then be compiled and run with little or no change." Emscripten documentation discusses their file system implementation here and explains how to package the files that populate the virtual file system here. It sounds like this might be a good solution that allows support for reading in external data files by the WebAssembly code running inside a browser sandbox.

Do you have any plans or timeframe to support the use of external data files in models converted to WebAssembly?

We would be happy to help debug the WebAssembly converter feature by trying to use it to convert the EPS to WebAssembly, and reporting to you on any errors or issues we encounter. If you can get your WebAssembly converter to successfully convert the EPS, it will probably work on almost all Vensim models, so the EPS would make a good test case for this new Vensim feature.

Jeff

[tomfid]

This is definitely very high on the roadmap list. The functions are generally no problem; just the external constants and data have been a challenge. I can't say what the timing is at this moment, but we'll know a lot more in a couple weeks.

[Administrator]

We've been working on supporting external data. We've got to cut the code out from Vensim that handles all the different data formats and re-compile to WASM which is proving to be quite a task.

[Travis]

Jeff -- why aren't you and Todd going with SDEverywhere? Is it not working with your model? It already supports external data files.

[jrissman]

Todd indicates that SDEverywhere doesn't include support for all the Vensim functions that the Energy Policy Simulator uses.

It is easier for Ventana Systems to add support for Vensim functions because they possess the original Vensim source code that shows how the function is implemented, whereas SDEverywhere must try to recreate the function from scratch and get the results to match Vensim. That may be simple in the case of straightforward functions, like MAX(), but it may be difficult in the case of more intricate functions, such as ALLOCATE AVAILABLE().

Also, Todd has indicated that SDEverywhere's method for handling external input data files is to do it in JavaScript at compile time, but a better implementation would be for the C runtime to do file I/O the same way that Vensim desktop does it. WebAssembly support for virtual file systems is relatively new, so there might be new possibilities for handling file I/O cleanly and with high performance that were not possible even a couple years ago.

That said, we would likely look into using SDEverywhere if Ventana Systems' official WebAssembly export capability turns out to be buggy, doesn't produce a high-performance Wasm app, or isn't compatible with the Energy Policy Simulator. We aren't in a big hurry and can wait for Ventana Systems to finish file I/O support and the official WebAssembly export capability before evaluating it and deciding if we can use it, or if we must look at alternatives.

So we are glad SDEverywhere exists as a possible fallback option, but we think the ideal thing would be for the model export capability to have first-party support from Ventana Systems and be built into Vensim.

[Travis]

Yep -- Todd knows the pain of implementing all the Vensim functions. And I agree that first-party support of model export could be better and is sorely needed. It would have been great if WASM (and full C export) were added to Vensim years ago.

The bright side is that SDE is allowing Climate Interactive to start continuous integration checks and other quality control measures. We can do this easily in the cloud (GitHub Actions) because we don't need the Vensim DLL. We'll be bringing this functionality to SDE and/or releasing this open source to the community "soon". I hope to have it working and ready for community input by the end of the year.