Ventana software support forum

Dear All,
I am starting a new project that would include many iterations and adjustments of a complex model over coming months and need tools to ensure changes are not breaking other parts of the model, likely using an automated suite of tests. I am wondering about the value of Vensim's built in reality checks vs. SDeverywhere for having such an automated set of tests. Any insights on best practices here? I know SDeverywhere is doing a bunch of other things for online integration and user interface, but the question is if reality checks add significant relative value beyond what is feasible/easy to do in SDeverywhere (I have not used reality checks for a long time and don't know the sdeverywhere capabilities much). Any insights are much appreciated!
thanks,
hazhir

I cannot comment on SDEverywhere (I totally failed to make it work the one time I tried, but that was quite a while back). I believe the SDEverywhere checks are more geared towards checking that it is working correctly rather than checking a model is doing what it should.

A really useful check I always build into my models is a mass-balance check. It's very easy to drain too much from a stock, or drain from a stock and move things to two other stocks. Mass-balance just makes sure you don't gain/lose anything in the system. Sadly I don't see them in many models, probably because they need implementing rather than being automatic (like the units check, saying that, I see too many models that don't even have units, a lot of users don't even bother with the basics). Especially on large models, a mass balance check will show quickly if you are double counting anywhere or losing things in the system.

Reality checks are obviously good to use. The only drawback is that you can easily create hundreds of checks.

One other thing that might be handy. The new run-config tool in Vensim (https://youtu.be/GLrZ-Xl59tA). With this you can set up a number of scenarios and run all of them with a single click. Could be useful if you make a change in the model and need to run it with small changes to see the results. You can tag a command script onto the end of a run that can export results and do a runs-compare. One model I'm currently working on has three scenarios, running all three each time I make a change used to be a chore (even though I used a command script).

Hope this helps. It's more general thoughts than actual advice though.

Great thoughts and suggestions, thank you!

The OP was right that SDEverywhere [sdeverywhere.org] does have functionality to track changes across model evolution. It is really powerful.

I'll give a short summary here.

MODEL COMPARE
* when you build a model, you can save off a version that is the version to compare to. Sometimes you think of this as the "last known good state", if the model is more mature.
* The tool can compare between the current model results and this saved state.
* The tool can also compare between any two saved states
* You can use this feature with or without Git integration
* The results are presented on a webpage (local page, if not using GitHub) with the difference in model results sorted by the variables that have changed the most.
* These variables can be clicked so you can see graphs and browse the model changes, going down the causal rabbit hole.

UNIT TESTS
* In addition to comparing the behavior over time of two different runs, there are "unit test", which are similar to reality checks.
* You can define these numerous ways, like "Value should be X in 2030" or "Value should be between X and Y always". Lots of different conditions and tolerances.
* The results from these test are also presented in the web page, with green checks (pass) or red crosses (fail).

BTW, all these tests are written in plaintext files with idea that, if Vensim or another SD tool wanted to use them as a portable way to have checks -- something that is tool-independent so the SD community could share a model, tests, data, etc., in a open manner.

There isn't currently great documentation of the process yet (forthcoming), but the default project "QuickStart" guide gives you the basic structure.

Happy to answer more questions. And anyone interested in a webinar to see this? Maybe we should do one.

Hi

I have been using 'reality checks' for years. They have two disadvantages; they are interpreted then slow and at the time, they could not be run in batch mode for example by launching multiple runs from Excel using DLLs. It may be possible that we can now run reality checks in batch mode with new commands using them.

If I want to build tests I now prefer to write them in Vensim simply, I have better control of what I do and it works just as well. Regarding the choice of tests to be done, it depends a lot on how the model is built. I now build my models in a strictly 'top doxn' way and at each step, I analyze the model, its usefulness and its reliability by building at each step 'tests' if I have any doubts about the model.

I think that if you build a model to the end without stopping at multiple steps, it is impossible to build an efficient set of tests, especially if the model has hundreds of equations and eventually heavily subscripted.

JJ

Thank you Travis and JJ for thoughtful comments.
Travis, the SDEverywhere functionalities you note (and some of which I had seen a while back in your talk) are really helpful, and any additional documentation/demo/seminar to enable regular users to benefit from those would be a great step.