Display code overhaul to support multiple people working together and version control for .mdl files

[jrissman]

Hello Ventana team,

I am trying to train other people at my organization to work on the Energy Policy Simulator model. As with most modern software, the model is tracked in a version control system (specifically Git / GitHub). We've run into an issue with the format Vensim uses to store display code inside the .mdl file that makes merging changes made by different modelers almost impossible.

- Simply going to an existing sheet and panning around to look at different parts of the sheet, or zooming in and out, creates many changes in Vensim’s display code for that sheet. It is pretty much impossible to avoid doing these things while working on the model. Even if a developer is confining his/her new work to a new sheet, to avoid editing any old sheets, he/she still needs to at least be able to examine old sheets without causing changes in their display code when the .mdl file is next saved.

- Editing one variable on a sheet or simply adding a comment to a sheet using the "Comment" tool can change hundreds of lines of display code for the variables and comments already on that sheet. This means that two people can't work on different parts of the same sheet, even if they don't edit any of the same variables as each other. (Even work on different sheets isn't safe because you are likely using shadow variables from other sheets, and renaming them will rename them across all sheets, changing the display code on all sheets.)

- Even in the simplest possible case where two developers put their work on two different, brand new sheets and don't pan, zoom, or otherwise look at any existing sheet, the success of the merge of the two .mdl files depends on the Git client you use. The client "Fork" wasn't able to merge files successfully because Vensim expected the following two lines to be repeated above the title of every new sheet:

Code: Select all

\\\---/// Sketch information - do not modify anything except names
V300  Do not put anything below this section - it will be ignored

The lines didn't change, so Fork didn't know that they needed to be duplicated before every sheet. There is no reason why these lines should need to be repeated before each sheet, since the leading asterisk is already a signal that a line denotes the start of a new sheet, as in:

Code: Select all

*Cost Outputs

Sub-models can help by isolating chunks of display code in different files but don't really address the core issue, which is that the display code format itself is not compatible with modern version control systems or allowing multiple people to work on the same .mdl file together.

I'd like to request that Ventana Systems strongly consider adopting a new format for the display code that enables multiple people to truly collaborate in the same .mdl file. I would suggest the following changes:

1. Most importantly, each visible thing (such as a variable or comment) would be fully defined on a single line of code (giving its coordinates, color, arrow connections, etc.) and would not be affected when other variables change, even if those other variables are on the same sheet.

2. There should be no changes to any code in the .mdl file if a user simply views the existing model structure (such as by panning, zooming, or clicking on variables, or producing graphs or tables). Only actual changes to the model such as moving, adding, recoloring, or deleting variables, arrows, or comments should cause changes to the display code.

3. The following lines should not be required, or if required, should only be required once at the beginning of the display code section and not before each sheet name:

Code: Select all

\\\---/// Sketch information - do not modify anything except names
V300  Do not put anything below this section - it will be ignored

4. Anything about the user's state such as the last sheet the user most recently was viewing (denoted with "19:" in the display code) or which variable was most recently selected (denoted with "5:" in the display code) should not be saved in the .mdl file. These cause merge conflicts. When the model is opened, you could simply default to displaying the first sheet, and either start with no variable selected, or the alphabetically first variable starts selected. That is, the most recent user's actions aren't part of the model structure and therefore shouldn't be a part of the actual model code. (If you feel the need to save these things, you could save them in a separate file with a different extension, which we can exclude from source control.)

I know this might seem to be asking for a lot, but being able to work together with others on a single model is a very important capability. Besides, the display code format Vensim currently uses doesn't make much sense anyway, since there shouldn't be dozens or hundreds of changed lines every time a user pans/zooms existing sheets or adds a variable to an existing sheet. A clean, modern display code format that only pays attention to true changes to the model and isolates each variable/comment/etc. to a single line would be so wonderful and important for making Vensim a tool teams can use together.

If you do create new display code along the lines above and need someone to test it, I can run a set of tests for you and report any parts of the code that fail to merge correctly and why. I'm happy to help.

-Jeff

[Administrator]

Do you have the models set up with the GIT/SVN settings? It sounds as if you don't.

Model->Settings->File format, there is a small button top right that fills in the best settings to work with SVN/GIT.

[jrissman]

Hi Ventana team,

Thanks for the guidance! That greatly helps to resolve most of the issues. However, even when using the new settings in Vensim 9.3.5 and the new GUI, there are two problems that remain that you should be aware of.

First, merging still fails if a person adds a new sheet in Git/SVN branch A, and a different person adds a different new sheet in Git/SVN branch B. This is because Vensim requires the following lines need to be repeated before the name of each new sheet, but Git/SVN doesn't realize that there need to be two copies of these lines. (That is, the version control system sees that these lines were added for one new sheet in each branch and think these additions are identical (overlapping), rather than realizing it needs to add these lines twice.)

Code: Select all

\\\---/// Sketch information - do not modify anything except names
V301  Do not put anything below this section - it will be ignored

It looks like the first line here is a comment and the second line specifies the file format (e.g. "V301").

To allow people to cleanly merge models where new sheets have been added in different branches, it would be great if Vensim didn't require these lines to be repeated before every sheet name. This can easily be done with perfect compatibility with older models by adding a rule that says that if Vensim encounters a new sheet name in the display code denoted with an asterisk, like this:

Code: Select all

*Cost Outputs

And if there are no immediately preceding lines showing the file format, then Vensim assumes this sheet uses the same file format as the previous sheet (or the first sheet) in the model (e.g. "V301"). This would even preserve compatibility with old models that do use different file formats on different sheets, if there are any such models. It simply allows Vensim to elegantly handle the case where a new sheet exists but hasn't been preceded by the format lines.

Or, if you want, you could include those two lines only before the first sheet and all sheets refer to the same format lines, assuming there is no reason for a model to have some sheets on V300 and other sheets on V301.

---

Second, your recommended Git/SVN settings include switching equation format to "terse." It turns out, this isn't necessary for merging - we can get merging working fine with equation format set to "As originally entered."

If we do change the equation format to "terse," save the model, quit Vensim, and then open the model in Vensim, it opens in text view and can't enter sketch view. It appears that Vensim's process of converting our equations from their current format to "terse" format introduces syntax errors.

Since "terse" is not necessary for proper merging, this isn't a roadblock for us, but I thought you might want to be aware that switching to the "terse" setting messes up our model equations in a way that prevents the sketch view from being used.

[LAUJJL]

Hi

I do not understand what you mean by display code inside a model. Is it a Venapp code or a script code? Can you give a simplistic model example illustrating a model with display code inside?

Regards.

JJ

[jrissman]

JJ,

By "display code," I mean the section of code that defines the appearance of the variables and comments in the flowchart view, such as their color, coordinates, font style, etc. on each sheet. It comes after the section of code that defines the mathematical equations. As far as I'm aware, every Vensim model contains display code.

Best,
Jeff

[LAUJJL]

Hi Jeff

The display code appears in the flowchart view if you click on a variable; you can modfy it ; it appears too in the model as text at the end of the list. It cannot be modified. Do you agree? What do you name a sheet? Is it what Vensim names a view? Or is it what appears when you clck on a variable where you can change the display code or is it the model as text?
I do not understand either why other people cannot view the model without modifying the display code. Is it because of the Github?

Regards.

JJ

[jrissman]

JJ,

No, the display code is not viewable in Vensim flowchart view. You need to view the model as text to see the display code.

"Sheet" and "View" are synonyms.

No, GitHub doesn't modify the model code. The Vensim application modifies the display code. Earlier versions of Vensim would modify the display code at inappropriate times (such as when a view was panned and zoomed) and in more places than necessary (such as modifying the display code for every variable on a view, when only one variable was edited), but these issues seem to have been largely fixed as of version 9.3.5.

The remaining issue is a small edge case where people add a new sheet in two different branches, and the diff tool doesn't flag the entire block of text for the new sheet correctly because two lines are repeated verbatim. This is a very small issue, since most of the larger problems had already been fixed as of 9.3.5 (and after selecting the correct file format settings). I am suggesting a tiny and backward-compatible tweak to the .mdl file format that would allow Vensim to understand that a view exists even if the name of the view is not preceded by these lines:

Code: Select all

\\\---/// Sketch information - do not modify anything except names
V301  Do not put anything below this section - it will be ignored

[LAUJJL]

I will look at yout post to-morrow. Too late for me to think. Good day.

JJ

[LAUJJL]

Hi

The display code is not viewable in the flowchart view, but if I right clkck on a variable of the view, you can view the entire display code of the variable and can change it, something that I often do; i never change anything in the text model below the warnings Sketch information etc..
What is a 'branch' and the 'diff tool'?
I had sometimss problem with display codes but I do not remember having changes on the display code of other variables when changing the dispolay code of a specific variable and certainaily not if i was just viewing the code by right clicking on the variable.

It would be good if you could post a simplistic model that ilustrates your problem and explaining the different actiions that generate a problem.

Regards.

JJ

[jrissman]

Hi JJ,

Thanks for trying to help on this issue. However, it would be necessary for you to become familiar with Git clients, GitHub, and the process of merging changes to code files from different Git branches before it would be possible to understand the issue I am reporting.

The real purpose of this thread is to flag for Ventana Systems that their file format improvements to support version control systems are excellent, but there is at least one lingering problem pertaining to new views defined in different branches, which could be solved if Ventana Systems updated their file format. I've suggested a backward-compatible way a future version of Vensim could better support this specific case. Unfortunately, this is not something another user can assist with, as it requires Ventana Systems' action on Vensim's file formats.

Best,
Jeff

[LAUJJL]

Hi Jeff

Thank you for your explanation. Working alone, I do not have to manage different people simultaneously modifying a source code that has some advantages and some drawbacks too.
Anyhow good luck for your project

Regards.

JJ

[Administrator]

The remaining issue is a small edge case where people add a new sheet in two different branches, and the diff tool doesn't flag the entire block of text for the new sheet correctly because two lines are repeated verbatim. This is a very small issue, since most of the larger problems had already been fixed as of 9.3.5 (and after selecting the correct file format settings). I am suggesting a tiny and backward-compatible tweak to the .mdl file format that would allow Vensim to understand that a view exists even if the name of the view is not preceded by these lines:

Code: Select all

\\\---/// Sketch information - do not modify anything except names
V301  Do not put anything below this section - it will be ignored

What client are you using to compare files?

I'm not sure we can actually do anything about this. Can you merge one view at a time? Would that work?

[jrissman]

I am using the Git client "Fork" to compare files. The website for Fork is https://git-fork.com/

Here are the steps to reproduce the issue:
* Start with a model in the source control system and make a new branch from the latest commit.
* Activate the new branch, then open the model in Vensim. Add a new view. I called the view "Pet Adoptions" and I added some variables and arrows.
* Save the changes to the model, then commit the updated .mdl file to the new branch.
* Activate the original branch, then open the .mdl file in Vensim. The "Pet Adoptions" view doesn't exist here because this branch doesn't contain that edit.
* Add a new view. I called the view "Bakery Simulation" and I added some variables and arrows.
* Save the changes to the model, then commit the updated .mdl file to the original branch.
* In the Git client, merge the old branch into the new branch.

There will be no merge conflicts in the equations section of the code, assuming all your variables are uniquely named. But there will be a merge conflict in the display code that looks like this:

MergeConflict1.png (195.72 KiB) Viewed 6803 times

The Git client thinks these might be different versions of the same code. We want to tell it to include both code blocks, because "Pet Adoptions" and "Bakery Simulation" are completely different sheets, and we want to keep them both. To do that, we simply check both boxes (on the left and the right in the git client's merge tool), which tells the client you want to include both code blocks in the merged file (at the bottom). The result looks like this screenshot:

MergeConflict2.png (250.22 KiB) Viewed 6803 times

This almost works! It's so very, very close to working perfectly. The problem is that Vensim won't see the second of the views listed here (the "Pet Adoptions" view) because its code block (highlighted in pale purple) doesn't have the required two lines repeated above it.

It will work if we manually edit the code to add the missing lines. They are the ones I've marked with red arrows in this screenshot:

MergeConflict3.png (262.26 KiB) Viewed 6803 times

Most users will not know that they need to manually edit the sketch information in this way. They will simply include both code blocks and complete the merge. Then they will discover that Vensim does not see the "Pet Adoptions" sheet, and they won't know how to fix the problem. They will think that merging is not possible, when it really was as simple as repeating these two little lines:

Code: Select all

\\\---/// Sketch information - do not modify anything except names
V301  Do not put anything below this section - it will be ignored

The funny thing is that these lines do not look important. Almost everything in them is a comment meant for humans, not a meaningful piece of code used by Vensim. The only part of these lines that appears to be intended for Vensim is the "V301" text at the beginning of the second line. This appears to be an indicator of the file format version used, since using file format from pre-9.2 Vensim puts "V300" here instead of "V301".

Generally, a single model has to use the same file format on all views of that model. That is, I would not expect any model to use "V300" on some views and "V301" on other views in the same .mdl file. Therefore, it seems like one could include these lines just once, at the top of the sketch information section, and assume that the "V300" or "V301" there applies to all views in that .mdl file.

But suppose there is some wacky model out there that has some views using V300 file format and other views using V301 file format. You can still support that wacky model. Vensim can continue to check for V300/V301 ahead of every new view declaration in the sketch information section. All that is needed is to change Vensim's error handling behavior: that is, What does Vensim do if the V300/V301 information for a particular view isn't found?

Today, what Vensim does is that it ignores the view declaration entirely, so the new view ("Pet Adoptions") and all the variables in that view don't show up anywhere in the model sketch. We know that's not the result the user wanted. Instead, if a new view declaration in the sketch information section is missing the V300/V301 file format designator, Vensim could assume that the new view uses the same file format as the previous view. This would allow Vensim to gracefully handle cases where those two lines are missing, a situation that can arise because of Git client merge tools.

Alternatively, one could envision simply including the file format information on the same line as the new view declaration. For example, instead of this to declare a new view:

Code: Select all

\\\---/// Sketch information - do not modify anything except names
V301  Do not put anything below this section - it will be ignored
*Pet Adoptions

You could instead have a one-line solution like this:

Code: Select all

*V301*Pet Adoptions

Merge tools should be able to handle the one-line version better. It also avoids repeating wordy comments ahead of every view. The downside of this approach is that it might not be backwards-compatible. I was assuming you might want to maintain backward compatibility rather than to update Vensim's .mdl file format, so changing Vensim's error handling behavior when it encounters a view declaration that lacks the preceding file format version info seemed like an easy way to achieve that.

That said, I don't know Vensim's code internals or what constraints you may have when parsing the sketch information, so if it's not possible, then it's not possible.

Jeff