Page 1 of 1

Precision question

Posted: Fri May 12, 2017 10:35 pm
by billh
I've attached a simple model that replicates an issue I'm having in a larger model. It's got three stocks, initialized to 315400, 5810, and 600 units. There's a Total variable that does what it implies.

However, it says the total is 321800, while my calculator and my brain say it should be 321810.

Is that the single-precision DLL at work, or is there something else going on? I'm running DSS 6.4E (x32).

As far as policy goes, this is a non-issue. I found it as part of a mass balance check to make it easy to see that I was initializing stocks to get the desired total. I knew the total I wanted (321800), and so I thought the initialization was working until I added the numbers manually.

Thanks,

Bill

Re: Precision question

Posted: Fri May 12, 2017 10:38 pm
by Administrator
If you right click on the table tool and select "Scientific" instead of "Pretty" for the number format, does the correct value show up?

Re: Precision question

Posted: Fri May 12, 2017 10:43 pm
by billh
Thanks--much better.

I learned something!

Bill