random seeds and infrequent events

[nhoward]

A model generates an event using Random Poisson, multiplies it by an amount sampled from Random Normal and sends it into a flow. Sometimes there might only be a handful of such events. The average ‘amount’ per event varies between runs using different seeds and can have values consistently greater or less than those of another stream. In the attached, Chart 1 shows average request amount of two runs (from a group of ten) of 100 Monte Carlo simulations each.
Run 1 used a seed of value 1 and Run 3 used a seed of value 3.

A total of 13 requests were made in each simulation in both runs. (Only one other run of the ten produced a different number of requests – it shared a random seed with another variable in that particular run.) The amount of material made in the two above runs described a similar pattern, with run 1 producing a significant amount more than Run 3, in Chart 2, attached.

Was it a quirk of low numbers that one run sampled lower amounts than the other? The input average amounts in Chart 3, attached, reveal Run 3 had the lower values, not Run 1.

Run 3 averaged 0.19 per sample and Run 1 averaged 0.47 per sample, with the mean for the ten runs coming to 0.42. In fact, Run 3 is below the 5% significance point. Yet for means of amounts per request made, Run 3 averages 0.59 per request and Run 1 at 0.41 per request, with the overall mean at 0.50 per request: both Run 1 and Run 3 lie outside the 5% significance points.

But are the pseudo-random streams not quite random enough? In Chart 4, attached, is output from three streams of Random Uniform function between 0 and 1, for seeds 1, 8 and 9. There three runs (of seeds 1, 8 and 9) average 0.19, 0.77 and 0.51, respectively. The amplitudes of the sequences of each stream are for the most part within relatively narrow ranges. A group of ten runs (seeds 0-9) averaged 0.51. The means of runs 1 and 8 lie outside the 5% significance points.

I hope this is clear enough. The charts wouldn't paste above but I can post a doc with them within the text if you prefer. Thanks.

[tomfid]

The uniform random results look particularly wrong, in that they should all have the same ranges, means & variances, unless the parameters (other than the seed) differ. I've tested sequences of 2 billion uniform random draws with the NIST test suite, and they were OK.

I think I understand what you're trying to do, but not sure I follow the implementation. Can you post a model, or equations for the random draws?

[nhoward]

Please find a copy of the model used attached. Results were got using version 6.3 PLE+ and exported final time data to tab file, parsed and saved in Excel, and have replicated them today. Perhaps I've missed something but can't see it. Good luck with it.

[tomfid]

Got it - I have to hop on a plane shortly but will have a look ASAP.

[tomfid]

Are the "average" results from your word doc from a Monte Carlo sim, with different seeds?

[nhoward]

Each of the first three charts shows results from two Monte Carlo runs using different seeds for 'Random Seed 2' in the model. Each line in a chart is of 100 points, an average value for each simulation in the run.

[tomfid]

Ahh ... now I get it.

[tomfid]

... or not. Looks OK to me.

What's in your .vsc sensitivity control file?

[nhoward]

Hi

.vsc file attached. It's as per this morning, repeating yesterday's results. Is this ok or do you need more data?

[tomfid]

That should do it.

[tomfid]

It's not a bug thankfully, but you've discovered a rather non-intuitive interaction among the random streams. Here's what's happening:

Since your .vsc varies the req input mean size, but not the random seeds, I'd normally expect your graph for the uniform draws (and probably the others, though I haven't looked at the equations in detail) to be flat lines. If you want the random stream to vary for each sim in the ensemble, you can add something like
random seed 2 = VECTOR(1,100,1)
to the VSC.

In fact, the random streams are almost, but not quite, the same. For example, for 2 runs out of a monte carlo, I see:

Code: Select all

0.132893	0.71575	0.971305	0.292402	0.868787	0.426666	0.277285	0.59238	0.956184	0.685163	0.266494	0.503936	0.264241	0.498228	0.00711332	0.407121
0.132893	0.136858	0.401645	0.0204532	0.517486	0.635454	0.707747	0.590935	0.884842	0.954989	0.81546	0.889405	0.54257	0.241733	0.675958	0.407121

Notice that the sequences start and end the same, but vary a bit in between.

This occurs because the request size = RANDOM NORMAL(...) is also using the same seed. Because the normal distribution is truncated at 0, the number of draws needed to yield a result within bounds can vary. Your .vsc is varying the req input mean size parameter, which is changing the number of draws needed, and that's spilling over to the uniform series.

You can see the effect easily if you do two ordinary simulations with different values of req input mean size, and look at the uniform var.

Hope this helps.

If you're varying the seed via the .vsc, the interaction isn't really a concern, because each draw is (quasi) independent. If you don't want to do that, then I'd suggest isolating the NORMAL variable from the UNIFORM variable by using different seeds.

[nhoward]

Thank you for looking at this. I half expected you to say I was using the seeds wrongly - there's quite a lot about how the seeds work I don't understand!
So there is interaction between the two streams. I think I can see this might happen if they both use the same seed.

I added the Uniform draw on "random seed 2" only to try and explain the results I was getting from Normal. It seems that doing that has made it worse! Giving Uniform a different sequence of seed values and not varying the mean of Normal - varying request size in the Monte Carlo setup - are worth trying, thanks

Making a low number of draws must still risk getting an outlier, though, particularly when Normal is truncated. I suppose you have to do enough runs without using the same seeds on different variables. ok, I'll get on with some more testing...

[tomfid]

The seed is really an ID # for a random stream. Internally, Vensim creates a separate generator for each stream. Each time a function is called, it looks up the appropriate generator by ID and returns the next draw.

Two RANDOM functions that use the same ID interact, because they're both consuming draws out of the same generator. If you use RANDOM UNIFORM, or other functions without truncation (wide min/max), it's not really an issue, because they always use the same number of draws. Even in the scenario you created, I think it's not an issue as long as the seed is varied. The NORMAL call does influence which draw is next for the UNIFORM call, but for any given seed, the UNIFORM results are still random, because the next draw is still independent of the prior history (at least as far as any pseudorandom generator can manage that).

I think it's OK to vary the request size in the MC setup. But to make it work, you'd have to do one of two things:
- use a different seed for the NORMAL and UNIFORM calls, or more generally, use unique seeds for RANDOM functions that might vary in number of calls, due to a parameter that could influence resampling needed for truncation
- vary the seed in every MC run, along with the other parameters (e.g. seed = VECTOR(1,1000,1)).

A small number of runs does risk an outlier - this recently happened to me in a real situation. I was working with an agent model of a health care system - essentially a discrete event simulation - and getting plausible answers with single simulations. When I varied the noise seed in an MC run, everything went down the toilet. It turns out that there were problems, but my base single simulation happened to be a big outlier, masking the problems. In this case, the problem arose because the distributions are very heavy tailed (worse than lognormal). In general, I don't recall many problems with this kind of thing.

This might be of interest: http://models.metasd.com/category/technical/stochastic/

[nhoward]

Hi

Thanks for this and apologies for not seeing it until today. Am applying different seeds to each parameter I vary in Monte Carlo runs now.

One other thing, following a sort of PC catastrophe that wiped out all my emails and reset Windows - some kind of update it seems - a Vensim model with excel inputs now opens together with the spreadsheet, which was slightly annoying as there doesn't seem to be a way to prevent it happening. However, a benefit is that Vensim does now allow excel based inputs to be changed with the Vensim model still open - previously it flagged an error.

[tomfid]

Strange. GET XLS should require that the spreadsheet opens, and grab inputs "live". GET DIRECT should read from the file without having it open (and Excel will generally lock out other apps if it is open). Is this an xls or xlsx?

[Administrator]

One other thing, following a sort of PC catastrophe that wiped out all my emails and reset Windows - some kind of update it seems - a Vensim model with excel inputs now opens together with the spreadsheet, which was slightly annoying as there doesn't seem to be a way to prevent it happening. However, a benefit is that Vensim does now allow excel based inputs to be changed with the Vensim model still open - previously it flagged an error.

Did you have to reinstall Vensim? It may be that you had the setting to read Excel files set to "Direct" (the setting is under Options->Options, GET XLS Protocol).

[nhoward]

Ok, thanks. Somehow the Get XLS Protocol in Options got switched to "OLE" from "Direct". Switching it back to "Direct" prevents the Excel sheet with the data inputs being opened. I hadn't reloaded Vensim. There had been some changes made in Windows.