Problem with MCMC posterior sampling

Use this forum to post Vensim related questions.
Post Reply
rodrigo?solis
Member
Posts: 22
Joined: Mon Jan 21, 2019 11:41 pm
Vensim version: DSS

Problem with MCMC posterior sampling

Post by rodrigo?solis »

Hi, I have been running a model through MCMC to obtain the posterior distribution of 7 parameters.

The problem is that, if I set an upper and lower limit for the search range of the parameter, for example:

Code: Select all

10<=Temperature<=50
the posterior seems to be getting 'artificially' oversampled in those limits. So, my histograms look more like a valley (edged by the search limits) rather than a bell curve.
Rplot.jpg
Rplot.jpg (41.02 KiB) Viewed 5213 times
Two solutions that I can think of, but I am not sure how valid they are:
  • To not set a limit for my search parameters. Unfortunately, that makes convergence extremely hard since the algorithm spends a lot of time in low likelihood regions (even after using quite informative priors) and comes up with some bizarre parameter sets.
  • Artificially remove from the posterior sample, vectors that have at least one of their parameters with a value equal to the limits set. I am not sure how valid this is from the standpoint of maintaining the properties of the sample. I've tried doing that, and the histograms immediately took a lovely bell shape (not sure how useful is that, though)
Any help or comment would be highly appreciated.

Thanks
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Problem with MCMC posterior sampling

Post by tomfid »

How many parameters are hitting the boundary? Just this one, or several of the 7? (If several, it might be getting stuck in a corner.)

I'm a little surprised that wider ranges, or no ranges, have convergence problems. The dimensionality isn't that great here. Is the model shareable for testing?

A hybrid option might be a prior with a flat, uninformative plateau, but soft shoulders.
rodrigo?solis
Member
Posts: 22
Joined: Mon Jan 21, 2019 11:41 pm
Vensim version: DSS

Re: Problem with MCMC posterior sampling

Post by rodrigo?solis »

Hi Tom,

Thanks for your reply. This is an updated version of a model that I already shared with you in the past (Monarch Butterfly). It seems that the xlsx file with all my external data is too large to upload here. could you provide me with an email to send you that file please?

And yes, it seems that at least 6 out of 7 of my parameters are showing that same problem. You mention that it could be getting stuck in a corner. How to 'unstuck' it?
Rplot.png
Rplot.png (19.13 KiB) Viewed 5195 times


Interestingly enough, if I run a sensitivity with that posterior sample, the resulting sims are acceptably fitting my observed data


Rplot01.png
Rplot01.png (26.27 KiB) Viewed 5195 times


and sorry, I am still struggling quite a bit with understanding how to include priors. I think I did it correctly here, but I definitively do not know how that hybrid option that you describe would be implemented.

Thanks in advance
Attachments
X2 nl BKP.mdl
(116.35 KiB) Downloaded 150 times
rodrigo?solis
Member
Posts: 22
Joined: Mon Jan 21, 2019 11:41 pm
Vensim version: DSS

Re: Problem with MCMC posterior sampling

Post by rodrigo?solis »

By the way: The convergence metrics with that odd-looking posterior sampling seem to be very good
Attachments
Rplot02.png
Rplot02.png (22.68 KiB) Viewed 5190 times
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Problem with MCMC posterior sampling

Post by tomfid »

This all looks pretty good, except for the corner problem. You can send a copy to vensim at vensim.com.
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Problem with MCMC posterior sampling

Post by tomfid »

One possible trick I'm thinking about is to log-transform one or two of the parameters that have large scale variations. That might make things behave better.
rodrigo?solis
Member
Posts: 22
Joined: Mon Jan 21, 2019 11:41 pm
Vensim version: DSS

Re: Problem with MCMC posterior sampling

Post by rodrigo?solis »

OK, thanks. I will try using log-transform. Also, I just emailed you the files of the model.

Thanks a lot for your rime
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Problem with MCMC posterior sampling

Post by tomfid »

Got it.
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Problem with MCMC posterior sampling

Post by tomfid »

Just getting to this now, and it appears that the .voc and .vpd are not in the archive. Would you mind posting those?
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Problem with MCMC posterior sampling

Post by tomfid »

OK, I see two issues, both easy to fix:

The payoff is intended to be a log likelihood, and the individual prior terms are log likelihoods, so the Prior Payoff equation should be a sum, not a product.

The range on overwintering density in the voc is set to 16-18 (roughly), but the actual parameter value should be in millions. This forces the estimate to immediately move to very bad values. I would try a broad range, like 1e6 to 100e6, or maybe 10e6 to 30e6 if the 16-18 is pretty reliable.

In addition, I'd recommend creating a control panel for manual exploration of the fit. Your Indicators view is close to what's needed - a combination of the Main custom graph plus the seven params in the voc. The ranges on the params should be set so that synthesim sliders work reasonably well.

I would define units for the whole model. https://metasd.com/2019/11/seven-deadly ... structure/
rodrigo?solis
Member
Posts: 22
Joined: Mon Jan 21, 2019 11:41 pm
Vensim version: DSS

Re: Problem with MCMC posterior sampling

Post by rodrigo?solis »

Hey Tom,

Thanks for the tips, I will correct the prior payoff as you recommend.

Regarding the VOC file, I am sorry, I sent you the file already updated with your previous recommendation (log-transforming some of my parameters). So the range you see is Log (OW Density).

I will run the MCMC now with the updated payoff, the log transformation, and an improved control panel. I will keep you posted.

Thanks
tomfid
Administrator
Posts: 3804
Joined: Wed May 24, 2006 4:54 am

Re: Problem with MCMC posterior sampling

Post by tomfid »

Ah ... the log transform makes sense. I suspect the fix to the prior payoff will make a significant difference.
Post Reply