MCMC for Credible Interval Estimation in Vensim

[rthompson42]

Hello! Our team is running an MCMC optimization for the first time to produce credible intervals for our SD model. None of us are very familiar with this method, and we have some questions about things we are seeing.

The SD model in question is a large health model that is being calibrated using 19 sets of time series data with 134 parameters in total being estimated.

Up until now, we have been running model calibrations using the Powell method for maximum likelihood estimation, with a log-likelihood payoff constructed from the weighted sum of squared errors between the data series and the model outputs.

In our first MCMC optimization run, we ran 2,500,000 total simulations with a burn-in period of 1,500,000. We left the rest of the MCMC optimization settings as the defaults. Here are some questions we have:

1) The payoff values became very large, with the final payoff being 2.56984e+14. In our Powell optimizations, the best payoff value we got was -168.086. Would a payoff value that large be expected from a large SD model using the MCMC optimization? Is the payoff calculated differently for Powell vs. MCMC optimization?

2) After running 2,500,000 simulations our PSRF stats were pretty awful (Max PSRF 7.57057, 11/135 < 1.2.), and over the course of the many simulations, these fluctuated up and down (ranging from 0/135 < 1.2, to 115/135 < 1.2). I take this to mean that the MCMC search process has not reached convergence even with the 2,500,000 runs. I have started a new MCMC optimization from scratch running 5million iterations total with a burn-in of 4million, but I'm not sure that will be enough. Do you have any suggestions for determining how long to run the MCMC optimization to reach convergence? Or is it just a process of trial and error?

Thanks much!

[rthompson42]

After writing this I have been playing around with the Powell optimization settings since I was concerned maybe our payoff wasn't appropriately weighted before taking the model into the MCMC optimizer. We previously were assuming normally distributed errors and coming up with weights from the standard errors between the data series and the simulation results. Now, I've specified Gaussian errors, applied a log transformation to the errors, and have been running Powell optimizations, also including StDev parameters in the payoff definition. Our payoff values are now around 6,800 - which doesn't look great since from my understanding "good" payoff should be something near the number of data points (for us, this is about 150 total).

Not sure where we are going wrong here. Could this be a overfit/underdetermined model issue? Or something else?

[rthompson42]

Issue was resolved after working out some issues with my payoff definition. Now payoff is looking "normal" (a number close to the number of data points used for calibration). Powell calibration had good fit to the data and MCMC ran normally.