MCMC for Credible Interval Estimation in Vensim
Posted: Fri Sep 22, 2023 7:38 pm
Hello! Our team is running an MCMC optimization for the first time to produce credible intervals for our SD model. None of us are very familiar with this method, and we have some questions about things we are seeing.
The SD model in question is a large health model that is being calibrated using 19 sets of time series data with 134 parameters in total being estimated.
Up until now, we have been running model calibrations using the Powell method for maximum likelihood estimation, with a log-likelihood payoff constructed from the weighted sum of squared errors between the data series and the model outputs.
In our first MCMC optimization run, we ran 2,500,000 total simulations with a burn-in period of 1,500,000. We left the rest of the MCMC optimization settings as the defaults. Here are some questions we have:
1) The payoff values became very large, with the final payoff being 2.56984e+14. In our Powell optimizations, the best payoff value we got was -168.086. Would a payoff value that large be expected from a large SD model using the MCMC optimization? Is the payoff calculated differently for Powell vs. MCMC optimization?
2) After running 2,500,000 simulations our PSRF stats were pretty awful (Max PSRF 7.57057, 11/135 < 1.2.), and over the course of the many simulations, these fluctuated up and down (ranging from 0/135 < 1.2, to 115/135 < 1.2). I take this to mean that the MCMC search process has not reached convergence even with the 2,500,000 runs. I have started a new MCMC optimization from scratch running 5million iterations total with a burn-in of 4million, but I'm not sure that will be enough. Do you have any suggestions for determining how long to run the MCMC optimization to reach convergence? Or is it just a process of trial and error?
Thanks much!
The SD model in question is a large health model that is being calibrated using 19 sets of time series data with 134 parameters in total being estimated.
Up until now, we have been running model calibrations using the Powell method for maximum likelihood estimation, with a log-likelihood payoff constructed from the weighted sum of squared errors between the data series and the model outputs.
In our first MCMC optimization run, we ran 2,500,000 total simulations with a burn-in period of 1,500,000. We left the rest of the MCMC optimization settings as the defaults. Here are some questions we have:
1) The payoff values became very large, with the final payoff being 2.56984e+14. In our Powell optimizations, the best payoff value we got was -168.086. Would a payoff value that large be expected from a large SD model using the MCMC optimization? Is the payoff calculated differently for Powell vs. MCMC optimization?
2) After running 2,500,000 simulations our PSRF stats were pretty awful (Max PSRF 7.57057, 11/135 < 1.2.), and over the course of the many simulations, these fluctuated up and down (ranging from 0/135 < 1.2, to 115/135 < 1.2). I take this to mean that the MCMC search process has not reached convergence even with the 2,500,000 runs. I have started a new MCMC optimization from scratch running 5million iterations total with a burn-in of 4million, but I'm not sure that will be enough. Do you have any suggestions for determining how long to run the MCMC optimization to reach convergence? Or is it just a process of trial and error?
Thanks much!