Internal convergence checking

Like any MCMC application, the results produced can not be relied upon without careful visual inspection of the posterior distributions produced from several independent runs. BCal attempts to perform a limited amount of convergence checking automatically. It is essential, however, that you understand that it is currently impossible to reliably detect convergence automatically. The convergence checking facility provided in BCal is intended to complement reproducibility tests conducted by you (see the BCal tutorial). It is by no means a replacement for such tests.

Convergence checking sensitivity

BCal utilises the Raftery & Lewis convergence diagnostic as implemented in CODA and thus supports a variety of different levels for the convergence checking sensitivity. A high level of sensitivity will result in a very strict convergence diagnostic (the ideal case) and a low level of sensitivity will results in a very lenient convergence diagnostic (the least ideal case). The higher the sensitivity, the greater the system requirement will be (on the BCal server machine). BCal provides five levels of sensitivity within the internal convergence checker. We strongly recommend you always use the highest level possible (level 5 is the most sensitive). However, the demands of the convergence checker, in terms of system resources, increases as the project size increases. Consequently, it is currently not possible to perform convergence checking on very large projects whilst using the highest level of sensitivity. When calibrating a very large project, the most appropriate sensitivity level must be selected. This will depend upon the number of parameters defined within the project. The following table shows the maximum number of parameters you can have for each sensitivity level, as well as the minimum sample size that is required to obtain anything like reliable results.

Sensitivity level	Sample size required	Maximum parameters
level 1	4000	340
level 2	7000	200
level 3	11000	120
level 4	18000	70
level 5	26000	50

When you submit a project for calibration BCal will always ensure that the sensitivity level chosen is suitable with regard to the size of the project. BCal will also check that the minimum number of samples you have defined is suitable, based upon the sensitivity level you have selected.

How the convergence checker operates

When you submit a project for calibration you specify the minimum sample size. This becomes the first check point. Once the number of iterations that have been simulated exceeds this check point, BCal runs the internal convergence checker for the first time. If the MCMC data passes the convergence check at the selected sensitivity level BCal will stop the calibration and make the results available to you. If the check is unsuccessful, BCal will set a new check point based upon the results obtained from the Raftery & Lewis convergence diagnostic.

The convergence checking process is repeated until either the MCMC data passes the test, or the maximum number of iterations is reached (currently 1,000,000). If the maximum number of iterations is reached, the calibration is deemed to have a convergence problem.

The Raftery & Lewis convergence diagnostic checks for two different symptoms of convergence failure. The first of these simply estimates whether sufficient simulations have been obtained given the number of parameters and the complexity of the problem under investigation (if this happens your project status will read "Convergence Problem"). The second is used to warn of high dependence between simulated values in your MCMC data (if this happens your project status will read "High Dependence factor"). In both of these cases BCal will not allow you access to the results of your calibration as they likely to be unreliable.

What to do if you have a convergence problem

If BCal informs you that your project has a "Convergence problem" it means that BCal has computed 1,000,000 iterations in an attempt to estimate the calendar dates of interest, but has been unable to obtain reliable results. If this is a very large project (say more than 50 radiocarbon determinations) we recommend that you try to subdivide the project into several smaller chunks. This has the benefit that you will get results more quickly while you investigate what the problem is. If this is not a very large project or you have already subdivided it, it may be that you have imposed some very strict constraints using prior information. You could try relaxing some of these to see if this solves the problem. Problems of this sort are often caused by the fact that you have placed some restrictions on the calendar dates that are conflicting with one another. Have a good look at all the constraints placed on the simulations (both in the project definition and the prior information) and consider whether you have included things that might be contradictory or over restrictive in some way.

If your calibration fails due to a "High dependence factor", it is because our internal checking suggests that you have a problem with high dependence between the simulated values for at least one of the parameters of your model. This means that the results computed are not completely reliable and so have not been made available. This problem usually arises when you provide prior information that is extremely restrictive or use model conditions that are too restrictive for the problem to hand. We suggest that you try relaxing one or more of the constraints imposed and try again. The sort of things to consider are:

your use of abutting phase boundaries - are you certain that these shouldn't perhaps be modelled as overlapping or with hiatuses between them?
your use of prior information - have you imposed very strict boundaries on something that could be overly restrictive?

If after experimentation to try to resolve convergence problems you are still unable to get BCal to provide useful results, please feel free to contact a member of the BCal team (info@bcal.shef.ac.uk) and we will do our best to find some time to help you out.