BCal help system: The Tutorial

Step D12: What is reproducibility and why is it important?

In addition to considering the sensitivity of your results to changes in the way you model archaeological information, you must also be aware that MCMC simulation methods (such as those used in BCal) cannot always he relied upon to provide results that are reproducible. There are lots of technical reasons why this is so and we do not propose to discuss them in detail here (there are a number of texts that discuss such issues in particular, from our bibliography, try Litton and Buck 1996 and Buck et al. 1996). In the remainder of this tutorial, what we hope to do is to draw you attention to what you need to do to establish that your results are reproducible before sharing them with others. This is important because if you don't check your results before reporting on them you may find that when other researchers try to analyse the same data they get different results or that when you try again you can't reproduce them either!

What you need to try to ensure is that results between runs with exactly the same model conditions are reproducible. In order to investigate whether or not this has been achieved, we recommend that users compute several runs for each set of results they propose to report upon and then compute and compare 95% HPD regions for all parameters. In the case of our fishpond example, we calibrated our initial definition (not the modified definition used for sensitivity analysis) twice more. The following points summarise the steps needed for each independent calibration we wish to perform:

Make a copy of the project by selecting "Copy" for that project from the project list page.
Name the copied project something like "The Tutorial run2".
Select the "Submit" button for this new project from the project list page.
We are then presented with a page (select "Show me" to see it now) that allows us to make changes to the calibration parameters BCal will use. The default parameters given by the software are often, but by no means always, suitable for achieving reproducible results (for more information see the references above, and the BCal manual page about project submission).
Once you have chosen the calibration parameters, select the "Submit" button to submit the new project for calibration.
Once the calibration is complete select the "Results" button for that project from the project list page.
Select the "individual parameters" button to view the results as posterior probability distributions.
Select HPD regions for the format type and also select each parameter for which we wish to view results.

When considering reproducibility issues, users should be particularly aware of two calibration parameters over which they have control. One is the MCMC long run size (or burn in) and the other is the MCMC sample size. Both of which are discussed a little more on the next page.