BCal help system: The Tutorial
Step D12: What is reproducibility and why is it important?
In addition to considering the sensitivity
of your results to changes in the way you model archaeological information,
you must also be aware that MCMC simulation methods (such as those used
in BCal) cannot always he relied upon to provide results that are reproducible.
There are lots of technical reasons why this is so and we do not propose
to discuss them in detail here (there are a number of texts that discuss
such issues in particular, from our bibliography,
try Litton and Buck 1996 and Buck et al. 1996). In
the remainder of this tutorial, what we hope to do is to draw you attention
to what you need to do to establish that your results are reproducible
before sharing them with others. This is important because if you
don't check your results before reporting on them you may find that when
other researchers try to analyse the same data they get different results
or that when you try again you can't reproduce them either!
What you need to try to ensure is that results between runs with exactly
the same model conditions are reproducible. In order to investigate
whether or not this has been achieved, we recommend that users compute
several runs for each set of results they propose to report upon and then
compute and compare 95% HPD regions for all parameters. In the case
of our fishpond example, we calibrated our initial definition (not the
modified definition used for sensitivity analysis) twice more. The
following points summarise the steps needed for each independent calibration
we wish to perform:
When considering reproducibility issues, users should be particularly aware
of two calibration parameters over which they have control. One is
the MCMC long run size (or burn in) and the other is the MCMC sample size. Both of which are discussed a little more on the next
Make a copy of the project by selecting "Copy" for that project from the
project list page.
Name the copied project something like "The Tutorial run2".
Select the "Submit" button for this new project from the project list page.
We are then presented with a page (select "Show me" to see it now) that
allows us to make changes to the calibration parameters BCal will use.
The default parameters given by the software are often, but by no means
always, suitable for achieving reproducible results (for more information
see the references above, and the BCal manual page about project submission).
Once you have chosen the calibration parameters, select the "Submit" button
to submit the new project for calibration.
Once the calibration is complete select the "Results" button for that project
from the project list page.
Select the "individual parameters" button to view the results as posterior
Select HPD regions for the format type and also select each parameter for
which we wish to view results.