Reproducibility Checklist

This tool responds to the GC Data Strategy for the Federal Public Service (2023-2026) Mission 3.3 for responsible, transparent and ethical data stewardship to maintain trust. This tool is suitable for a first screening level estimation of the reproducibility of a particular data asset and associated code.

There are two companion tools which you may also find useful:

FAIRER-Aware-SELF-Assessment (responds to the GC data strategy Priority 4.2.b., assessment of data skill needs)
FAIRER-Aware-DATA-Assessment (responds to the GC data strategy Priority 2.2.b., assessment of existing data for reuse).
FAIRER-Aware-AI Safety-Checklist (responds to he GC data strategy Mission 3.3 for responsible, transparent and ethical data stewardship to maintain trust).

Do you work with data? Are you looking to make it future proof? The FAIRER data principles will help you!

In 2023, the international consortium, Common Infrastructure for National Cohorts in Europe, Canada, and Africa (CINECA) stated that: " While the FAIR principles have become a guiding technical resource for data sharing, legal and socio-ethical considerations are equally important for a fair data ecosystem . ... FAIR data should be FAIRER, including also ethical and reproducible as key components .”

FAIRER principles refer to the Findability, Accessibility, Interoperability, Reusability, Ethics and Reproducibility of data assets, including related code. Applying these principles to your data assets will help others to find, verify, cite, and reuse your data and code more easily.

This tool helps you to assess the reproducibility of a data asset and associated code, and get tips on how you could increase their value and impact.

The tool is discipline-agnostic, making it relevant to any scientific field.

The checklist will take 15-30 minutes to complete, after which you will receive a quantitative summary of the level of reproducibility of your data and code, and tips on how you can improve the level of reproducibility. No information is saved on our servers, but you will be able to save the results of the assessment, including tips for improvement, to your local computer and add notes for future reference.

CRediT Author statement

REPRODUCIBILITY:

Reproducible data and code means that the final data and code are computationally reproducible within some tolerance interval or defined limits of precision and accuracy, i.e. a 3rd party will be able to verify the data lineage and processing, reanalyze the data and obtain consistent computational results using the same input raw data, computational steps, methods, computer software & code, and conditions of analysis in order to determine if the same result emerges from the reprocessing and reanalysis. “Same result” can mean different things in different contexts: identical measures in a fully deterministic context, the same numeric results but differing in some irrelevant detail, statistically similar results in a non-deterministic context, or validation of a hypothesis. All data and code are made available for 3rd-party verification of reproducibility. Note that reproducibility is a different concept from replicability. In the latter case, the final published data are linked to sufficiently detailed methods and information for a 3rd-party to be able to verify the results based on the independent collection of new raw data using similar or different methods but leading to comparable results.

Checklist Questions

My code is:
Deterministic	Yes No
Non-Deterministic	Yes No
Executable only on a high performance computing (HPC) system or a super computer.	Yes No
Quantum Code	Yes No
For all models and algorithms, I provided a link to:
A clear description of the mathematical setting, algorithm, and/or model.	Yes No Partial N/A
A clear explanation of any assumptions.	Yes No Partial N/A
An analysis of the complexity (time, space, sample size) of the algorithm.	Yes No Partial N/A
A conceptual outline and/or pseudocode description.	Yes No Partial N/A
For any theoretical claims, I provided a link to:
A clear statement of the claim.	Yes No Partial N/A
A complete proof of the claim.	Yes No Partial N/A
A clear formal statement of all assumptions.	Yes No Partial N/A
A clear formal statement of all restrictions.	Yes No Partial N/A
Proofs of all novel claims.	Yes No Partial N/A
Proof sketches or intuitions for complex and/or novel results.	Yes No Partial N/A
Appropriate citations to theoretical tools used are given.	Yes No Partial N/A
An empirical demonstration that all theoretical claims hold.	Yes No Partial N/A
All experimental code used to eliminate or disprove claims.	Yes No Partial N/A
For all datasets used, I provided a link to:
A downloadable version of the dataset or simulation environment.	Yes No Partial N/A
The relevant statistics (e.g., the number of examples).	Yes No Partial N/A
The details of train / validation / test splits.	Yes No Partial N/A
An explanation of any data that were excluded, and all pre-processing steps.	Yes No Partial N/A
A complete description of the data collection process for any new data collected, including instructions to annotators and methods for quality control.	Yes No Partial N/A
A motivation statement for why the experiments are conducted on the selected datasets.	Yes No Partial N/A
A licence that allows free usage of the datasets for research purposes.	Yes No Partial N/A
All datasets drawn from the existing literature (potentially including authors’ own previously published work) are publicly available.	Yes No Partial N/A
A detailed explanation, where applicable, as to why datasets used are not publicly available, and why publicly available alternatives were not used.	Yes No Partial N/A
A complete description of the data collection process (e.g., expt’l setup, device(s) used, image acquisition parameters, subjects/objects involved, instructions to annotators, and qa/qc methods.)	Yes No Partial N/A
Ethics approval.	Yes No Partial N/A
For all code used, I provided a link to:
The specification of dependencies.	Yes No Partial N/A
The training code.	Yes No Partial N/A
The evaluation code.	Yes No Partial N/A
The (pre-)trained model(s).	Yes No Partial N/A
A ReadMe file that includes a table of results accompanied by precise command to run to produce those results.	Yes No Partial N/A
Any code required for pre-processing data.	Yes No Partial N/A
All source code required for conducting and analyzing the experiment(s).	Yes No Partial N/A
A licence that allows free usage of the code for research purposes.	Yes No Partial N/A
A document with comments detailing the implementation of new methods, with references to the paper where each step comes from.	Yes No Partial N/A
The method used for setting seeds (if an algorithm depends on randomness) described in a way sufficient to allow replication of results.	Yes No Partial N/A
A description of the computing infrastructure used (hardware and software), including GPU/CPU models; memory; OS; names/versions of software libraries and frameworks.	Yes No Partial N/A
A formal description of evaluation metrics used and and explanation of the motivation for choosing these metrics.	Yes No Partial N/A
A statement of the number of algorithm runs used to compute each reported result.	Yes No Partial N/A
An analysis of experiments that goes beyond single-dimensional summaries of performance (e.g., average; median) to include measures of variation, confidence, or other distributional information.	Yes No Partial N/A
A description of the significance of any improvement or decrease in performance, judged using appropriate statistical tests (e.g., Wilcoxon signed-rank).	Yes No Partial N/A
A list of all final (hyper-)parameters used for each model/algorithm in the for each of the experiments.	Yes No Partial N/A
A statement of the number and range of values tried per (hyper-) parameter during development, along with the criterion used for selecting the final parameter setting.	Yes No Partial N/A
A ReadMe file with a table of results accompanied by precise commands to produce those results	Yes No Partial N/A
For all reported experimental results, I provided a link to:
The range of hyper-parameters considered, method to select the best hyper-parameter configuration, and specification of all hyper-parameters used to generate results.	Yes No Partial N/A
The exact number of training and evaluation runs.	Yes No Partial N/A
A clear definition of the specific measure or statistics used to report results.	Yes No Partial N/A
A description of results with central tendency (e.g. mean) & variation (e.g. error bars).	Yes No Partial N/A
The average runtime for each result, or estimated energy cost.	Yes No Partial N/A
A description of the computing infrastructure used.	Yes No Partial N/A
A document that clearly delineates statements that are opinions, hypothesis, and speculation from objective facts and results.	Yes No Partial N/A
A description of the range of hyper-parameters considered, method to select the best hyper-parameter configuration, and specification of all hyper-parameters used to generate results.	Yes No Partial N/A
Information on sensitivity regarding parameter changes.	Yes No Partial N/A
Details on how baseline methods were implemented and tuned.	Yes No Partial N/A
A clear definition of the specific evaluation metrics and/or statistics used to report results.	Yes No Partial N/A
A description of results with central tendency (e.g. mean) and variation (e.g. error bars).	Yes No Partial N/A
An analysis of statistical significance of reported differences in performance between methods.	Yes No Partial N/A
A description of the average runtime for each result, or estimated energy cost.	Yes No Partial N/A
A description of the memory footprint.	Yes No Partial N/A
An analysis of situations in which the method failed.	Yes No Partial N/A

Your Notes

Add any notes you may have here. These notes will be included when you print and save your results to your local computer. No information will be saved to our server. Feel free to capture any thoughts or insights that you'd like to remember or revisit later.

Your Data are 0% Reproducible

Data - Definition and Context

SOURCE: ECCC Science & Technology Branch, Scientific and regulatory data strategy and action plan (Draft).

DATA are a set of values of subjects with respect to qualitative or quantitative variables representing facts, statistics, or items of information in a formalized manner suitable for communication, reinterpretation, or processing (TBS Policy on Service and Digital, 2019).

Data are facts, measurements, recordings, records, or observations about the world collected by scientists and others with a minimum of contextual interpretation. Data may be in any format or medium taking the form of writings, notes, numbers, symbols, text, images, films, video, sound recordings, pictorial reproductions, drawings, designs or other graphical representations, procedural manuals, forms, diagrams, work flow charts, equipment descriptions, data files, data processing algorithms/code/scripts, or statistical records (CODATA-IRiDiuM (2018)  -International Research Data Management glossary).

The word “data” may be used very broadly to comprise data (in the strict sense) and the ecosystem of digital things that relate to data, including metadata, software and algorithms, as well as physical samples and analogue artefacts - and the digital representations and metadata relating to these things. (CODATA 2019 - Beijing Declaration on Research Data). There are dozens of other definitions of data that may be useful depending on the context¹.

SCIENTIFIC DATA are data that are used by scientists as primary sources to support technical and regulatory development or scientific enquiry, research, or scholarship, and that are used as evidence in the scientific process and/or are commonly accepted in the scientific community as necessary to validate scientific findings and results. All other digital and non-digital content have the potential of becoming research data. Examples of scientific data include data arising from: experiments, research and development, ‘citizen science’, surveys, operations, surveillance, monitoring, field analyzers or data-loggers, instruments, laboratory analyses, inventories, modeling and simulation output, processed data, and repurposed data (IRiDiuM - International Research Data Management glossary). The scientific nature of the data is demonstrated when the process of creating, maintaining and quality-proofing the data comply with commonly recognized scientific standards.

Although scientific data share many aspects in common with other types of data (e.g., administrative data, financial data, business data), their processing frequently requires more complex software and infrastructure. The data themselves may also be:

more complex (e.g., associated accuracy, precision, detection limits, confidence intervals, quality assurance/quality control procedures, etc.);
more tightly controlled;
held to higher standards;
retained for a longer period of time, often indefinitely;
documented more carefully and in greater detail (e.g., description of methods used to obtain measurements);
used as evidence, including in court, and therefore require a higher level of credibility, reliability, and accessibility.

Author Statement

Esther Liu (Data curation, Software, Visualization) ;
Dominique Charles (Data curation, Validation) ; and,
Claire C. Austin (Conceptualization, Supervision, Writing).

All authors reviewed, discussed, and agreed to all aspects of the final work.

All views and opinions expressed are those of the co-authors, and do not necessarily reflect the official policy or position of their respective employers, or of any government, agency or organization.

Cite as: Liu, E., Charles, D., & Austin, C. C. (2024). FAIRER Aware Reproducibility Checklist Tool (Version 1.0) [Computer software]. https://github.com/FAIRERdata/FAIRER-Aware-REPRODUCIBILITY-Assessment/.

SOURCES (built upon and added to):
Reproducibility Checklist – Association for the Advancement of Artificial Intelligence
AI-reproducibility-2020.pdf (McGill University and University of Waterloo)
MICCAI_2023_Repro_Tuto.pdf

Unanswered Questions