ORSO workshop 2020

Session 1: Detailed introduction & aims of this working group – (26th 16:30/27th 08:00)

  • The chair should briefly introduce the working group
  • The working group is focused on Reproducibility, in measurement and analysis
  • We all (hopefully) believe that reproducibility is important, however it is frequently overlooked
  • We want to develop strategies to ensure that our measurements are conducted, reduced, and analysed in a reproducible fashion
  • We have a presence on GitHub (https://github.com/reflectivity/reproducibility), which includes some “Goals”
  • We want to generate a resource explaining why reproducibility in NR and XRR are important (session 2)
  • We have started building a folder of “examples” of reproducible analysis
  • We want a list of papers that represent a high standard of reproducibility (for others to aim for)
  • Does anyone have other ideas for achievable goals?

AIMS

  • Define a list of requirements for a reproducible measurement standard and how it would be documented
  • Define a list of requirements for the reproducible analysis of data
  • Identify individuals who will continue to work on these projects (item one; move towards a definition of this measurement standard, item two; develop a standardised protocol for producing such an analysis “workflow”)

Session 2: How to communicate the importance of reproducibility? – (26th 22:30/28th 10:00)

  • Reproducibility is important but showing if can be difficult
  • Most users are too busy chasing the “science” to worry about showing their analysis is reproducible
  • Few will worry about their data reduction being reproducible
  • How can we show the importance of this

AIMS

  • Develop a list of reasons why reproducibility is important in neutron and X-ray reflectometry
  • Give reasons why these aspects are important to users
  • Identify individuals who will continent this project (ideal situation is a publication showing the benefits of reproducibility to the user and presents problems from a lack of it).

Session 3: What are our expectations for reproducibility? – (27th 11:00/27th 22:30)

  • If we are going to evangelise reproducibility, we need to set reasonable expectations
  • We should be the bar that we expect, for example when we peer review a publication?
  • Should there be a “reproducibility expectation” when beamtime is applied for?
  • The bar needs to be shared openly so that people can aim for it
  • How should we advertise the bar (publication?)?

AIMS

  • Revise the lists from session 1 to find a “reasonable expectation”
  • Detail why items have been added/removed
  • Identify individuals who will continue (ideally resulting in a page for the website that gives our bare minimum to be considered “reproducible”.