SoundSoftware.ac.uk Prizes for Reproducibility in Semantic Audio Research

in

The SoundSoftware Project, in collaboration with the Audio Engineering Society's 53rd Conference on Semantic Audio, will be sponsoring a Prize for Reproducibility in the field of Semantic Audio published at this conference.

This edition of the Reproducible Research prizes follows our earlier SoundSoftware.ac.uk Prizes for Reproducibility in Audio and Music Research.

News and important dates

Eligibility

Authors of papers submitted to the Audio Engineering Society's 53rd Conference on Semantic Audio may present their submitted papers for this edition of the SoundSoftware.ac.uk Prizes for Reproducibility in Semantic Audio Research. Please read below for details on how to submit your work.

Both researchers from UK institutions eligible for EPSRC support and non-UK researchers will be considered for this award.

Categories

Category A - Fully reproducible work
Awarded for a paper whose results can be reproduced using the datasets, software and methodologies described in the paper;
Category B - Reproducibility-enabling work
Awarded for a paper presenting infrastructure, datasets, or standards intended to enable future reproducible work from other authors.

Within both categories we will separately consider regular and student-led submissions. Participants will be expected to choose a category on submission.

Prizes

Prizes will consist of a travel bursary of up to 1000 GBP (inclusive of conference registration costs) to present your work at this conference.

If you are a researcher from a UK institutions eligible for EPSRC support, in addition to the travel bursary the prize will also cover the Article Open Access Publication Fee - please see http://www.aes.org/openaccess/.

In addition, entrants may be invited to present their work at a SoundSoftware workshop.

How to enter

To submit your application to this edition of the SoundSoftware.ac.uk Prizes for Reproducibility in Semantic Audio Research, please complete the following form: application form.

Please take note that the deadline for submission is October 11, 2013 September 30, 2013.

Submission Deadline

Please note that the submissions are now closed.

The deadline was extended: the new deadline for submission is October 11, 2013 September 30, 2013.

Evaluation

The entries will be evaluated a panel of advisers to the SoundSoftware project. The panel will consider the following factors:

  • Ease of reproducibility of the results: a straightforward baseline replicability test in which we will evaluate whether each submission's published results can be regenerated using the software and data associated with the paper (where applicable).
  • Quality of sustainability planning: each submission's sustainability (of associated software and data) will be evaluated based on the Software Sustainability Institute's sustainability evaluation criteria (see http://www.software.ac.uk/online-sustainability-evaluation).
  • Potential to enable high quality research in the UK audio and music research community: all submissions will be sent to (at least) two external reviewers in order to assess their potential to enable further high quality research in their field

Some advice on how to prepare your submission

To give you the best chance of winning the prize, we suggest that you test both the sustainability and the reproducibility of your entry before submitting.

To test for sustainability, you can refer to the Software Sustainability Institute's Online Sustainability evaluation tool: http://www.software.ac.uk/online-sustainability-evaluation. Of particular note are whether the code and data are available in a suitable public repository, whether version control is used, which tools for community involvement are available, and whether the work includes a useful README file, copyright and licence information.

To test for reproducibility, we suggest that you simply test your software on a different machine. You might, for instance, ask a colleague to test it for you, preferably without standing over them explaining what to do as they do it! Usually this is enough to detect many significant oversights, such as missing files, references to nonexistent paths (eg /home/myusername/data/) or even incomplete or (completely missing) instructions.

If for any reason you can't ask a colleague to test your software for you, we'd advise you to test it on a virtual machine. For example (if you are not using Matlab) try running your code within this Linux image originally provided for use by entrants to an IEEE AASP acoustic scene challenge.

Ideally, a submission would contain individual scripts that regenerate all results presented in your paper. We are aware that many results (either in the form of tables or plots) are generated a few hours before the submission deadline ends; writing scripts that generate the results on your paper at the same time you are writing the paper itself is usually the best option! Don't forget that the person most likely to want to reproduce your results is you, at a later date - so why not prepare for that from the start?

If you wish to know more about the previous edition's evaluation process, please see the following page: http://soundsoftware.ac.uk/rr-prize-how.

Notes

While we encourage fully open access and open source submissions, research that can reasonably be reproduced by other audio or music researchers using common non-open tools or data is also acceptable (e.g. using Matlab or the RWC dataset).

If no submissions of suitable quality are received for a particular category, that prize may not be awarded.

For further advice or assistance contact the SoundSoftware project at info@soundsoftware.ac.uk.