Winners of the AES53 Reproducible Research Prizes

The SoundSoftware project this week sponsored Prizes for Reproducibility in Semantic Audio Research at the Audio Engineering Society's 53rd Conference.

These prizes were awarded to papers, submitted and accepted for the conference and also submitted to our prize evaluation process, in two categories: fully-reproducible works and reproducibility-enabling works. The evaluation process was broadly the same as described in the evaluation process page from our previous round of Reproducible Research prizes, although for this round we carried out the sustainability evaluation ourselves and were able to ask the conference reviewers to evaluate for potential to enable further work.

Before we announce the winners, we'd like to thank everyone who entered their paper for these prizes. We'll be providing further feedback to all entrants during the coming week.

We're delighted to be able to announce the winners now!

Fully-Reproducible Works

Winner

Son N. Tran, Daniel Wolff, Tillman Weyde, and Artur Garcez, Feature Preprocessing with RBMs for Music Similarity Learning (paper, site, code)
Submitted by Daniel Wolff

Panel's comments: This submission is well-documented and made available in a version-control system. Its MATLAB code clearly breaks out the scripts used to generate the figures in the paper, and external dependency locations are configured in a dedicated file with appropriate documentation. The repository contains a README and a suitable licence (GPLv3). We found the code easy to run. Some suggestions for even further improvement include: using a public repository hosting site instead of (or as well as) a departmental repository; supplementing the random feature generation step with a copy of the exact features used for the figures in the paper; and clarifying, at the top level, the licence status of dependent toolboxes that are aggregated into the source tree.

Reproducibility-Enabling Works

Winner

Dan Stowell and Mark D. Plumbley, An open dataset for research on audio ﬁeld recording archives: freeﬁeld1010 (paper, data)
Submitted by Dan Stowell

Panel's comments: This submission received very high scores for sustainability planning and potential to enable high-quality research. The authors have taken a pragmatic approach to availability and portability of the dataset in terms of data formats and packaging, have used redundant hosting, and have documented the dataset well. External reviewers described this submission as lowering the threshold for sound classification and auto-tagging research.

Honourable Mention

Joren Six, Olmo Cornelis, and Marc Leman, TarsosDSP, a Real-Time Audio Processing Framework in Java (site, code)
Submitted by Joren Six

Panel's comments: This submission scores especially highly on the criterion of planning for sustainability. It has a well-maintained public website that includes examples and documentation, and it is managed under version control in a recognised public repository (github). The repository includes very comprehensive README documentation and a clearly stated licence (GPLv3). External reviewers described TarsosDSP as a useful toolkit. Although building the software was not quite an out-of-the-box experience for us, the Java code itself is portable and clear.

Honourable Mention

Kevin R. Page, Raúl Palma, Piotr Hołubowicz, Graham Klyne, Stian Soiland-Reyes, Daniel Garijo, Khalid Belhajjame, and Rudolf Mayer, Research Objects for Audio Processing: Capturing Semantics for Reproducibility (data, code)
Submitted by Raúl Palma

Panel's comments: This submission presents an application to audio processing and classification of a comprehensive existing experimental framework. The ambition to enable modular, configurable workflows in a collaborative experimental environment is a desirable one. Although we found a few hitches in making immediate use of the system in the short term, external reviewers appreciated the potential of this work to be significant in the longer term.

Tweets by @soundsoftwareuk

Recent notes

MLSP Prizes for Reproducibility: Winners announced!

Announcing the winners of the MLSP 2014 and SoundSoftware.ac.uk Prizes for Reproducibility in Signal Processing, organised by SoundSoftware.ac.uk in conjuction with the IEEE Signal Processing Society for the 2014 IEEE International Workshop on Machine Learning for Signal Processing.

SoundSoftware 2014: Videos now available!

The SoundSoftware 2014 workshop, our third annual workshop on software and data in audio and music research, was just as enjoyable as the previous two. Because so much research in this field ends up being expressed through software, a software workshop turns out to be all about the means by which research becomes useful and relevant to people other than the original researchers—fertile ground for interesting and thought-provoking talks.

The workshop videos are now available online at http://soundsoftware.ac.uk/soundsoftware2014, so if you weren't able to make it in person, catch up here!

Our third annual one-day workshop on Software and Data for Audio and Music Research takes place on the 8th of July 2014 at Queen Mary, University of London. The workshop includes talks on issues such as robust software development for audio and music research, reproducible research in general, management of research data, and open access. Read more here, clear your calendar, and register now!