June 2019 Summaries

Robert Alonso and Alfredo De La Fuente, both with backgrounds in machine learning, participated in the Reproducibility Challenge, collaborating remotely from different countries to replicate and extend a paper on Variational Sparse Coding. The paper, which lacked some details on batch size and weights initialization, was reproduced successfully after communication with the original authors. The team suggested improvements such as using disentanglement metrics and emphasized the importance of reproducibility by recommending practices like setting random seeds and documenting environment details. They tackled challenges such as hyperparameter tuning and numerical stability, and leveraged resources like GPUs from Robert's university to manage computational demands. Their experience highlighted the need for more reproducible research practices and suggested that conferences consider reproducibility as a criterion for paper acceptance.

Arnout Devos and his team, comprising Sylvain Chatel and Matthias Grossglauser from the Swiss Federal Institute of Technology in Lausanne (EPFL), participated in a Reproducibility Challenge driven by an interest in the importance of reproducibility and meta-learning. Motivated by a presentation from Joelle Pineau on reproducibility, they selected a paper on meta-learning, specifically focusing on the R2D2 algorithm, and attempted to reproduce its results despite challenges such as missing parameters and time constraints. They based their approach on the Model-Agnostic Meta-Learning (MAML) framework, though they encountered initial difficulties in aligning with previous results due to variability in seeding and dataset acquisition. In their efforts, they communicated with the original authors, who subsequently clarified and updated their paper, exemplifying the iterative nature of scientific research. The experience underscored the importance of clear documentation and reproducibility in research, influencing Arnout to prioritize these aspects in his future work.

Aniket Didolkar's participation in the ICLR Reproducibility Challenge involved reproducing a method to tackle the exploding gradient problem in LSTM models, showcasing the complexities and importance of reproducibility in machine learning research. Despite being a solo participant, Aniket managed to replicate most experiments from the original paper, except one due to time constraints, emphasizing that access to detailed documentation, including hyperparameters and data preprocessing steps, is crucial for successful reproduction. The challenge underscored the importance of organized code repositories and transparent sharing of both successful and failed hyperparameter settings to enhance reproducibility. Aniket's experience highlighted the computational challenges, such as the prolonged runtime of experiments requiring him to use platforms like Google Colab, which has limitations on session durations, necessitating frequent restarts and data reloads. He pointed out that open-source code and comprehensive documentation could significantly ease the reproducibility process, suggesting that future research should include these elements to facilitate broader application and understanding. Aniket's perspective did not change about the field; however, he emphasized that research papers should ideally be accompanied by code or an explanatory section to aid understanding and replication, especially in complex fields like reinforcement learning, which he found particularly challenging to reproduce compared to areas like NLP or computer vision.