The State of Reproducibility Stamps for Visualization Research Papers

Description:

I analyze the evolution of papers certified by the Graphics Replicability Stamp Initiative (GRSI) to be reproducible, with a specific focus on the subset of publications that address visualization-related topics. With this analysis I show that, while the number of papers is increasing overall and within the visualization field, we still have to improve quite a bit to escape the replication crisis. I base my analysis on the data published by the GRSI as well as publication data for the different venues in visualization and lists of journal papers that have been presented at visualization-focused conferences. I also analyze the differences between the involved journals as well as the percentage of reproducible papers in the different presentation venues. Furthermore, I look at the authors of the publications and, in particular, their affiliation countries to see where most reproducible papers come from. Finally, I discuss potential reasons for the low reproducibility numbers and suggest possible ways to overcome these obstacles. This paper is reproducible itself, with source code and data available from github.com/tobiasisenberg/Visualization-Reproducibility as well as a free paper copy and all supplemental materials at osf.io/mvnbj.

Paper download: (2.3 MB)

Additional material:

All images, an author version of the paper, and a snapshot of the code and data is available on OSF: osf.io/mvnbj.

Code/data:

The source code of the analysis including the whole paper sources are also available at github.com/tobiasisenberg/Visualization-Reproducibility.

Pictures:

(these images as well as all others from the paper are available under a CC-BY 4.0 license, see the license statement at the end of the paper)

Reference:

Tobias Isenberg (2024) The State of Reproducibility Stamps for Visualization Research Papers. In Anastasia Bezerianos, Michael Correll, Kyle Hall, Jürgen Bernard, Dan Keefe, Mai Elshehaly, and Mahsan Nourani, eds., Proceedings of BELIV. IEEE Computer Society, Los Alamitos, 2024. To appear.

BibTeX entry:



@INPROCEEDINGS{Isenberg:2024:SRS,
  author      = {Tobias Isenberg},
  title       = {The State of Reproducibility Stamps for Visualization Research Papers},
  booktitle   = {Proceedings of BELIV},
  OPTeditor   = {Anastasia Bezerianos and Michael Correll and Kyle Hall and J{\"u}rgen Bernard and Dan Keefe and Mai Elshehaly and Mahsan Nourani},
  year        = {2024},
  publisher   = {IEEE Computer Society},
  address     = {Los Alamitos},
  oa_hal_url  = {https://hal.science/hal-04697010},
  preprint    = {https://doi.org/10.48550/arXiv.2408.03889},
  osf_url     = {https://osf.io/mvnbj/},
  github_url  = {https://github.com/tobiasisenberg/Visualization-Reproducibility},
  url         = {https://tobias.isenberg.cc/p/Isenberg2024SRS},
  pdf         = {https://tobias.isenberg.cc/personal/papers/Isenberg_2024_SRS.pdf},
}

This work was done at the AVIZ project group of Inria, France.