Why Question Answering Modality Matters in Visualization Studies and Tests

Description:

In this position paper, we argue that answer modality choices can affect the validity of visualization evaluations. Visualization evaluation studies frequently rely on specific answer modalities such as multiple-choice questions, open-ended numerical estimates, verbal explanations, or spatial interactions. Yet, answer modality is often treated as a neutral reporting mechanism rather than as a factor that may influence reasoning strategies, representational transformations, and ultimately measured performance and the conclusions researchers can draw from their observations. We discuss two broad classes of modality effects on participant performance: changes in participants’ task-solving strategies and changes induced by representational transformations between the visual stimulus and the expected response format. We further argue that answer modality determines what evidence about participants’ reasoning is preserved for analysis. We ground our reflections in observations from prior empirical work and propose practical guidance for reasoning about answer modality as part of visualization evaluation design.

Paper download: (8.2 MB)

Additional material:

We make several items of additional material available in the following OSF repository: osf.io/muvrh.

Pictures:

All figures in the paper that are our own are available under a CC-BY 4.0 license, see the license statement at the end of the paper.

Main Reference:

Anne-Flore Cabouat, Lily W. Ge, Tobias Isenberg, Samuel Huron, Matthew Kay, and Petra Isenberg (2026) Why Question Answering Modality Matters in Visualization Studies and Tests. In Proceedings of BELIV. IEEE Computer Society, Los Alamitos, 2026. To appear.

BibTeX entry:



@INPROCEEDINGS{Cabouat:2026:WQA,
  author      = {Anne-Flore Cabouat and Lily W. Ge and Tobias Isenberg and Samuel Huron and Matthew Kay and Petra Isenberg},
  title       = {Why Question Answering Modality Matters in Visualization Studies and Tests},
  booktitle   = {Proceedings of BELIV},
  year        = {2026},
  publisher   = {IEEE Computer Society},
  address     = {Los Alamitos},
  oa_hal_url  = {https://hal.science/hal-05717323},
  osf_url     = {https://osf.io/muvrh/},
  url         = {https://tobias.isenberg.cc/p/Cabouat2026WQA},
  pdf         = {https://tobias.isenberg.cc/personal/papers/Cabouat_2026_WQA.pdf},
}

This work was done at the AVIZ project group of Inria, France.