Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design

Author name not available (Why is that?)

Publication date: 13 August 2021

Abstract: Many scientific conferences employ a two-phase paper review process, where some papers are assigned additional reviewers after the initial reviews are submitted. Many conferences also design and run experiments on their paper review process, where some papers are assigned reviewers who provide reviews under an experimental condition. In this paper, we consider the question: how should reviewers be divided between phases or conditions in order to maximize total assignment similarity? We make several contributions towards answering this question. First, we prove that when the set of papers requiring additional review is unknown, a simplified variant of this problem is NP-hard. Second, we empirically show that across several datasets pertaining to real conference data, dividing reviewers between phases/conditions uniformly at random allows an assignment that is nearly as good as the oracle optimal assignment. This uniformly random choice is practical for both the two-phase and conference experiment design settings. Third, we provide explanations of this phenomenon by providing theoretical bounds on the suboptimality of this random strategy under certain natural conditions. From these easily-interpretable conditions, we provide actionable insights to conference program chairs about whether a random reviewer split is suitable for their conference.

Has companion code repository: https://github.com/sjecmen/multistage_reviewing_bounds

This page was built for publication: Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6375199)