Katherine Kuchenbecker debates about the peer-review process at ICRA 2024

Yokohama – Katherine Kuchenbecker, Director of the Haptic Intelligence Department at MPI-IS and a regular attendee of the IEEE International Conference on Robotics and Automation (ICRA), took part in a popular debating format the conference maintains. At this year’s five-day event in Yokohama, Japan, she and other leading scientists from the robotics community kick-started the conference by arguing the pros and cons of three controversial topics: single-blind peer review, generative AI, and the relevance of universities for modern robotics. As every year, the format created quite a buzz: Seats became scarce in no time, with people standing in the hallway outside and even sitting on the ground behind the debaters at the end.

“This is the third time I took part in these debates,” Kuchenbecker says. “The first time, I was a debater, the second time, I was a moderator, and now I was a debater again. I love the format and couldn’t believe how much attention the discussions got this year.”

For 80 minutes, Kuchenbecker argued the for-proposition of the debate "Peer review is a too-noisy process biased by big names in the case of single-blind reviews at ICRA/IROS". Joining her on the pro-side was Georgia Chalvatzaki, a professor from TU Darmstadt. Meanwhile, on the against-side, Geoffrey Hollinger and Javier Civera defended their arguments, while Michael Milford moderated. Debaters are asked to adopt a strong position for or against their assigned proposition, aiming to provoke critical thinking in the audience by sharing perspectives that might differ from their personal viewpoints.

Kuchenbecker made her case clear in her opening arguments, saying, “Specifically, I argue that letting reviewers see the names, the institutions, and the countries of the authors of the submission that they are reviewing, this introduces significant noise into all aspects of the review process. The outcomes that I care about are not just whether a particular submission is accepted or rejected, but also whether a particular reviewer accepts the invitation to review this manuscript, the ratings and the comments that they share with the authors and later whether that paper is considered for any awards. I base my arguments on a core assumption that it is only the content of the submission – and here I mean the text, figures, the results, the videos, the citations, and the research environment at the time of submission – that should dictate what a reviewer writes about that submission. The authors and the affiliations listed on the first page should not affect the outcome. But currently, in our single-blind review process, they do.”

She continued, “That is understandable because we humans are social creatures and we’re very good at finding perceptual patterns. Especially, we use these heuristics, these biases, when we have limited information and or when we have limited time. We don’t have all the time in the world, and even when we are aware of these biases, even if I know that I might be more inclined to rate a paper more highly if it’s someone like me, a woman, an American, someone who studied at Stanford University, even if I know that I can’t get rid of my biases, I might actually be harsher on papers from people I think I have a bias towards. So, the only way to remove these biases from the decision is to remove the author information from submissions. This is a system called double-blind review or double-anonymous review. Here we write our papers so that the reviewers don’t know who wrote the paper. That means authors have to consciously hide their identities, remove names, don’t include pictures of your university logo – you blur them out – and you don’t say the name of your institution. You write about your research in the third person. If you pay the cost of anonymizing your research, then you reap three important benefits.”

Watch the full video to hear from Katherine Kuchenbecker what these benefits are, as well as all the arguments shared by the eleven other participating debaters:

https://www.roboticsdebates.org/

Meanwhile, Jeannette Bohg, an MPI-IS alumna and now an Assistant Professor at Stanford University, also took part in the third debate at ICRA 2024. She gave her views on generative AI, arguing that it will make a lot of traditional robotics approaches obsolete. A favorite part of these debates is that each reveals their actual personal views at the end of their debate; listen to the end to find out what Bohg really thinks about generative AI for robotics.

The organizers of these debates believe that “it is better to debate a question without settling it than to settle a question without debating it,” a quote from Joseph Joubert. Kuchenbecker finds this debate format so compelling that she is planning to introduce it to the 2025 IMPRS-IS Boot Camp, which will likely be held in September 2025 in Stuttgart and Tübingen. Our IMPRS-IS Scholars will have a chance to propose debate topics, and IMPRS-IS Faculty and other community members will take the stage to debate both sides of each selected proposition.