Introduction
This post draws from my personal experience writing rebuttals for top-tier machine learning conferences (NeurIPS, ICLR, and ICML). While the specific advice here focuses on ML conferences, many of the principles, particularly around organizing reviewer comments and planning responses can be valuable for researchers in other fields. However, the actual construction and writing style of rebuttals may differ significantly for other disciplines like computer vision (CVPR, ICCV, ECCV) or natural language processing (EMNLP, ACL, NAACL).
Understanding the Review Process
The review process typically begins when you receive comments and scores from 3-5 reviewers. Based on these initial reviews, you’ll need to evaluate your position:
- Very Low Scores: Consider resubmitting to another conference if all scores are below the borderline or if multiple reviewers show fundamental misunderstandings.
- Strong Scores: Use the rebuttal to aim for spotlight or oral presentation slots.
- Borderline Scores: This is the most common case where your rebuttal becomes critical for acceptance.
Conferences typically use one of two rebuttal formats. The first is immediate discussion, where you can address reviewer concerns and engage in ongoing dialogue until the rebuttal deadline. The second is a two-stage process, beginning with initial responses (hidden from reviewers) followed by an open discussion period.
The waiting period after rebuttal varies. Some reviewers may indicate score changes during discussions, while others adjust scores after the rebuttal period closes. Remember that acceptance isn’t solely about absolute scores, your paper’s relative standing compared to other submissions often matters more.
Crafting an Effective Rebuttal
1. Organizing Reviewer Comments
Create a shared document (e.g., Google Doc) to track essential information
- For each reviewer, record:
- Their identity and score
- Verbatim comments and concerns
- Your initial response ideas and required experiments
Share this document with all collaborators immediately to begin coordinating your response strategy.
2. Planning Additional Experiments
When planning experiments during the rebuttal period, careful prioritization and time management are crucial. Meet with your collaborators to develop a strategic experimental plan:
- Prioritize experiments based on:
- Issues raised by multiple reviewers
- Concerns from reviewers who gave lowest scores
- Feasibility within the time constraint
- Strategic considerations:
- Consider simplified versions of experiments that can still effectively demonstrate your point
- Be cautious about showing dramatically improved results -> reviewers might question why these weren’t in the original submission
- Set realistic timelines with buffer for unexpected issues
Remember that the rebuttal is like a strategic game with reviewers. If you present substantially better results during rebuttal, some reviewers might prefer to reject the paper, suggesting you submit the improved version to another conference. It’s often better to show targeted improvements that address specific concerns while maintaining the core contribution of your original submission.
Discuss with your collaborators:
- Which experiments are absolutely necessary versus “nice to have”
- What experimental settings would be reasonable given the time constraints
- How to present results most effectively in the limited rebuttal space
Having a clear timeline and backup plans is essential. Some experiments might not work out as expected, so prioritize what’s most important and have alternatives ready if needed.
3. Writing the Rebuttal
Craft your responses thoughtfully and professionally, addressing each reviewer individually even when responding to similar concerns. This individual attention shows respect and helps prevent discussions from becoming disorganized.
For discussion-based rebuttals, engaging actively with reviewers is crucial. Many reviewers are more likely to adjust their scores after multiple rounds of constructive dialogue. Don’t wait until the last minute to respond, as this limits the opportunity for meaningful discussion.
Draft your responses in a shared document and regularly remind collaborators to review them. If collaborators aren’t responding, set a deadline for yourself to ensure you have time to post the response carefully. Using markdown-friendly platforms like HackMD for final formatting can be helpful, and tools like Table to Markdown can easily convert experimental results into the proper format.
Response Structure:
Response to Reviewer [NICKNAME] -- Part {i}
[Your detailed response]
[R{i}] References
We appreciate the reviewer's feedback and are happy to address any additional questions.
Best Regards,
Authors
Consider setting reminders during the discussion phase. Since reviewers themselves are often authors with their own submissions, they may be busy and forget to participate in ongoing discussions. Regular reminders to check the platform and polite follow-ups for non-responsive reviewers can significantly increase engagement. This is particularly important when aiming for score increases, as meaningful discussions can not only improve your paper’s chances but sometimes lead to concrete directions for future research.
We appreciate the constructive feedback from the reviewer. As the the discussion dealine coming to an end, we haven't received additional feedback. We look forward to engaging with you through OpenReview's interactive discussion feature, and we're confident our response addresses your concerns.
[Summary of the rebuttal response]
For Score Increases:
We thank the reviewer for the score increase and constructive feedback. We will incorporate these suggestions in the final version.
4. Communicating with Area Chairs
Emergency Situations
During the review process, certain serious issues require immediate attention from the area chair:
- When to Contact Immediately:
- Automated or AI-generated reviews without human oversight
- Offensive or inappropriate reviewer comments
- Unresponsive reviewers or circular discussions
- Reviews that clearly don’t match your paper’s content
Remember that some conferences don’t allow direct email contact during the rebuttal phase. Instead, use the review platform (e.g., OpenReview) to send a message visible only to authors and area chair. Clearly explain the situation and provide specific evidence of the issue.
End-of-Discussion Summary
When the discussion phase is winding down and reviewers seem to have no further questions, it’s beneficial to send a summary to the area chair. This final message should be visible only to authors and area chair, not reviewers. Focus on synthesizing the most significant points of discussion and how you’ve addressed key concerns, making it easier for the area chair to make their final decision.
Sample Area Chair Communication:
Dear Area Chair,
We appreciate the thoughtful feedback received from all reviewers. To facilitate your decision-making, we summarize the main points and our responses:
1. [Major Point 1] raised by Reviewers X, Y
Our response and improvements...
2. [Major Point 2] raised by Reviewer Z
Our clarification and additional results...
We hope our summary is helpful in the final discussion phase.
Best Regards,
Authors
Learning from Examples
Here are three exemplary rebuttals that I personally enjoy reading:
- “Understanding In-Context Learning in Transformers and LLMs” demonstrates clear, focused responses while maintaining a professional yet persuasive tone.
- “LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models” shows excellent organization and strategic presentation of new experimental results.
- “Interpreting CLIP’s Image Representation” exemplifies how to skillfully clarify misunderstandings while maintaining a constructive dialogue.
Acknowledgement
I am especially grateful to receive support and feedback from Pin-Yu Chen, Chia-Hsiang Kao, Yu-Hsuan Li, Yun-Hsuan Lien, Tung-Yu Wu and Yu-Lin Tsai in perfecting this post.