FAQ: Reviewing for AAMAS 2025

On this page we, the program chairs of AAMAS 2025, respond to many frequently asked questions by the members of the program committee.  The latter includes regular PC members, senior PC members, and area chairs.  If you have a question not (yet) addressed here, in the Call for Papers, or Submission Instructions, then you are welcome to contact us: [email protected]. 

General Information for Reviewers (PC Members)

1. I have previously served as senior PC member and now got an invitation to serve as regular PC member. Is that a demotion?

No, of course not. AAMAS has a rule saying that, as much as possible, nobody should serve as senior PC member (or area chair) for more than two years in a row. This is both about fairness (younger people should have the chance to grow into these senior roles) and quality (experienced people should keep writing reviews). Of course, there are a few instances where this rule was not followed to the letter in the past (mostly due to imperfect bookkeeping), but the intention is to stick to it as much as possible.

It’s also helpful to keep in mind that we need around 10 times as many regular PC members as senior PC members, so everyone should expect that in most years their contribution to AAMAS will take the form of a regular PC membership (i.e., reviewing).

2. I received a submission to review I did not bid for. Why is that?

This is impossible to avoid, as papers vary dramatically in popularity, but all must ultimately be reviewed by at least 3 PC members. The OpenReview matching system still optimizes the quality of matches even in the absence of bids.  However, this, too, is imperfect.  Some submissions may not be a great match for most reviewers.  In addition, your previous publications, which OpenReview uses to determine your expertise in paper matching, may not provide an ideal indicator in this regard.  At the end of the day, the system is imperfect.  However, we still expect you to do your best in reviewing all of the assigned papers.

3. Why are there two different review deadlines?

We ask each regular PC member to deliver the reviews for at least 50% of the papers assigned to them one week ahead of the final reviewing deadline. The main reason for this deviation from what you might be used to is that we want to make the work of our senior PC members a bit more manageable.

While the vast majority of AAMAS reviewers manage to deliver high-quality reviews on time, there are always a few exceptions that require extra attention from senior PC members and, in some cases, force them to solicit additional emergency reviews from other members of the community. The challenge is that, as a senior PC member, you do not know in advance which reviews will turn out to be the problematic ones that take up most of your time.

Given that typically around half of all reviews arrive on the day of the deadline—and many in the final hours—this core part of the work of a senior PC member (and the emergency reviewers they approach) traditionally happens over the course of a single sleepless weekend just after the final review deadline. This clearly is suboptimal. By requiring half of the reviews to arrive one week earlier, we not only allow the senior PC members to spread their work a little, but we also will find out one week earlier which reviewers might need some extra support.

Finally, we of course also think that it is healthy for a regular PC member to not cram all their reviewing assignments into those last couple of days just before the final deadline. We all know this to be true for other people; we just sometimes need a bit of a nudge to apply this golden rule also to ourselves. The staggered review deadline is such a nudge. 

4. I do not yet have a lot of experience with reviewing. How can I make sure that I do a good job?

First of all: welcome, great that you are getting involved! The fact that you have been included in the PC means we believe that you are qualified for the job. In our experience, newcomers often are particularly thorough and put a lot of effort into their reviews (that’s great, try to keep that for as long as you can manage!). But they sometimes are also particularly harsh in their assessment (please try to resist that temptation!). Ulle Endriss has written some fantastic notes about reviewing a while ago, which you may find helpful.

5. I heard that the information on submissions I receive as a PC member is confidential. What does this mean?

You are expected to treat all the information regarding submissions you get access to as a PC member as confidential information. This includes the papers assigned to you for review, but also the (anonymized) list of all submissions you get to see during the bidding phase.

Specifically, you should never tell an author that you reviewed their paper, independently of whether the paper was accepted or rejected.

You also cannot make use of the information you have obtained as a PC member to advance your own research. Of course, once a paper becomes publicly available (e.g., if it has been accepted and gets published in the AAMAS proceedings, or if it eventually appears elsewhere), you are free to make full use of the results in that paper.

6. I think I know who wrote one of the papers I’m reviewing/handling. What should I do?

First of all, please try to adhere to the principles of double-blind peer reviewing as much as possible. Specifically, we ask you to not actively try and de-anonymize the submissions assigned to you (e.g., by googling for the paper titles and so forth). Yes, we know that this can be difficult, but this level of self-discipline certainly is expected.

Of course, despite your and the authors’ best efforts to ensure the integrity of peer review, sometimes you simply do know or are easily able to guess who wrote a given paper. In those cases, please try to ignore this information as much as possible when forming your judgment about the quality of the paper.

If you find out who wrote one of the papers you are reviewing/handling and this happens to be someone you should have declared a conflict of interest for, then please get in touch with us (the program chairs) right away.

7. As a PC member, can I delegate a reviewing task to an auxiliary reviewer?

While we hope that you will review most of the papers you receive yourself, you also have the option of outsourcing some of these reviews to trusted colleagues in your direct vicinity (so as to not accidentally violate any conflicts of interest). Importantly, if you choose to do this, you still remain responsible for the timely submission and quality of those reviews. You might want to outsource a review if you happen to know someone with specific expertise relevant to a given paper. You might also want to do this in your role as the supervisor of an advanced PhD student who has published at AAMAS or comparable venues before and who now needs to receive training in writing reviews (before they get to fulfill this important duty on their own, after graduation). Involving a PhD student in this manner is fine, as long as you remain actively involved yourself and can vouch for the quality of the review submitted.

8. AAMAS attracts very different kinds of papers (from theory to applications). How should I account for this in my reviews?

While every paper accepted for publication in the AAMAS proceedings should meet the same high standards, there are important differences in how to apply those standards when reviewing different kinds of papers. For example, the expectations we have for a paper on game theory will be very different from those for a paper on virtual agents.

When reviewing a given paper please take into account the specific requirements for papers in the area the paper has been submitted to. You most likely will be familiar with those requirements, but if you are unsure, please ask the area chair assigned to the paper for advice.

Keep in mind that not every paper needs to come with theoretical results in the form of theorems, not every paper needs to present a new algorithm, not every paper needs to discuss an implementation, not every paper needs to contain experimental results, and not every paper needs to demonstrate its immediate relevance to a concrete application.

Specifically, for papers reporting on deployed or emerging applications using agent-based technology, we do not expect authors to necessarily present new theoretical results or new algorithms. Instead, authors might focus on the challenges associated with deploying agent-based technology in the real world, on how users experience a particular application, or on how agent-based technology can be integrated with other technologies. For deployed applications, we would expect that applications have been deployed over a reasonable amount of time, so that evidence of impact is available and lessons learned for the autonomous agents and multiagent systems community about what worked and what did not can be included in the discussion. For emerging applications, it is important that the applications discussed really are novel applications, which have not been tackled with agent-based approaches before.

9. Am I supposed to recommend acceptance (rejection) for the best (worst) paper in my batch?

No. Please try to judge each paper on its own merits, independently from the other papers you happen to be reviewing for this one edition of this one conference. Your batch of papers is unlikely to be representative: as a regular PC member you will get to see fewer than 1% of all submissions, while as a senior PC member you will get to see fewer than 2%.

General Information for SPC

1. Can you explain the mapping between the typical AAMAS terminology (PC, SPC, AC) and the terms used by OpenReview?

This is indeed confusing.  Here is how the mapping works. PC (i.e., program committee members) are referred to by OpenReview simply as “Reviewers”.  What we typically call SPC (senior program committee member), OpenReview calls “Area Chair”.  Finally, what we commonly call AC (Area Chair) in typical AAMAS program committee terminology is referred by the OpenReview as “Senior Area Chairs” (SAC).

2. Can SPC (“Area Chairs” according to OpenReview) change reviewer assignments to papers in their batch?  If so, how?

Yes, we have enabled this functionality! The instructions for doing this can be found here.

Rebuttals and Discussions with Authors

1. Do I have to actively respond to the authors after they post their rebuttal during the rebuttal phase?

While we do not require this, insofar as the authors put a significant effort in addressing your comments and criticisms, it is proper to carefully consider their responses, and engage with the authors should you need further clarification.  Such discussions can at times be rewarding, and help clarify any misunderstanding, which is the main purpose of both the rebuttal phase, and any discussion that follows.

Participating in the PC Discussion

1. Why is it important that I update my review during the PC discussion?

First, it is simply the polite thing to do: to acknowledge that you have read the response of the authors and considered it.

In most cases, you will agree with some of the responses and disagree with other parts. Similarly, you probably will understand some of the response and find other parts unclear. Especially when some of the responses are unclear, you can actively engage with the authors during the rebuttal period to further clarify any potential misunderstanding.  Sometimes, certain remarks will have made you change your mind on some aspect of the paper, while for other aspects your opinion remains unchanged, even in the face of a seemingly good response. Finally, in some cases you may find a specific part of a response rather helpful but ultimately irrelevant to your overall assessment of the paper, while in other cases a response may really have changed your opinion in important ways. Indeed, sometimes a response actually makes you view the paper more negatively.  Please let the authors know about all of this! This kind of feedback can be very helpful.

2. Suppose a new problem with a paper, not noted in the initial reviews, emerges during the discussion. How should we deal with this?

Of course, this can happen, but you should realize that this can easily look unfair to the authors. This applies in particular to points that could have easily been discovered during the initial reviewing phase and that maybe could have been resolved by means of a simple clarification question that the authors could have answered during the rebuttal phase. Nevertheless, if the point is important, it is proper to update your review.  We encourage doing so as empathetically as possible.

Naturally, any new point of criticism emerging during the PC discussion should be included in one of the reviews (possibly the metareview). Please make it clear whether the new point you are adding ended up playing a crucial role in arriving at your final recommendation or whether you are simply providing additional information that you hope will be useful to the authors.

3. Do I need to change my score to reflect the consensus reached during the PC discussion?

Not necessarily. It is perfectly possible for a group of PC members to agree about whether to recommend acceptance or rejection without all of you reporting the same score.

Indeed, if a paper ends up getting accepted but your own review is more critical than the others, then it is helpful for the authors to have clarity on this matter and see it reflected in your score. Similarly, if a paper ends up getting rejected but your own review is more positive than the others, then it is nice for the authors to see that too.

Of course, sometimes it will be appropriate to update your score. If you do, then you must also change the text of your review. An unmotivated change of score is never acceptable.

4. When is it appropriate to recommend acceptance in the form of an extended abstract?

All papers are submitted as full papers of up to 8 pages (plus references), but some of these submissions will only be accepted as extended abstracts of up to 2 pages (plus references).

This might be a good option for a submission that is not yet sufficiently mature for publication as a full paper but that includes interesting and promising ideas that are worth sharing with the community.

A necessary condition for accepting a submission as an extended abstract is that its main message can be communicated in the space of 2 pages (plus references). This is a matter that is orthogonal to scientific quality: some strong papers can be condensed in this way, while some cannot; likewise, some weak papers can be, while some cannot.

5. How do I write a good metareview?

The metareview typically consists of a single paragraph of text. Its core purpose is to explain the decision taken by the program committee to the authors.

Additionally, you may also use the metareview to communicate further information to the authors, such as relevant points arising during the discussion that none of the reviewers included in their regular reviews, or some of your own feedback regarding the paper. (The latter—particularly if you have a lot of things to say—you can also do by submitting a regular review yourself.) The remainder of this answer only touches on the aforementioned core purpose of the metareview.

If the decision to be taken is obvious given the reviews, then a single sentence stating this fact is sufficient. For example, if all reviewers recommend strong acceptance, the paper should obviously be accepted.

But for many submissions, the decision will not be entirely obvious and some explanation will be called for. For example, if one reviewer was arguing for rejection and you, with the support of the other reviewers, decided to overrule this objection, you should say something about this in your review (recalling the main arguments in favor of the paper, then acknowledging the criticism but explaining why it was given less weight).

If you feel that one of the reviews did not fully live up to the standards you would have liked to see, it is ok to signal to the authors that the final decision was largely based on the other reviews. If you overlooked something that’s really a little inappropriate in the initial reviews, it is ok to apologize for that oversight to the authors and, again, to assure them about the process that has led to the final decision. Of course, all of this needs to be done with due consideration for the reviewers concerned.

If you recommend that a paper be accepted as an extended abstract, then it is great if you can suggest what part of the work the authors might want to focus on in that extended abstract. In addition, you should only recommend acceptance as an extended abstract if the authors did not opt out of this possibility.

Never mention numerical scores in your metareview. Scores are just an imperfect and overly simplistic reflection of the content of a review. Instead, justify your decision in terms of the content of the reviews.

Note that there is no need to include phrases such as “I recommend to accept this paper” in your review. First, that sounds like a message directed at us (the program chairs), not the authors, while it is the authors you should be addressing. Second, by the time the authors will get to your metareview, they already know what the acceptance decision is.

For clear-cut or otherwise well-argued decisions, it is extremely unlikely that we would see any need to overrule your decision. But, of course, there will be borderline cases where you only very slightly prefer one decision over another, and we eventually will go for that other decision due to considerations of a more “global” nature (say, comparisons with other borderline papers that are not in your batch). In such a case, it is best if you phrase your metarview as highlighting both strengths and weaknesses, but that does not commit to a particular recommendation that you will actually make.

6. I heard that PC discussions are confidential. What does this mean?

Just as the content of the anonymous submissions you are reviewing is confidential, so is the content of the PC discussion you are witnessing and contributing to. It goes without saying that it would be a clear betrayal of the trust put in you if you were to reveal to an outsider (such as an author) what a given reviewer said about a given paper.

Dealing with Problems

1. I will be unable to complete my tasks as a member of the PC or expect to miss a deadline due to other commitments or unforeseen problems in my personal or professional life. What should I do?

In general, we rely on everyone who accepts our invitation to join the PC to ensure they are able to dedicate the time required to their duties and meet all relevant deadlines. If you think you might not be able to do this, then we prefer it if you decline our invitation. But of course, we understand very well that unforeseen problems can occur. In such a case, if you are a regular PC member, please inform the SPC members of all the papers for which you may miss a review deadline as soon as possible. If you are an SPC member, please inform the ACs of the areas for which you are handling papers. If things are really bad and you can only manage to write a single email, please make sure you send it to us (the program chairs). 

2. I do not feel qualified to review one of the papers assigned to me. What should I do?

Please understand that it is not always possible to arrive at a perfect paper assignment. PC members for AAMAS are experienced representatives of our research community and should be able to provide useful feedback also on papers outside their own immediate area of research. Indeed, a carefully crafted outside assessment can be particularly useful and may help make a paper more accessible and more broadly relevant. But in extreme cases, if you let the SPC (“Area Chair” in OpenReview) know within 72 hours of the paper assignment having been sent to you, they will try to find somebody else to review the paper (or escalate it further up the program committee chain). 

3. I have a conflict of interest (CoI) regarding one of the papers assigned to me. What should I do?

Please send your SPC (“Area Chair” in OpenReview) know immediately by sending them a message, mentioning the submission number and title of the paper, and briefly outlining the nature of the CoI. If you cannot reach your SPC, please contact the program chairs.  Please carefully check all the papers assigned to you within 72 hours of the assignment having been sent to you, as it will be very difficult for us to reassign a paper while still ensuring a high-quality reviewing process if we only find out later than that.

4. I’m concerned about a possible case of plagiarism, fraud, ethical misconduct, or some other issue that may warrant a paper getting rejected without review. What should I do?

Please send us (the program chairs) a message (or email), mentioning the paper submission number and title, and briefly outline your concern.

5. I noticed that a paper I’m handling/reviewing has been previously published. What should I do?

If the paper has appeared in an archival venue (a journal or a conference/workshop with formally published proceedings), then this would be a clear violation of the terms stated in the Call for Papers. Please send us (the program chairs) a message, mentioning the submission number, paper title, and details of the other publication (a link to the official publication would be most helpful).

Please note that presentations at conferences/workshops without formally published proceedings are acceptable (even if the paper is, for example, available from the conference/workshop website). Manuscripts uploaded to arXiv or other preprint servers are also acceptable.

6. I believe that (a version of) a paper I’m handling/reviewing is currently under submission at another conference or journal. What should I do?

Please send us (the program chairs) a message, mentioning the submission number, paper title, and whatever information you have about the other submission.

7. A paper I’m handling includes an appendix. What should I do?

The authors of the paper may have misunderstood the instructions regarding supplementary material and included an appendix at the end of their paper. Instead, they should have submitted this supplementary material as a separate document.

Provided the appendix starts after the references, so the first 8+1 pages look like a correct submission, we will not impose any sanctions in such cases. Please treat this kind of supplementary material just as you would have done if it had been submitted correctly—which means, feel free to simply ignore it.