- Stay tuned
International Society for Music Information Retrieval (ISMIR)
As soon as you are notified of your assigned paper, please check all of them to make sure that:
Please, if any issues arise regarding these points, notify them immediately to the Program Committee. You should schedule in advance your review work, and not leave it for the last few days near the deadline. Adhering to the deadline is essential, since the Program Committee and Chairs still have a huge amount of work to do after the review process has been completed. It is advisable to read papers well in advance before the deadline in order to have time to think about them over a sufficiently long timespan before writing your reviews. This is essential in order for you to make thoughtful decisions about your assigned papers, and to provide helpful suggestions for the authors.
You will be asked to provide an evaluation of the papers assigned to you according to the following criteria:
Keep in mind that minor flaws can be corrected, and should not be a reason to reject a paper. However, accepted papers have to be technically sound and make an original and substantial contribution to the field. Also, regarding the topics of the conference, please familiarize yourself with the information in the Call for Participation.
When deciding your recommendation for a paper, do not be shy. Use the whole spectrum of evaluation scores: if you think a paper is outstanding, give it the highest score and no less than that. Similarly, if you think a paper is really bad (and can convincingly support your opinion), then give it the lowest score and no more than that. Very often reviewers tend to use intermediate scores, because they are not entirely confident on their judgment, and/or because they did not have time to read their papers thoroughly. This attitude causes really good papers not to stand out, and very bad papers not to be “caught”.
Ensure that your scores are consistent with your comments to the authors. In particular, receiving good comments and a poor score is frustrating, and often causes the authors to request clarifications or rebuttals.
Your comments for the authors are probably the most important part of your reviews. They will be returned to the authors, so you should include any specific feedback which help improve the papers. Thorough comments also help the Program Committee decide which papers to accept, sometimes more than your score. Remember that your reviews are evaluated by the Program Committee. Moreover, after the completion of the review process, they will be available to other reviewers of the same papers. Therefore your good work will help generating a positive trend in the research community.
Short reviews are not helpful to either the authors and the Program Committee. Please be as specific and detailed as you can. When discussing related work and references, simply saying “this is well known” or “this has been common practice for years” is not appropriate. You should cite publications, or other public disclosures of techniques, which can support your statements. Be specific also when you suggest improvements in the writing. If there is a particular passage in the text that is unclear, point it out and give suggestions for improvements. On the other hand, if the whole paper is poorly written, you are not expected to rewrite it for the authors.
Be generous about providing new ideas for improvement. You may suggest different techniques or tools to be used in the applications presented in a paper. You may also suggest the authors a new application area that might benefit from their work. You may suggest them a generalization of their concept, which they have not considered. If you think that the paper has merits but does not exactly match the topics of the ISMIR conference, please do not simply reject the paper but communicate this to the Program Committee. These papers will then be looked at and discussed with specific care. Suggestions for alternative publication options that are more appropriate in your opinion (journals, conferences, workshops) are welcome.
Remember that academic careers and reputations rely on scientific publications. Therefore you have to be seriously committed to your work as a reviewer.
In the past there have been complaints about some reviews being too sketchy and superficial, so that it looked like the reviewer did not take the time to read the paper carefully. A sketchy or casual review is a lack of respect to the authors who have seriously submitted their paper, and in the long run damages the ISMIR conference. If you have agreed to review a paper, you should devote enough time to write a thoughtful and detailed review. If you think you cannot review properly your assigned papers because you are too busy, you should not commit to your assignment. However, please do communicate this to the Program Committee as soon as possible so that the papers can be reassigned in time. Acting that way you are helping much more than doing a superficial review.
Keep in mind that belittling or sarcastic comments are not appropriate. Even if you think that a paper is really bad, you should be constructive and still provide feedback to the authors. If you give a paper a low score, it is essential that you justify the reason for that score in detail. Just saying “I do not like this approach because I am a guru in this area” is not constructive. Also keep in mind that directly talking about the authors can be sometimes perceived as being confrontational, even though you do not mean it this way. For this reason, you may want to avoid referring to the authors by using the phrase “you” or “the authors”, and use instead “the paper”.
As a reviewer you have the responsibility to protect the confidentiality of the ideas represented in the papers you review. Submissions to the ISMIR conference have not (or should not have) been published before.
Although the authors’ ultimate goal is to publish and disseminate their work, a percentage of submitted papers will not be accepted and published in the ISMIR conference proceedings, and will most likely be submitted to some other journal or conference. Sometimes a submitted paper is still considered confidential by the author’s employers or funding sources. In order to comply with confidentiality requirements,
Although some reviewers like to disclose their identity to authors, it is advisable not to do so. One of the most common ways of inadvertently disclosing your identity is asking the authors to cite your past work and several of your own papers. This should be avoided. Besides, this attitude may have a negative effect on your review: it may be seen as if you just want to gain more citations, and may ultimately result in the authors just ignoring your review (and possibly the Program Committee too).
Even though you would judge impartially any paper assigned to you, there has to be no doubt about the impartiality of your reviews. Therefore, if there is a potential conflict of interest with one of your assigned papers, you should inform the Program Committee. Although in general you should use your judgment, examples of situations of potential conflicts of interest are the following:
In case you have any doubt about a potential conflict of interest, then rather decide that there is such a conflict and contact the Program Committee.
This document closely follows the guidelines of the ISMIR2012 and SMC2011 conferences and is partially based on the reviewers’ guidelines of the CVPR2010 and SIGGRAPH2008 conferences: