acl-org / aclrollingreview Goto Github PK

There is no link to modify reviewers, see below:

For comparison, see January 2022 ARR:

Hello,

When resubmitting a paper, we find this:

The title and the subtitle/explanation are contradictory. I assumed that the right interpretation is the long explanation, but I don't know.

Thank you

https://aclrollingreview.org/reviewform does not mention a Soundness score, but I see it is in the review form for the October cycle. Should it be added and explained in a blog post?

Also, will there be any changes to the metareview form?

(moved from openreview/openreview#14 )

Describe the bug
Rating, Review, and Area Chair are missing in the ACL ARR Reviewer Console overview (see screenshot)

ACL ARR January 2022

Screenshots

melisabok commented 1 hour ago
These fields "rating" and "review" are not requested in ARR review forms. Please contact the ARR organizers, they should be able to show any other fields of the review.

Bug description: The JSON format of the current commitment file provided by the ARR website is not formatted for API 2, employed by all commitment venues.

Suggested fix: In collaboration with the OpenReview team, I created a PR with the suggested revision.

From @nschneid:

• What about resubmissions? Is it important to assign the original reviewers if possible (or is it OK to replace, for example, a positive but less informative reviewer)? Note that the platform provides access to the new paper as well as authors' summary of changes. Any other special considerations for handling resubmissions?

I think some tutorials are out there that walk through how author responses are used and give more guidance—e.g., that the author response should be respectful of reviewers' efforts and aim to provide helpful clarifications/corrections.

For explanations of revisions, it would be helpful to provide an example or template (#23). https://transacl.org/index.php/tacl/about/submissions has a template for TACL.

(moved from openreview/openreview#15 )

Describe the bug
I'm not sure if this is intentional or not, but as as reviewer, I can see other reviewers' names, e.g.:

ACL ARR 2022 January PaperXXX Reviewer XXX FirstName SecondName (privately revealed to you)

Expected behavior
I believe reviewers should stay anonymous among themselves. I don't see any reason for doing otherwise. Maybe double-check with ARR editors.

melisabok commented 2 hours ago
It seems the ARR organizers specified this, please check with the ARR organizers.

ACL ARR June 2022

Role: Reviewer

Bug: Ethics review Yes/No -> Radio button only for Yes (but No missing), see below

Fix: Either checkbox or radio buttons Yes + No

The discussion format of the author response period is mentioned as a new feature of the Oct 2023 cycle, but there is no elaboration on whether it is a long-term design choice or how, substantively, people are supposed to engage in it (what kinds of interactions are suggested?).

https://x.com/complingy/status/1820239363222966677

I'm AE for Feb 2023 ARR, and for some papers I can see previous reviewers (?) with some wierd IDs (NaN etc.)... When opening detail page of the paper, there's no reviews visible.

ACL has rescinded the policy on anonymous preprints as applied to future submissions. This affects the CFP, Authors page, Reviewing page, and likely some of the subpages linked from those.

Is your feature request related to a problem? Please describe.
Is there a login link on the home page? I cant find it. It's frustrating to find where to login for reviewing. Adding a Login link at *the top would be helpful.

Describe alternatives you've considered
Alternative is to memorize that I need to login to Openreview.net not aclrollingreview.org

June 2023, AC-ing

after clicking on "Show reviewers", it shows an input box instead of a list of reviewers (?)

Summary

I think that there's a need for identifying reviewers who would have a conflict of interest in reviewing a submission (but whom you don't have a general COI with).
Also, it would make sense to implement structure to nominate reviewers who would be particularly well-suited to review a submission. This is a pretty common interface in a lot of social scientific research venues to help guide reviewer selection.

Per paper COIs

It can be relevant with per-paper COIs for instance in cases where authors are directly addressing issues with other authors' work or where some specific reviewers would have a vested interest in seeing those papers not be published (disproves, refutes, highlights issues or is competing in time with the would-be COIs work).
This would be on a per-paper basis, it isn't a matter of relationship between authors but between reviewers and submitted paper.

Per-paper nominated reviewers

This can be useful, particularly for new tasks/methods/theories that are introduced into the ACL community, where it is unclear who a good reviewer is, even to a well informed AE or SAC (if submitted at conference deadline time). In these cases, it could serve as an aid to AEs and SACs in selecting reviewers if some reviewers are already nominated, as AEs and SACs can select reviewers whose body of work/profile is similar to that of the nominated reviewers, if the nominated reviewers aren't available or there's a COI.

Would it be worth having a suggested template, or an example to demo how to structure the document to respond to each reviewer concern clearly and concisely? Perhaps something like this sample.

It currently looks like this:

The way it wraps across lines is ugly.

Moreover,
a) not all users are necessarily familiar with what "CFP" stands for
b) there is a separation of Reviewers vs. AEs, but nothing about SAEs
c) The vast majority of visitors will be authors, so it is perhaps unnecessary to devote a huge chunk of the navbar to "Reviewers" "Action Editors" "Venue Organizers"

How about:

where "Review Team" is a catch-all for the various roles, including venue PCs. One page would explain the different roles and link to further information.

What do you think?

Questions from @nschneid:

• Is it recommended to always open the floor for discussion, or just when there is disagreement among reviews?
• Are there things AEs should be on the lookout for that would be inappropriate in reviews, and ask the reviewers to edit? E.g. vague criticisms (ask the reviewer to be more concrete), overly harsh tone....

Reviews will be made accessible to authors after the "review release" date. However, I don't know whether it is true also for any threads (discussions) between reviewers and ACs.

For example, this is the original review (with some group access rights which do not contain authors yet, I guess?)

And here is my answer to the review as AC where I tried to clarify some potential misunderstandings:

Both have the same access groups now, but will they all be released to the authors? (which would be favorable, as they add more transparency to the decision process)

Currently the style files are accessed by clicking (taking the LaTeX style file as an example):

"CFP" -> "here" -> "latex" -> "acl.sty"
"Authors" -> "Submission templates" -> copy and paste URL -> "latex" -> "acl.sty"

I would be happy to make the link at https://acl-org.github.io/ACLPUB/formatting.html clickable. I'd also be happy to make a PR here, but want to know if there are any guidelines to follow. For example, do we want there to be a single entry point, and if so, should it be https://acl-org.github.io/ACLPUB/formatting.html or https://github.com/acl-org/acl-style-files?

@mjpost

See the screenshot, should be self-explaining. (ARR January 2022)

Authors should be provided with the option to never be available to review for ARR under specific conditions.
For instance, authors who are not within the NLP field are unlikely to be qualified reviewers for a majority of submitted papers.
To account for such instances, the review unavailability form should include an option to indicate that although one has submitted work to ARR (as a co-author most likely), one is not qualified to review.

The lack of such a possibility creates unnecessary barriers to interdisciplinary work to appear in ACL venues.

Currently, the only possibility to mark yourself as not qualified to review is marking yourself as unavailable far into the future - which subverts the purpose of unavailability form.

See the screenshot. I would expect that "AC Recommendation:" would display the given score (which is called "Overall Assessment" in the meta review), for example

"AC Recommendation: 3" or better "AC Recommendation: 3 (There are major points that may be revised)"