Anonymous Peer Review Process

CNTAC joins the large variety of peer-review panels in double-anonymous review of proposals. This paradigm requires both proposers and referees to have no information about the other, and there is an extensive literature supporting that this format helps avoid conscious and unconscious biases in peer-review processes.

For semester 2023B, CNTAC encourages proposers to write anonymously according to the guidelines below. However, there will be no enforcement yet. Future semesters will see changes in policies and in the proposal submission interface moving towards a full dual-anonymous review.

Guidelines for applicants

This section provides the guidelines the applicants must follow for anonymizing their proposals and is adapted (with permission) from ESO guidelines at: https://www.eso.org/sci/observing/phase1/dual-anonymous-guidelines.html.

These apply to the entire proposal except sections “current status of the project and team involvement”. These fields as well as any identification of proposers will not be included in the material distributed to the reviewers during the proposal review phase and will only be accessible to them after the ranking phase is completed.

Here follow the main criteria the users must comply with:

• Do not include applicant names or affiliations anywhere in the title, abstract, scientific or technical justification. This refers to all anonymous parts of the document, including diagrams, figures, captions, watermarks, etc.
• When citing literature and providing references, especially in the case of self-referencing, third person neutral wording must be used. For instance, the sentence “as we have demonstrated in Galilei (1610)” must be rephrased as follows: “as shown by Galilei (1610)”.
• Future time awarded at other facilities for the same science case (and which has not produced any data yet) must be cited in an impersonal way, and referred to using the «(private communication)» expression. For example, the sentence: «We have obtained 30 orbits of HST time (HST001) for…» must be replaced by: «Thirty orbits of HST time were awarded for… (private communication)».
• Acknowledgments must be avoided, and so must references to possible grant funding (the latter can be listed in team involvementsection of the proposal).

Citing own work, data and models

One of the main difficulties encountered by the applicants while quoting in an anonymous way their previous work, data, data-reduction tools and/or modelling capabilities, is the fear that, in the lack of a proper and explicit citation, the reviewers would not trust them. Also, applicants may refrain from using the first person (though anonymous) fearing that this would violate to some extent violate the anonymization guidelines.

In those cases in which the applicants wish to use the first person forms (e.g. «As we have shown (private communication)«, «The data will be processed with our pipeline«, etc.), they should keep in mind that:

• The reviewers are instructed not to try to guess the team’s identity by explicit searches on the web. This constitutes a violation of the reviewer’s guidelines. For the same reason, team identities resulting from active searches by the reviewers, will not be considered as anonymization violations.
• The reviewers are instructed to trust the statements made by the applicants even in the cases in which, because of the anonymization, it is not possible for them to check directly in the literature.
• In case of doubts, the reviewers can ask CNTAC to verify the correctness of one or more statements. This will be done by CNTAC coordinator and chairperson checking the content of the “current status and team involvement” It is therefore important that the applicants fill this section with all the details they reckon are necessary to allow CNTAC to verify the team’s qualifications with respect to the proposed observations (expertise, data reduction, analysis, modelling).

Instructions for the specific cases are provided in the next sections.

Propietary/unpublished work/data/codes

Here follows a summary on how to properly quote proprietary/unpublished work/data from the proposing team. For each case an example of anonymized reference is provided.

• Published Proprietary models/codes (published/advertised, developed by the applicants but not publicly accessible)

«Calculations run with FANTASTIC code (private communication) predict that…»

• Unpublished Proprietary models/codes(developed by the applicants, not published/advertised, not publicly accessible)

«Based on a model we have developed (in preparation) we predict that…»

• Public data, unpublished results (data publicly accessible to everybody, no publication)

«After inspecting FORS2 archival data (ESO programme ID 105.20AB), we have detected emission lines in…»

• Proprietary data, unpublished results (data accessible only to the proposing team, no publication)

«From the analysis of an existing MIKE@Magellan spectroscopy data set we have selected a sample of suitable candidates…»

• Proprietary data, published results (data accessible only to the proposing team, published)

If the statement refers to results contained in a publication:

«As shown by Author et al (2019), NGC300 is a perfect laboratory for …»

If the statement refers to new results, not contained in a publication:

«The analysis of existing DECAM data has revealed that…»

All these formulations are sufficiently ambiguous and can be safely used in the text. Of course, you can (and you should) be explicit in the non-anonymized sections (previous results and team involvement). This will give the reviewers the opportunity to check the correctness of the anonymized statements, once the review is completed, should they reckon this is critical for the specific case.

Continuation/follow-up proposals

In many cases a proposal is a continuation or a follow-up of a recently approved project, led by the same team. The data and the results from that proposal (especially in the case of a pilot program) may play a crucial role for the new submission and hence need to be properly referred to. This is a particularly tricky case, one of those in which reaching 100% anonymity is hard. Depending on the status of the data (proprietary/public) and the results (unpublished/published), the considerations presented in the previous section do apply to this case as well.

The applicant should put all non-anonymous details on “Previous usage”, but refrain from reveal their identity elsewhere, a few examples on how previous allocations and corresponding results should be referred to in the Scientific or Technical Justification section of the proposal.

Unpublished results

1. «An analysis of existing TripleSpec data (private communication) has shown that…«
2. «Based on the analysis of data obtained in previous semesters, we have now refined the sample…«
3. «Recently, a pilot study has revealed that … (private communication). We propose follow-up observations to…«

Published results

(in the following examples, Explicit et al. are both the authors of the paper/s and the applicants)

1. «A study of  a sample of  17 galaxies has shown that 90% them are the products of major mergers (Explicit et al. 2019) . With this proposal we plan to extend that study to a sample of 231 galaxies. This will enable…»
2. «Explicit et al. (2020) have shown that HeI6678A can be used to determine … We propose to exploit this technique to…»

Caution in using the explicit reference/s has to be exerted depending on the sentences that preceed and follow the citation. If an obvious connection can be made between the publication and the time already allocated to the same project (if specified in the dedicated section), it is more appropriate to use the implicit version suggested for the unpublished results:

1. «A pilot study of a sample of 17 galaxies has shown that 90% them are the products of major mergers (private communication). With this proposal…»
2. «It has been recently demonstrated that HeI6678A can be used to determine … (private communication). We propose to…»

Generic references to previous allocations

1. «Time was already allocated to this project for the RA range 0-12. This proposal extends the request to the RA range 12-24.»
2. «Time was already allocated to this project. Only 50% of the observations were carried out. With this proposal we plan to complete the sample.»

Complying with the above guidelines requires working on the grammar and structure of the scientific rationale. As a consequence, it will not be possible to re-cycle previous material without reviewing it in the light of the anonymization requirements. The proposing teams should take this into account when planning their submission/s, because text anonymization requires some extra effort.

Team Involvement

The applicants are required to fill in the Team Involvement section. This is meant to be a short descriptino of the background, expertise and roles of the various team members in the context of the science case discussed in the proposal. Chilean involvement in the proposal should be highlighted, specially when the proposing team consists mainly of foreign collaborators. 

Important note

Users must not add comments with PDF editors to their rationales as, depending on the editing software, the author/s of the comments is/are visible when displaying the uploaded PDF, hence disclosing their identities.

The applicants are required to fill in the Team Involvement section. This is meant to be a short description of the background, expertise and roles of the various team members in the context of the science case discussed in the proposal. Chilean involvement in the proposal should be highligthed, specially when the proposing team consists mainly of foreign collaborators.

Important note

Users must not add comments with PDF editors to their rationales as, depending on the editing software, the author/s of the comments is/are visible when displaying the uploaded PDF, hence disclosing their identities.

Guidelines for reviewers

Starting with Period 2023B, anonymization by the applicant is encouraged and the dual-anonymous procedure will be enforced on the reviewers’ side. The reviewers should follow these guidelines:

• The review must aim at selecting the most promising proposals, not the best proposing teams;
• The ranking is purely based on the scientific merit of the proposals: the pre-meeting review and the panel discussions must focus on science only;
• The background and expertise of the applicants with scientific facilities is not to be considered during the evaluations;
• The reviewers should not try to guess the PI’s or the team’s identity. Actions taken in this sense by the reviewers will be considered as violations of these guidelines. For the same reason, team identities resulting from active searches by the reviewers, will not be considered as anonymisation violations;
• The chairs of the panels must refocus the discussion whenever this moves to the team identity, expertise or publication record;
• The main purpose of the anonymization is to reduce the sources of «distraction», which may influence the objectiveness of the process;
• The reviewers should flag to the CNTAC coordinator all the cases that they reckon do not comply with the anonymization rules spelled out above;
• In principle, only open violations will be considered for proposal rejection by the CNTAC chairperson. Although the specific cases will require a closer scrutiny, open violations are those which clearly indicate that the proposing team did not make any effort to obfuscate their identity;
• In general, the applicants have done a good job when the identity of the team is reasonably ambiguous;
• Cases in which the identity of the team cannot be derived directly from the proposal, but it can be determined by deliberate web searches by the reviewers (which are anyway highly discouraged), cannot be considered as a violation of the rules;
• The reviewers must consider that it is impossible to reach 100% anonymity in all cases. There will always be certain cases in which it is impossible to conceal the identity of the teams. This is a fact that has to be accepted.
• As a rule, the reviewers should trust the statements made by the applicants about their declared achievements, data reduction, analysis and modelling capabilities, although these are kept anonymous. In case of doubt, the reviewers may ask CNTAC to verify the correctness of the statements made by the applicants.

Types of anonymization violations

When scrutinizing the anonymization violation reports submitted by the reviewers CNTAC will categorize them in three classes:

1. Major violations– those revealing directly and unambiguously the identity of the proposing team (e.g. «As we have shown (Galilei et al. 1609)»). Major violations are flagged to the CNTAC chair and lead to the disqualification of the proposals. If identified in advance, these proposals will not be evaluated by the review panel.
2. Minor violations– those revealing unambiguously but indirectly the team’s identity. In these cases the identity cannot be deduced from the proposal’s content, but may be derived by secondary sources (e.g. including project names which are quoted in cited publications). Minor violations do not lead to proposal disqualification, but will turn into a warning to the PI, asking them to fix the issue/s for future submissions.
3. No violations:
   1. The team’s identity remains ambiguous (e.g. there is more than one possibility);
   2. The team has thoroughly followed the anonymization guidelines and it is impossible to fully hide their identity (e.g. very well-known teams working on very specific science cases);
   3. Cases in which the identity is found by explicit searches by the reviewer.