Our manuscript received 5 reviews. This is an unusual number of reviews. As a member of the ISMB Program Committee I participate in reviews, and the papers that I have been assigned generally get three reviews.
Dear Casey S.,
Thank you for your submission to the ISMB/ECCB 2020 Proceedings Track. We regret to inform you that we are unable to accept your paper for publication. There were many strong submissions this year and only a small fraction could be accepted.
A total of 329 papers were submitted. Of these, 64 manuscripts have been conditionally accepted, giving the final acceptance rate of 19.4%. Each paper was reviewed by several members of the Program Committee, overseen by the Senior Program Committee. Papers were individually reviewed and subsequently discussed by their reviewers to encourage a robust debate and possible resolution of discrepancies between reviewers. Based on the reviews and the outcomes of these discussions, papers were nominated for acceptance into the program. A discussion of the Area Chairs, moderated by the Proceedings Chairs, brought us to a final selection of papers for continued consideration.
We hope you will be able to participate at ISMB 2020 in Montreal, Canada, July 12-16. You may wish to consider submitting to the Abstract Track which includes talks and posters. Submission details are available at:
https://www.iscb.org/ismb2020-submit/abstracts.
We understand that the current public health situation is leaving summer travel plans uncertain at this time, but we are considering virtual options for those who may not be able to attend in person.
Conference registration is open at:
https://www.iscb.org/ismb2020-registration
Sincerely,
Nadia El-Mabrouk and Donna Slonim
ISMB 2020 Proceedings Chairs
SUBMISSION: 189
TITLE: Analysis of ISCB honorees and keynotes reveals disparities
----------------------- REVIEW 1 ---------------------
SUBMISSION: 189
TITLE: Analysis of ISCB honorees and keynotes reveals disparities
AUTHORS: Trang T. Le, Daniel S. Himmelstein, Ariel A. Hippen Anderson, Matthew R. Gazzara and Casey S. Greene
----------- Overall evaluation -----------
SCORE: -2 (reject)
----- TEXT:
The paper describes an analysis of origin honorees and keynote speakers in the main bioinformatics conferences compared with the publication in three related journals using a number of methods to assign the names of the main authors to a gender and geographical origin (honors as the term used is debatable in this context)
At the technical level the paper describes, first; a significant effort to recover the names of speakers and awardees from the conferences and second; a number of methods to assign names to gender and geographical origin.
#38 There have been quite a lot of discussion on the methodological weakness of this paper following its release on bioRxiv. The Race and Ethnicity chapter was particularly controversial, for example mixing it with religious terms was clearly an important error. Therefore, even if interesting, the technology proposed in the paper has still to mature to produce reliable results.
#39 Furthermore, classifying names might reveal an origin but not a nationality and certainly not where the work was carried out. All these consideration seems to scape the analysis (i.e. considering the scientists with a greek name - that is easy to recognise as such- and got some of the ISCB award recently represent a minority when they have developed almost all their profesional time in the USA is very unclear).
#40 It is also unclear why this is the right strategy for the identification of the origin of the awardees/speakers. Direct searches for the names in bibliographic databases or in the society affiliations could be alternative possibilities.
#41 A second problem with this approach is the use of the first/corresponding author of papers as background. Honors are given to scientists with significant contributions, where significant can be highly quoted, or original, or correspond to a long trajectory. Aspects, that might not be necessarily correspond to publications in the mentioned three journals. Indeed, many keynote invitations are to authors in related fields not necessarily published in bioinformatics journals. The logic of the comparison is unclear to say the least.
#42 Additionally, the two main conclusions of the paper are so obvious that is difficult to understand the need of the paper. First, conferences and societies in the field are doing an effort to maintain a healthy gender balance, even if clearly far from perfect the interpretation of the results seems to show a positive trend. Instead of analysing the results he authors go into a very long discussion of the causes and consequences, a discussion that is potentially more appropriate for an opinion paper than for a scientific paper in the conference. The second aspect, is the geographical bias. Yes, it is obvious that most of the speakers are white, the important question to understand the origin of the bias is if there are other more influential authors in the literature that the ones that have been selected in these conferences . The paper does not provide the necessary data to assess is this is the case. To assume that the ideal situation will be to have a number of invitatio!
ns/honors proportional to the number of papers by region does not make any sense from a scientific or from the conference organisation point of view.
#43 In summary, at the scientific level the problem of normalizing names in terms of gender and origin is interesting and even if this paper makes some progress it is still not sufficient for a publication. As an opinion paper, the topic is obviously relevant and the conclusions known (gender balance is difficult and other biases are even more challenging) but the selection of the background (published papers in three bioinformatics journals) is misleading respect to the goal, i.e. selection of speakers and awardees, that is based on other criteria.
----------------------- REVIEW 2 ---------------------
SUBMISSION: 189
TITLE: Analysis of ISCB honorees and keynotes reveals disparities
AUTHORS: Trang T. Le, Daniel S. Himmelstein, Ariel A. Hippen Anderson, Matthew R. Gazzara and Casey S. Greene
----------- Overall evaluation -----------
SCORE: -1 (weak reject)
----- TEXT:
The paper has an eye-catching title, partly because of the nature of the content and partly due to
its implication to ISMB as the flagship conference of ISCB.
Most of the technical content is around associating names to genders/geographic origin or curating publications.
#44 Figs 1 and 2 does not strongly support the title. The over-representation and under-representation has
not been quantified, with the appropriate statistical measures.
A few questions:
#45 1. Nationality/region definition: Mixing of religion and geographic regions is somewhat
confusing for the uninitiated.
I would have certainly benefited by understanding a basis for this classification.
Even in "Nameprism" that the authors cite, the basis is unclear.
#46 2. I was not able to parse the subtle implications of
"We suggest that considering equity may be more appropriate than strictly diversity" that the authors
offer in the Conclusion section.
#47 Any comments on LGBTQ, disability representation in STEM or the field of computational biology ?
Finally, while I think the paper is interesting addressing the social/demographics of the
ISCB community, it is outside the scope of the technical program of ISMB.
----------------------- REVIEW 3 ---------------------
SUBMISSION: 189
TITLE: Analysis of ISCB honorees and keynotes reveals disparities
AUTHORS: Trang T. Le, Daniel S. Himmelstein, Ariel A. Hippen Anderson, Matthew R. Gazzara and Casey S. Greene
----------- Overall evaluation -----------
SCORE: 1 (weak accept)
----- TEXT:
#48 This is an important reflective effort to undertake, and I am glad that it was performed. The analysis on the whole seems reasonable. It would be good to see more citations of this type of work in other disciplines and a comparison of the results here to what has been found in other scientific organizations. It also seems that the work could have been a bit more statistically rigorous in spots, and it would be nice to see more analysis of non-Asian ethnicities.
Some specific comments below.
#49 p. 1, last paragraph: I feel that this first paragraph of the introduction is not the strongest and doesn't add much to the paper overall. I would suggest removing or reworking.
#50 p. 3, paragraph 2: research advisors don't seem to be the best proxy for senior faculty who would be invited for keynotes or honored as fellows
#51 p. 3, penultimate paragraph: if the breakdown is approximately 40/60 on first/last authors being corresponding, why take a majority rule? Seems more rigorous to randomize the selection process according to this "weighted die"
#52 p. 4, paragraph 2: It would be great to see citations for Genderize, especially ones that provide insights on accuracy/reliability. It would also be nice to know how inclusive this app is in terms of names from across the world. Without that, the 1578 missing forenames could be biased by any bias in this app. This discussion is had with respect to wru lower down on the page, so it's natural to provide this standard here.
#53 p. 4, paragraph 4: The terms "race" and "ethnicity" are used in the paper, but there isn't really a definition of these terms in terms of categories. It would be nice to see that explicitly laid out, perhaps with a citation.
#54 p. 4, paragraph 4: Why is using the average demographic distribution a reasonable assumption to make?
#55 p. 7, Figure 2: It would be better to see statistics provided as part of this analysis. It strikes me that the problem at hand here is similar to sample-to-sample microbiome population analysis, which use rigorous statistics, and the same types of methods could be used for analysis in Fig. 2. This level of statistical rigor is present as part of Figure 3, which was good to see.
#56 p. 9, Figure 4: The same point can be made here as for Figure 2. It would be nice to see a statistical analysis and a p-value or two to justify claims beyond appealing to figures.
#57 p. 9, Figure 4: The term "other categories" is used but these are never really defined (see previous point). It would be good to do so, perhaps even in a supplement.
----------------------- REVIEW 4 ---------------------
SUBMISSION: 189
TITLE: Analysis of ISCB honorees and keynotes reveals disparities
AUTHORS: Trang T. Le, Daniel S. Himmelstein, Ariel A. Hippen Anderson, Matthew R. Gazzara and Casey S. Greene
----------- Overall evaluation -----------
SCORE: -3 (strong reject)
----- TEXT:
This manuscript looks at the representation issues when honoring scientists in the field of computational biology by a computational biology society (ISCB). It establishes a background distribution based on the authors of relevant journal publications and then studies subgroup representation of keynotes at three of the ISCB-affiliated conferences (ISMB, RECOMB, PSB) and ISCB fellows against that distribution. It analyzes predicted gender, nationalities, ethnicities, and race of honorees. There are several findings but the major and only stated conclusion in Abstract is that there is recognition overrepresentation of white scientists and underrepresentation of non-white scientists.
#58 This topic is out of scope for the proceedings of ISMB, at least by how I interpret the scope. It has its place in the published literature (I recommend with major revisions), but there is no methodology for or the analysis of molecular and biological data. Instead it is a study about the authors of such papers and broadly speaking it characterizes the recognition and reward system for their contributions to the field.
To give feedback to the ISMB senior committee and also the authors about their work, I offer a critique below assuming the manuscript is within scope of the ISMB proceedings.
Overall this is a clearly written manuscript on a clearly important topic. The work has merits and it presents simple conclusions, but it also has deficiencies in analysis and presentation of results that need to be addressed.
#59 Positives:
-
Social trends in science are important to track and corrective measures are necessary when disparities are identified. The manuscript brings attention to these important issues.
-
Societies should be kept in check and it is the responsibility of the members (and non-members) to appropriately expose problems and fix problems. It is difficult to strike the right tone and for most of the manuscript the authors did it well.
-
The analysis is solid for the most part. The focus on corresponding authors is appropriate. I see no better way to address the issues of gender and race/ethnicity but to employ prediction. I'd assume this prediction is roughly correct for gender. I am less sure about race, ethnicity, religion, etc because a few queries at NamePrism did not convince me of a very high accuracy or satisfy me with its groupings. The authors use their own tool, but I have not queried it. Overall though, I am willing to accept that this works well on average to not impact broad conclusions. The authors recognized that.
-
Although results might appear simple, substantial work has been completed.
Comments on the analysis:
#60 * The analysis focuses on ISCB, its fellows and its conferences. ISCB conferences are listed at: https://www.iscb.org/iscb-conferences however, justification was not provided as to why some conferences made it and others did not. PSB for example was assessed as international but this is hard to justify given its location patterns. GIW/ISCB-Asia is omitted, but no justification was given. This is of course of significant importance for the conclusions. Either justification or expanded analysis is needed.
#61 * The background in the study is created from the publications in well-selected journals (Bioinformatics, BMC Bioinformatics and PLOS Computational Biology). That background is appropriate and provides a useful characterization but is not necessarily the most appropriate. When one talks about a society, an alternative appropriate background would be the professional membership in that society (a society has responsibilities to recognize the work of its members well). The gender information in ISCB should be publicly available so the authors can check the trends and agreements with the current background. The authors could even be able to work out a solution with ISCB to apply the predictors to the data and more deeply test their hypotheses. The conclusions could hold or not and new patterns could emerge.
#62 * Related to above, the selected journals for the background are of global reach (especially Bioinformatics because of rankings and $0 closed-access publication charges) whereas selected ISCB conferences (with their honorees) are more localized in Europe and North America for financial reasons. Considering the field as a whole, there are conferences that are limited to Asia, such as APBC or ISCB-Asia/Genome Informatics Workshop that are both of good quality and well known. APBC 2020 web site also shows that BMC Bioinformatics is among journals for their special issue. This suggests that the journals indeed better represent the field and that conferences are more geographically fragmented, including potentially ISCB conferences. This in turn means that the selection of conferences is critical for the outcome and appropriateness of this analysis. I think it would be appropriate to see a recognition of this fact in the manuscript and adjust the analysis.
#63 * Figure 1. There is no statistical analysis of this data. By eyeballing the graphs it looks like that women are not underrepresented as honorees, and might even be overrepresented. This should be discussed more formally (e.g. statistical tests) and confidence intervals or statistical significance should be provided much like in Figure 2, whatever the conclusions are.
#64 * This may or may not be a problem in this study, but suppose 10 random papers from group A are published by 5 different corresponding authors and another 10 random papers from group B by a single member of that group. The chances that the top scientist is from group A is not 50%. How was this potential problem accounted for? If not, the authors could state how this might impact the outcomes.
#65 * Minor: the keynote speakers for ISMB can be found on ISCB's web site and AAAI web site prior to 2002.
Comments on the presentation:
#66 * The abstract lists only one conclusion in the last sentence: white scientists are overrepresented and non-white scientists are underrepresented. This strikes me as cherry picking of findings. The paper, if I understood correctly, also found that female scientists are not underrepresented. Given the current climate, wouldn't it also be a major conclusion that the honorees are reasonably distributed gender-wise? Instead, the conclusion is "but the proportion has not reached parity". This suddenly changes the focus from ISCB practices relative to the field to ISCB practices relative to the entire society, which is a different background distribution and one where ISCB has little influence on.
#67 * Introduction: "finding that minority scientists tend to apply for awards on topics with lower success rates [1] could be interpreted either as minority scientists select topics in more poorly funded areas or that majority scientists consider topics of particular interest to minority scientists as less worthy of funding." It might look inflammatory to use terminology "less worthy" here. Maybe this was meant in a narrow technical sense, but a reader might decide to go for a broad interpretation and assess that the whole system is discriminatory.
#68 * Figure 4 and the category selection for the analysis are highly problematic. I understand that the categories were selected according to NamePrism, but it is ultimately the responsibility of the authors to justify choices. What is compared is a continent (Europe, Africa), a religion (Muslim), a country (Israel), a part of the continent (East Asian, South Asian), a group of different races or ancestries that spans continents (Hispanic), and then there are Celtic English folk. This analysis needs a substantial revision to the broader and more parallel categories, even compared to previous Figures in the manuscript. The genetics community has been successful discussing continental and subcontinental populations though the problem here is to infer those from names that reflect many things. Singling out Israel as overrepresented sends a tricky message when listed right next to Muslim as underrepresented. It surprised me to see this insensitivity in a study that is meant to asse!
ss where we are as a community and to promote sensitivities.
#69 * I would like to see discussion on the topic of how ISCB should adjust to identify scientists that merit recognition but have fallen through the cracks so far? Can a data-driven help be of use?
----------------------- REVIEW 5 ---------------------
SUBMISSION: 189
TITLE: Analysis of ISCB honorees and keynotes reveals disparities
AUTHORS: Trang T. Le, Daniel S. Himmelstein, Ariel A. Hippen Anderson, Matthew R. Gazzara and Casey S. Greene
----------- Overall evaluation -----------
SCORE: 1 (weak accept)
----- TEXT:
This is an analysis of 411 ISCB honorees who were keynote speakers at ISCB-associated conferences (ISMB, 2002-2019, RECOMB 1997-2019, and PSB 1999-2020) as well as ISCB Fellows named (2009-2019).
The profile of awardees was compared to a corpus of articles in Bioinformatics, BMC Bioinformatics from (2002-), and PLOS Computational Biology (2005-). PMC corresponding author information when it was available (20,696 articles) and the PubMed last author as a fallback when corresponding author information was missing (9,053 articles). It is unknown how many of these articles were published by the name authors, or if the were attributed uniquely to authors. For example, it is custom in some institutes that the dept chair is a senior or corresponding author, and thus some authors might be over-represented by chance in the data. Recent Increases in Asian corresponding authors was primarily driven by recent publications in Bioinformatics and BMC Bioinformatics. The % of Asian authors estimated from these journals suggests a lack of Asian scientists among keynote speakers and honorees.
The authors created the pubmedpy Python package to parse names from articles and used https://genderize.io to predict gender. Race and ethnicity of honorees and authors was inferred using the R package wru and were associated with race/ethnicity category based on US census data
#70 Of 34,050 corresponding authors, over 25% (8,770) had a last name for which wru did not provide predictions. Do the author know if these were random or non-random in distribution.
#71 There are assumptions and analysis issues. Did the author check if the association between predicted race/ethnicity and author Affiliation? Was Asian bias by country or ethnicity? For example, are American-Asians represented in ISCB honorees but Asians who are geographically located in Asia under-represented. Is this associated with geographic attendance at ISCB meetings? One might expect local awardees in the host country. For example, the authors note that ISMB keynotes had more probability attributable to Israel, while RECOMB had more attributable to East Asian countries.
#72 Given this small pool of 411, some names of which were duplicated (fellows and keynotes etc), were they manually checked for accuracy.
#73 There are non-random missing data. European Spanish/Portuguese/Italian are called Hispanic. Assignment of the race (white/black/Asian) based surname is difficult to extrapolate to a multi-cultural society where last name and skin color are maybe discordant.
I will break these out into component issues and edit the block quotes above with the GitHub identifiers.