egouldo / manyanalysts Goto Github PK

SM3: Table C5, change 'mixed model' to 'Random Included' or some version, so that is consistent with other language throughout.

Line 938 I think a formula for $Zr$ is warranted here.

and/or add citation. I think I struggled to find this in the literature but ended up finding it in Mengersen's meta-analysis for ecology handbook back when I double checked the calculation originally when coding the analysis.

Bullet the following rather than having separate paragraphs.

Chunk descriptive-munging is no longer executing, giving the following error on knit:

processing file: index.qmd
  |....                           |  11% [descriptive-munging]                 Error in `UseMethod()`:
! no applicable method for 'inner_join' applied to an object of class "character"
Backtrace:
 1. dplyr::inner_join(BTSorensen, effect_Ids)
                                                                                                                                 


Quitting from lines 2418-2789 [descriptive-munging] (index.qmd)

Execution halted

Also, I note that this code is duplicated in SM2_EffectSizeAnalysis.qmd, however in the latter document, the objects are mostly subset.

• Shift funs into ManyEcoEvo (egouldo/ManyEcoEvo#24)
• Shift code into ManyEcoEvo targets pipeline (egouldo/ManyEcoEvo#24)
• Retain manipulation only needed to extract required data in both index.qmd and M2_EffectSizeAnalysis.qmd, culling the remainder
• Ensure dependent object calls execute (specifically in-text R expressions and tables etc. #32 )

• @parkerth to identify which analyses to drop
• @egouldo BT forest plot, with dropped analyses that have substantial collinearity

Currently only can navigate to Appendices but would be better to also navigate around main text as is super long.

• The data and code to reproduce the results in the manuscript is provided in their R package (hosted at Zenodo and GitHub account of first author) called ManyEcoEvo.

• Add link to Zenodo
• Add link to GH

• provide the link (https://github.com/egouldo/ManyAnalysts/tree/main) (not just to the rendered HTML manuscript at https://egouldo.github.io/ManyAnalysts/, which I note is in the supplementary file)
• cite the software they used for writing their manuscript in their references, i.e. Quarto, gt and ggplot2
• The formatting in HTML manuscript was more pleasurable and it would have been nice to have been alluded to this earlier rather than discovering it later

• Agreed the word doc is hell! Let's point readers there earlier. Suspect the formatted published article will be better than the word doc but will still be a challenging read
• #18

Each additional post-hoc analysis that is included nullifies every positional reference in in-text R expressions. After updating for #27 will need to ensure that EVERY single in-text expression is matched semantically (by column and row values) rather than by position.

Next, we will assess the relationship between the inclusion of random effects in the analysis and the deviation from the mean effect size. We anticipate that most analysts will use random effects in a mixed model framework, but if we are wrong, we want to evaluate the differences in outcomes when using random effects versus not using random effects. Thus if there are at least 5 analyses that do and 5 analyses that do not include random effects, we will add a binary predictor variable random effects included (yes/no) to our set of univariate analyses and will add this predictor variable to our multivariate model described below.

• update tense

Thanks Tim, I think I was just relying on the two headers being under their respective sub-headers for the precise meaning of each sub-heading being conveyed. But it’s probably best to specify exactly what outputs these belong to, so I’ll update them.

Elliot.

On 1 Jun 2024, at 6:34 am, Tim Parker [email protected] wrote:

Note this redundancy between these two supplement sections

A.2.3 Post-hoc analysis: Exploring the effect of removing analyses with poor peer-review ratings on heterogeneity

B.1.1.2 Post-hoc analysis: Exploring the effect of removing analyses with poor peer-review ratings on heterogeneity

Line 988 should the “meta-analytic mean Zr or yi” have bars over them? I think so, to indicate mean value.

I don't think the following statement is correct:

The blue tit dataset had a total of r length(unique(var_names[[1,2]][[1]])) candidate variables, each of which were used in at least one analysis.

Based on what I observed when removing bugs in descriptive munging chunk to incorporate collinearity analyses.

For some reason, in the rendered version, two of Shinichi's citations are including his first name in the rendered text. In this line, it appears as:
linear models [GLMs; Shinichi Nakagawa and Cuthill (2007).

Shinichi's 2022 paper also has this problem. I have looked that the citations in the .bib file, and I cannot see any difference in formatting compared to other citations. I have not seen this problem arise for any other citations.

I didn't quite understand Supp A.1.3 where it said that XX candidate variables were used in a mean of YY Zr analyses. Does that mean YY analyses used all XX candidate variables?

SD misleading (larger than mean) if don't know that distributions are skewed. We will add ranges for all descriptive statistics.

Note, these are already calculated in object Table2 as min and max.

Hey Tim,

I think this may potentially be a result of the way I repeated the analyses over each of the different data subsets, i'll look into it.

So, the models may have been fitted automatically, but I never extracted the results of the cases where we had < 5 analyses with random effects included.

1. I’ll make sure these cases are removed from the manuscript first,
2. And then can remove these from the analysis pipeline.

Elliot.

On 1 Jun 2024, at 6:14 am, Tim Parker [email protected] wrote:

Hi Elliot,

In the supplement
C.6.1 Effect Sizes Zr
also
C.6.2 Out of sample predictions yi

It appears that the presence of mixed effects (whether or not the analyst included a random effect) was included in both the BT and the Euc models, but I thought that we only included this effect for Euc since there were too few BT analysts that did not include a random effect.

-Tim

Where did this figure come from??

r All_Teams+28 call in Summary Statistics section

https://github.com/egouldo/ManyAnalysts/blob/28c4a40fa016deae5837cf59b2b22f8e18b8f170/index.qmd#L3789\

I included a placeholder citation for the new forest plot that excludes the effects from blue tit analyses containing the two highly collinear predictors.
This placeholder is "(@fig-forest-plot-?????)"

The table captions (but not the figure captions) are centered instead of left justified. Also, there narrow lines above and below most of the specially formatted text.

Page 45, Line 1005-1006 suggest that the ReviewerID and AnalysisID have the same variance. I think the authors meant this to be different so $\sigma^2$ should have different notations. Same for Page 49, Line 1043-1044.

For the univariate analysis of the Deviation Score, I don't see mention of weighting based on the inverse variance however looking at the analysis, it looks like the authors did this for continuous rating (https://github.com/egouldo/ManyEcoEvo/blob/main/R/fit_boxcox_ratings_cont.R#L44) but it's commented out for categorical rating (https://github.com/egouldo/ManyEcoEvo/blob/main/R/fit_boxcox_ratings_cat.R#L52). I'm wondering why?

Can't recall exactly why, and maybe this is an error, but from memory I think there were some problems with model fit for the categorical ratings and inverse variance?

• Check the git blame and notes for this line
• Rerun with what was preregistered - model fit etc. ok??

Model not included in that parameter table.
Check that model parameter estimates for Sorensen's models are included somewhere in that document.

In the Supp Mat (SM3_ExplainingDeviation.qmd), Subsection C.4.1 Effect Sizes (deviation scores as explained by distinctiveness of variables in each analysis) is empty...?

Is this because this content occurs elsewhere in the manuscript?