metafor update: random effect structure does not work any more about metalab HOT 10 OPEN

Hi all, some new info:

1. Generally, we would want ES- and study-level random effects. So that can be a general approach.
2. Some MAs have non-independent ES for two reasons:
   a) The same participants have been measured in multiple conditions (I count first versus second half of the test phase here, too, although we might want to reconsider).
   b) The same participants have been measured in multiple ways (eg overt and looking responses).

Both cases can be found in the IDS-pref MA.

So far, we treated both reasons the same, but that's not ideal. We should add a way to separate these cases, which I think can be done by checking whether there are differences in response_mode

So we can do a number of things now:

• Overspecify the random effects structure, which can have unwanted consequences (but in most cases will not dramatically affect fixed effects estimates)
• Have a check that eg more than 5% of ES are affected in a MA and then switch to a different random effects structure (with less, the exercise might be futile anyhow)
• Look into cluster-robust variance estimators, as implemented in an rma.mv() compatible way in ClubSandwich

Any preferences?

from metalab.

from metalab.

Yep, sounds like a plan.

from metalab.

Hi all, I am going ahead and implementing the updated random effects structure as follows:
random = 1 | short_cite / same_infant_calc / unique_row

Left to do:

• cluster-robust inferences methods as sensitivity check
• Assess impact on estimates, moderators
• Figure out how to deal with datasets where 2 different DVs come from the same participant (IDS preference for example)

from metalab.

note that random = 1 | short_cite / same_infant_calc / unique_row + exposure_phase is interpreted as random = 1 | short_cite / same_infant_calc / unique_row / exposure_phase

from metalab.

@alecristia But for the general format, we only want random = 1 | short_cite / same_infant_calc / unique_row right?

from metalab.

I think so... it's been a while ;)

from metalab.

by the way, I though we were recommending the use of robumeta (I know 2 of my team members have tried/succeeded in using it), but robu doesn't appear anywhere in our code up here, which seems to be all metafor rma.mv:
https://github.com/langcog/metalab2/search?q=same_infant_calc&unscoped_q=same_infant_calc

Nothing to do if that was intentional!

from metalab.

from metalab.

Sharing these resources from ESMAR 2021 in case they help when we get back to this Issue
https://osf.io/vyfcj/
https://youtu.be/Q9Nce5pxebY?t=50

from metalab.