Missing values for confounds result in incorrect number of volumes about fitlins HOT 16 CLOSED

This might be a bug with pybids's resampling. My confound variable came in sparse, and was forced dense in get_design_matrix. My guess is the resampling does its job wrong if there are n/as. Will investigate and open issue there.

from fitlins.

Is there a consensus on what to do if you have confounds missing for some volumes?

from fitlins.

Doubt it, but replacing the with mean value seems like a reasonable solution (the "proper" BIDS model solution I discussed earlier).

Either way, I think the re-sampling function has a bug, because its returning the wrong number of volumes.

from fitlins.

You can set skip_empty=False in pybids to keep the column around even though it's all N/As. I'm not a fan of mean imputation in this context, and I don't think it would work in any case given that adding a constant value throughout a run would be collinear with the intercept (which is presumably already included).

I think the right thing to do in this case is (a) drop the run/subject, (b) leave the confound out for that run but include it for everyone else, or (c) not include the confound in the model for anyone. I think (a) is probably the best option in most cases.

from fitlins.

Oh, but if the issue is just with a few volumes within the scan, and the n/as are occurring for sparse data, then mean imputation is probably okay. There should probably be some fairly conservative cut-off though--e.g., more than 5% missing values, and the run gets trashed.

If pybids doesn't currently have any options for ignoring n/as, feel free to open an issue for that. I don't know that mean imputation should be part of pybids though; that does feel like a transformation thing to me, even if it's kind of a pain. Otherwise we lose on the reproducibility front.

from fitlins.

It's only a single missing value in this instance (first volume). I'll open a separate issue in pybids because I think I know what the problem is

Also, pybids in some ways already has mean imputation, if you use scale correctly.

from fitlins.

Is it just that FramewiseDisplacement is defined on the diff of two volumes, and thus is undefined either at the first or last volume? Maybe zero-filling is appropriate, instead.

from fitlins.

Ah, that makes sense, and yes we should zero fill. I guess the larger question remains: which component should handle this, and in which way (silently? by default? for which variables?)

from fitlins.

@chrisfilo Do you have an opinion here? My inclination is that this should be at the Nipype/fMRIPrep level.

from fitlins.

I think n/a is a fair value. Zero is technically incorrect. And this issue is more general and could crop up elsewhere.

from fitlins.

There is also a fillna argument that can be passed down from get_design_matrix which is an option for dealing with this. The problem right now is a bug which omits the value entirely rather than returning n/a.

from fitlins.

Now that that bug is fixed, I would vote for fitlins filling zeros in in confounds by default, but warning if it's doing so. Adding a transformation for this seems really clunky, although it would be more reproducible.

from fitlins.

Agreed, but I would do it only for FramewiseDisplacement and DVARS. Also mean of non-zero values is a better value to impute than zero.

from fitlins.

Mean is better to impute for FD and DVARS? Or 0 for the first volume of FD and DVARS, and mean of non-zeros in general for n/as found in other confounds?

from fitlins.

Mean is always always a better imputation value than an arbitrary constant (such as zero).

Imputing for all regressors comes with risks. I am worried about hiding issues with regressors. For FD and DVARS we are expecting this missing value to it makes sense to impute it. For other regressors I would not do imputation (at least until someone complains).

from fitlins.

Looks like right now we're always imputing 0 for all confounds:

So that needs fixing.

from fitlins.