Ideas for reducing file sizes about scpca-nf HOT 6 CLOSED

Tagging @allyhawkins, @sjspielman and @jaclyn-taroni for comments

from scpca-nf.

This can be implemented with a change in write_rds() from compress='gz' to compress='bz2', and will be completely transparent to users as far as code goes, as the result is still an rds file

To me, this seems like a no-brainer - definitely do it.

One other idea I had was to remove the miQC_model object from the "processed" files. At that point, the data in the object can't really be used as the rejected cells have already been removed. This object is also fairly large (100MB or more) and does not appear to compress well, so removing it would save a significant amount of space, and having it in both the "filtered" and "processed" data seems redundant. This change would require a docs update as well.

I wasn't sure how I felt about this at first, but you've convinced me with At that point, the data in the object can't really be used as the rejected cells have already been removed. As long as we keep it around somewhere (in filtered) I'm ok removing from processed.

from scpca-nf.

In my testing, this takes about 50% longer (e.g. 35 seconds vs 21 seconds, though so not much really), but results in files ~50% smaller. However, the read time increases substantially, from ~1.4 to 9.4 seconds in the example I tested. I tend to think the tradeoff is likely worth it for our use case, but others may disagree, so we should discuss this!

My concern would be read time for the larger merged objects, not necessarily the smaller individual objects. I think at least for the individual RDS files then this proposal makes sense.

One other idea I had was to remove the miQC_model object from the "processed" files. At that point, the data in the object can't really be used as the rejected cells have already been removed. This object is also fairly large (100MB or more) and does not appear to compress well, so removing it would save a significant amount of space, and having it in both the "filtered" and "processed" data seems redundant. This change would require a docs update as well.

I'm totally onboard with removing it from the processed objects.

from scpca-nf.

My concern would be read time for the larger merged objects, not necessarily the smaller individual objects. I think at least for the individual RDS files then this proposal makes sense.

Good middle ground!

from scpca-nf.

closed by #712

from scpca-nf.

Though docs updates are still pending in AlexsLemonade/scpca-docs#273

from scpca-nf.