fill missing GT about hail HOT 5 CLOSED

Hi @EugeneEA , if you want to convert every missing GT to homozygous reference, try this:

mt = mt.annotate_entries(GT = hl.coalesce(mt.GT, hl.call(0, 0)))

from hail.

Thanks a lot!

from hail.

This does not work for me unfortunately. I started with a table in coordinate format. I then converted to a matrix table. If I do mt.show() of course some values are correctly missing. But if I then do mt = mt.annotate_entries(GT = hl.coalesce(mt.GT, hl.call(0, 0))) nothing is filled. The NA values remain NA. If I do mt.entry.take(5) or mt.GT.take(5) there are no "None" values in the list. It's as if the mt.GT list only contains the non-missing data. Similarly if I do mt.annotate_entries(test=1), This entry is added only to the coordinates that have GT non-missing and test=NA for everything else! Any help would be appreciated

from hail.

Hey @ag14774 !

Short answer: add an mt.unfilter_entries() before you annotate_entries.

This is a bit confusing but there are two distinct ideas in Hail:

1. filteredness
2. missingness

A row can be filtered, a column can be filtered and an entry can be filtered. When rows or columns are filtered, all their entries are also filtered and the row or column key is excluded entirely. When entries are filtered, its just the individual entry that is removed.

In contrast, any particular value can be "missing". A field, like GT, can be missing, but so can a particular element of an array, or a particular field of an hl.struct.

We think of "filtered" data as not even existing. It's not included in the denominator of, say, hl.agg.mean.

We think of "missing" data as something that exists but is hidden from us. When we do statistics, we need to decide how to treat that data, sometimes we mean-impute, sometimes we convert to hom-ref, sometimes we use a sophisticated model that handles missingness directly.

from hail.

Thanks @danking that worked!

from hail.