Predicate to filter conversion. about adam HOT 9 CLOSED

Just curious, why do we need to convert the predicate to a Spark filter rather than just not ignoring the parquet predicate in the BAM/SAM/VCF code path?

from adam.

Ah, please disregard. I thought the predicate was being ignored for BAM/SAM data inside ADAM files (the name adamLoad is confusing in this respect)

from adam.

I took an attempt at this here: https://github.com/hammerlab/adam/compare/predicate-bam?expand=1 Would love some feedback or alternative approaches - right now this only supports boolean equality filters.

from adam.

This looks good! Are you intentionally only starting with booleans for the
value to sketch out the broader approach, or am I missing something and
that will encompass all possible filters?

On Wed, Apr 16, 2014 at 2:54 PM, Arun Ahuja [email protected]:

I took an attempt at this here:
https://github.com/hammerlab/adam/compare/predicate-bam?expand=1 Would
love some feedback or alternative approaches - right now this only supports
boolean equality filters.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/62#issuecomment-40637624
.

from adam.

Yea it was for simplicity, pushed an update now for more general equality support. I hadn't realized that Parquet only supported equality filters. @fnothaft I saw you were the last contributor to Parquet on that - is there support elsewhere for other types of ColumnFilters?

Also, I see what I did only supports AND's of conditions as well.

from adam.

Good stuff. I wonder if it would be worth it to generalize and let people
pass in arbitrary boolean functions rather than specific values to match
against. The function could take as input the Parquet value. Would that
work? Then you could express arbitrary computation as a condition for the
record to pass.
Neal

On Wed, Apr 16, 2014 at 7:32 PM, Arun Ahuja [email protected]:

Yea it was for simplicity, pushed an update now for more general equality
support. I hadn't realized that Parquet only supported equality filters.
@fnothaft https://github.com/fnothaft I saw you were the last
contributor to Parquet on that - is there support elsewhere for other types
of ColumnFilters?

Also, I see what I did only supports AND's of conditions as well.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/62#issuecomment-40665152
.

from adam.

Nice! Indeed, this does look cool, and like a nice approach.

@arahuja I've actually added code to Parquet that supports arbitrary predicate functions. These are the "applyFunctionTo*" predicates. If you've got any questions about the predicate functions, or would like any changes made to them, let me know. I'd be glad to help on this, or to work on anything that'd make the predicates easier to use.

from adam.

@arahuja @fnothaft Is this issue still live, or can it be closed?

from adam.

Closed by #234.

from adam.