NULL values on indexed columns support about qbeast-spark HOT 5 CLOSED

Merged in #67

from qbeast-spark.

Right now we throw an exception for this, but for further enhancement #3 we should investigate how to deal with null values on columns (when schema evolution happens and a new column is added, the old files would have null values for that column)

from qbeast-spark.

At the moment the columns which are not indexed can have null values. The reason is that indexed columns must not have null values right now, and OTreeAlgorithm prefer to use non-nullable columns to calculate the hash. This should be tested again when we implement null value support for the indexed columns.

from qbeast-spark.

Some random thoughts on possible solution.

The idea is represent the NULL values with some special numerical values to let the indexing work as in case of non null values. Space revision should help to avoid collision between the user defined values and special values.

When a space revision is created we fix the boundaries for the data coordinates. If later new data violates the boundaries then we create a new space revision. As soon as a space revision is created we can define a linear transformation which maps the data into a space like [0 + eps, 1]^n, where n is the number of dimensions (currently we map into the space [0,1]^n). Then null coordinate values can be represented by 0. The indexing can be done as we do it now.

The next challenge is to handle the queries correctly. As soon as we have cubes containing the query results we should filter out those elements which does not satisfy where clause because of null values. I hope it is not a problem for Spark, after all indexes are unlikely to return the correct data, they are likely to return some small superset of data, so I expect that all the elements retrieved from the index blocks will be tested to satify the query.

@osopardo1 @cugni what do you think? Thanks.

from qbeast-spark.

First, indeed, we don't care so much about those null values for schema evolution. Since we will be working with different space revisions, the null values of the new columns do not affect older versions.
Then, for the normal datasets with null values, I think you are right. We can treat them with 0.

Yes, on Spark, all elements retrieved from index blocks will be filtered again on memory to satisfy the where clause.

from qbeast-spark.