Comments (7)
Have custom logic for unions that looks up the child array to determine if the value is null
+1 for second option. I think we should check the children's nullability.
from arrow-datafusion.
I've proposed a fix in #11321.
from arrow-datafusion.
@alamb any idea on where I would start looking to try and fix this?
from arrow-datafusion.
I would suggest writing a standlone test case / reproducer as the first step
Then I suspect we can either help you find the code needed to be fixed (or maybe even someone would be interested in fixing it themselves)
from arrow-datafusion.
See #11314 as a demonstration of the problem for both dense and sparse unions.
After a bit of investigation, the issues lies in the first instance with
datafusion/datafusion/physical-expr/src/expressions/is_null.rs
Lines 74 to 84 in 08c5345
Then with this code in arrow-rs
:
/// Returns a non-null [BooleanArray] with whether each value of the array is null.
/// # Error
/// This function never errors.
/// # Example
/// ...
pub fn is_null(input: &dyn Array) -> Result<BooleanArray, ArrowError> {
let values = match input.logical_nulls() {
None => BooleanBuffer::new_unset(input.len()),
Some(nulls) => !nulls.inner(),
};
Ok(BooleanArray::new(values, None))
}
And then with this code
/// Union types always return non null as there is no validity buffer.
/// To check validity correctly you must check the underlying vector.
fn is_null(&self, _index: usize) -> bool {
false
}
Ultimately with the spec
Unlike other data types, unions do not have their own validity bitmap. Instead,
the nullness of each slot is determined exclusively by the child arrays which
are composed to create the union.
Basically arrow is saying "we're not going to tell you if a union is null, you need to look in the child arrays", but datafusion isn't listening and is just asking the union if it's null in the naive way.
Two options to move forward as far as I can tell:
- Decide unions in DF can never be null — I'll need to abandon unions in
datafusion-functions-json
and just return strings everywhere - Have custom logic for unions that looks up the child array to determine if the value is null
If (as I hope) we go for the second option, there's also the issue (as demonstrated by #11314) that the representation of "null" union items doesn't match other types, it shows {A=}
instead of an empty string.
from arrow-datafusion.
I suppose there's a third option of updating arrow-rs to correctly calculate if a UnionArray
is null, but I presume that works take much longer
from arrow-datafusion.
I suppose there's a third option of updating arrow-rs to correctly calculate if a
UnionArray
is null, but I presume that works take much longer
It would likely take longer
Note there is a method that takes into account child nullability that perhaps we could use instead of is_null
:
https://docs.rs/arrow/latest/arrow/array/trait.Array.html#method.logical_nulls
Update: it turns out logical_nulls
is incorrect for UnionArray: apache/arrow-rs#6017
from arrow-datafusion.
Related Issues (20)
- Link is outdated
- Proposal: Create `dfdb`, a new CLI different than `datafusion-cli` with pre-built integrations HOT 10
- The batch_size selection for `CoalesceBatches` doesn't account for cases with a limit HOT 2
- order_by not respected for window functions using udaf HOT 2
- Window functions create unwanted projection HOT 8
- Improve lpad udf by using a GenericStringBuilder HOT 1
- DataFusion weekly project plan (Andrew Lamb) - Aug 12, 2024
- Disable `create_default_catalog` when the exist session state has default catalog for `SessionStateBuilder` HOT 1
- Improve performance of `REPEAT` functions HOT 2
- Internal error: Empty iterator passed to ScalarValue::iter_to_array HOT 1
- `optimize_projections` fails when eagerly selecting from a join with ambigious column names HOT 1
- Speed up rpad implementation to use StringBuilder HOT 1
- Remove unused config value `scalar_update_factor` HOT 1
- 在文件prost.rs文件中定义的"pub struct Schema"等结构体,需要使用"#[derive(Serialize)]"属性生成序列化的方法。 HOT 2
- Improve error message for invalid aggregate queries
- Support different aggregate expression (outside SELECT list) in ORDER BY list HOT 2
- Allow suppling a table schema to ParquetExec HOT 4
- Panics in `MIN()/MAX()` aggregate functions (SQLancer) HOT 3
- Internal error in `approx_percentile_cont()` aggregate function (SQLancer)
- Invalid aggregate SQL query with `HAVING` can be executed without error (SQLancer-TLP)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from arrow-datafusion.