Comments (3)
Yes, DataFusion should decide what null semantics it wants / make this configurable, and implement it using make_comparator if appropriate
from arrow-rs.
I'm not sure we want to support this within these kernels, in particular the null behaviour is not well-defined. I added support to the comparator for this, which will allow systems like DataFusion to define an ordering for nulls based on a config setting, as Spark and Postgres use different ordering.
There is an example of this here - https://docs.rs/arrow-ord/latest/arrow_ord/ord/fn.make_comparator.html#postgres-compatible-nested-comparison
This will also make it more obvious that the nested comparison is not vectorised
Edit: Filed #5942 to make this more discoverable
from arrow-rs.
I'm not sure we want to support this within these kernels, in particular the null behaviour is not well-defined. I added support to the comparator for this, which will allow systems like DataFusion to define an ordering for nulls based on a config setting, as Spark and Postgres use different ordering.
There is an example of this here - https://docs.rs/arrow-ord/latest/arrow_ord/ord/fn.make_comparator.html#postgres-compatible-nested-comparison
This will also make it more obvious that the nested comparison is not vectorised
Edit: Filed #5942 to make this more discoverable
Are you suggesting that if datafusion requires nested comparison, we could implement them in datafusion with make_comparator
like what I had done? Arrow-rs only support the make_comparator
kernel for nested type?
This is the code that utilize compare_op
, and nested type is missing
from arrow-rs.
Related Issues (20)
- ByteView should be using signed types? HOT 2
- Use reconstructed ListBlobs marker to provide list offset support in `MicrosoftAzure` store HOT 1
- Improve ability for users to create their own writers using the low level parquet APIs. HOT 1
- Is cargo fmt no longer working properly in parquet crate HOT 4
- [DISCUSSION] Proposal move `object_store` to its own github repo? HOT 19
- Remove test duplication in parquet statistics tets HOT 1
- mismatch between parquet type `is_optional` codes and comment
- Special-case `nullif` kernel for scalar using `Datum`
- ParquetReader should check creator version before applying row group statistics HOT 2
- Adding sub day seconds to Date64 is ignored. HOT 1
- Make it easy to write parquet to object_store -- Implement `AsyncFileWriter` for a type that implements `obj_store::MultipartUpload` for `AsyncArrowWriter` HOT 1
- Add benchmarks for `BYTE_STREAM_SPLIT` encoded Parquet `FIXED_LEN_BYTE_ARRAY` data
- Using a take kernel on a dense union can result in reaching "unreachable" code HOT 1
- Using unreleased arrow-rs in DataFusion HOT 1
- arrow/parquet-rs writes `MapArray` types as-is without taking Parquet spec into account HOT 1
- How to specify a custom `ArrayType` .
- Parquet Statistics null_count does not distinguish between `0` and not specified HOT 4
- Look into optimizing reading FixedSizeBinary arrays from parquet HOT 12
- Error in IPC serializer or reader when using dictionary columns HOT 2
- lexsort_to_indices should not fallback to non-lexical sort if the datatype is not supported
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from arrow-rs.