Comments (12)
@yingsu00 I have started to understand the code. I am already up to speed with developer docs on Velox. But still wrapping my head around the implementation details.
In terms of time commitment, I should easily be able to spend ~4-5 hours daily going up to 10-12 hours/day if required. However, being new to the code base, I may need some guidance to speed up.
from velox.
@pedroerp Now I mainly use online use cases to do some performance tests. First, I will convert the format to parquet format. If the performance is good, I will develop orc format support, because online Hive tables are mainly in orc format.
from velox.
@yingsu00 I would like to attempt this and get it done. Still pretty new to the library but I hope to be able to contribute with guidance. Let me know if this is okay.
from velox.
@ankur4369 Thank you for offering help! ORC reader is crucial and performance critical. Some functionality may already be supported in DWRF reader but many are not. It will take several months to make it work and performant. How's your time commitment on it?
@mbasmanova @pedroerp What's Meta's plan on this? Will XLDB folks work on ORC reader?
from velox.
@ankur4369 It would be very nice if you could help add support for the ORC format. ORC is similar to DWRF and we have support for DWRF already. I hope it would be just a handful of tweaks to add support for ORC. We should be able to help by answering questions, brainstorming and reviewing the PRs. I'm CC-ing Zhenyuan and Orri who should be able to provide guidance as well.
I suggest to first figure out how to make ORC work without worrying too much about performance. Perf optimizations can come after basic support is in place.
from velox.
@ankur4369 Is there any update on this?
from velox.
Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
from velox.
@mbasmanova @pedroerp Does Velox have any plan to support ORC reader? I saw the above message, seems this feature didn't finish, right?
from velox.
@wpleonardo It would be nice to add support for ORC. Would you like to help build that?
from velox.
Re-opening since ORC support is still relevant.
from velox.
Hi, what's the status now?
from velox.
I don't know of anyone actively working on this. @wubiaoi, is this something you would be interested in contributing?
from velox.
Related Issues (20)
- Enhance WindowFuzzer to support k-range and k-rows frames with column boundary HOT 5
- Build issue with ABFS adapter HOT 2
- Velox write to ABFS storage fail with FileSink is not registered issue
- Add background daemon in Velox runtime for periodic runtime metric reporting HOT 3
- ‘FOLLY_MAYBE_UNUSED’ does not name a type
- Add Long Divide Spark function
- CI 'Ubuntu debug with resolve_dependency' is failing with The repository '...ubuntu/22.04...' is no longer signed HOT 2
- Native worker segmentation fault on makeExprSetFromFlag HOT 2
- date_format returns wrong results in Spark
- Parquet reading failed to decompress LZO files
- Failure in Fuzzer Jobs / Build (pull_request) HOT 3
- Add support for GenericViews in ContainerRowSerde HOT 3
- Support Spark Generate with outer HOT 8
- Extend Simple Function API to allow reporting errors without throwing HOT 9
- Velox can't open the ORC files created by spark-sql HOT 6
- Free uninitialized variable-width key column in abandon partial agg mode. HOT 3
- Query execution tracing and replay tool HOT 3
- Make evaluateOnce() work with extended types
- Complete tpch q2, q4 and q11 in TPC-H micro benchmark suite HOT 2
- ParquetReader needs to implement columnStatistics()
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from velox.