Comments (7)
Hi @moonkev,
I am ver happy that you appreciate my effort! It incites me even more to make this project better and more mature.
I like your idea, especially because one of the goals of this project is to give flexibility to developers to use various data types according to their needs.
I checked your commit. Changes in the TimeValueCodecs look fine! However, I have small doubts about how you handle the TimeZone setting. I didn't think much about the solution when I created #21. Having mutable variable in object of TimeValueCodecs indeed seems as the easiest solution and for sure it is better than what is now. Unfortunately, it still makes impossible for reading different types with different time zone in parallel. And I am not a big fun of global-like mutable variables.
What do you think about having time zone as an optional setting for reader and writer? Writer already has Options
parameter. We can add similar one to the reader. Writer and reader can translate it to implicit variable TimeZoneFor[T]
and TimeValueCodecs can use it. May sound as over-engineering, still, the whole project is based on type classes so it would be consistent. I am open to your suggestions!
from parquet4s.
I think that definitely makes sense. I have removed the timezone change and squashed down the commit here so now it only contains the change to add the Java 8 date/time API types. If this looks good to you I can create a pull request and we can bring the timezone discussion back into your original ticket #21
from parquet4s.
from parquet4s.
Thank you, I have opened the above reference PR.
EDIT: If you are curious as to the force push on the PR, it is because somehow the additions in the schema resolver got backed out on accident (maybe when I was removing the timezone change), so that was just me adding those back in.
from parquet4s.
Your PR is merged!
I just noticed that CircleCI was disabled for forked PRs. It should work for next ones from now.
Please tell me how soon you want have your changes released. Do you want to handle #21 first? I can have a look at it myself today evening (CET, speaking about time zones...).
from parquet4s.
Thank you. If you are willing to look at #21 that would be great. I wouldn't mind helping on it, but it seems you have a concrete idea on how you would like to implement it (which sounds good by the way). So I'm in no hurry for the release, I can wait for this change as well as #21 as your time permits. Also, I would be glad to help with any testing needed for that item. Thank you again!
from parquet4s.
OK, so I am closing this issue, thanks for contributing! I will ask you for checking #21 when I am ready
from parquet4s.
Related Issues (20)
- An example for writing a required or optional value codec for custom type with more than one field. HOT 6
- Is schema backwards compatibility on projections possible? HOT 1
- Options `ParquetFileWriter.Mode.OVERWRITE` not deleting old parquet files in S3 HOT 1
- failed to read parquet generated by pandas HOT 5
- Add Pekko support HOT 4
- Is it possible to write a file without Akka or Fs2 integration? HOT 2
- `ParquetReader.projectedGeneric` does not work when selecting more than one column from a same group HOT 5
- Reading from gcs bucket HOT 1
- Do not publish a pekko/akko versions of scapapb module HOT 1
- missing tail records of large(~193M) parquet files HOT 4
- Protobuf enums deserialisation HOT 3
- compatible parquet-hadoop with spark3.1 HOT 3
- Unsure how to use for 'semiauto' approach HOT 2
- ParquetSchemaResolver test fails on recent JVMs HOT 1
- [akka/pekko] Too many paths created during record partitioning HOT 2
- [RFC] Refactor timestamp codecs HOT 2
- Feature request: Expose partitions as a `Stream[F, Stream[F, Record]]` for FS2 HOT 5
- Incorrect value after reading parquet HOT 7
- [Question] get a listing of parquet files? HOT 4
- [Question] Is there a mechanism to detect when the `rotatingWriter` finishes writing to a file and to be notified of the file that was written? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from parquet4s.