Comments (3)
This looks strange, indeed. It looks like a bug.
from parquet4s.
Hi @struong
Given proto
syntax = "proto3";
option java_package = "com.github.mjakubowski84";
message LoginEvent {
enum EventType {
LOGIN_EVENT_UNSPECIFIED = 0;
LOGIN_SUCCESSFUL = 1;
}
int64 timestamp = 1;
EventType eventType = 2;
}
and Scala code:
package com.github.mjakubowski84
import com.github.mjakubowski84.parquet4s._
import ScalaPBImplicits._
import java.time.Instant
object PBTest extends App {
val outFile = InMemoryOutputFile(initBufferSize = 4800)
ParquetWriter.of[LoginEvent].writeAndClose(
file = outFile,
data = Seq(
LoginEvent(
timestamp = Instant.now().toEpochMilli,
eventType = LoginEvent.EventType.LOGIN_SUCCESSFUL
)
)
)
val inFile = InMemoryInputFile.fromBytes(outFile.take())
val protoResult = ParquetReader.as[LoginEvent].read(inFile)
// prints LoginEvent(1701085450638,LOGIN_SUCCESSFUL,UnknownFieldSet(Map()))
println(protoResult.head)
val genericResult = ParquetReader.generic.read(inFile)
// prints Some(LOGIN_SUCCESSFUL)
println(genericResult.head.get[String]("eventType", ValueCodecConfiguration.Default))
}
The created record is RowParquetRecord(timestamp=LongValue(1701085521641),eventType=BinaryValue(Binary{16 constant bytes, [76, 79, 71, 73, 78, 95, 83, 85, 67, 67, 69, 83, 83, 70, 85, 76]}))
with eventType
being a binary encoding a String value of the enum.
Everything looks good to me with enums and deserialisation.
It seems that you have a bug somewhere else.
from parquet4s.
@mjakubowski84 thank you for your prompt reply!
I believe it's an issue with my VSCode plugin to read parquet files, I used an online reader and indeed the rows look good. Thank you for investigating!
from parquet4s.
Related Issues (20)
- An example for writing a required or optional value codec for custom type with more than one field. HOT 6
- Is schema backwards compatibility on projections possible? HOT 1
- Options `ParquetFileWriter.Mode.OVERWRITE` not deleting old parquet files in S3 HOT 1
- failed to read parquet generated by pandas HOT 5
- Add Pekko support HOT 4
- Is it possible to write a file without Akka or Fs2 integration? HOT 2
- `ParquetReader.projectedGeneric` does not work when selecting more than one column from a same group HOT 5
- Reading from gcs bucket HOT 1
- Do not publish a pekko/akko versions of scapapb module HOT 1
- missing tail records of large(~193M) parquet files HOT 4
- compatible parquet-hadoop with spark3.1 HOT 3
- Unsure how to use for 'semiauto' approach HOT 2
- ParquetSchemaResolver test fails on recent JVMs HOT 1
- [akka/pekko] Too many paths created during record partitioning HOT 2
- [RFC] Refactor timestamp codecs HOT 2
- Feature request: Expose partitions as a `Stream[F, Stream[F, Record]]` for FS2 HOT 5
- Incorrect value after reading parquet HOT 7
- [Question] get a listing of parquet files? HOT 4
- [Question] Is there a mechanism to detect when the `rotatingWriter` finishes writing to a file and to be notified of the file that was written? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from parquet4s.