Comments (2)
hi, @jancona
As i known, the s3 file system doesn't support file random access(#19).
So some other project like gleam firstly download the file to local.
Do you have some good suggestions or methods?
from parquet-go.
You're right that S3 isn't a file system. It's more of an object store. But it does support using HTTP range requests to retrieve part of an object. See: https://godoc.org/github.com/aws/aws-sdk-go-v2/service/s3#example-S3-GetObjectRequest-Shared01 and https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGET.html
Unfortunately S3 doesn't support partial updates, so the only option for writes is to build the file on the client (in memory or on local disk) and then upload it.
I'm going to work on solving the problem within my app for now. If I come up with something that I think can be contributed back, I'll let you know.
from parquet-go.
Related Issues (20)
- Inefficient Parquet Conversion with columnify (parquet-go) compared to pyarrow #93
- Release 1.7.0? HOT 1
- write down and then just 1 successful HOT 1
- Unrecognized tag 'converted type' HOT 1
- Does csv_writer has stream write pattern?
- Reading parquet files written by spark returns unexpected values
- Incompatibilities in v1.6.0? What are they? HOT 1
- Need help in reading parquet file in chunks from GCS
- Memory usage increased by more than 4x until OOM(140G) when upgrading from v1.5.4 to the latest commit HOT 2
- Library may be trying to convert the byte array into a string representation instead of preserving the raw byte data HOT 2
- Break backward compatibility HOT 1
- Unable to read parquet file HOT 1
- Go v1.22 cannot detect PAR_GO_PATH_DELIMITER as "." HOT 1
- Corrupted Parquet Statistics in Trino SQL HOT 1
- A parquet format question: reading compressed pages
- [Question] Read Perfomances
- [Question] it it possible to use an io.Reader as source without dumping it to a buffer or file? HOT 1
- INT96 is convert to datetime
- panic: runtime error: invalid memory address or nil pointer dereference
- ничего не работает
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from parquet-go.