Amazon S3 support? about parquet-go HOT 2 CLOSED

hi, @jancona
As i known, the s3 file system doesn't support file random access(#19).
So some other project like gleam firstly download the file to local.
Do you have some good suggestions or methods?

from parquet-go.

You're right that S3 isn't a file system. It's more of an object store. But it does support using HTTP range requests to retrieve part of an object. See: https://godoc.org/github.com/aws/aws-sdk-go-v2/service/s3#example-S3-GetObjectRequest-Shared01 and https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGET.html

Unfortunately S3 doesn't support partial updates, so the only option for writes is to build the file on the client (in memory or on local disk) and then upload it.

I'm going to work on solving the problem within my app for now. If I come up with something that I think can be contributed back, I'll let you know.

from parquet-go.