Code Monkey home page Code Monkey logo

go-storage's Introduction

go-storage

Go dev License

Matrix Slack Telegram

Build Test Cross Build Unit Test

A vendor-neutral storage library for Golang.

Vision

Write once, run on every storage service.

Goal

  • Vendor agnostic
  • Production ready
  • High performance

Examples

package main

import (
    "log"

    "github.com/beyondstorage/go-storage/v5/services"
    "github.com/beyondstorage/go-storage/v5/types"

    // Add fs support
    _ "github.com/beyondstorage/go-storage/services/fs/v4"
    // Add s3 support
    _ "github.com/beyondstorage/go-storage/services/s3/v3"
    // Add gcs support
    _ "github.com/beyondstorage/go-storage/services/gcs/v3"
    // Add azblob support
    _ "github.com/beyondstorage/go-storage/services/azblob/v3"
    // More support could be found under BeyondStorage.
    _ "github.com/beyondstorage/go-storage/services/xxx" 
)

func main() {
    // Init a Storager from connection string. 
    store, err := services.NewStoragerFromString("s3://bucket_name/path/to/workdir")
    if err != nil {
        log.Fatalf("service init failed: %v", err)
    }

    // Write data from io.Reader into hello.txt
    n, err := store.Write("hello.txt", r, length)

    // Read data from hello.txt to io.Writer
    n, err := store.Read("hello.txt", w)

    // Stat hello.txt to check existence or get its metadata
    o, err := store.Stat("hello.txt")

    // Use object's functions to get metadata
    length, ok := o.GetContentLength()
    
    // List will create an iterator of object under path.
    it, err := store.List("path")
    
    for {
    	// Use iterator.Next to retrieve next object until we meet IterateDone.
    	o, err := it.Next()
    	if errors.Is(err, types.IterateDone) {
    		break
        }
    }

    // Delete hello.txt
    err = store.Delete("hello.txt")
}

More examples could be found at go-storage-example.

Features

Widely native services support

16 stable services that have passed all integration tests.

3 beta services that implemented required functions, but not passed integration tests.

4 alpha services that still under development.

More service ideas could be found at Service Integration Tracking.

Complete and easily extensible interface

Basic operations

  • Metadata: get Storager metadata
meta := store.Metadata()
_ := meta.GetWorkDir() // Get object WorkDir
_, ok := meta.GetWriteSizeMaximum() // Get the maximum size for write operation
  • Read: read Object content
// Read 2048 byte at the offset 1024 into the io.Writer.
n, err := store.Read("path", w, pairs.WithOffset(1024), pairs.WithSize(2048))
  • Write: write content into Object
// Write 2048 byte from io.Reader
n, err := store.Write("path", r, 2048)
  • Stat: get Object metadata or check existences
o, err := store.Stat("path")
if errors.Is(err, services.ErrObjectNotExist) {
	// object is not exist
}
length, ok := o.GetContentLength() // get the object content length.
  • Delete: delete an Object
err := store.Delete("path") // Delete the object "path"
  • List: list Object in given prefix or dir
it, err := store.List("path")
for {
	o, err := it.Next()
	if err != nil && errors.Is(err, types.IterateDone) {
        // the list is over 
    }
    length, ok := o.GetContentLength() // get the object content length.
}

Extended operations

  • Copy: copy a Object inside storager
err := store.(Copier).Copy(src, dst) // Copy an object from src to dst.
  • Move: move a Object inside storager
err := store.(Mover).Move(src, dst) // Move an object from src to dst.
  • Reach: generate a public accessible url to an Object
url, err := store.(Reacher).Reach("path") // Generate an url to the object.
  • Dir: Dir Object support
o, err := store.(Direr).CreateDir("path") // Create a dir object.

Large file manipulation

  • Multipart: allow doing multipart uploads
ms := store.(Multiparter)

// Create a multipart object.
o, err := ms.CreateMultipart("path")
// Write 1024 bytes from io.Reader into a multipart at index 1
n, part, err := ms.WriteMultipart(o, r, 1024, 1)
// Complete a multipart object.
err := ms.CompleteMultipart(o, []*Part{part})
  • Append: allow appending to an object
as := store.(Appender)

// Create an appendable object.
o, err := as.CreateAppend("path")
// Write 1024 bytes from io.Reader.
n, err := as.WriteAppend(o, r, 1024)
// Commit an append object.
err = as.CommitAppend(o)
  • Block: allow combining an object with block ids
bs := store.(Blocker)

// Create a block object.
o, err := bs.CreateBlock("path")
// Write 1024 bytes from io.Reader with block id "id-abc"
n, err := bs.WriteBlock(o, r, 1024, "id-abc")
// Combine block via block ids.
err := bs.CombineBlock(o, []string{"id-abc"})
  • Page: allow doing random writes
ps := store.(Pager)

// Create a page object.
o, err := ps.CreatePage("path")
// Write 1024 bytes from io.Reader at offset 2048
n, err := ps.WritePage(o, r, 1024, 2048)

Comprehensive metadata

Global object metadata

  • id: unique key in service
  • name: relative path towards service's work dir
  • mode: object mode can be a combination of read, dir, part and more
  • etag: entity tag as defined in rfc2616
  • content-length: object's content size.
  • content-md5: md5 digest as defined in rfc2616
  • content-type: media type as defined in rfc2616
  • last-modified: object's last updated time.

System object metadata

Service system object metadata like storage-class and so on.

o, err := store.Stat("path")

// Get service system metadata via API provides by go-service-s3.
om := s3.GetObjectSystemMetadata(o)
_ = om.StorageClass // this object's storage class
_ = om.ServerSideEncryptionCustomerAlgorithm // this object's sse algorithm

Strong Typing Everywhere

Self maintained codegen definitions helps to generate all our APIs, pairs and metadata.

Generated pairs which can be used as API optional arguments.

func WithContentMd5(v string) Pair {
    return Pair{
        Key:   "content_md5",
        Value: v,
    }
}

Generated object metadata which can be used to get content md5 from object.

func (o *Object) GetContentMd5() (string, bool) {
    o.stat()
    
    if o.bit&objectIndexContentMd5 != 0 {
        return o.contentMd5, true
    }
    
    return "", false
}

Server-Side Encrypt

Server-Side Encrypt supports via system pair and system metadata, and we can use Default Pairs to simplify the job.

func NewS3SseC(key []byte) (types.Storager, error) {
    defaultPairs := s3.DefaultStoragePairs{
        Write: []types.Pair{
            // Required, must be AES256
            s3.WithServerSideEncryptionCustomerAlgorithm(s3.ServerSideEncryptionAes256),
            // Required, your AES-256 key, a 32-byte binary value
            s3.WithServerSideEncryptionCustomerKey(key),
        },
        // Now you have to provide customer key to read encrypted data
        Read: []types.Pair{
            // Required, must be AES256
            s3.WithServerSideEncryptionCustomerAlgorithm(s3.ServerSideEncryptionAes256),
            // Required, your AES-256 key, a 32-byte binary value
            s3.WithServerSideEncryptionCustomerKey(key),
        }}
    
    return s3.NewStorager(..., s3.WithDefaultStoragePairs(defaultPairs))
}

Sponsor

go-storage's People

Contributors

abyss-w avatar aeinrw avatar beyondrobot avatar burntcarrot avatar ccamel avatar dependabot-preview[bot] avatar dependabot[bot] avatar hqc19907228 avatar jinnyyi avatar joey-1445601153 avatar joey2132 avatar junaire avatar juneezee avatar kondanta avatar prnyself avatar renovate[bot] avatar rhnsharma avatar xuanwo avatar xxchan avatar zooltd avatar zu1k avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

go-storage's Issues

tests: Some errors in standard library doesn't implement xerrors.Wrapper

strconv.NumError as an exmaple: we need to write following code for unittest:

e := &strconv.NumError{}
ok := errors.As(err, &e)
if ok {
	assert.True(t, errors.Is(e.Err, tt.err))
} else {
	assert.True(t, errors.Is(err, tt.err))
}

After go 1.14 released, we can change them into:

assert.True(t, errors.Is(err, tt.err))

Roadmap to v1.0.0

Roadmap

  • After 2020-03-13, all new proposals that could lead to breaking changes will be postponed to next major release (v2.0.0)
  • After 2020-03-20, all proposals that not implemented will be postponed to next minor release (v1.1.0)

Proposals

Following proposals are sure to be included in v1.0.0 (a.k.a v0.9.0)

Services

There are no ensurement for any new service to be included in v1.0.0

Steps

There will be two minor release before v1.0.0: v0.8.0 and v0.9.0

v0.8.0 will be released shortly after 2020-03-06, and v0.9.0 expected to be released in 2020-03-20.

After v0.9.0, no new proposals will be implemented until v1.0.0 released, and I will focus on bug fix and code cleanup.

services/dropbox: meta.Name used incorrectly

o := &types.Object{
	ID:         meta.Id,
	Type:       types.ObjectTypeFile,
	Name:       filepath.Join(path, meta.Name),
	Size:       int64(meta.Size),
	UpdatedAt:  meta.ServerModified,
	ObjectMeta: metadata.NewObjectMeta(),
}

filepath.Join(path, meta.Name) is not the valid name for this file, use path directly?

services/fs: Size and Offset in Read handled incorrectly

if opt.HasSize && opt.HasOffset {
	return iowrap.SectionReadCloser(f, opt.Offset, opt.Size), nil
}
if opt.HasSize {
	return iowrap.LimitReadCloser(f, opt.Size), nil
}
if opt.HasOffset {
	_, err = f.Seek(opt.Offset, 0)
	if err != nil {
		return nil, fmt.Errorf(errorMessage, s, path, handleOsError(err))
	}
}
if opt.HasReadCallbackFunc {
	return iowrap.CallbackReadCloser(f, opt.ReadCallbackFunc), nil
}

should be like

if opt.HasSize {
	r = iowrap.LimitReadCloser(f, opt.Size), nil
}

Proposal: Move id to object type instead of metadata

PR #53 adds a new metadata Id.

This inspires me that we need an Id across all services:

For local file system: Id should be whole path towords root.
For object storage: Id should be the whole key.
For dropbox alike SaaS: Id should be their Id in business.

So let's move Id to object type instead of in metadata.

Proposal: Support service init via config string

Background

This project intents to be a unified storage layer for Golang, but different storage layers' configuration are so different we can't unify them well.

For posixfs: we only need to specify the workdir.
For object storage: we need to specify host, port, protocol, access key id and others.

We used to support them by type and options, like we did in qscamel:

Every service(endpoint in qscamel) should handle their own options:

type Client struct {
	BucketName          string `yaml:"bucket_name"`
	Endpoint            string `yaml:"endpoint"`
	Region              string `yaml:"region"`
	AccessKeyID         string `yaml:"access_key_id"`
	SecretAccessKey     string `yaml:"secret_access_key"`
	DisableSSL          bool   `yaml:"disable_ssl"`
	UseAccelerate       bool   `yaml:"use_accelerate"`
	PathStyle           bool   `yaml:"path_style"`
	EnableListObjectsV2 bool   `yaml:"enable_list_objects_v2"`
	EnableSignatureV2   bool   `yaml:"enable_signature_v2"`
	DisableURICleaning  bool   `yaml:"disable_uri_cleaning"`

	Path string

	client *s3.S3
}

func New(ctx context.Context, et uint8, hc *http.Client) (c *Client, err error) {
	...
	content, err := yaml.Marshal(e.Options)
	if err != nil {
		return
	}
	err = yaml.Unmarshal(content, c)
	if err != nil {
		return
	}
    ...
}

Developer who want to use this service should handle the type:

switch t.Src.Type {
...
case constants.EndpointS3:
    src, err = s3.New(ctx, constants.SourceEndpoint, contexts.Client)
    if err != nil {
        return
    }
...
default:
    logrus.Errorf("Type %s is not supported.", t.Src.Type)
    err = constants.ErrEndpointNotSupported
    return
}

User should set them in config directly:

source:
  type: s3
  path: "/path/to/source"
  options:
    bucket_name: example_bucket
    endpoint: example_endpoint
    region: example_region
    access_key_id: example_access_key_id
    secret_access_key: example_secret_access_key
    disable_ssl: false
    use_accelerate: false
    path_style: false
    enable_list_objects_v2: false
    enable_signature_v2: false
    disable_uri_cleaning: false

It works, but it doesn't meet our goal. To address this problem, we split endpoint and credential in PR services: Split endpoint and credential into different pair. In this PR, we can init an object service like:

srv := qingstor.New()
err = srv.Init(
    pairs.WithCredential(credential.NewStatic(accessKey, secretKey)),
    pairs.WithEndpoint(endpoint.NewStaticFromParsedURL(protocol, host, port)),
)
if err != nil {
    log.Printf("service init failed: %v", err)
}

It's better, but not enough. We need a general way to init all service like:

srv := storage.SomeCall("<type>", something)

Proposal

So I propose following changes:

Introduce the concept of "config string"

config string is widely used in db connections:

mysql: user:password@/dbname?charset=utf8&parseTime=True&loc=Local
postgres: host=myhost port=myport user=gorm dbname=gorm password=mypassword
sqlserver: sqlserver://username:password@localhost:1433?database=dbname

Like we did in URL, we can use different part in a formatted string to represent different meaning.

Config string in storage would be like:

<type>://<config>
             +
             |
             v
<credential>@<endpoint>/<name>?<options>
     +            +                 +
     |            +------------+    +----------------------+
     v                         v                           v
<protocol>:<data>   <protocol>:<host>:<port>   <key>:<value>[&<key>:<value>]
  • credential: <protocol>:<data>, static credential could be static://<access_key>:<secret_key>.
  • endpoint: <protocol>:<host>:<port>, qingstor's valid endpoint could be https://qingstor.com:443, 80 and 443 can be emitted with matched protocol.
  • name: a valid storager name for this services
  • options: multiple <key>=<value> connected with &

So a valid config string could be:

  • qos://static:<access_key_id>:<secret_access_key>@https:qingstor.com:443/<bucket_name>?zone=pek3b&work_dir=/storage
  • posixfs:///<path>

proposal: Go version policy

Start from go 1.13, we will test target the latest two major version.

  • At go1.13, we will test on go1.13
  • After go1.14, we will test on go1.13 and go1.14
  • After go1.15, we will test on go1.14 and go1.15

Any install/setup/build error report on not tested version will be marked as wontfix

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.