wpnpeiris / nats-gateway Goto Github PK

S3 API compatible NATS Object Store

License: Apache License 2.0

Go 99.22% Makefile 0.78%

nats-gateway's Introduction

NATS

NATS is a high performance distributed system including at its core a Publish/Subscribe system and a built-in persistent system that enables Streaming, Key-Value Store and Object Store. It is a complete system with built-in authentication/authorization, and multi-tenancy models, with various other features.

NATS-S3

Most of the modern Object Storage systems such as MinIO, SeaweedFS, JuiceFS, AIStore, etc supports S3 API which simplifies the integration over HTTP. We take the same approach here with NATS-S3 enabling access to NATS Object Store over S3 protocol.

Usage

Once NATS-S3 is running connected to NATS servers, NATS Object Store can be access over S3 API. The following shows how NATS Object Store is accessible over AWS CLI.

List NATS Buckets

$ aws s3 ls  --endpoint-url=http://localhost:5222

List content of NATS Bucket, bucket1

$ aws s3 ls s3://bucket1 --endpoint-url=http://localhost:5222

List content of NATS Bucket, bucket1

$ aws s3 ls s3://bucket1 --endpoint-url=http://localhost:5222

Upload an object to a NATS bucket

$ aws s3 cp file1.txt s3://bucket1 --endpoint-url=http://localhost:5222

Download an object from a NATS bucket

$ aws s3 cp s3://bucket1/file1.txt file1_copy.txt --endpoint-url=http://localhost:5222

nats-gateway's People

Contributors

Stargazers

Watchers

Forkers

insidegreen gedw99

nats-gateway's Issues

check https://github.com/minio/mc also works as cli client

https://github.com/minio/mc

Its lighter than the Amazon one.

HTTP endpoint for file upload / download and changs

this is really cool. Been meaning to make something like this.

Would like to add a HTTP endpoint to this thats not S3. It can then cal the S3 Api if we want.

For file up and down.

then a very simple golang template html for Files up and down via the HTTP gateway.

panic serving 127.0.0.1:56069: runtime error: invalid memory address or nil pointer dereference goroutine 29 [running]:

Mac darwin environment....

AWS cli

aws s3 ls s3://bucket1 --endpoint-url=http://localhost:8080

Connection was closed before we received a valid response from endpoint URL: "http://localhost:8080/bucket1?list-type=2&prefix=&delimiter=%2F&encoding-type=url".

All the was cli calls fails with the same error message BTW.

NATS Gateway server

nats-gateway -listen localhost:8080 -nats nats:/127.0.0.1:4222
2024/06/16 14:40:06 Starting NATS Gateway...
2024/06/16 14:40:06 Listening for HTTP requests on localhost:8080
List Objects in bucket bucket1
2024/06/16 14:40:12 http: panic serving 127.0.0.1:56069: runtime error: invalid memory address or nil pointer dereference
goroutine 29 [running]:
net/http.(*conn).serve.func1()
        /opt/homebrew/opt/go/libexec/src/net/http/server.go:1898 +0xb0
panic({0x1068f8b40?, 0x107ac52c0?})
        /opt/homebrew/opt/go/libexec/src/runtime/panic.go:770 +0x124
github.com/wpnpeiris/nats-gateway/internal/s3.(*S3Gateway).ListObjects(0x140001c2f20, {0x106c3e690, 0x140007cc000}, 0x1066bcc90?)
        /Users/apple/workspace/go/src/github.com/nats-gateway/internal/s3/s3-object-handlers.go:43 +0xf4
net/http.HandlerFunc.ServeHTTP(0x140007aa480?, {0x106c3e690?, 0x140007cc000?}, 0x104d8df68?)
        /opt/homebrew/opt/go/libexec/src/net/http/server.go:2166 +0x38
github.com/gorilla/mux.(*Router).ServeHTTP(0x14000394480, {0x106c3e690, 0x140007cc000}, 0x140007aa000)
        /Users/apple/workspace/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0x194
net/http.serverHandler.ServeHTTP({0x140007841e0?}, {0x106c3e690?, 0x140007cc000?}, 0x6?)
        /opt/homebrew/opt/go/libexec/src/net/http/server.go:3137 +0xbc
net/http.(*conn).serve(0x1400079e000, {0x106c44078, 0x140007840f0})
        /opt/homebrew/opt/go/libexec/src/net/http/server.go:2039 +0x508
created by net/http.(*Server).Serve in goroutine 104
        /opt/homebrew/opt/go/libexec/src/net/http/server.go:3285 +0x3f0
List Objects in bucket bucket1
2024/06/16 14:40:12 http: panic serving 127.0.0.1:56071: runtime error: invalid memory address or nil pointer dereference
goroutine 32 [running]:
net/http.(*conn).serve.func1()
        /opt/homebrew/opt/go/libexec/src/net/http/server.go:1898 +0xb0
panic({0x1068f8b40?, 0x107ac52c0?})
        /opt/homebrew/opt/go/libexec/src/runtime/panic.go:770 +0x124
github.com/wpnpeiris/nats-gateway/internal/s3.(*S3Gateway).ListObjects(0x140001c2f20, {0x106c3e690, 0x140007cc380}, 0x1066bcc90?)
        /Users/apple/workspace/go/src/github.com/nats-gateway/internal/s3/s3-object-handlers.go:43 +0xf4
net/http.HandlerFunc.ServeHTTP(0x140007aa360?, {0x106c3e690?, 0x140007cc380?}, 0x104d8df68?)
        /opt/homebrew/opt/go/libexec/src/net/http/server.go:2166 +0x38
github.com/gorilla/mux.(*Router).ServeHTTP(0x14000394480, {0x106c3e690, 0x140007cc380}, 0x140007aa120)
        /Users/apple/workspace/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0x194
net/http.serverHandler.ServeHTTP({0x14000784360?}, {0x106c3e690?, 0x140007cc380?}, 0x6?)
        /opt/homebrew/opt/go/libexec/src/net/http/server.go:3137 +0xbc
net/http.(*conn).serve(0x1400079e120, {0x106c44078, 0x140007840f0})
        /opt/homebrew/opt/go/libexec/src/net/http/server.go:2039 +0x508
created by net/http.(*Server).Serve in goroutine 104
        /opt/homebrew/opt/go/libexec/src/net/http/server.go:3285 +0x3f0
List Objects in bucket bucket1
2024/06/16 14:40:14 http: panic serving 127.0.0.1:56073: runtime error: invalid memory address or nil pointer dereference
goroutine 114 [running]:
net/http.(*conn).serve.func1()
        /opt/homebrew/opt/go/libexec/src/net/http/server.go:1898 +0xb0
panic({0x1068f8b40?, 0x107ac52c0?})
        /opt/homebrew/opt/go/libexec/src/runtime/panic.go:770 +0x124
github.com/wpnpeiris/nats-gateway/internal/s3.(*S3Gateway).ListObjects(0x140001c2f20, {0x106c3e690, 0x140007cc460}, 0x1066bcc90?)
        /Users/apple/workspace/go/src/github.com/nats-gateway/internal/s3/s3-object-handlers.go:43 +0xf4
net/http.HandlerFunc.ServeHTTP(0x140007aa7e0?, {0x106c3e690?, 0x140007cc460?}, 0x104d8df68?)
        /opt/homebrew/opt/go/libexec/src/net/http/server.go:2166 +0x38
github.com/gorilla/mux.(*Router).ServeHTTP(0x14000394480, {0x106c3e690, 0x140007cc460}, 0x140007aa5a0)
        /Users/apple/workspace/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0x194
net/http.serverHandler.ServeHTTP({0x14000784780?}, {0x106c3e690?, 0x140007cc460?}, 0x6?)
        /opt/homebrew/opt/go/libexec/src/net/http/server.go:3137 +0xbc
net/http.(*conn).serve(0x1400079e3f0, {0x106c44078, 0x140007840f0})
        /opt/homebrew/opt/go/libexec/src/net/http/server.go:2039 +0x508
created by net/http.(*Server).Serve in goroutine 104
        /opt/homebrew/opt/go/libexec/src/net/http/server.go:3285 +0x3f0

nats server

nats-server --jetstream --store_dir /Users/apple/workspace/go/src/github.com/nats-gateway/.data --net 127.0.0.1 --port 4222
[12687] 2024/06/14 20:33:19.208988 [INF] Starting nats-server
[12687] 2024/06/14 20:33:19.209577 [INF]   Version:  2.10.16
[12687] 2024/06/14 20:33:19.209581 [INF]   Git:      [not set]
[12687] 2024/06/14 20:33:19.209583 [INF]   Name:     NAO77KBSDDZ5VEQ6SRFQQ7ZO7ZXBMG6TTVRYGLB545MULKFXDICDRYBQ
[12687] 2024/06/14 20:33:19.209588 [INF]   Node:     wBs3sdxq
[12687] 2024/06/14 20:33:19.209590 [INF]   ID:       NAO77KBSDDZ5VEQ6SRFQQ7ZO7ZXBMG6TTVRYGLB545MULKFXDICDRYBQ
[12687] 2024/06/14 20:33:19.210510 [INF] Starting JetStream
[12687] 2024/06/14 20:33:19.211560 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[12687] 2024/06/14 20:33:19.211565 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[12687] 2024/06/14 20:33:19.211567 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[12687] 2024/06/14 20:33:19.211568 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[12687] 2024/06/14 20:33:19.211569 [INF] 
[12687] 2024/06/14 20:33:19.211572 [INF]          https://docs.nats.io/jetstream
[12687] 2024/06/14 20:33:19.211573 [INF] 
[12687] 2024/06/14 20:33:19.211574 [INF] ---------------- JETSTREAM ----------------
[12687] 2024/06/14 20:33:19.211577 [INF]   Max Memory:      12.00 GB
[12687] 2024/06/14 20:33:19.211578 [INF]   Max Storage:     78.63 GB
[12687] 2024/06/14 20:33:19.211580 [INF]   Store Directory: "/Users/apple/workspace/go/src/github.com/nats-gateway/.data/jetstream"
[12687] 2024/06/14 20:33:19.211583 [INF] -------------------------------------------
[12687] 2024/06/14 20:33:19.212010 [INF] Listening for client connections on 127.0.0.1:4222
[12687] 2024/06/14 20:33:19.212021 [INF] Server is ready

Using this to replicate blobs to Cloudflare s3 or AmazonS3 in many regions.

At the moment the system replicates the blobs to many NATS Object stores, and imitates an S3 stores. So the API is an S3 API.

I am wondering if we can use this as a way to replicate blobs to and S3 provider ?

Say you have 10 servers around the world, each running NATS S3 Gateway and any of them can take a blob mutation, which NATS then holds, and then replicates to different S3 storage systems.

When you want to read the bob, the S3 API is used still. All that happens is that it checks if it's in the NATS Object store. If it is then it returns that, but if it's not then it returns back the URL to the blo in the nearest S3 store. The client then redirects to that URL.

So in a way the NATS Object Store is acting as a Registry and Write Mutation cache of where the blobs really is in S3. If the Blobs has made it to an S3 Store, then its not acting as a Cache, but returned the real location to the S3 blob.

upgrade https://github.com/aws/aws-sdk-go ?

is it worth upgrading to https://github.com/aws/aws-sdk-go-v2 ?

https://github.com/aws/aws-sdk-go still has 1 year left until EOL...

https://aws.amazon.com/blogs/developer/announcing-end-of-support-for-aws-sdk-for-go-v1-on-july-31-2025/

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.