Code Monkey home page Code Monkey logo

s5's Introduction

S5

Content-addressed storage, but fast.


Ethos

At its core, S5 is a content-addressed storage network similar to IPFS and also uses many of the formats and standards created in the IPFS project. It just builds upon them to be much more lightweight and scalable. Read the docs for more info on the nitty gritty.

Usage

docker run -it --rm -p 5050:5050 -v /local/path/to/config:/config --name s5-node ghcr.io/s5-dev/node:latest

A basic config file is generated for you, just make sure the path to the directory exists.

Or run it with docker compose

version: '3'
services:
  s5-node:
    image: ghcr.io/s5-dev/node:latest
    volumes:
      - /local/path/to/config:/config
    ports:
      - "5050:5050"
    restart: unless-stopped

To add file stores edit the config as described in the docs.

Supported Storage Backends

  • S3 (Any cloud provider supporting the S3 protocol, see https://s3.wiki)
  • Local filesystem (needs additional configuration to make a http port available on the internet)
  • Sia (experimental and cheap, https://sia.tech/)
  • Arweave (expensive, permanent storage)
  • Pixeldrain (affordable, https://pixeldrain.com/)
  • Estuary.tech (experimental)

Development

Dependencies

Build

  • git clone https://github.com/s5-dev/s5.git
  • cd s5/rust
  • cargo build --release
  • cp target/release/librust.so ..
  • cd ..
  • dart compile exe bin/s5_server.dart

Run the node with ./bin/s5_server.exe config.toml

License

This project is licensed under the MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

s5's People

Contributors

lukehmcc avatar redsolver avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

s5's Issues

If the email exists when creating an account, an error message that is too long and unclear appears

Hello @redsolver,

When creating an account with "/s5/admin/accounts?email=[email protected]" the "Account Id" is returned as a response.

If I run the same command again, a much too long and unclear error message comes back as a response.

If it is feasible, can you shorten the error message and return a clearer error message as an answer.
Such as "The email has already been registered." or similar.

Here is the long error message:

SqfliteFfiException(sqlite_error: 2067, , SqliteException(2067): while executing statement, UNIQUE constraint failed: Account.email, constraint failed (code 2067)
  Causing statement: INSERT INTO Account (created_at, email, tier) VALUES (?, ?, ?), parameters: 1685832816753, [email protected], 1}) DatabaseException(SqliteException(2067): while executing statement, UNIQUE constraint failed: Account.email, constraint failed (code 2067)
  Causing statement: INSERT INTO Account (created_at, email, tier) VALUES (?, ?, ?), parameters: 1685832816753, [email protected], 1) sql 'INSERT INTO Account (created_at, email, tier) VALUES (?, ?, ?)' args [1685832816753, [email protected], 1] {details: {database: {path: D:\8888\2\S5\data\accounts\accounts.db, id: 1, readOnly: false, singleInstance: true}}}

when complling subfolder rust from version 0.5.0 "warnings" appear

Hello @resolver

There are problems installing S5 version 0.5.0.
I cloned the repo with git, and ran "dart pub get" which also works without errors.
Then I executed the command "cargo build" in the subfolder "rust" and "warnings" came up:

At the beginning:
warning: dependency (flutter_rust_bridge_codegen) specified without providing a local path, Git repository, or version to use. This will be considered an error in future versions

Just before the end:

   Compiling rust v0.1.0 (D:\d3\1\S5\rust)
warning: unused import: `Hasher`
 --> src\api.rs:1:20
  |
1 | use blake3::{Hash, Hasher};
  |                    ^^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: unused import: `flutter_rust_bridge::frb`
 --> src\api.rs:3:5
  |
3 | use flutter_rust_bridge::frb;
  |     ^^^^^^^^^^^^^^^^^^^^^^^^

warning: unused import: `std::sync::Arc`
 --> src\api.rs:7:5
  |
7 | use std::sync::Arc;
  |     ^^^^^^^^^^^^^^

warning: unused `Result` that must be used
   --> src\api.rs:126:9
    |
126 |         encoder.write(&buffer[..count]);
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: `#[warn(unused_must_use)]` on by default
    = note: this `Result` may be an `Err` variant, which should be handled

warning: `rust` (lib) generated 4 warnings
    Finished dev [unoptimized + debuginfo] target(s) in 1m 49s

The warnings come with all installaions of my 4 s5 nodes on "linux 32bit, ubuntu 64bit and windows10 64bit".
But it compiles all files so that there are no errors, except for the warnings.

My question:
Is this ok or is this a bug?
Or maybe something is missing?

What can I do, or is everything ok?

Accounts DB Creation Issue

So when trying to initialize application for the first time it launches just fine and is accessible generating the seed and all, but when you attempt to add the [accounts] and [accounts.database] it appears to fail to create it, and stops the application from fully launching.

I have attempted to use the path in the docs and a unique path for (one not inside the /db directory) neither results in a change. Here is the log output.

Config Section

[accounts]
enabled = true
alwaysAllowedScopes = [
    'account/login',
    'account/register',
    's5/registry/read',
    's5/metadata',
    's5/debug/storage_locations',
    's5/debug/download_urls',
    's5/blob/redirect',
]
[accounts.database]
path = "/accounts"

SQLite Error on Boot

s5box-s5-node-1  | [sqlite] init
s5box-s5-node-1  | 2023-09-07 14:13:19.093995 - info - HTTP Server listening on port 8989
s5box-s5-node-1  | [cache] total used cache size: 0 B / 4 GB
s5box-s5-node-1  | [cache] doing nothing because max cache size is not reached yet
s5box-s5-node-1  | [warn] SqfliteFfiException(error, Invalid argument(s): Failed to load dynamic library 'libsqlite3.so': libsqlite3.so: cannot open shared object file: No such file or directory}) DatabaseException(Invalid argument(s): Failed to load dynamic library 'libsqlite3.so': libsqlite3.so: cannot open shared object file: No such file or directory)
s5box-s5-node-1  | #0      responseToResultOrThrow (package:sqflite_common_ffi/src/method_call.dart:123)
s5box-s5-node-1  | #1      SqfliteIsolate.handle (package:sqflite_common_ffi/src/isolate.dart:33)
s5box-s5-node-1  | <asynchronous suspension>
s5box-s5-node-1  | #2      _isolateHandle (package:sqflite_common_ffi/src/database_factory_ffi_io.dart:94)
s5box-s5-node-1  | <asynchronous suspension>
s5box-s5-node-1  | #3      ffiMethodCallhandleInIsolate (package:sqflite_common_ffi/src/database_factory_ffi_io.dart:49)
s5box-s5-node-1  | <asynchronous suspension>
s5box-s5-node-1  | #4      _SqfliteDatabaseFactoryImpl.invokeMethod (package:sqflite_common/src/mixin/factory.dart:22)
s5box-s5-node-1  | <asynchronous suspension>
s5box-s5-node-1  | #5      SqfliteDatabaseMixin.openDatabase (package:sqflite_common/src/database_mixin.dart:777)
s5box-s5-node-1  | <asynchronous suspension>
s5box-s5-node-1  | #6      SqfliteDatabaseMixin.doOpen (package:sqflite_common/src/database_mixin.dart:879)
s5box-s5-node-1  | <asynchronous suspension>
s5box-s5-node-1  | #7      SqfliteDatabaseOpenHelper.openDatabase (package:sqflite_common/src/database.dart:46)
s5box-s5-node-1  | <asynchronous suspension>
s5box-s5-node-1  | #8      SqfliteDatabaseFactoryMixin.openDatabase.<anonymous closure> (package:sqflite_common/src/factory_mixin.dart:110)
s5box-s5-node-1  | <asynchronous suspension>
s5box-s5-node-1  | #9      ReentrantLock.synchronized.<anonymous closure> (package:synchronized/src/reentrant_lock.dart:37)
s5box-s5-node-1  | <asynchronous suspension>
s5box-s5-node-1  | #10     BasicLock.synchronized (package:synchronized/src/basic_lock.dart:33)
s5box-s5-node-1  | <asynchronous suspension>
s5box-s5-node-1  | #11     SQLService.init (package:s5_server/service/sql.dart:22)
s5box-s5-node-1  | <asynchronous suspension>
s5box-s5-node-1  | #12     AccountsService.init (package:s5_server/service/accounts.dart:197)
s5box-s5-node-1  | <asynchronous suspension>
s5box-s5-node-1  | #13     S5Node.start (package:s5_server/node.dart:146)
s5box-s5-node-1  | <asynchronous suspension>

error: No store configured, uploads not possible. But how?

Hello @redsolver,

cool project.

I took a look at it and immediately tried to install a test node. it also worked, but when I want to upload something to "http://127.0.0.1:5522/s5/upload" I get an error message:
No store configured, uploads not possible

hence my first question:

  • How can I set up the storage backends: for local file system and S3 (how do the [store.entries] have to be in the config.toml?)?

Second question:

  • What do I have to change to make the s5-node publicly available and not only on "localhost:5522"?
    I've also tested it on linux and windows, but it only says "127.0.0.1:5522" when it starts.
    Can this also be set in the config.toml?

I urgently need help with this.

Is there documentation anywhere?

If my S5 test and development environment is installed and running, then I can also help you with development and programming.
I also helped "SkynetLabs" with porting "skynet-js" to "nodejs-skynet" or rather tried to keep it up to date.

Link directory to previous versions

A directory should have a means of storing the previous version on an update to enable a sort of Windows defrag so if app devs do not do things properly and assume they link the directory versions, you can track history and be able to unpin files and prevent them from cogging up any portal account.

Msgpack Standardization

A lot of the current metadata formats have some awkward formatting that need to be resolved.

This should serve as a tracking issue for them all

"image.jpg" is uploaded as a raw file and is not displayed in the browser.

Hello redsolver and s5-dev team,

I have extensively tested the latest version 0.5.0 from the master.
The installation in Windows10 and Ubuntu was a little bit complicated and there were 4 small warnings, but it installed in both systems and it works.

Will test it today also on linux 32bit.
The previous problems with "p2p" version 0.3.0 are gone. awesome

But there are other problems, such as:

  • With a single upload of image.jpg or video.mp4 via "/s5/upload", the file is uploaded, but as type "RAW file" and not as type "image-file" or type "media-file ".
    If I want to call up the cids in the browser, the cid is downloaded and not displayed in the browser.

What am I doing wrong or is this a bug?

In the previous version 0.3.0 this worked great. In the new version 0.5.0 I can only display the individual files in the browser if I copy them to a directory before uploading and then upload the directory with the one file via "/s5/upload/directory".

I hope you can help.

In "0.13.1 - 0.13.3" of "s5-dev/S5" is a bug when retrieving web apps with base32 CID

Hello @redsolver,

I found a bug in the latest versions "0.13.1" - "0.13.3" of "s5-dev/S5".

When I open my webapp via "http://bleptij2lj2e7i3g4msarql3at2pkhohrwj4af3e4znyi6zs36gw4poq.localhost:5050/"
If you want to retrieve it, you get this error message:

HandshakeException: Handshake error in client (OS Error: 
    CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(../../third_party/boringssl/src/ssl/handshake.cc:393))

In the s5-node console I can see this message:

[dns] resolveName bleptij2lj2e7i3g4msarql3at2pkhohrwj4af3e4znyi6zs36gw4poq.localhost

What's going on there?

With version "0.12.0" of "s5-dev/S5" everything works great.
There I can see these messages in the console on the s5 node:

[try] http://localhost:8989/1/d4/2c/os/2o/rh2gzxdeqemc6ye6t2r3r4nspabozhglochwmw7rvxd3u
2023-09-28 22:59:10.006374 - info - GET - /1/d4/2c/os/2o/rh2gzxdeqemc6ye6t2r3r4nspabozhglochwmw7rvxd3u

Add flag to disable p2p

Some users may wish to simply use an S5 node as a way to do multi-backend remote storage. Having a flag to remove p2p would be useful for this usecase.

S5 version 0.3.0 there are problems with the p2p. CRASHING all S5-nodes.

Hello @redsolver,

with the S5 version 0.3.0 there are problems with the p2p.
I installed 4 completely new s5-nodes on 4 computers in the local network.
On the first node I have in the config.toml under

[network.peers]
initial = []
entered and in Node2 / Node3 and Node4 I have under:
[network.peers]
initial = [ 'tcp://[email protected]:4444',
]

registered.

important information:
Before all new tests, I delete the "tmp" and "data" directories on all nodes, so that all nodes are installed as new for all tests, and I can understand where the errors are occurring.

First problem:
when I try to connect all nodes it keeps crashing. Sometimes all connected nodes support, but sometimes only a part of the nodes, but all those that don't crash no longer react.

Second problem:
If I test it through several times and by chance some nodes
get online, and then shut down a single node, after that all other connected ones crash too.

Third problem:
If I go to all 4 nodes under

[network.peers]
initial = [ 'tcp://[email protected]:4444',
'tcp://cid_from_node2@ip_node2:4444',
'tcp://cid_from_node3@ip_node3:4444',
'tcp://cid_from_node4@ip_node4:4444',
]

enter, all S5 nodes are only crashing when booting up.

What am I doing wrong or is this a big bug in version 0.3.0?
fallow help so that I can get my test environment up and running without everything collapsing.

@redsolver can you help.
Where can I download the S5 version 0.4.0 or 0.5.0 for testing? or is the development not on github?

there are problems when building s5-node with rust 1.70.0

Hello @redsolver,

I just reinstalled one of my PCs, I noticed that "s5-node" doesn't work with the latest rust "1.70.x", there are errors with "cargo build --release".
And I couldn't start the s5-node.

I had to install rust "1.65.0" to avoid errors.
Now start the "s5-node" again.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.