fluencelabs / aqua-ipfs Goto Github PK

IPFS API bindings for Aqua scripts

License: Apache License 2.0

Shell 5.39% Rust 78.62% TypeScript 6.57% Python 8.96% JavaScript 0.46%

ipfs p2p aqua

aqua-ipfs's Introduction

Aqua IPFS bindings

Aqua-ipfs is a native IPFS integration to Aqua language. It lets one call the API of an IPFS daemon, e.g., to transfer files between peers & services or to orchestrate IPFS nodes.

Quick Installation and Usage

First, make sure you have Fluence CLI installed. After this, installation is as simple as:

fluence dep npm i @fluencelabs/aqua-ipfs

Next, run:

import "@fluencelabs/aqua-ipfs/ipfs-api.aqua"
import "@fluencelabs/aqua-lib/builtin.aqua"

const PERIOD_SEC = 10

func install_scheduled_script_from_ipfs_to_peer(from: PeerId, to: PeerId, cid: string) -> ?string:
  script_id: ?string

  ipfs_maddr <- get_external_api_multiaddr(from)
  if ipfs_maddr.success:
    get <- get_from(to, cid, ipfs_maddr.multiaddr)
    if get.success:
      script_id <- Script.add_from_vault(get.path, ?[PERIOD_SEC])

  <- script_id

To run it in Marine REPL (mrepl), one needs to install mrepl via cargo install mrepl first. After that:

cd service
mrepl
new Config.toml
call ipfs_pure set_local_api_multiaddr ["/ip4/134.209.186.43/tcp/5001/p2p/12D3KooWEhCqQ9NBnmtSfNeXSNfhgccmH86xodkCUxZNEXab6pkw"]

call ipfs_pure set_external_api_multiaddr ["/ip4/134.209.186.43/tcp/5001/p2p/12D3KooWEhCqQ9NBnmtSfNeXSNfhgccmH86xodkCUxZNEXab6pkw"]

call ipfs_pure get_from ["QmfBRabun4FpaHV4wVXtnqtopUTro93XJHiWhNZscViCaq", "/ip4/134.209.186.43/tcp/5001/p2p/12D3KooWEhCqQ9NBnmtSfNeXSNfhgccmH86xodkCUxZNEXab6pkw"]

You can use interface and help inside mrepl to further discover what's possible.

A simple example of using aqua-ipfs in TypeScript is available here.

Documentation

Comprehensive documentation including API and usage examples can be found in Aqua Book.

Repository Structure

aqua is Aqua API of Aqua-ipfs. See its README for details.
builtin-package Files necessary to use Aqua-ipfs on peers
example A simple example of how to use IPFS adapter from TypeScript
local-network contains Docker Compose YAML config to run a local Fluence network of 3 nodes
service contains Rust service that implements all IPFS calls by using ipfs CLI mounted binary

Support

Please, file an issue if you find a bug. You can also contact us at Discord or Telegram. We will do our best to resolve the issue ASAP.

Contributing

Any interested person is welcome to contribute to the project. Please, make sure you read and follow some basic rules.

License

All software code is copyright (c) Fluence Labs, Inc. under the Apache-2.0 license.

aqua-ipfs's People

Contributors

Stargazers

Watchers

Forkers

forkkit eddiejjefferson pholophus iqbalbaharum catcatm200 maushish

aqua-ipfs's Issues

allow to pass single string to file as parse from and cid out of it

func get_and_cache(
    node: PeerId,
    cid: CID,
    from: Multiaddr,
    error: string, string -> ()
) -> ?CID:

https://docs.ipfs.io/how-to/address-ipfs-on-web/

func get_and_cache(
    node: PeerId,
    cidAddress: Multiaddr,,
    error: string, string -> ()
) -> ?CID:

so that when client uploaded it can use same link to file he got to pass to aqua, without parsing

ipfs swarm connect error is ignored

# ipfs swarm connect /ip4/134.209.186.43/tcp/5001
Error: found no ipfs peers at /ip4/134.209.186.43/tcp/5001

[2021-07-02T18:12:52.694733Z INFO  ipfs_pure] get called with hash: QmZBoTfAmcfaQTWJU62xD3v54LCxnhhUHuGdW4eAwKqFa8
[2021-07-02T18:12:52.696940Z INFO  ipfs_effector] connect called with multiaddr /ip4/134.209.186.43/tcp/5001
[2021-07-02T18:12:52.777554Z INFO  ipfs_effector] get called with hash QmZBoTfAmcfaQTWJU62xD3v54LCxnhhUHuGdW4eAwKqFa8
[2021-07-02T18:12:52.777619Z INFO  ipfs_effector] ipfs get args ["get", "--timeout", "1s", "-o", "/.fluence/v1/stepper/particles/vault//204f2059-f7dc-4803-882b-6964a0a054cd/QmZBoTfAmcfaQTWJU62xD3v54LCxnhhUHuGdW4eAwKqFa8", "QmZBoTfAmcfaQTWJU62xD3v54LCxnhhUHuGdW4eAwKqFa8"]
[2021-07-02T18:12:53.861753Z INFO  ipfs_effector] ipfs get output: error: , stderr: Error: context deadline exceeded

Builtin deploy failed: error loading plugins: open /config/ipfs/config: permission denied

I've seen this on krasnodar's node,

Error: builtins deploy failed

Caused by:
    on_start call failed: [String("get_peer_id: ipfs cli call failed: error: , stderr: Error: error loading plugins: open /config/ipfs/config: permission denied\n\n\nLocation:\n    effector/src/effector.rs:40:106")]

Location:
    crates/builtins-deployer/src/builtins_deployer.rs:87:18

services.json:

# cat /services.json
{
  "aqua-ipfs": {
    "version": "v0.5.0",
    "url": "https://github.com/fluencelabs/aqua-ipfs/releases/download/v0.5.0/aqua-ipfs.tar.gz",
    "sha256": "ce4c85f3bed33b0ae880306e6c8394e6f932e03ee25404bcd8eebac7bd1bbee7"
  },
  "aqua-dht": {
    "version": "v0.2.3",
    "url": "https://github.com/fluencelabs/aqua-dht/releases/download/v0.2.3/aqua-dht.tar.gz",
    "sha256": "809af04204624ae686c1b2ec3dd79dc9d2fe6fb0e4289cf75c396be36dde0d11"
  }
}

container: fluencelabs/node:latest_v259

set_external_swarm_maddr

Need to be able to communicate IPFS's Swarm address (4001) as well as API address (5001).

set_external_swarm_maddr
set_external_api_maddr

set_local_multiaddr

Add ability to set multiaddr to use with ipfs --api, so users can point their ipfs-adapter services to custom hosts & ports

function to return IPFS's address configuration

Seems like get_from requires to specify a multiaddr of IPFS's swarm (i.e. these that by default listen on 4001). So get_external_maddr isn't enough. We need to return full-blown address configuration, something like this:

{
   external_maddr_api: .../5001,
   external_maddr_swarm: .../4001/p2p/123D...,
   local_maddr: ??? not sure if needed
}

All paths that trespass API must be relative to `/tmp/vault/$particle-id`

Currently, we're exposing too much information through the API of this service. We should trim prefix /tmp/vault/$particle-id from all paths returned from the API, and prepend that prefix to all paths given to the ipfs-adapter's API.

rename Ipfs.put to Ipfs.start_copy_to_local_node

and allow for checking if download status

Rename hash to CID

I think we should name it CID as it's a term from IPFS domain

on_start.air: EXTERNAL_MADDR & LOCAL_MADDR variables to configure ipfs-adapter

It's crucial to be able to configure ipfs-adapter without rebuilding docker container and/or repackaging builtins.

To allow that, introduce the following variables:

FLUENCE_IPFS_ADAPTER_EXTERNAL_MADDR to be passed to set_external_maddr to use in Addresses.Announce of ipfs config
FLUENCE_IPFS_ADAPTER_LOCAL_MADDR to be passed to set_local_maddr to use with --api in ipfs cli

add api to map cid to multiadress suffexed with cid on any node

so if i have cid, i can map to adress to download from peer (via external proxy adress or other)

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Warning

These dependencies are deprecated:

Datasource	Name	Replacement PR?
npm	`@fluencelabs/cli`

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

chore(deps): update dependency pytest-xdist to v3.6.1
chore(deps): update rust crate itertools to 0.13.0
fix(deps): update dependency @fluencelabs/aqua-ipfs to ^0.6.0
fix(deps): update rust crate marine-rs-sdk to 0.14.0
fix(deps): update rust crate marine-rs-sdk-test to 0.16.0
chore(deps): update dependency kubo-rpc-client to v4
chore(deps): update dependency @fluencelabs/fluence-network-environment to v1.2.1
🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

Detected dependencies

cargo

service/Cargo.toml

marine-rs-sdk 0.13.0

marine-rs-sdk-test 0.14.0

service/distro/Cargo.toml

maplit 1.0.2

built 0.7.0

service/effector/Cargo.toml

multiaddr 0.11.2

log 0.4.20

eyre 0.6.8

serde_json 1.0.64

itertools 0.12.0

service/pure/Cargo.toml

multiaddr 0.11.2

log 0.4.20

eyre 0.6.8

toml 0.8.8

serde 1.0.188

bs58 0.5.0

service/types/Cargo.toml

eyre 0.6.8

github-actions

.github/workflows/e2e.yml

.github/workflows/lint.yml

amannn/action-semantic-pull-request v5

actions/checkout v3

reviewdog/action-actionlint v1

.github/workflows/release.yml

google-github-actions/release-please-action v3

actions/checkout v3

actions-rust-lang/setup-rust-toolchain v1

baptiste0928/cargo-install v2.0.0

actions/setup-node v3

stefanzweifel/git-auto-commit-action v4

actions/checkout v3

hashicorp/vault-action v2.4.3

actions-rust-lang/setup-rust-toolchain v1

baptiste0928/cargo-install v2.0.0

fluencelabs/setup-marine v1

actions/setup-node v3

fluencelabs/setup-fluence v1

nahsi/setup-ipfs v1

softprops/action-gh-release v1

lwhiteley/dependent-jobs-result-check v1

hashicorp/vault-action v2.4.3

ravsamhq/notify-slack-action v2

.github/workflows/run-tests.yml

actions/checkout v3

actions-rust-lang/setup-rust-toolchain v1

fluencelabs/setup-marine v1

baptiste0928/cargo-install v2.0.0

actions-rs/cargo v1

actions/checkout v3

actions/setup-node v3

fluencelabs/setup-fluence v1

actions/checkout v3

actions-rust-lang/setup-rust-toolchain v1

actions-rs/cargo v1

.github/workflows/snapshot.yml

actions/checkout v3

actions-rust-lang/setup-rust-toolchain v1

fluencelabs/setup-marine v1

hashicorp/vault-action v2.5.0

.github/workflows/tests.yml

hashicorp/vault-action v2.4.3

actions/checkout v3

actions-rust-lang/setup-rust-toolchain v1

fluencelabs/setup-marine v1

actions/setup-node v3

docker/login-action v2

isbang/compose-action v1.4.1

actions/setup-python v4

nahsi/setup-ipfs v1

jwalton/gh-docker-logs v2

npm

aqua-tests/package.json

@fluencelabs/aqua-lib 0.10.1

@fluencelabs/cli 0.15.1

@fluencelabs/fluence-network-environment 1.1.2

aqua/package.json

@fluencelabs/aqua-lib 0.10.1

example/package.json

@fluencelabs/aqua-ipfs ^0.5.29

@fluencelabs/aqua-lib ^0.9.0

@fluencelabs/js-client ^0.4.2

kubo-rpc-client ^3.0.2

@multiformats/multiaddr ^12.1.1

it-all ^1.0.5

uint8arrays ^2.1.5

typescript ^4.0.0

pip_requirements

aqua-tests/requirements.txt

delegator.py ==0.1.1

pytest ==7.3.0

pytest-xdist ==3.2.1

pytest-repeat ==0.9.1

Check this box to trigger a request for Renovate to run again on this repository