Autoretrieve is a standalone gateway server / Bitswap provider that bridges Filecoin Graphsync content over to Bitswap clients.

When a Bitswap client requests data from Autoretrieve, Autoretrieve asks either Estuary or an STI indexer which Filecoin service providers are hosting it, starts a retrieval deal with a selected SP, and streams the incoming blocks to the Bitswap client.

Autoretrieve uses Docker with Buildkit for build caching. Docker rebuilds are quite fast, and it is usable for local development. Check the docker-compose documentation for more help.

$ DOCKER_BUILDKIT=1 docker-compose up

You may optionally set FULLNODE_API_INFO to a custom fullnode's WebSocket address. The default is FULLNODE_API_INFO=wss://api.chain.love.

By default, config files and cache are stored at ~/.autoretrieve. When using docker-compose, a binding is created to this directory. This location can be configured by setting AUTORETRIEVE_DATA_DIR.

Internally, the Docker volume's path on the image is /root/.autoretrieve. Keep this in mind when using the Docker image directly.

Some CLI flags and corresponding environment variables are available for basic configuration.

For more advanced configuration, config.yaml may be used. It lives in the autoretrieve data directory, and will be automatically generated by running autoretrieve. It may also be manually generated using the gen-config subcommand.

Configurations are applied in the following order, from least to most important:

• YAML config
• Environment variables
• CLI flags

advertise-endpoint-url: # leave blank to disable, example https://api.estuary.tech/autoretrieve/heartbeat (must be registered)
advertise-endpoint-token: # leave blank to disable
lookup-endpoint-type: indexer # indexer | estuary
lookup-endpoint-url: https://cid.contact # for estuary endpoint-type: https://api.estuary.tech/retrieval-candidates
max-bitswap-workers: 1
routing-table-type: dht
prune-threshold: 1GiB # 1000000000, 1 GB, etc. Uses go-humanize for parsing. Table of valid byte sizes can be found here: https://github.com/dustin/go-humanize/blob/v1.0.0/bytes.go#L34-L62
pin-duration: 1h # 1h30m, etc.
log-resource-manager: false
log-retrieval-stats: false
disable-retrieval: false
cid-blacklist:
  - QmCID01234
  - QmCID56789
  - QmCIDabcde
miner-blacklist:
  - f01234
  - f05678
miner-whitelist:
  - f01234
default-miner-config:
  retrieval-timeout: 1m
  max-concurrent-retrievals: 1
miner-configs:
  f01234:
    retrieval-timeout: 2m30s
    max-concurrent-retrievals: 2
  f05678:
    max-concurrent-retrievals: 10

$ autoretrieve --help
NAME:
   autoretrieve - A new cli application

USAGE:
   autoretrieve [global options] command [command options] [arguments...]

COMMANDS:
   gen-config    Generate a new config with default values
   print-config  Print detected config values as autoretrieve sees them
   check-cid     Takes a CID argument and tries walking the DAG using the local blockstore
   help, h       Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --data-dir value               [$AUTORETRIEVE_DATA_DIR]
   --lookup-endpoint-url value   Indexer or Estuary endpoint to get retrieval candidates from [$AUTORETRIEVE_LOOKUP_ENDPOINT_URL]
   --lookup-endpoint-type value  Type of endpoint for finding data (valid values are "estuary" and "indexer") [$AUTORETRIEVE_LOOKUP_ENDPOINT_TYPE]
   --disable-retrieval           Whether to disable the retriever module, for testing provider only (default: false) [$AUTORETRIEVE_DISABLE_RETRIEVAL]
   --routing-table-type value    [dht|fullrt|disabled] [$AUTORETRIEVE_ROUTING_TABLE_TYPE]
   --log-resource-manager        Whether to present output about the current state of the libp2p resource manager (default: false) [$AUTORETRIEVE_LOG_RESOURCE_MANAGER]
   --log-retrievals              Whether to present periodic output about the progress of retrievals (default: false) [$AUTORETRIEVE_LOG_RETRIEVALS]
   --help, -h                    show help (default: false)