ipfs-shipyard / cohosting Goto Github PK

View Code? Open in Web Editor NEW

25.0 14.0 1.0 29 KB

A set of light tools and conventions for cohosting public websites and files on IPFS

License: Creative Commons Zero v1.0 Universal

Shell 100.00%

cohosting's Introduction

Experiment in MFS-based cohosting

Exploration around simple tools and conventions for cohosting existing websites with IPFS

Motivation

Make it easier for people to contribute storage and bandwidth to sites and datasets they care about
- Support IPNS (libp2p keys) and DNSLink roots (human-readable)
- Switch tools like ipfs-cohost and ipfs-companion from raw pins to MFS
Periodically detect updates to and preload them to a local node
Experiment in userland: make it easy to implement, no new APIs, reuse existing ones
- Identify constraints of the mutable filesystem (MFS) and propose ways to improve it

Scope vs. Use Cases

See this analysis.

Specification

See SPEC.md

🚧 note: this is a draft, an early, exploratory experiment 🚧

Feedback is welcome. Fill an issue!

Potential Implementations

This specification is not implemented yet. The following IPFS Shipyard projects could implement it:

🍎 = Not started
🍊 = In progress
🍏 = Complete

🍊 cohosting.sh

MVP bash script that can be used for cli

add rm for adding / removing sites to cohosting list via commandline
sync a command to run cohosting check (for use in crond etc)
gc [n] drop all old snapshots (if n is provided, keeps that many snapshots per site)
Lazy cohosting

🍏 ipfs-cohost

NPM-based interactive cli tool

switch from pin to MFS-based spec

🍎 ipfs-companion

browser extension

provides UI for adding / removing sites via browser action menu
runs cohosting check periodically

🍎 ipfs-desktop

desktop app and GUI for managing go-ipfs

runs cohosting check periodically

🍎 ipfs-webui

web frontend for IPFS node

provides UI for adding / removing sites to cohosting list as an experiment on Settings page

cohosting's People

Contributors

Stargazers

Watchers

cohosting's Issues

ipfs files cp does not ensure recursive content

While working on cohosting.sh and ipfs-cohost, I noticed ipfs files cp does not ensure recursive content. It just links the hash to a certain place in MFS and only fetches it if the user actually goes there.

Even though having a directory on MFS prevents it from being garbage collected, it does not get recursively fetched when copying for the first time.

/cc @lidel @autonome

Rename `gc` to something else

We should rename gc operation to something else to avoid confustion with ipfs gc command.

How about prune ?

ipfs-cohost - a cli to cohost sites on IPFS

It's a thing https://github.com/olizilla/ipfs-cohost

If your domain has a DNSlink to a CID, then ipfs-cohost will let others pin it to their IPFS node.

Lazy cohosting

Technical Background

Things in MFS are implicitly pinned (won't be garbage collected).

Copying IPFS content paths to MFS (ipfs files cp /ipfs/{CID} /path/in/mfs) does not ensure recursive content is fetched and store in local repo. All cp does is to create entry in MFS and only fetches new blocks if the user actually goes there and requested block is not in the local repo.

That is why additional ipfs refs -r call is needed to ensure all blocks are prefetched nad present in the local repo (#4, #5)

Potential Use

Here's an idea: the lack of recursive fetch could be a feature that enables regular people to "cohost" parts of big websites such as https://ipfs.io/ipns/en.wikipedia-on-ipfs.org

Without explicit preload (#5) only the parts that were actually visited are stored in local repo, saving disk space and enabling cohosting of more content overall.

Challenges

The key challenge here is UX. Initial ideas:

how to communicate the difference between "full" and "lazy" cohosting?
how to enable user to make the decision?
- when adding website via ipfs-companion user could be prompted if they want to cohost in "full" ("eager") or "lazy" fashion (while showing current total size of a website)
- for adding website via CLI or programmatic interface, there should be a separate addLazy command that does not run ipfs refs -r

cc @meiqimichelle @hacdias @autonome