Code Monkey home page Code Monkey logo

fs-over-http's Introduction

time tracker Docker Pulls Docker Build CodeFactor

A filesystem interface over http.

NOTE: I wrote this when I was still learning Go, and as such many improvements can be made. I have detailed what I would like to improve in the TODO section, with Partial Content, better error handling and response syntax being the main focus.

Contributing

Contributions to fix my code are welcome, as well as any improvements.

To build:

git clone [email protected]:5HT2/fs-over-http.git
cd fs-over-http
make

To run:

# I recommend using genpasswd https://gist.github.com/5HT2/30f98284e9f92e1b47b4df6e05a063fc
AUTH='some secure token'
echo "$AUTH" > token

# Change the port to whatever you'd like. 
# Change localhost to your public IP if you'd like.
./fs-over-http -addr=localhost:6060

Usage

Please see USAGE.md for examples of interacting with a fs-over-http server.

IPv6

IPv6 is supported, do note that you need to format the addr flag differently.

# Example IPv4
./fs-over-http -addr "10.0.1.1:6060"
# Example IPv6
./fs-over-http -addr "[2fb1:e540:13a7:3fa1:37bc:80b4:0b96:dbb8]:6060"

Production

I recommend using Caddy for automatic renewal + as a reverse proxy.

# Caddyfile example
i.l1v.in {
  header Server Caddy "Nintendo Wii"
  reverse_proxy localhost:6060
}

There is also a docker image available with the following command, or checkout the update.sh script for automatically updating a live docker image.

docker pull l1ving/fs-over-http:latest

TODO

  • Binary file support
  • Allow marking a folder as public
  • Custom shell for interacting
  • Partial Content support (docs)
  • Switch X-File-Content to using forms
    • eg: curl -X POST -H "Auth: $TOKEN" -d 'content=File content' localhost:6060/file.txt
    • Switch folder creation to same syntax with empty content
    • Read 512 bytes at a time like so.
  • Move error handling to ListenAndServe instead of individually sending the error
    • Switch to using X-Error-Message instead of printing it out, add a newline end of normal responses
  • Refactor use of JoinStr to fmt.Sprintf/Sprintln and +
  • Set ReadTimeout and WriteTimeout to prevent abuse
  • Add Docker image
    • Add CI service
  • Add Caddyfile example
    • Maybe with rate limit options and the such
    • Refactor docs about TLS
  • Encoding of uploading text-based files (eg the โ— character)
  • Cleanup README
  • Fix scripts to use new format

fs-over-http's People

Contributors

5ht2 avatar xwashere avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

fs-over-http's Issues

OOM killer kills it for serving large files

It seems that anything above ~1GB gets foh killed very quickly. I think this has something to do with either how the file is being read or how fasthttp works

A possible workaround is using a byte stream instead

impl: Redirect paths starting with `/public/` when not Authed

Ideally, when an Auth header is not provided, if a user is given a link such as https://cdn.frogg.ie/public/pdfs/, or a specific file, it should redirect to https://cdn.frogg.ie/pdfs/ (or the specific file), if it exists.

This is preferable over being handed a 404, and if you have the URL of an existing file you could already remove the /public/ from the URL yourself.

The only downside I could see this having is it being confusing in some cases?? Maybe? Like, if you thought you had provided an Auth and weren't expecting it to serve non-Authed content.

  • The only cases where this really matters could easily be mitigated by only redirecting on GET and nothing else, to avoid rewriting real edit requests.

  • Maybe this be configurable??

  • Maybe config options allow enabling this only for folder paths?

Rewrite in Rust - insane latency on localhost

I just realized the minimum latency for serving one JSON file is ~30ms on localhost and it should not be this big.

The disk read for the file barely takes a millisecond, it's all in our handling logic and negotiating a localhost connection.

[Issue moved] Switch to a real DB

I'm having way too many issues with redis persistence, and honestly, a beats key in a redisjson database with 265020 values (it is literally a json array with 265,020 values in the root key) is not the best way to go about this.

This is exactly the sort of thing SQL is for and I wish I knew that when I decided to write this project 2 years ago, so it's time to fix it (after Christmas, ideally).

Switch to GH Containers

Github has support for storing your Docker images on Github, instead of Docker Hub. The GH Action that we're using currently does not need you to change much to add support for this.

The reasoning behind this is that Docker now only allows 1 personal access token for pushing your containers, and org plans are paid. Given this push to pay for Docker Hub (which enforces poor security practices), it would be better to switch to GH for hosting the images.

  • Switch Action and Makefile
  • Update update.sh script

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.