nfnitloop / feoblog Goto Github PK

As a server admin, I should be able to use the CLI to:

• See how much data users are using.
• Delete and/or ban particular content/users I find objectionable.
• Delete data that is no longer used. (ex: users that are no longer followed)
• Find out why a particular user/Item is on my server. (Who follows this person?)

In the meantime you have to make DB calls yourself.

Right now, FeoBlog uses an SQLite database. This simplifies maintenance, and should scale enough for individual use, or use among a small group of friends. But SQLite does have limitations with high write loads.

Currently, database operations are all abstracted through the Backend type. It should be easy to add a new implementation, but I'd need to plan for a few things.

• Do I want to maintain multiple backends?
• If not, how do I allow backend implementations to be maintained by others without requiring lock-step development?
• Config file support so database passwords don't need to be specified as command-line arguments. (Which is also good to have in general, I suppose.)

The UI of the JS and plain HTML versions of the site has diverged a bit, and TBH the JS version is much nicer. For a better initial experience, redirect browsers that have JavaScript enabled to the corresponding URL in the client.

• Should include an option to disable the redirect.

Right now, user administration is performed via the CLI. That's not terrible for a server run by a programmer for a few of his friends. But others might prefer a web-based UI.

How do we do this?

• Re-use nacl crypto, and create a special "server admin" UID?
• How do we prevent replay attacks?

FeoBlog "blogs" are organized as an unordered¹ collection of Items so that an individual server may deny storing certain Items. However, this means that, when looking at data stored on a single server, it's not possible to determine whether any of a user's data is missing. To do so, one would have to find a list of Item on some other server and compare it to the one in question.

To enable this type of checking against a single server, we could add optional metadata to the Item type which lists previous Item IDs published by this user ID. Like the rest of the Item, this data would be cryptographically signed by the user's key, so unable to be modified by servers. Clients could walk that DAG and report on any missing items.

IMO, server sync is going to be enough to detect censored Items. During sync, the client compares lists of Item IDs from two servers and copies over missing Items. If a server denies POSTing an item, the client will notify the user, and the user can detect the censorship.

1. Though posts/Items are commonly ordered by their timestamp, there is nothing in the data structure that enforces that, unlike a DAG.

In addition to #25, it would be nice if the server would periodically sync data for "server users" so that their content would be available without first having to manually sync.

May also want to do this in conjunction with #2 to allow enabling/configuring this option?

To control spam, objectional content, and abuse, and to comply with legal requirements, server admins should be able to:

• Block a user ID
• Block an Item ID
• Block an attachment (via its content hash)

Blocking one of those should:

• Prevent it from being served
• Disallow POSTing it again.
• Optionally, delete it from the database.

Additionally, we should have commands to find out the reason that blocked content showed up on our instance in the first place:

• Command to list which users have attached a particular file. (To detect all users responsible for a particular offense.)
• Command to list users who have followed a particular user ID. (To detect users who use follows to allow abuse.)

Currently, URLs to posts on here are incredibly long, and it would be nice to have "shortcuts" (ex. t.co/XJK98Y)

I should compose a list of servers running FeoBlog, once there are some.

• What's the server address?
• Does the server have a specific purpose/community?
• Is the server accepting new users?
• If so, how (to whom) does one request access?

Right now, AFAIK, this is just https://blog.nfnitloop.com/. If you're running another server, please do comment and let me know! :)

I added Luxon, and it seems to be a smaller/better library. But because moment is scattered around the code I keep accidentally using more of it. :p

See:

Right now I'm just using the webbrowser crate to open a browser, but on Linux that might open a blocking browser, which will block the server from actually starting.

There's an open issue for that here:
amodm/webbrowser-rs#18

--open still might work for a user, depending on what they have set up as their default browser, and which launcher it can find, so it's not necessarily fully broken, but it would be nice if this were consistent with Windows & MacOS behavior.

One of the first people I shared FeoBlog with asked "Does it support ActivityPub?" and I had to say no.

I should create a FAQ or somewhere to document these kinds of things, and this question in particular.

The Short Version

Until then, here's why FeoBlog doesn't support ActivityPub:

1. ActivityPub is federated, not distributed. Your userID is tied to a particular server (DNS hostname). That does not work with the data structures and security measures that FeoBlog was built upon.
2. The ActivityPub standard doesn't have a way to verify the contents of a posts, so accepting ActivityPub data would mean that FeoBlog clients wouldn't be able to verify all of their data.
3. ActivityPub relies on the concept of posting messages to users' "inboxes". But in FeoBlog, a user may exist on multiple servers, so there may be no known (and online) "inbox" for ActivityPub to post to.

See https://www.w3.org/TR/activitypub/#security-considerations for some of the unanswered security concerns.

In particular, I don't see there how servers or clients can validate that data they received is actually from the supposedly originating user.

Meanwhile, FeoBlog only accepts signed data. As a result, its data model doesn't need an "inbox". Being passed data for a user operates the same whether it's from that user, or from some other service on their behalf.

I had fun implementing them. But it's not immediately obvious how to use them. Just do a JS confirmation dialog instead.

If I'm writing a post and navigate away from that page then return to it, it should restore my previous post. Losing long posts this way would be pretty annoying.

Browsers try to save form data for you these days, but I think because I'm running a single-page webapp which just removes the element when you navigate away, the browser doesn't know how to save/restore that state. (Plus I think my Svelte code is explicitly starting out with empty state.)

• Servers should be able to set data quotas for particular users.
• Servers should be able to set default/maximum data quotas for followed users.
• Users should be able to set quotas for blogs they follow. (Ex: I want to follow X, but only cache the latest 10MiB on my behalf.)
• There should be a command to show quota usage.
• There should be a command to clean up over-quota data. (Right now, we don't preemptively delete it.)

Advanced, possibly future features:

• Separate quotas for Items and file attachments?

• Users should be able to comment on any Item (including other comments).

• Clients should have a way to query for comments in response to an Item.

• By default, the displayed comments should be limited to:

  • Those followed by the author of the original Item.
  • (optionally) Additionally those followed by the logged-in user.

• Comments should not be rendered on the main HTML-only home page.

• Comments may optionally be rendered in clients' feeds (chronologically, detached, but linking to the Item they respond to.)

• Comments should be rendered when viewing an individual item.

• Comments are not Posts. In particular, they lack:

  • Titles
  • The ability to attach files (once that's available)
  • The ability to inline images. (TBD)

Right now the server makes users' posts and feeds available as plain HTML, and as data accessible via the FeoBlog REST API.

However, to allow consuming a FeoBlog blog from third party systems like feed readers, we should also expose the feeds as RSS, and/or Atom, and/or JsonFeed.

• should link to the plain HTML version of the page. (But see #22)
• Should probably only serve blogs of "server users", so that folks can't accidentally start following a feed that doesn't actually get regularly updated on this server.
• Maybe even further limit to only users on the homepage? Or add as a per-user option? Need some consideration here.

When I visit blog.nfnitloop.com, the thing I usually want to see is the feed of all my follows' (and my) posts. But it currently takes me to the "Home" view by default:

This is just because we redirect to the client from /, and I'm trying to mirror the URL structure from the web site in the client. However, I think we can make an exception for the home page, give it an explicit /client/#/home URL, and redirect /client/#/ to the user's feed.

Users should be able to attach files to a post. This would make sharing images, PDFs, etc. possible.

Rough plans are:

• Implement a content-addressable store.
• Re-use whatever hash-algorithm NaCL signatures currently use, since we're already using it.
• File names, hashes & sizes are stored within the protobuf.
• Standard URL sub-directory where files live.
• File uploads to the file names can be performed by anyone.
• File attachment quotas may disallow attaching large files.

Additional work:

• Update the client to know how to sync file attachments along with their Items.

If I write a post or otherwise create an Item that's larger than accepted by the server, the client could/should warn me before I hit submit.

This project reminds me a lot of tumblr, so it would be cool if you could bring back some of tumblrs features into this.

Out of the box, FeoBlog uses its embedded styles.css and client.css files to render the plain HTML view and client view.

Users will probably want a way to provide their own styles for their own sites.

FeoBlog should include a robots.txt

Immediately after sharing a link to a live server for the first time, it got hit by several bots, and they generally requested a robots.txt.

I should read up on the modern format of robots.txt and include an appropriate one.

Can they do wildcard matching? Do I want to allow/deny specific sections? (Maybe deny /client/*?)

Support audio recording in the client.

https://blog.addpipe.com/recording-audio-in-the-browser-using-pure-html5-and-minimal-javascript/ has some example code for reference.

Depends on File attachments working: (#4)

I'll probably also need to add some Item metadata to say that "this attachment is audio meant to be played inline", since unfortunately .ogg and .webm file extensions are not exactly unambiguous there.

I just posted minutes ago, but somehow the post is backdated 18 hours. !?!?

I think I figured out what happened. Since the drafts feature was introduced, if there was something in the text field (and there was in this case), we load the text and the timestamp associated w/ that text.

I cleared the text, wrote a new post, and posted it without even looking at the timestamp. So it ended up getting posted w/ an old timestamp. Oops.

I think the common case is going to be posting with the current timestamp, so I should probably make that the default behavior for both new posts and saved drafts.

Maybe I can just allow the timestamp to be blank to mean "now" and leave it blank by default?

Since there's there are raw (/u/_/i/_/) links and client (/client/#/u/_/...) links, it ends up being easier to share the client link, though the other one would likely result in better SEO and discoverability. Add a share button. It could:

• Share to Twitter
• Copy Link
• ???

Currently, servers are pretty simple.

1. Has a list of users/follows.
2. May allow POSTs from those users.
3. Serves that content.

All of the sync logic is written in the client.

It would be nice if I as a user could run a command from the CLI to sync posts. That way I could easily make a copy to bootstrap a new server, or create a backup, or just grab stuff before I go offline for a while.

I thought I'd need support for this in rust-embed, but as I wrote up pyrossh/rust-embed#140 I realized a workaround would be simple:

Create some wrapper that calculates an ETag (and also Last-Modified date?) on server startup or lazily on the first request. Then serve static files w/ that ETag so browsers will cache them longer.

Use the keyboard to navigate through items in the client.

ex: j/k for next/previous items.

Right now you have to drag/drop attachments into the web UI to add them. This doesn't work on iOS (and probably not Android?)

I suppose I should add some UI element that's more discoverable than drag & drop to enable that.

Related to #20 -- maybe allow attaching multiple versions of an image with different sizes?

Hmm, this might be a bit complicated.

First thought: Add some "metadata" to the Attachment protobuf message that allows alternative sizes.

But ... wouldn't I want all attachments listed in the top-level repeated Attachments.file? Maybe there should be an Attachments.meta? But then clients that naively copy all Attachments may get redundant copies of images? Hmm. I'll need to think about this one a bit more.

Right now the client has you "Log In" as a user. But you don't provide any password to do so. Really you're just configuring the client to act as if you are logged in.

It feels a bit like a kludge. Is there some better language / UI to represent this?

I recently implemented filtering for the user's feed.

Eventually, when there are many users on a server, a user might want to filter which comments are shown in reply to posts. For example:

• Include users I follow
• Include users followed by OP
• Include all (other) users on the server.

I'm not going to bother with this until I get some more users, though, and this actually becomes an issue.

I wrote in the v0.1.0 README.md:

Reblogging [is an unplanned feature]. I [believe] there should be a higher barrier to sharing others' content. You'll need to comment, post, or "Reply" post to share content to your followers.

But, the more I think about it having a lot of posts like

⬆️ OMG This

is probably even more annoying. If I implement a reblog functionality, I could limit it to the feed, and add an option to not display reblogs for people that don't want to see that.

I really want to use this service, but right now there seems to be only one user and one website, with the UI being not the best. Is there a detailed guide on how to set up your own instance?

See: https://www.reddit.com/r/rust/comments/o80zh4/feoblog_v040_released_today/h3367dw/?utm_source=reddit&utm_medium=web2x&context=3

What are the obligations servers have under GDPR?

My layman's understanding is that under GDPR users must be able to:

1. Request any data a server has about them.
2. Be able to request that their data be deleted from a server. (within some timespan?)

The REST API can already list all of a user's Items, so that may take care of point 1.

And eventually I'll implement #5 -- is that enough to handle point 2?

If folks know other GDPR obligations, please leave comments, IANAL.

Maybe just think about branding in general? In particular "blog" is not quite accurate. And "Feo" was a funny pun for a while but may not really attract new users.

Example names: Disapora, Mastodon, Scuttlebutt. Single word, unique, not necessarily related to what it is.

I'm writing a new rss->feoblog client for Deno. I was getting a 500 "Server Error" from the server. That's not super helpful. Should expose what failed validation.

I saw this on my actix web console:

[2021-07-04T02:05:33Z ERROR actix_http::response] Internal Server Error: Error { inner: ValidationError { message: "Timestamp is required" } }

Make sure that gets returned via HTTP.

Summary

When Snowpack generates files via snowpack build, it ends up generating files that have colons in the name.

Initially, these seem to work when I'm running the rust server in dev. mode, which passes through file requests to the underlying file system. However, when I attempt to do a release build, which bundles files up into the executable, the required files do not exist.

This also breaks the ability for rollup to do a release build. It itself can't find the files it generated. 😣

Steps to reproduce

• Check out the latest version of the development branch (028941e)
• cd feoblog/web-client
• npm install
• npm run build

Take a look in build/web_modules/common

• On Linux/MacOS, you'll likely see a file named __node-resolve:empty-1777c4cc.js
• ⚠️ But on Windows the file will be called __node-resolve. It will be 0 bytes.

This is because file names can't have a colon in them on Windows. However, file paths can have a colon in them, which causes the bit after the colon to be interpreted as a stream when using NTFS file systems. You can see this in this output:

C:\Users\Cody\code\feoblog\web-client\build\web_modules\common>dir /r __node-resolve
 Volume in drive C has no label.
 Volume Serial Number is 3894-D9B7

 Directory of C:\Users\Cody\code\feoblog\web-client\build\web_modules\common

02/20/2021  09:41 PM                 0 __node-resolve
                                   358 __node-resolve:empty-1777c4cc.js:$DATA
               1 File(s)              0 bytes
               0 Dir(s)  357,425,332,224 bytes free

This issue still exists even if I upgrade to Snowpack v3.

I'm not sure if this is a Snowpack bug or a Rollup bug, so I've just reported it here against my own code until I find the culprit. :)

A bit more detail

This seems to work in dev mode because the expected file name is directly requested via the browser. RustEmbed then asks the filesystem for that file path, which also ends up opening the NTFS data stream.

However, bundling the files likely just lists files on disk and tries to include them in the binary. Likewise, copying or zipping the files would probably break this functionality. Right now this is a blocker to having a working feoblog web client on Windows. 😢

See:

• Me, scratching my head about this earlier today on Twitter
• Asking about it in Snowpack's Discord server

Current Status

To do:

• Work with the Rollup team to merge and release rollup/rollup#3972
• Ping the Snowpack team to create a release w/ that version.
• Update Feoblog to use that version of Snowpack
• Abandon Snowpack, switch to esbuild.
• Confirm it works on Windows.
• Enable Windows build automation.
• Release a Windows build

Now that I've got some "blogs" with lots of "posts", the initial sync can be a bit slow.

For example:

There's probably room to parallelize the sync more than it is already.

I think I can also walk down the list of items instead of fetching the full list then doing set comparisons. Though I doubt that's the slow part, it's probably just the latency overhead of copying items from and to a server one-by-one.

Right now, profiles only contain an identicon based on their public ID.

• Add the ability to attach a profile photo w/ a profile.
• Add the ability to attach more than one profile photo.
• Add the ability to reorder attachments. (The first photo will be the main one.)
• Maintain previous profile photos (attachments) when editing a profile.
• Document which photo formats are accepted. (JPG, PNG. GIF?)

My current thinking is that profile photos will just be attachments to the Profile object. Since those are backed by a content-addressable store, editing profiles won't require re-uploading new photos.

This is like a comment (#1) , but is a fully-fledged post. It would appear on a feed or the home page like other posts.

However, it also has additional metadata that marks it as a reply to some other Item.

Having that metadata available would allow some nice UI features:

• Optionally displaying the original post inline
• The ability to display forward references from an Item to any "Reply posts" that reply to it.

When you attach a very large image, or paste an image (which seems to always paste as a PNG), the file size can be quite large.

Offer an option to re-encode the image as a smaller jpeg (or webp) image to save space.

If a userID is compromised, or its private key is lost, there should be some tool(s) for migrating content to a new userID (pub/priv key pair).

• The tool should scan all content for a key, and re-post it under a new userID.

Optional features:

• Preview items of content before reposting them.
• Server-side integration to allow doing this for userIDs that have been revoked? #5

When a UserID becomes compromised, or a user just wants to delete their account, they should be able to create a revocation Item. Servers should honor this item by:

• Refusing to accept any new items for that user.
• Refusing to serve any existing Items for that user (except the revocation Item itself).

The server should also delete the content for that user, but may do so asynchronously. (ex: via a clean command at some later time.) This could allow someone whose key has been compromised a chance to recover their content and publish it under a new key.

• The /u/:userID/proto3 endpoint should list 1 Item, which is the revocation item, so that other servers that sync from this one will learn of the revocation.

There's a TODO here:
https://github.com/NfNitLoop/feoblog/blob/develop/protobufs/feoblog.proto#L114
... but the more I think about it, the more I think a Revocation should be its own item_type. I don't want the option of specifying a Profile.display_name, about, servers, or follows.

Need to get some automated builds and releases on this.

• Automatic builds for develop branch. I don't care about warnings at this stage. OMG Rust has so many warnings.
• Automatic builds for pull requests.
• Automatic builds & release for release stage. OK I should probably clean up warnings for that.

Oh, right, also, some tests. 😬

Once there's a release for protoc-gen-ts that contains thesayyn/protoc-gen-ts#20 it should have better support for oneof fields. Update to it and remove any hacky workarounds I made in the meantime.

Right now, following a user, and knowing whether you already follow a user, is a bit error-prone. You've got to copy/paste the userID into your Edit Profile page.

The UI should:

• Show you that you're following a user when you view their user profile.
• Offer a Follow button
• The Edit Profile page should accept a user ID and prompt the user asking whether they want to add that ID to their list of follows.