Code Monkey home page Code Monkey logo

pith's Introduction

Pith

This repository contains the code for running a Pith server and client interface.

You can try out the project at pith.is. You can run your own Pith server following the instructions below.

Development

There are two primary components to this project:

Frontend

The client-side portion is a Svelte project using the Sapper application framework and the socket.io client library for interfacing with the backend. You can find the client code in the /frontend directory.

Backend

The server-side portion of Pith. It uses python-socketio running on the aiohttp server, Arq for task queues, Redis as a database for task and message queues, and MongoDB as a database for storing content. You can find it in the /backend directory.

Containers

We use Docker containers to facilitate development and testing. All of the services needed to run the project are defined in docker-compose.yml. These are:

  • static: serves the Sapper client
  • app: the socketio interface that's used by the client.
  • worker: a worker script using the arq task queue for executing assorted long-term tasks such as archiving discussions when they're complete.
  • redis: a Redis database used by the task queue (worker) and as a message queue by socketio (app).
  • mongo: the MongoDB database used to store the discussion content.
  • tests: a container that runs the unit tests and connects to app to test the interface.

The containers use volumes to easily facilitate development. You can edit any code in /backend/src and /frontend/src and containers that rely on that code will automatically reload.

Run it

If you are running on a remote machine, copy the .env.test file to a .env file. Change HOST_NAME within the .env file to the external IP address of the remote machine. Use .env instead of .env.test in the following commands.

To run the development build:

$ docker-compose --env-file .env.test up --build -d

Once the build completes, the client can be accessed from http://localhost:3000. The socketio api is running at http://localhost:8080.

If you're using a cloud-based Redis or MongoDB database, you can set the respective environment variables in .env with the connection information.

MONGODB: mongodb+srv://<user>:<password>@<cluster>.mongodb.net/<dbname>?retryWrites=true&w=majority
MONGO_NAME=<dbname>
REDIS=redis

Testing

To run the tests, connect to the tests container:

$ docker exec -it pith_tests_1 /bin/bash

Then, run the tests:

$ ./test.sh

Creating new boards from the command line

From the tests container, you can create new boards by running:

$ python3.8 make_fake.py

This should give output including a new board ID:

made board id 5yo96fq8xr1m

Now you can join the board from the client by visiting in the browser:

http://localhost:3000/b/5yo96fq8xr1m/

Deployment

Don't run Pith in production! It's early in development and things will break. Consider these instructions as inspiration for how one might run the server after a few more versions...

Quickstart

The best way to set up a Pith server will require having:

  • A computer running Linux (ideally an Ubuntu VM)
  • Docker
  • A web domain
  • An SSL certificate for the domain
  • An S3 bucket

A barebones install will require:

  • A computer running Linux
  • Docker

Prerequisites

First, you'll probably want a Linux server running Ubuntu. The most basic $5 tier virtual machine offerings from Linode or Digital Ocean should work fine in most cases.

Ensure you've installed Docker and Docker Engine and installed Docker Compose on the machine and set up a Docker group.

Next, you'll likely want to obtain a domain (or create a sub-domain), otherwise you can access the server directly albeit without SSL. If you're using a domain, we recommend that you also obtain a free SSL certificate from a CA such as Cloudflare or Let's Encrypt.

Finally, you'll also need an S3 bucket, which can be obtained from AWS. If you opt not to use one, be aware that the document export will not function.

Clone the project

To get started, clone this repository to your server:

$ git clone https://github.com/rainflame/pith.git
$ cd pith

Create an ENV file

You will need to create a .env file in the root of this repository containing a few key environment variables. We'd recommend making a copy of .env.test:

$ cp .env.test .env

Then populate these variables in .env with your values:

MONGO_USER=<a default user for MongoDB>
MONGO_PASS=<a default user's password for MongoDB>
HOST_NAME=<your domain name, otherwise the IP of your server>
AWS_WORKER_ACCESS_KEY=<access key for your S3 bucket>
AWS_WORKER_SECRET_KEY=<secret key for your S3 bucket>

Add the SSL certificate

If you're using an SSL certificate, create a combined certificate file for your domain. You'll need the private key generated by the CA, the certificate generated by the CA, and the CA's origin root certificate (if your CA has one; Cloudflare's can be found here). Create a file called cert.pem with the keys in this format:

-----BEGIN CERTIFICATE-----
<your certificate>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<the CA's origin root certificate>
-----END CERTIFICATE-----
-----BEGIN PRIVATE KEY-----
<your private key>
-----END PRIVATE KEY-----

Then, in haproxy.cfg, uncomment these lines so the certificate file is loaded by the hapoxy load balancer:

bind *:443 ssl crt /cert.pem  # ssl cert goes here
redirect scheme https if !{ ssl_fc } # redirect to ssl

Run the Docker stack

To deploy the project, run the Docker stack:

$ docker-compose -f docker-compose.prod.yml --env-file .env up --build

The project will now be accessible from the server's IP address on port 443 if you're using an SSL certificate or port 80 otherwise. If you're using a domain, set its DNS to resolve to the server's IP address with an A record.

Load Balancing

If you'd like, you can add additional instances of the socketio interface to handle higher traffic. Determine the number of instances of the interface you'd like to run. In haproxy.cfg, add additional servers as needed:

server app01 127.0.0.1:5000 check cookie app01
server app02 127.0.0.1:5001 check cookie app02
# add more here as needed

In docker-compose.prod.yml, adjust the port range of the app to accommodate the servers you've added:

services:
  app:
    ports:
      - "5000-5001:8080"

Then, start the services, specifying how many app instances to run:

$ docker-compose -f docker-compose.prod.yml --env-file .env up --build --scale app=2

Helpful Links

AWS Open Ports

pith's People

Contributors

tlz9472 avatar cbroms avatar slz4025 avatar dependabot[bot] avatar

Stargazers

Sergey Zakharov avatar Ben Tsai avatar Brendan Schlagel avatar tomguim avatar

Watchers

James Cloos avatar  avatar  avatar

Forkers

slz4025

pith's Issues

Push updates to the board

Rather than having the client constantly creating requests to refresh the board, have the server push updates to clients every 10 seconds or so. Only send updates if there's new content. This is much more efficient as it removes unnecessary requests from the client and sends just the data that's been changed.

Handle deleting a board unit

How should we deal with this? Might be able to automatically unfocus the unit from any attached discussions... Alternatively could prevent deleting while discussions are attached.

Redesign loading of the board.

We are potentially loading a lot of data on the board. Need to think of a better prefetching/caching startegy. Detangle routes from loading perhaps to avoid issue of leaving discussion, going to board, and loading it back to a "beginning state"?

Duplicate units in pinned

Backend allows for you to add already pinned units to the pinned section, this should be prohibited and fail silently.

Make loading chat more performat

Backend: only send past 25 messages. Add a pagination route so we can request page 0, 1, 2 and so on infinitely.

Frontend: better loading indicator for chat. Detect when top of element is hit and make next page request. Keep track of current page in discussionDisplayStore.

Search with multiple filters.

  • Must always filter on current board.
  • Use keywords.
  • Can also search within current discussion. Search within just chat or block units.
  • For discussion units, can filter on author/time.

Reactions

Allow people to show an "indicator" in real-time, somewhat similar to Zoom.
Have to consider where this indicator appears (i.e. next to user icon?)

Consider later.

Improve linking affordance

Make it clearer you can click on any unit on the board in order to form a link. Also allow for canceling creating a link once started.

Make transclusion keys double brackets [[ ]]

Use double brackets, allow spaces inside but not inside brackets.

Valid:

[[ feugswd53yb5 ]]
[[feugswd53yb5]]
[[feugswd53yb5 ]]

Invalid:

[ [ feugswd53yb5 ]]
[ [feugswd53yb5 ] ]
[[feugswd53yb5] ]

Standardize all valid to:

[[feugswd53yb5]]

Redimension layout in discussion mode.

In board mode, the whole space can be dedicated to the 2D board.

In discussion mode, first divide the screen in half so the left is for discussion and the right is for board. On the left side, the half-section is further halved into a chat section and a summary section. It is important each feels equal in weight, that the summary feels as important as the ongoing discussion. On the right side, put the focus on the top, such that the board feels like a square. The square shape will give the psychological feeling that we see "more at once".

Render board in 2D

Existing solutions, like in React, are too big, or others are too specific.
May need to cook-up something ourselves.

Need:

  • seemingly infinite space, with origin (in center of div), and maybe bounding box
  • panning => library?
  • zoom in/out => library?
  • rendering of basic shapes, like rectangles and arrows (arrows could get messy!)
  • drag and drop, but on a grid => will have to worry a bit about arrow rendering

How do existing methods do it, like mind-mappers or design spaces? Excalidraw, kinopio.club, Figma.
Excalidraw and others use a more heavy solution, which feels rather slow.

For now, maybe use kinopio solution. Origin is at top-left, uses native scrollbars and zooming. For "arrows", there is a line that draws from one point to another. This is a pretty smart idea. The nodes are always rendered above the lines. The lines don't have directionality. Maybe instead of having an arrow point at the end, the line itself has chevrons on it indicating direction. Also, requires dragging to make line. Would be nicer to do a click-click.

When copy unit ID, nicer feedback.

Get rid of the dialog. Maybe do the banner thing, like the one for errors but in a different color (i.e. green), to say they copied it successfully.

Create new board from the project index?

Should we let people create new boards someplace other than the command line? Could be nice to have to get people to create boards themselves rather than just joining existing ones...

Beams

Is a type of unit.
Add two new fields: beam (boolean), purpose (str).

Beams are treated the same as unit in the store. However, they are rendered specially, maybe with a special color, and the purpose should be displayed.

When creating a unit, can choose whether it is a beam. Thus, modify add_unit from add_unit(board_id, text) to add_unit(board_id, text, beam=False, beam_purpose=None).

Unit Ids should be pseudo-random for discussion units

Linking a unit in a discussion chat or document results in a very long ID. It would be better if these IDs were short and unique to just the discussion. For example, a chat unit would be "c003" or "c023" and a document unit might be "d532". It helps tell the unit type apart and promotes readability in the unit editor.

Post transclusions should be returned as an array

Currently, the backend sends a map of transcluded IDs for every post that has transclusions. This should be changed to an array so the order transclusions appear is preserved.

Currently:

{
    "k5rxb5t2owyu": "hello, this is a post",
    "ocy83cof8964": "some idea"
}

Ideally:

[
    { "id": "k5rxb5t2owyu", "pith": "hello, this is a post" },
    { "id": "ocy83cof8964", "pith": "some idea" }
]

Support markdown in editor

It would be nice to be able to support markdown formatting in the editor. Right now the only way to format text is with meta + i/u/b, which uses document.exec to insert the proper html tag around the selected text. In ContentEditable.jsx we should add "" or "*" chars automatically around bold or em tags, and do the opposite when someone adds "" chars around a selection of text.

Rework pinned unit section

  • Rename to "summary"
  • Increase width, reducing chat and board width to accommodate
  • Increase height so it takes 2/3 rather than 1/2 of vertical space

Typing indicators

We need something. Probably not "jim is typing...", but perhaps just some little animated ellipses when typing is happening.

This will require creating some client logic to determine if someone is typing and sending start and stop events to the backend. The backend can then send updates to all clients to indicate typing is or is not taking place.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.