Code Monkey home page Code Monkey logo

lpp's Introduction

Lemmy Post Purger (LPP)

Latest documentation available at: https://nowsci.com/#/lemmy/lpp/?id=lemmy-post-purger-lpp

Instances can grow fast, and that can be a problem if you have limited disk space. If you don't mind losing history that your users have not interacted with, LPP can help. It will purge posts/media older than a specified time if they have not been posted, marked read, voted on, commented on, had comments voted on, or saved by users on your instance.

Pict-rs processing is optional. If there is concern that Lemmy is not purging from Pict-rs, or that Pict-rs is not removing files from the file system, then this extra processing will ensure that occurs. Supplying PICTRS_RM_OLDER_THAN_DAYS along with a URL and server admin token will run a purge command directly against Pict-rs for media that falls within the above criteria of no users on your instance interacting with them. In addition, supplying a PICTRS_FOLDER will also scan the folder for images that were not properly removed from the file system during the purge and force remove them.

Usage

In docker-compose:

version: '2.1'

services:

  lpp:
    image: nowsci/lpp
    container_name: lpp
    volumes:
      - /path/to/pictrs/mnt:/mnt
    environment:
      LOCAL_URL: https://lemmy.domain.ext
      LOCAL_USERNAME: top_level_admin_user
      LOCAL_PASSWORD: password
      PURGE_OLDER_THAN_DAYS: 365
      PICTRS_RM_OLDER_THAN_DAYS: 365
      PICTRS_SERVER_API_TOKEN: abcdefg
      PICTRS_URL: http://lemmy-pictrs:8080
      PICTRS_FOLDER: /mnt
      HOURS_BETWEEN_PURGES: 24
      PG_HOST: lemmy_db_host
      PG_PORT: 5432
      PG_USERNAME: lemmy
      PG_PASSWORD: lemmy_db_password
    restart: unless-stopped

Manually:

cd src
export LOCAL_URL=https://lemmy.domain.ext
export LOCAL_USERNAME=top_level_admin_user
export LOCAL_PASSWORD=password
export PURGE_OLDER_THAN_DAYS=365
export PICTRS_RM_OLDER_THAN_DAYS=365
export PICTRS_SERVER_API_TOKEN=abcdefg
export PICTRS_URL=http://lemmy-pictrs:8080
export PICTRS_FOLDER=/mnt
export HOURS_BETWEEN_PURGES=24
export PG_HOST=lemmy_db_host
export PG_PORT=5432
export PG_DATABASE=lemmy
export PG_USERNAME=lemmy
export PG_PASSWORD=lemmy_db_password
npm install
npm start

Variables

Variable Description
LOCAL_URL The URL of your instance
LOCAL_USERNAME Username of a user for your instance
LOCAL_PASSWORD Password of a user for your instance
PURGE_OLDER_THAN_DAYS Posts must be older than this many days to be purged
PURGE_BATCH_SIZE Optional: Process this many posts at a time. Default: 100
PICTRS_RM_OLDER_THAN_DAYS Optional: Pictures older than this many days that failed to be purged by lemmy will be purged directly in Pict-rs
PICTRS_SERVER_API_TOKEN Optional: The server API key (PICTRS__SERVER__API_KEY) for Pict-rs
PICTRS_URL Optional: The URL for your Pict-rs instance
PICTRS_FOLDER Optional: The path to the Pict-rs folder if you're using local storage where any media that failed to purge will be force removed
HOURS_BETWEEN_PURGES How long to wait between runs
PG_HOST Lemmy Postgres host
PG_PORT Lemmy Postgres port
PG_DATABASE Lemmy Postgres database name. Optional, and if not specified the username will be used as the database name
PG_USERNAME Lemmy Postgres user name
PG_PASSWORD Lemmy Postgres password

lpp's People

Contributors

fmstrat avatar muddybulldog avatar andrewmunsell avatar

Stargazers

Tomas avatar Emhl avatar Thomas Fournier avatar MSTCL avatar Pål Sollie avatar Mohamed Elashri avatar  avatar Hudson Newey avatar Russell avatar Arif Widi Nugroho avatar  avatar shofhna avatar Maor Kadosh avatar Nιƈσ 🐉🔥 avatar Adam Curtis avatar  avatar badblocks avatar DevSH01 avatar Óliver García Albertos avatar Jose V Herrera Gallegos avatar

Watchers

 avatar  avatar

lpp's Issues

also remove items only down voted?

Seems like spam items might be interacted with via down-votes, thus not being removed, correct?

Would be nice to have an option to also remove posts that are only down voted or predominantly down voted.

ReferenceError: aliasURL is not defined

lpp  |
lpp  | > [email protected] start
lpp  | > node index.js
lpp  |
lpp  | (node:22) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
lpp  | (Use `node --trace-deprecation ...` to show where the warning was created)
lpp  | Purging 0 posts older than 7 days
lpp  | Purging images older than 7 days
lpp  | Processing 414 posts with thumbnails
lpp  | /app/pictrs.js:168
lpp  |         console.error(`Failed on ${aliasUrl}`)
lpp  |                                    ^
lpp  |
lpp  | ReferenceError: aliasUrl is not defined
lpp  |     at purgePictrs (/app/pictrs.js:168:36)
lpp  |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
lpp  |     at async main (/app/index.js:146:7)
lpp  |
lpp  | Node.js v21.1.0
lpp exited with code 0

Removes legitimate files from pict-rs

After running this script, I noticed the web interface has broken images in the "trending communities" section, and the favicon was missing.

The second part of purgePictrs(pool) scans the filesystem and any files that don't match posts are deleted.

Unfortunately I found this removes

  • Lemmy instance logo/thumbnail
  • Community logos/thumbnails
  • Embedded images in comments
  • Embedded images in community/instance descriptions
  • User avatars

For now, I'm adjusting the script to exit after the first part.

Feature to only purge "orphaned" media from Pict-rs

Hi there! I'd love to see a feature in LPP that allows you to configure only purging media from pictrs that is not tied to a post (or comment, if possible).

My instance is currently on a VM that is limited to 128GBs of space, and at the current moment it'd be a bit difficult to expand it. I'm not super concerned with old posts that contain only pure text / links to external stuff, but media that no longer is attached to a post/comment would be nice to remove.

It'd also be even more convenient if this could apply to posts that have been removed (either by a mod or the user themselves) since posts tend to get removed, but not necessarily purged.

Super appreciative of LPP altogether though!

lpp crashes with db error 'error: relation "post" does not exist'

Running Lemmy 0.18.4

k logs -f deploy/lpp

> [email protected] start
> node index.js

/app/node_modules/pg-protocol/dist/parser.js:287
        const message = name === 'notice' ? new messages_1.NoticeMessage(length, messageValue) : new messages_1.DatabaseError(messageValue, length, name);
                                                                                                 ^

error: relation "post" does not exist
    at Parser.parseErrorMessage (/app/node_modules/pg-protocol/dist/parser.js:287:98)
    at Parser.handlePacket (/app/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/app/node_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.<anonymous> (/app/node_modules/pg-protocol/dist/index.js:11:42)
    at Socket.emit (node:events:512:28)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Readable.push (node:internal/streams/readable:234:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23) {
  length: 103,
  severity: 'ERROR',
  code: '42P01',
  detail: undefined,
  hint: undefined,
  position: '25',
  internalPosition: undefined,
  internalQuery: undefined,
  where: undefined,
  schema: undefined,
  table: undefined,
  column: undefined,
  dataType: undefined,
  constraint: undefined,
  file: 'parse_relation.c',
  line: '1392',
  routine: 'parserOpenTable'
}

Node.js v19.5.0
npm notice 
npm notice New minor version of npm available! 9.3.1 -> 9.8.1
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v9.8.1>
npm notice Run `npm install -g [email protected]` to update!
npm notice 

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.