Code Monkey home page Code Monkey logo

spacechop's Introduction

SpaceChop is a Docker container for processing your images through HTTP requests.

πŸ”’ Secure - Dont allow everyone to create arbitrary image transformations and overload your server.

πŸ“¦ Storage - Store original and transformed images on the infrastructure you are already using. For example Amazon S3, HTTP server, or local volume.

πŸ‘¨β€πŸ’» For developers - Set up available transformations via yaml files.

At spacechop.com, you can get for the hosted solution of SpaceChop which includes CDN, storage, caching, database, statistics and analytics for your service.

To get started with SpaceChop, go to our documentation website.

Installation

The recommended way to use SpaceChop is via docker-compose:

docker-compose.yml

version: '3'
services:
  spacechop:
    container_name: spacechop
    image: spacechop/spacechop:latest
    ports:
      - 8080:3000
    volumes:
      - ./config.yml:/config.yml

config.yml

paths:
  - /:preset/:image(.*)
sources:
  - http:
      root: https://upload.wikimedia.org/wikipedia/commons/:image
presets:
  # fill 200x200 with type jpeg and compress with quality 0.9
  t_200:
    steps:
      - $fill:
          width: 200
          height: 200
      - $format:
          type: jpeg
      - $compress:
          quality: 90
# start the service using docker-compose
docker-compose up -d

# check the logs for help with your configuration and see what's happening
docker-compose logs -f spacechop

Go to http://localhost:8080/t_200/c/c4/Photo_Wallet_product.jpg in your browser and you should see an image fetched from Wikimedia and transformed with the above preset t_200. You just successfully transfomed your first image using SpaceChop!

Getting started

You will at least need a path, source and preset to get started.

paths and sources

Paths and sources are interlinked and work together using path-to-regex with support for ()-matching groups. Use the path express-route-tester with version 2.0.0 to see what works.

Basically how it works is that the path you request at http://localhost:8080/t_200/your-unique-path/with-slashes/to-and-image-url.jpg will look through your sources until it finds an image for your root url (if you are using the http source).

i.e.
http://localhost:8080/t_200/your-unique-path/with-slashes/to-and-image-url.jpg will in the above configuration look at: http://commons.wikipedia.org/your-unique-path/with-slashes/to-and-image-url.jpg

Notice the usage of :image in paths and sources, you can change this to whatever makes sense for you in your matching. You can use your own variable names and matching in your sources.

Each source has its own set of configuration to be able to lookup and download originals. You can read more about each source in our documentation.

presets

As you might have seen there is a :preset variable in the path in the example configuration above, this is the only required variable in path that there is, and this will match the exact name of your preset in the configuration as you might have guessed.

In a preset there are steps which are composed of a list of operations with their unique set of options. You can read more about each operation in our documentation.

Contributing

Build Status codecov Greenkeeper badge PRs Welcome

Read the SpaceChop Contributor Guidelines.

Running tests locally:

sh test.sh

or manually using docker-compose

docker-compose up -d --build
docker-compose exec spacechop sh -c "npm run test:watch"

This project uses TypeScript for static typing and TSLint for linting. You can get both of these built into your editor with no configuration by opening this project in Visual Studio Code, an open source IDE which is available for free on all platforms.

spacechop's People

Contributors

ericwenn avatar greenkeeper[bot] avatar ppa-covis avatar rjnienaber avatar timbrandin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

spacechop's Issues

Investigate: how we handle large files

I've found that we load all bytes into memory, and if the file is larger than the memory available the process crashes. My best guess is to look into how to limit and manage memory using for our subprocesses or just raise the amount of memory available on the machines.

An in-range update of @types/node is breaking the build 🚨

The devDependency @types/node was updated from 10.12.8 to 10.12.9.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

@types/node is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

An in-range update of nodemon is breaking the build 🚨

The devDependency nodemon was updated from 1.18.5 to 1.18.6.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

nodemon is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Release Notes for v1.18.6

1.18.6 (2018-11-05)

Bug Fixes

Commits

The new version differs by 1 commits.

  • 521eb1e fix: restart on change for non-default signals (#1409) (#1430)

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

An in-range update of js-yaml is breaking the build 🚨

The dependency js-yaml was updated from 3.12.1 to 3.12.2.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

js-yaml is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Commits

The new version differs by 3 commits.

  • e4267fc 3.12.2 released
  • 7231a49 Browser files rebuild
  • 99c0bf9 Fix for issue #468 includes passing test (#469)

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

An in-range update of winston-daily-rotate-file is breaking the build 🚨

The dependency winston-daily-rotate-file was updated from 3.3.3 to 3.3.4.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

winston-daily-rotate-file is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Commits

The new version differs by 2 commits.

  • e104440 3.3.4
  • bf0a7d5 fix(gzip): rotate file uncaughtException when node is in cluster mode (#185)

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

An in-range update of jest-image-snapshot is breaking the build 🚨

The devDependency jest-image-snapshot was updated from 2.8.0 to 2.8.1.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

jest-image-snapshot is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Commits

The new version differs by 4 commits.

  • 50ed18b chore(version): iterate to v2.8.1
  • 9d7fc6a fix(options): Fix broken diffDirection option (#125)
  • 6272412 docs(readme): Update jest peer dep version (#128)
  • f8e4d34 chore(deps): update jest and peerDeps

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

An in-range update of @types/node is breaking the build 🚨

The devDependency @types/node was updated from 10.12.0 to 10.12.1.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

@types/node is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

The hash parameter means that old images are not removed/updated in storage

Currently when a transformation is stored in storage it is never deleted.
Using the S3 source it could be possible to set an expiry for the file, when it will be removed by S3/DO Spaces. This however does not include possible future storages (HTTP/Volumes etc).

If anyone has any potential solution for this, this is the thread.

Set config.yml location via environment variables

I'm trying to mount my config file in kubernetes via a ConfigMap, and cant mount it directly in root folder /.

I propose a environment variable CONFIG_PATH that controls where config file is read from.

Possibility to put a picture in another picture

Example:
In a picture of a room/environment/setting, place another picture with a dynamic painting/artwork/photo that a user uploaded.

Typical use-case is for customers of a print-shop how their artwork will look like in some example room/environment/setting that we have prepared and mapped out perfect positions and width/height to avoid bleed and to for example match where the wall is within a room for that painting to be placed.

Strip does not always work

Error: Can't delete all CanonDR4 information from a DR4 file - -

at ChildProcess.onExit (/src/node_modules/duplex-child-process/index.js:116:12)
at Object.onceWrapper (events.js:273:13)
at ChildProcess.emit (events.js:182:13)
at ChildProcess.EventEmitter.emit (domain.js:442:20)
at maybeClose (internal/child_process.js:962:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:251:5)

Emitted 'error' event at:
at ChildProcess.onExit (/src/node_modules/duplex-child-process/index.js:120:12)
at Object.onceWrapper (events.js:273:13)
[... lines matching original stack trace ...]
at Process.ChildProcess._handle.onexit (internal/child_process.js:251:5)

Disk usage is constantly growing

Hosting only the SpaceChop container on digital ocean droplet I can see the disk usage constantly growing:
image

This could be due to logs, but also temporary image not being deleted properly.

Analyze does not return the same output when the transformation is cached as the first time

When analyzing images with faces the first time faces is set on the output, but when the image is cached faces is not set any more.

So when fetching the image from the cache and doing analyze we should also include the required parameters from operations.
So these lines:

if ('analyze' in req.query) {
const { state } = await buildTransformation(fromCache.stream, []);
res.json(state);
return;
}

Should look a bit more like these:
https://github.com/spacechop/spacechop/blob/master/src/spacechop.ts#L113-L117

An in-range update of @types/node is breaking the build 🚨

The devDependency @types/node was updated from 10.11.0 to 10.11.1.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

@types/node is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

An in-range update of @types/jest is breaking the build 🚨

The devDependency @types/jest was updated from 23.3.5 to 23.3.6.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

@types/jest is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

An in-range update of jest-express is breaking the build 🚨

The devDependency jest-express was updated from 1.7.0 to 1.8.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

jest-express is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Release Notes for v1.8.0

1.8.0 (2018-09-20)

Features

  • Request: Allow for setting defaults in constructor (12e95bd)
Commits

The new version differs by 5 commits.

  • 6add245 Merge pull request #98 from sheerun/better-defaults
  • 2f73298 Add setHostname to docs
  • 4abde3b Update @sheerun as a contributor
  • d0d47b6 Update documentation
  • 12e95bd feat(Request): Allow for setting defaults in constructor

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

Dynamic presets based on path-parameters.

When serving images on a website the width of the device screen could be used to determine the width needed for the image.

The only way that I'm aware of to transport the wanted width of the image to spacechop is via url parameters. This introduces a security risk as now anyone can request the same image, but with a different width-parameter and overload the service.

One solution to this is to set rules for these parameters in the configuration. Say, in the previous example, that we want pretty high cache hits and therefore (on the client) rounds the device width to something divisible by 100. This validation can, in addition to the client, be done in spacechop.
How this might look in the configuration:

paths:
  - "/:preset/:_width/:image(.*)

dynamic_parameters:
  _width:
    divisible_by: 100

presets:
  my_dynamic_preset:
    steps:
      - $crop:
          width:
            dynamic: _width

divisible_by could be one of several validators that can be used in combination to lower the amount of possible values a dynamic parameter can take.

An in-range update of codecov is breaking the build 🚨

The devDependency codecov was updated from 3.1.0 to 3.2.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

codecov is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Commits

The new version differs by 3 commits.

  • e427d90 feat(services): add azure pipelines (#114)
  • 023d204 Use small HTTP dependency (#110)
  • 500f308 Update Readme

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

Handle configuration updates when storing transformations

If an image is transformed with preset A and then uploaded to storage, and then preset A changes configuration, the cached transformation should be discarded.

This also address the issue when using path parameters in a preset, as currently its up to the user to include all parameters in the storage path. So if a path parameter width is used but is not present in the s3 storage path, width is effectively ignored in the path, and the resulting image is no longer dynamic.

One possible solution is to hash the config object and use that as an identifier for the transformation. For S3 storage we could set arbitrary header that will be returned when doing headObject and therefor we can check if its the correct image. However, this would not allow storing multiple different transformations, for example if using width you cant store width=100 and width=200 as their path would conflict with each other.

Therefor we probably need to expose another path parameter hash that can be used in the path for storage. This would have to be required when using path parameters, but not if using only static parameters.

Service restarts to often


{ Error: spawn ENOMEM
    at ChildProcess.spawn (internal/child_process.js:358:11)
    at spawn (child_process.js:533:9)
    at Child_Process.spawn (/src/node_modules/duplex-child-process/index.js:58:19)
    at Function.Child_Process.spawn (/src/node_modules/duplex-child-process/index.js:15:30)
    at Object.<anonymous> (/src/dist/transform/index.js:34:43)
    at Generator.next (<anonymous>)
    at fulfilled (/src/dist/transform/index.js:4:58)
    at process._tickCallback (internal/process/next_tick.js:68:7) errno: 'ENOMEM', code: 'ENOMEM', syscall: 'spawn' }
events.js:167
      throw er; // Unhandled 'error' event
      ^

Error: Command failed:
    at ChildProcess.onExit (/src/node_modules/duplex-child-process/index.js:116:12)
    at Object.onceWrapper (events.js:273:13)
    at ChildProcess.emit (events.js:182:13)
    at ChildProcess.EventEmitter.emit (domain.js:442:20)
    at maybeClose (internal/child_process.js:962:16)
    at Socket.stream.socket.on (internal/child_process.js:381:11)
    at Socket.emit (events.js:182:13)
    at Socket.EventEmitter.emit (domain.js:442:20)
    at Pipe._handle.close (net.js:606:12)
Emitted 'error' event at:
    at ChildProcess.onExit (/src/node_modules/duplex-child-process/index.js:120:12)
    at Object.onceWrapper (events.js:273:13)
    [... lines matching original stack trace ...]
    at Pipe._handle.close (net.js:606:12)

An in-range update of winston-daily-rotate-file is breaking the build 🚨

The dependency winston-daily-rotate-file was updated from 3.4.1 to 3.5.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

winston-daily-rotate-file is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Commits

The new version differs by 2 commits.

  • 0fdb94e 3.5.0
  • dc9404b update definition to export the interfaces along with class

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

An in-range update of @types/node is breaking the build 🚨

The devDependency @types/node was updated from 10.12.21 to 10.12.22.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

@types/node is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

Upload transformed images to storage

After a transformation has been successfully completed, upload the transformation to Storage.
When a transformation is requested first check if it exists in configured storage and if so pipe that to response.

The configuration could be similar to Sources.
An initial version could support S3 storage, with a common interface making it easy to add new storages.

SVG support

SVG is currently an unsupported type, but most transformations should be applicable to svg as well.

An in-range update of @types/node is breaking the build 🚨

The devDependency @types/node was updated from 10.12.2 to 10.12.3.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

@types/node is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.