openbridge / ob_bulkstash Goto Github PK

Bulk Stash is a docker rclone service to sync, or copy, files between different storage services. For example, you can copy files either to or from a remote storage services like Amazon S3 to Google Cloud Storage, or locally from your laptop to a remote storage.

Home Page: https://www.openbridge.com

License: MIT License

Shell 87.47% Dockerfile 12.53%

rclone data-pipeline amazon-web-services google-cloud sftp-synchronisation docker s3 oracle-cloud google-cloud-storage storage-service

ob_bulkstash's People

Contributors

Stargazers

Watchers

Forkers

eltonrodrigues efrecon rzn siran shafiahmed hongkongkiwi ddiiwoong olivierdalang kurrik xbl3 hongshunyang vssharma79 kennethehmsen futuralogic catmeme pinkbluersglobal

ob_bulkstash's Issues

Having trouble getting image to run as Lambda function

Errors out with "Error: the working directory '' is invalid, it needs to be an absolute path",
"errorType": "Runtime.InvalidWorkingDir". What should I put as the WORKDIR override parameter?

Can I use the rclone GUI?

I think this container is set up for command line, but I wondered if there was a way to configure it to use the new GUI?

environment variable for time zone?

there's no environment variable to set the timezone used by crond.
I needed to mount a file to /etc/localtime. Is this the preferred way to do it?

input too short when revealing password

Having a bit of trouble getting rclone to create a file system for sftp... Since the container is running on my NAS inside my home network, and I am still a bit unfamiliar with docker's secrets functionality, I figured I would just put my password in the RCLONE_CONFIG_MYSFTP_PASS environment variable, but when I try this, I get the following error...

Failed to create file system for "MYSFTP:files/subdir": input too short when revealing password - is it obscured?

in my env file I have the following:

RCLONE_CONFIG_MYSFTP_TYPE="sftp"
RCLONE_CONFIG_MYSFTP_HOST="xxx.xxxxx.xxx"
RCLONE_CONFIG_MYSFTP_USER="user"
RCLONE_CONFIG_MYSFTP_PORT=22
RCLONE_CONFIG_MYSFTP_PASS="passwordhere"

Did I miss something? Or do I need to spend some time learning docker secrets before getting this to work?

Suppress monit output in docker entrypoint

In continuation of #5 I have a need to suppress monit output in docker entrypoint, which cames each time a container starts and looks like:

# docker run --rm --name=rclone-move-logs -v /path/to/logs:/data/path/to/logs -e RCLONE_CROND_SOURCE_PATH="" -e RCLONE_CROND_DESTINATION_PATH="" -e RCLONE_CONFIG_STORAGE_TYPE="s3" -e RCLONE_CONFIG_STORAGE_ACCESS_KEY_ID=$RCLONE_CONFIG_STORAGE_ACCESS_KEY_ID -e RCLONE_CONFIG_STORAGE_SECRET_ACCESS_KEY=$RCLONE_CONFIG_STORAGE_SECRET_ACCESS_KEY openbridge/ob_bulkstash:1.40 rclone move /data/path/to/logs/ --include "*.gz" STORAGE:some-storage/some-dir/ --quiet
...
 New Monit id: 41f5c471ffe84363ef17412f85fcd618
 Stored in '/root/.monit.id'
Starting Monit 5.25.1 daemon with http interface at [localhost]:2849

Unfortunately, I do not have any experience with monit. Fast googling did not find me any way to suppress the info output of this utility. Nevertheless, it's always possible to send all monit output to /dev/null. :)

expanded ampersand in /etc/crontabs/root

I am running bulkstash as: ( Btw. I can not get it work for Google Drive right now )

docker run -d \
--name rclone \
--restart unless-stopped \
-v /tmp/foobar:/host/tmp/foobar \
-e RCLONE_CROND_SOURCE_PATH="/host/tmp/foobar" \
-e RCLONE_CROND_DESTINATION_PATH="mygd:backup/foobar" \
-e RCLONE_SYNC_COMMAND="*/1 * * * * /usr/bin/env bash -c /rclone.sh run 2>&1" \
-e RCLONE_CONFIG_MYGD_TYPE="drive" \
-e RCLONE_CONFIG_MYGD_CLIENT_ID="clientID" \
-e RCLONE_CONFIG_MYGD_CLIENT_SECRET="clientSecret" \
openbridge/ob_bulkstash rcron start

and seems that ampesand ( stdout,stderr redirection ) will expand to {{RCLONE_SYNC_COMMAND}}

cat /etc/crontabs/root

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
*/1 * * * * /usr/bin/env bash -c /rclone.sh run 2>{{RCLONE_SYNC_COMMAND}}1

P.S. I am using default configuration, because of this #17 , where RCLONE_CROND_SCHEDULE does not work.

FTP support?

would it be possible to add good old fashioned FTP support?

RCLONE_CROND_SCHEDULE in docker-entrypoint.sh is not considered

There https://github.com/openbridge/ob_bulkstash/blob/develop/docker-entrypoint.sh#L32 should be RCLONE_CROND_SCHEDULE instead of RCLONE_SYNC_COMMAND, isnt it ?

Add quiet mode for docker entrypoint

First of all, I want to thank the contributors of this useful docker image for rclone. Unlike other images, this one can be used not only with sync command but also with all other rclone abilities. Coupled with a flexible configuration with environment variables, ob_bulkstash is an excellent instrument for everyday life of DevOps engineers.

While using ob_bulkstash in my life I severely lack a quiet mode for docker entrypoint. The reason here is that I use rclone in the everyday logrotate scenario which runs via cron. All of my servers are tuned to send me emails if some tool outputs anything in cron jobs (think, this is usual for the whole industry). This fact leads to receiving similar emails with info messages about ob_bulkstash normal work every day, when I should receive messages only if some command falls down.

Here is my example of ob_bulkstash run and its output which obstructs me:

# docker run --rm --name=rclone-move-logs -v /path/to/logs:/data/path/to/logs -e RCLONE_CROND_SOURCE_PATH="" -e RCLONE_CROND_DESTINATION_PATH="" -e RCLONE_CONFIG_STORAGE_TYPE="s3" -e RCLONE_CONFIG_STORAGE_ACCESS_KEY_ID=$RCLONE_CONFIG_STORAGE_ACCESS_KEY_ID -e RCLONE_CONFIG_STORAGE_SECRET_ACCESS_KEY=$RCLONE_CONFIG_STORAGE_SECRET_ACCESS_KEY openbridge/ob_bulkstash:1.40 rclone move /data/path/to/logs/ --include "*.gz" STORAGE:some-storage/some-dir/ --quiet
INFO: There is no CROND configs present. Skipping use of CROND
 New Monit id: 13d31271d4fd8f87bb346fc24e29e2bd
 Stored in '/root/.monit.id'
Starting Monit 5.24.0 daemon with http interface at [localhost]:2849
OK: All processes have completed. rclone service is starting...

I looked at docker-entrypoint.sh and saw that as for now there is no way to suppress printing of info messagess. Is it possible to add any run option which will disable any info output of container's entrypoint?

Tagging the image per the rclone version

As shown in a similar project, Docker hub hooks can be used to automatically generate images with all the official versions of rclone, tagged with the version number of rclone. This allow to pick up specific versions of rclone in production.

Would you be interested in a PR achieving this? For a longer discussion and history, see here.

Support for Secrets

According to the doc

you can use Docker secrets to centrally manage this data and securely transmit it to only those containers that need access to it. Secrets are encrypted during transit and at rest in a Docker swarm. A given secret is only accessible to those services which have been granted explicit access to it, and only while those service tasks are running.

PR #2 provides support for carrying secretive information such as tokens or passwords for the sources and destinations using secrets. This follows the idea of the image itself, an image with a strong focus on configuration through environment variables (and not a configuration file, even if this is still possible). The PR achieves this through looking into the content of environment variables and resolving those that follow a given template to the file that it points at. This is slightly different from the "regular" Docker way. Official images use _FILE at the end of environment variables to take information from a file instead. However, as the variables are read by rclone, the mechanism in the PR provides a solution that does not imply any modification of the rclone binary implementation, thus will be able to work in future version of rclone.

Would you mind considering taking in the PR?

problem building imag

When building the container from source I get ERROR: 'bash@community' is not a valid child dependency, format is name([<>~=]version). I provided a pull request:

My setup:
System: MacOS Sierra 10.12.6
Docker version:

$ docker --version
Docker version 18.06.1-ce, build e68fc7a

Full output:

$ docker build --tag crc .
Sending build context to Docker daemon  166.9kB
Step 1/12 : FROM alpine:latest
 ---> 196d12cf6ab1
Step 2/12 : MAINTAINER Thomas Spicer <[email protected]>
 ---> Using cache
 ---> 5d8311d59e6c
Step 3/12 : ENV RCLONE_VERSION="current"
 ---> Using cache
 ---> 242719035aaf
Step 4/12 : ENV RCLONE_TYPE="amd64"
 ---> Using cache
 ---> fa236e6082fe
Step 5/12 : ENV BUILD_DEPS       wget@community       linux-headers@community       unzip@community       fuse@communityc
 ---> Using cache
 ---> 12e6910e682f
Step 6/12 : RUN set -x     && echo "@community http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories     && apk update     && apk add --no-cache --virtual .persistent-deps         bash@community         curl@community         monit@community         ca-certificates@community     && apk add --no-cache --virtual .build-deps         $BUILD_DEPS     && cd /tmp      && wget -q http://downloads.rclone.org/rclone-${RCLONE_VERSION}-linux-${RCLONE_TYPE}.zip     && unzip /tmp/rclone-${RCLONE_VERSION}-linux-${RCLONE_TYPE}.zip     && mv /tmp/rclone-*-linux-${RCLONE_TYPE}/rclone /usr/bin     && addgroup -g 1000 rclone     && adduser -SDH -u 1000 -s /bin/false rclone -G rclone     && sed -i 's/#user_allow_other/user_allow_other/' /etc/fuse.conf 	  && mkdir -p /config /defaults /data     && rm -Rf /tmp/*     && rm -rf /var/cache/apk/*     && apk del .build-deps
 ---> Running in 7c0971e4e83c
+ echo '@community http://dl-cdn.alpinelinux.org/alpine/edge/community'
+ apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
v3.8.1-27-g42946288bd [http://dl-cdn.alpinelinux.org/alpine/v3.8/main]
v3.8.1-23-ga2d8d72222 [http://dl-cdn.alpinelinux.org/alpine/v3.8/community]
v3.8.0-2385-gd3e4872faa [http://dl-cdn.alpinelinux.org/alpine/edge/community]
OK: 13568 distinct packages available
+ apk add --no-cache --virtual .persistent-deps bash@community curl@community monit@community ca-certificates@community
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
ERROR: 'bash@community' is not a valid child dependency, format is name([<>~=]version)
The command '/bin/sh -c set -x     && echo "@community http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories     && apk update     && apk add --no-cache --virtual .persistent-deps         bash@community         curl@community         monit@community         ca-certificates@community     && apk add --no-cache --virtual .build-deps         $BUILD_DEPS     && cd /tmp      && wget -q http://downloads.rclone.org/rclone-${RCLONE_VERSION}-linux-${RCLONE_TYPE}.zip     && unzip /tmp/rclone-${RCLONE_VERSION}-linux-${RCLONE_TYPE}.zip     && mv /tmp/rclone-*-linux-${RCLONE_TYPE}/rclone /usr/bin     && addgroup -g 1000 rclone     && adduser -SDH -u 1000 -s /bin/false rclone -G rclone     && sed -i 's/#user_allow_other/user_allow_other/' /etc/fuse.conf 	  && mkdir -p /config /defaults /data     && rm -Rf /tmp/*     && rm -rf /var/cache/apk/*     && apk del .build-deps' returned a non-zero code: 99

Do not run rclone command on startup, only on cron

Firstly thank you for this awesome docker!

The docker image complains if I set the rclone schedule but do not want to set the command to run on startup.

For example, in the case where I only want to run on schedule and not immediately I have no option (docker-entrypoint complains that I havn't passed any arguments).

Can we make this an option? something like RCLONE_CROND_RUN_ON_STARTUP

docker-compose w/ crond doesn’t stay running

running docker-compose up with the RCLONE_CRONFILE env var set doesn’t run, a perpetual command was needed to keep it running like command: tail -f /dev/null

crond can't see drive contents

I've not been able to get this running with crond
Sticking some debug stuff in rclone.sh, shows that it can see that gdrive is a remote but it can't see its contents

...
  fi
  echo "Executing => $sync_command" >> /test
  eval "$sync_command" || send

  echo "rclone listremotes ===" >> /test
  rclone listremotes >> /test
  echo "===" >> /test
  echo "rclone ls ===" >> /test
  rclone ls gdrive: >> /test
  echo "===" >> /test

) 200>/run/rclone.lock
...

from the console it seems to work fine though

bash-4.4# rclone listremotes
2019/03/08 08:32:45 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
gdrive:
bash-4.4# rclone ls gdrive:
2019/03/08 08:33:00 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
  1560010 Getting started
bash-4.4# cat test
Executing => rclone copy --drive-shared-with-me /data GDRIVE:/Backups/testing
rclone listremotes ===
gdrive:
===
rclone ls ===
===