Code Monkey home page Code Monkey logo

filehitch's Introduction

Filehitch

Automate remote file synchronization by monitoring changes in HTTP and S3 resources based on a specified cron interval. Decrypt and trigger commands as needed.

Github release version Go report GoDoc License Code with hearth by Stnby

Install

To install the latest version of Filehitch from sources, run:

go install github.com/digilolnet/filehitch/cmd/filehitch@latest

To install Secure IO file encryption tool from sources, run:

go install github.com/minio/sio/cmd/ncrypt@latest

Features

  • Cron Expression Scheduling: Set up automated monitoring intervals using precise cron expressions, including seconds, minutes, hours, days of the month, months, days of the week, and multiple timezones support.
  • HTTP and S3 Resource Monitoring: Monitor remote files via HTTP endpoints and S3 resources, ensuring you stay updated with the latest changes.
  • Secure Encryption: Utilize the robust ChaCha20-Poly1305 algorithm for seamless decryption of encrypted resources, ensuring secure synchronization and access to sensitive data.
  • Customizable Command Triggers: Execute custom commands of your choice when changes in remote files are detected, enabling flexible automation.
  • Flexible Configuration: Intuitive YAML configuration for easy setup and maintenance.
  • User-Friendly Setup: Intuitive YAML configuration allows for easy and efficient setup and maintenance of the synchronization process.

Configuration

jobs:
  # Define a job named "HTTP example job"
  - name: HTTP example job
    schedule:
      # Set the cron expression for the job to run every hour
      expression: "0 0 * * * *"
      # Specify the timezone for scheduling (America/New_York)
      timezone: America/New_York
      # Introduce a jitter of up to 10 minutes to prevent synchronized execution
      jitter: 600
    resource:
      # Specify the resource type as HTTP for remote file monitoring
      type: http
      http:
        # Specify the URL of the remote file
        url: https://example.com/your-remote-file.txt
        # Specify the HTTP method (GET, POST, etc.)
        method: "GET"
        # Specify custom headers if needed
        headers:
          User-Agent:
            - "MyApp/1.0"
          Authorization:
            - "Bearer YOUR_TOKEN"
        # Specify the expected HTTP response code
        expect:
          code: 200
        # Set a timeout for the HTTP request
        timeout: 30s
        # Optionally include a request body, use base64 encoded string
        body: SGVsbG8sIFdvcmxkIQo=
        # Set the maximum number of attempts
        max_attempts: 3
    file:
      # Specify the local file path to save the downloaded file
      path: "/home/alice/your-other-file-path.txt"
      permissions:
        # Set file permissions to read and write for the owner only
        mode: "0600"
        # Specify the owner of the file
        owner: "alice"
        # Specify the group of the file
        group: "alice"
    trigger:
      # Define the command to be triggered upon file change detection
      command: ["/home/alice/scripts/run.sh", "--input", "your-other-file-path.txt"]
      # Specify the user for executing the trigger command
      user: "alice"
      # Specify the group for executing the trigger command
      group: "alice"
      # Set the current working directory for the trigger command
      cwd: "/home/alice"

  # Define a job named "S3 example job"
  - name: S3 example job
    schedule:
      # Set the cron expression for the job to run every 30 seconds
      expression: "*/30 * * * * *"
      # Specify the timezone for scheduling (Europe/Amsterdam)
      timezone: Europe/Amsterdam
      # Introduce a jitter of up to 5 seconds to prevent synchronized execution
      jitter: 5 
    resource:
      # Specify the resource type as S3 for remote file monitoring
      type: s3
      s3:
        # Specify the S3 endpoint (for AWS, use s3.amazonaws.com)
        endpoint: s3.amazonaws.com
        # Specify the AWS region for the S3 bucket
        region: your-aws-region
        # Specify your AWS access key ID
        access_key_id: YOUR_AWS_ACCESS_KEY_ID
        # Specify your AWS secret access key
        secret_access_key: YOUR_AWS_SECRET_ACCESS_KEY
        # Specify the name of the S3 bucket
        bucket: my-s3-bucket
        # Specify the path to the object within the bucket
        object: your-file-path.txt.enc
      decryption:
        # Specify the decryption password only if the object is encrypted
        password: "YourDecryptionPassword"
    file:
      # Specify the local file path to save the downloaded file
      path: "/home/alice/your-file-path.txt"
      permissions:
        # Set file permissions to read and write for the owner only
        mode: "0600"
        # Specify the owner of the file
        owner: "alice"
        # Specify the group of the file
        group: "alice"
    trigger:
      # Define the command to be triggered upon file change detection
      command: ["/home/alice/scripts/run.sh", "--input", "your-file-path.txt"]
      # Specify the user for executing the trigger command
      user: "alice"
      # Specify the group for executing the trigger command
      group: "alice"
      # Set the current working directory for the trigger command
      cwd: "/home/alice"

Encrypting a file

Make sure Secure IO file encryption tool is installed and run:

ncrypt -cipher C20P1305 your-file.txt > your-file.txt.enc

License

This project is licensed under the Apache License, Version 2.0 - see the LICENSE file for details.

filehitch's People

Contributors

stnby avatar dependabot[bot] avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.