Code Monkey home page Code Monkey logo

eodatadown's Introduction

EODataDown

A tool for automatically downloading Earth Observation imagery and products.

Dependencies

Google Cloud Services

For google cloud services the following dependencies need to installed.

  • google-api-core
  • google-cloud-core
  • google-auth
  • google-cloud-bigquery
  • google-cloud-storage

[https://github.com/GoogleCloudPlatform/google-cloud-python]

Installation

Using conda and conda-forge the dependencies can be installed within the following command:

conda install -c conda-forge pycurl postgresql psycopg2 sqlalchemy google-cloud-storage google-cloud-bigquery arcsi wget pyyaml planet

Configuration

There are a number of configuration files which need to created for the system to work

Overall System

An example is within the share directory of the source code.

Logging

EODataDown uses the python logging library. A general configuration suitable for most users is installed alongside the source <install_path>/share/eodatadown. This can be edited or the EDD_LOG_CFG variable used to use a different configuration file.

Environmental Variables

  • EDD_MAIN_CFG - specify the location of a JSON file configuring the system.
  • EDD_LOG_CFG - specify the location of a JSON file configuring the python logging system.
  • EDD_NCORES - specify the number of cores to use when running jobs which can use multiple cores.

Notes for Sensors

Landsat and Sentinel-2 from Google.

To use this sensor you need to set up a google account and project. While the data and download is free an account is still required. See https://cloud.google.com/resource-manager/docs/creating-managing-projects

Details of the account will need to be specified in the JSON config file for the LandsatGOOG sensor.

Setting up Google Accounts

See the following page for setting up a project:

https://cloud.google.com/resource-manager/docs/creating-managing-projects

And then the following page for setting up your credentials:

https://cloud.google.com/bigquery/docs/reference/libraries

Setting up Postgresql Database

New database system

To initialise a database use the initdb command.

initdb -D <path to data>

Start database

postgres -D <path to data>

Alternatively, you can start the database as a background service.

pg_ctl -D <path to data> -l logfile start

Create database

createdb eodd_dev_db

Setup User

psql
CREATE USER eodduser WITH PASSWORD '<password>';
GRANT ALL PRIVILEGES ON DATABASE eodd_dev_db TO eodduser;

eodatadown's People

Contributors

petebunting avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

danclewley

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.