getSpatialData is an R package in an early development stage that ultimately aims to provide homogeneous function bundles to query, download, prepare and transform various kinds of spatial datasets from open sources, e.g. Satellite sensor data, higher-level environmental data products etc. It supports both sf and sp classes as AOI inputs (see set_aoi in available functions). Due to the early development stage, the included functions and their concepts could be removed or changed in some cases.

For all public functions documentation is available. See also the list of data sources that are or will be implemented.

To install the current beta version, use devtools.

devtools::install_github("16EAGLE/getSpatialData")

The following functions are publicly available. They have been manually tested on Linux (Ubuntu 16.04 LTS, 17.10) and Windows 10 and are being constantly build and checked through Travis (R-devel and R-release). The get* function familiy allows to connect to a specific service to query for product names, query for records by time and AOI, preview images and download imagery and meta datasets. Session functions define session-wide settings that are visible for every function in the running session, e.g. to define one specific AOI that can be used for different queries without defining it with each function call.

• getSentinel_query() – querys the Copernicus Open Access Hubs for Sentinel-1, Sentinel-2 and Sentinel-3 data and returns a data frame containing the found records (rows) and their attributes (columns).
• getSentinel_preview() – uses the output of getSentinel_query() to preview (quick-look) a user-selected record even before downloading it. By default, the preview is displayed corner-georeferenced in a map viewer in relation to the session AOI.
• getSentinel_data() – uses the output of getSentinel_query() to download Sentinel data.

• getLandsat_names() – obtains available Landsat product names from USGS Earth Explorer, which can be optionally used with getLandsat_query() to narrow the search.
• getLandsat_query() – querys USGS Earth Explorer for Landsat data and returns a data frame containing the found records (rows) and their attributes (columns).
• getLandsat_preview() – uses the output of getLandsat_query() to preview (quick-look) a user-selected record. By default, the preview is displayed corner-georeferenced in a map viewer in relation to the session AOI.
• getLandsat_data() – uses the output of getLandsat_query() to order and download Landsat data.
  • supports order (on-demand processing) and download of higher-level products (all Landsat products), e.g. top-of-atmosphere (TOA), surface reflectance (SR) or different indices, from USGS-EROS ESPA.
  • supports direct download of Level-1 products (Landsat-8 only) via Amazon Web Services (AWS).
  • will support direct download of Level-1 products (all Landsat products) via USGS EarthExplorer (requires a USGS user profile with machine-to-machine download permission)

• getMODIS_names() – obtains available MODIS product names from USGS Earth Explorer, which can be optionally used with getMODIS_query() to narrow the search.
• getMODIS_query() – querys USGS Earth Explorer for MODIS data and returns a data frame containing the found records (rows) and their attributes (columns).
• getMODIS_preview() – uses the output of getMODIS_query() to preview (quick-look) a user-selected record. By default, the preview is displayed corner-georeferenced in a map viewer in relation to the session AOI.
• getMODIS_data() – uses the output of getMODIS_query() to order and download MODIS data from LAADS.

• prepSentinel() beta – makes downloaded Sentinel datasets ready-to-use by automatically inspecting, extracting, sorting and converting the relevant contents of the datasets to a user-defined format.
• cropFAST() beta – crops a raster file to a spatial extent using GDAL. It is useful when working with large-scale, memory-intensive datasets.

• login_CopHub() – define your Copernicus Open Access login credentials once for the present R session to be able to call each getSentinel* function without defining login arguments each time you use them.
• login_USGS() – define your USGS login credentials once for the present R session to be able to call each get* function that connects to a USGS service without defining login arguments each time you use them.

• set_archive() – define a getSpatialData archive directory to which all *_data functions will download data.
• set_aoi() - draw or define an AOI as sf, sp or matrix object for the running session that can be used by all query functions.
• view_aoi() - display the session AOI in an interactive mapview/leaflet map viewer.
• get_aoi() - get the session AOI you have defined or drawn before as sf, sp or matrix object.

The following universal semantics on data are used by getSpatialData (from smallest to biggest entity):

• image: An image of a specific time and spatial extent.
• record: A set of meta fields identifying and describing a specific image, being part of multiple records in a query.
• dataset: Smallest entity that is delivered by a service. Might consist of multiple files, including meta data and bandwise imagery. Covers a specific time and spatial extent.
• product: A data product offered by a specific service, consisting of multiple datasets over a period of time and a wide spatial extent. Might be differentiated by:
  • platform: A general platform design (e.g. "Landsat" or "Sentinel").
  • sensor: Type of sensor which acquired the data from which the product originates (e.g. "MODIS", "MSI" or "OLI").
  • collection: A product version.
  • level: Processing level of the product (e.g. "Level 2A" or "Surface Reflectance").
  • source: The service acquiring, processing or distributing the product (e.g. "ESA Copernicus" or "USGS").

The following universal semantics on computational steps are used by getSpatialData:

• get: Recieve data from different sources, named either by sensor or platform (whichever is used by the scientific community to referr to the derived products)
  • names: Result of searching available products (differs by source and platform), which might be differentiated further later on (e.g. by level).
  • query: Result of searching a source for data records of a specific or multiple products.
  • preview: Preview a record.
  • data: Result of recieving one or multiple dataset from a source.
• prep: Prepare/preprocess data obtained with get

The following code represents a working chain for querying, filtering, previewing and downloading Sentinel-2 data within R. The procedure can be done for Sentinel-1, Sentinel-2 or Sentinel-3.

## Load packages
library(getSpatialData)
library(raster)
library(sf)
library(sp)

## Define an AOI (either matrix, sf or sp object)
data("aoi_data") # example aoi

aoi <- aoi_data[[3]] # AOI as matrix object, or better:
aoi <- aoi_data[[2]] # AOI as sp object, or:
aoi <- aoi_data[[1]] # AOI as sf object
#instead, you could define an AOI yourself, e.g. as simple matrix

## set AOI for this session
set_aoi(aoi)
view_aoi() #view AOI in viewer, which will look like this:

_{Figure 1: Screenshot of the RStudio Viewer, displaying the previously defined session AOI using view_aoi()}

#instead of using an existing AOI, you can simply draw one:
set_aoi() #call set_aoi() without argument, which opens a mapedit editor:

_{Figure 2: Screenshot of the RStudio Viewer, displaying the mapedit editor allowing the user to draw a session AOI}

## After defining a session AOI, define time range and platform
time_range <-  c("2017-08-01", "2017-08-30")
platform <- "Sentinel-2" #or "Sentinel-1" or "Sentinel-3"

## set login credentials and archive directory
login_CopHub(username = "username") #asks for password or define 'password'
set_archive("/path/to/archive/")

## Use getSentinel_query to search for data (using the session AOI)
records <- getSentinel_query(time_range = time_range, platform = platform)

## Filter the records
colnames(records) #see all available filter attributes
unique(records$processinglevel) #use one of the, e.g. to see available processing levels

records_filtered <- records[which(records$processinglevel == "Level-1C"),] #filter by Level
records_filtered <- records_filtered[as.numeric(records_filtered$cloudcoverpercentage) <= 30, ] #filter by clouds

## View records table
View(records)
View(records_filtered)
#browser records or your filtered records

_{Figure 3: Screenshot of the View() display in RStudio, displaying a filtered records table produced by getSentinel_query()}

## Preview a single record on a mapview map with session AOI
getSentinel_preview(record = records_filtered[9,])

_{Figure 4: Screenshot of the RStudio viewer, displaying a corner-georeferenced Sentinel-2 preview and the session AOI using getSentinel_preview()}

## Preview a single record on a mapview map without session AOI
getSentinel_preview(record = records_filtered[9,], show_aoi = FALSE)

_{Figure 5: Screenshot of the RStudio viewer, displaying a corner-georeferenced Sentinel-2 preview using getSentinel_preview()}

## Preview a single record as RGB plot
getSentinel_preview(record = records_filtered[9,], on_map = FALSE)

_{Figure 6: Screenshot of the RStudio viewer, displaying a simple Sentinel-2 RGB plot preview using getSentinel_preview()}

## Download some datasets to your archive directory
datasets <- getSentinel_data(records = records_filtered[c(4,7,9), ])

## Finally, define an output format and make them ready-to-use
datasets_prep <- prepSentinel(datasets, format = "tiff")
# or use VRT to not store duplicates of different formats
datasets_prep <- prepSentinel(datasets, format = "vrt")

## View the files
datasets_prep[[1]][[1]][1] #first dataset, first tile, 10 m resolution
datasets_prep[[1]][[1]][2] #first dataset, first tile, 20 m resolution
datasets_prep[[1]][[1]][3] #first dataset, first tile, 60 m resolution

## Load them directly into R
r <- stack(datasets_prep[[1]][[1]][1])

The following products are being evaluated to be implemented within the package. This also includes sources which can be already accessed through existing packages that could be wrapped behind an standardized R function interface. Please feel free to contribute to the list, e. g. through a pull request:

Ideas on possible data sources to be included, technical ideas or other are welcome! Open an issue to start a discussion: https://github.com/16eagle/getSpatialData/issues