Code Monkey home page Code Monkey logo

rt_eqcorrscan's Introduction

EQcorrscan

A python package for the detection and analysis of repeating and near-repeating earthquakes.

Citation:

We have a manuscript on the development of EQcorrscan, if you make use of EQcorrscan please cite the following paper:

Chamberlain, C. J., Hopp, C. J., Boese, C. M., Warren-Smith, E., Chambers, D., Chu, S. X., Michailos, K., Townend, J., EQcorrscan: Repeating and near-repeating earthquake detection and analysis in Python. Seismological Research Letters 2017

If you want to you should also cite the version number: DOI

Installation

The easiest way to install EQcorrscan is through anaconda: Anaconda-Server Badge

Instructions for installing EQcorrscan and the required dependency, fftw are linked from the docs

Updates

If you want to be kept informed about releases, bug-tracking and enhancements without having to keep looking on github, subscribe to our google group.

Documentation

The full documentation for this package can be found here: Docs. Any errors including typos and just missing bits can either be fixed by you, or flagged in the issues tab here. We host our docs on readthedocs, which uses sphinx to scrape the docstrings in the codes, so it is simple to match the docs to the codes and change the docstrings.

Contributing

Please fork this project and work on it there then create a pull request to merge back to this main repository. Please create a branch from develop.

When you make changes please run the tests in the test directory to ensure everything merges with minimum effort. If there is not yet a test to cope with your changes then please write one.

Please document your functions following the other documentation within the functions, these doc-scripts will then be built into the main documentation using Sphinx.

Functionality

This package contains routines to enable the user to conduct matched-filter earthquake detections using obspy bindings when reading and writing seismic data, as well as subspace detection, brightness source-scanning, relative moment calculation using singular-value decomposition, and correlation pick-adjustment for similar events.

Also within this package are:

  • Clustering routines for seismic data;
  • Peak finding algorithm (basic, but appropriate for noisy data);
  • Automatic amplitude picker for local magnitude scale;
  • Obspy.core.event integration, which opens up lots of other functions (Seishub, hypoDDpy etc.);
  • Stacking routines including phase-weighted stacking based on Thurber at al. (2014);
  • Brightness based template creation based on the work of Frank et al. (2014);
  • Singular Value Decomposition derived magnitude calculations based on Rubinstein & Ellsworth (2010).

The code-base has grown to be quite large - it is probably worth having a look at the docs to check what functions we have. We are writing a series of tutorials included on the EQcorrscan API to highlight key functions.

A note on correlation precision EQcorrscan computes normalised cross-correlations in the frequency-domain using the fftw (Fastest Fourier Transform in the West). Internally the C routines enforce double-precision (64-Bit floating point numbers) for all aspects of the cross-correlations (despite requiring 32-Bit float input and output). Results in testing are accurate to within ~0.0001 of time-domain cross-correlation results.

Test status

Note that tests for travis and appveyor are run daily on master as cron jobs, and may reflect time-out issues.

Service tests Badge
CI checks test
Code coverage codecov

Licence

This package is written and maintained by the EQcorrscan developers, and is distributed under the LGPL GNU License, Copyright EQcorrscan developers 2018.

Funding

RCET

Continued development of the EQcorrscan package is directly supported by the RCET, Rapid Characterisation of Earthquakes and Tsunami programme funded by the New Zealand Ministry of Business, Innovation and Employment Endeavour fund.

Development is indirectly funded by grants from Toku TÅ« Ake: EQC and a Rutherford Discovery Fellowship.

rt_eqcorrscan's People

Contributors

calum-chamberlain avatar snyk-bot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

rt_eqcorrscan's Issues

Process-based parallelism for streaming only supported on Linux

Describe the bug
When using thread-based parallelism to run a streamer and detector concurrently the detector can interfere with the streamer collecting data. #13 fixes this for Linux by moving to Process-based parallelism, however this implementation currently relies on using the fork method of starting Processes, which is only supported on Linux. Windows and MacOS do not have a safe fork method and use spawn instead. spawn does not work with the implmentation in #13 and an adjustment needs to be made to make this compatible with spawn.

To Reproduce
Attempt to use multiprocessing rather than multithreading in streamer.py.

Expected behavior
The streamer should be able to collect data regardless of what the rest of the system is doing.

Desktop (please complete the following information):

  • OS: Windows and Mac)S

Additional context
#13 was somewhat rushed through as a fix to issues experienced by me, and I do not use MacOS or Windows for this, so the patch was not tested on these systems until running through CI systems. The changes in #13 were quite significant, and I didn't have time to correct these for spawn once I had made them. This issue is here to serve both as a reminder to me, and for anyone experiencing this issue on MacOS and Windows to let me know that it is urgent for them.

Use dequeue for data buffer

collections.dequeue implements a FIFO queue that I should probably use for the buffer in utils.seedlink. This should be more efficient that removing items from the left of the queue...

Doesn't cope well when data are slow to arrive

When data have a high latency detections are not computed, even when this high latency is only for one station. It would be good if detections were computed and the missing (offending) station was ignored for that period.

Simulate real-time server for old data for testing

To test the system on previous, known sequences it would be useful to simulate a real-time server with old data. This system should have the same interface as the seedlink server currently in rt_eqcorrscan, but will keep track of the servers internal time. It should also have some randomised latency associated with it to attempt to simulate some of the issues with slow data (#3).

To Do

A general to do list for real-time matched-filtering:

  • run sub detectors via mpisubprocess, write new templates to be added to tribe to disk and read in. This should stop memory duplication, and allow detectors to log to their own file.
  • add a build-db script to build the template database based on config and starttime
  • add a simulate real time script to simulate old events
  • seedlink client for real-time matched filtering
  • results visualisation (e.g. plot waveforms in real-time and mark detections) - Use PyQT directly, like this? Using bokeh to allow serving
    • Plot data as it comes in
    • Plot detection on those data
    • Note, QTThreads will likely be needed for this - as I have the plotting written at the moment the plot window complains.
  • catalogue clustering for template database Using obsplus - not clustering
    • Database management - reason I want to be able to search for all templates within n km of source
      • some kind of "event" server system?
      • Use pyASDF to store data, event data and provenance information together (Note that this might be better placed in EQcorrscan itself).
      • Run an SQL or similar server to hold limited metadata (event origin and path to HDF5 file?).
      • Or, make use of h5serv via hypyd? (video here)
      • Need a remote template server that workers can access via web (e.g. run template server continuously on some machine, then spool up AWS instances after earthquakes).
      • For these kind of long-running things - ensuring that template station info (response and location in particular) are consistent might be important (e.g. don't stack or correlate the wrong stations).
    • real-time updating
  • large earthquake or swarm triggering

Distribution to do:

  • conda-forge distribution
  • pypi distribution
  • docs
  • tutorials

RCET wishlist

The MBIE funded RCET project has specific additional needs to real-time matched filter analyses, and additional components that can be leveraged to enhance the RT-EQcorrscan flow. This issues is meant to be a wishlist for the RCET project for extensions and improvements that would be useful to the RCET project. These enhancements will probably be made on an RCET branch that is kept up to date with master, but may never merge into master.

  1. AWS spin-up capability
  2. Ability to take other, more accurate, magnitude information (e.g. from FinDER)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.