uncg-daisy / psi-collect Goto Github PK

View Code? Open in Web Editor NEW

4.0 6.0 3.0 293.63 MB

Collect Post-Storm Imagery from NOAA

License: MIT License

Python 75.31% GLSL 4.68% HTML 17.14% TeX 2.86%

noaa coastal hurricane storms imagery

psi-collect's Introduction

Post-Storm Imagery Collection

This package downloads, catalogs, and stores NOAA emergency response imagery

Quick Start

There are two ways to get started with psi-collect — using the version on PyPI or installing directly from this GitHub repository. The version currently on PyPI does not work with images from the two most recent Hurricanes (Hx Delta and Hx Zeta). The version here on GitHub is modified and works with these storms (but we still need to make a few more modifications before this new version is packaged for PyPI).

from PyPI:

Install Python 3.6, 3.7, or 3.8 (Download Here)
Run pip3 install psi-collect in your favorite terminal
Run pstorm collect -h for help on collecting images or pstorm catalog -h for help on cataloging local archives

from GitHub Repository

Install Python 3.6, 3.7, or 3.8 (Download Here)
Make sure you have dependecies installed (check them out in requirements.txt)
Run pip install git+https://github.com/UNCG-DAISY/psi-collect in your favorite terminal
Run pstorm collect -h for help on collecting images or pstorm catalog -h for help on cataloging local archives

Usage Guide

Check out the documentation for Collecting and Cataloging images to see usage.

An Example Use of `psi-collect`: Labeling Imagery

We labeled storm impacts for ~300 images from Hurricane Florence using the Coastal Image Labeler. The labels are available on figshare.

The specific NOAA imagery to link to these labels can be retrieved with psi-collect via:

pstorm collect -s Florence -a 20180917a_jpgs -d

Contributing / Developing

Code of Conduct

We hope to foster an inclusive and respectful environment surrounding the contribution and discussion of our project. Make sure you understand our Code of Conduct.

Code Conventions

Before committing to the repository please read the project Code Conventions.

Project Pre-Requisites

Python 3.6, 3.7, or 3.8 (Download Here)
Pipenv (Run pip install pipenv)
You will need to add the module path <parent dir>/psi-collect/psicollect/ to your PYTHONPATH (See Configuring Module)

If you get an error message that looks something like Module not found: "psicollect", then the PYTHONPATH is not configured correctly!

You will need this in order to run the project.

Installing Dependencies for Development

Change current directory (cd) to psi-collect/ (the project root)
Run pipenv install --dev to install dependencies
Run pre-commit install to install style checking when committing

Testing the Collect Script

Change current directory to the collector module (cd collector/)
Either use pipenv run collect.py <args> or pipenv shell then collect.py <args>
In addition tests can be run using pytest from the project root directory

The arguments for collect.py are listed here

NOAA Data 💾

NOAA landing page for the post-storm Emergency Response Imagery, here

Team Members and contributors

psi-collect's People

Contributors

Stargazers

Watchers

Forkers

conlin-matt katmratliff priyankabudavi

psi-collect's Issues

problem extracting .zip on windows

Describe the bug
I get the AttributeError message 'ZipInfo' object has no attribute 'name' when extracting .zip archives on my windows machine. This happens when downloading/extracting to the default directory or to a directory of my choice (e.g., on the desktop). Interestingly, the files do still seem to extract, but if you are trying to download and extract multiple archive files, this breaks the process so that only the first one gets downloaded and extracted.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
It is very possible that this is an issue with my network/machine (federal government). I suspect this is the case if other windows users/reviewers haven't had this issue. I have no trouble downloading/extracting .tar archives.

Note: Submitting this issue as part of the JOSS review process (openjournals/joss-reviews#2075)

implement `click` package for path wrangling

Using the click package can help tidy up some of the path wrangling that's being done in the pstorm.py file. Otherwise perhaps look at the argparse add_subparsers command. I'm not sure that using exec() to call the script is the best idea.

from @chrisleaman 's JOSS review: openjournals/joss-reviews#2075 (comment)

Cannot specify path for pstorm catalog

Describe the bug
pstorm catalog expects the downloaded data to only be at <user_home>/psi/collect/data/archives. If the user specifies another location to download the files to (pstorm collect --path C:\data\), there is no way to specify this path and pstorm catalog cannot find the data.

To Reproduce
Steps to reproduce the behavior:

pstorm collect -s Florence --path C:\data\ --download.
pstorm catalog -e tif

Expected behavior
Either one of two options could work:

Remove --path option from pstorm collect. Data will always be downloaded to <user_home>/psi/collect/data/archives
Add --path option to pstorm catalog so user can specify where data is located.

Cannot get archive variable from path when making catalog

Describe the bug
pstorm catalog exits without creating the catalog.

To Reproduce
Steps to reproduce the behavior:

Download images: pstorm collect -s Florence -a 20180916b_RGB.tar --download
Run catalog: pstorm catalog --debug -v 3 -e tif results in this output (the exit code might indicate some sort of recursion error?)

[Output, click to expand]

Parsing out current path to determine catalog variables to use ... Looking for storm in path: C:\Users\z5189959\psi\collect\data\archives
DONE
Getting a list of all valid images ... 
Searching through C:\Users\z5189959\psi\collect\data\archives for the pattern ".*" ...
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0793900w325700n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0793900w325830n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0794030w325400n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0794030w325530n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0794030w325700n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0794200w325230n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0794200w325400n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0794200w325530n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0794330w325100n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0794330w325230n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0794330w325400n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0794500w324930n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0794500w325100n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0794500w325230n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0794630w324800n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0794630w324930n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0794800w324800n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0794800w324930n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0794930w324800n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0795100w324630n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0795100w324800n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0795230w324500n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0795230w324630n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0795400w324500n.tif ... matches pattern!
C:\Users\z5189959\psi\collect\data\archives\Florence\20180916b_RGB\20180916bC0795400w324630n.tif ... matches pattern!
 1  Florence/20180916b_RGB/20180916bC0793900w325700n.tif
 2  Florence/20180916b_RGB/20180916bC0793900w325830n.tif
 3  Florence/20180916b_RGB/20180916bC0794030w325400n.tif
 4  Florence/20180916b_RGB/20180916bC0794030w325530n.tif
 5  Florence/20180916b_RGB/20180916bC0794030w325700n.tif
 6  Florence/20180916b_RGB/20180916bC0794200w325230n.tif
 7  Florence/20180916b_RGB/20180916bC0794200w325400n.tif
 8  Florence/20180916b_RGB/20180916bC0794200w325530n.tif
 9  Florence/20180916b_RGB/20180916bC0794330w325100n.tif
10  Florence/20180916b_RGB/20180916bC0794330w325230n.tif
11  Florence/20180916b_RGB/20180916bC0794330w325400n.tif
12  Florence/20180916b_RGB/20180916bC0794500w324930n.tif
13  Florence/20180916b_RGB/20180916bC0794500w325100n.tif
14  Florence/20180916b_RGB/20180916bC0794500w325230n.tif
15  Florence/20180916b_RGB/20180916bC0794630w324800n.tif
16  Florence/20180916b_RGB/20180916bC0794630w324930n.tif
17  Florence/20180916b_RGB/20180916bC0794800w324800n.tif
18  Florence/20180916b_RGB/20180916bC0794800w324930n.tif
19  Florence/20180916b_RGB/20180916bC0794930w324800n.tif
20  Florence/20180916b_RGB/20180916bC0795100w324630n.tif
21  Florence/20180916b_RGB/20180916bC0795100w324800n.tif
22  Florence/20180916b_RGB/20180916bC0795230w324500n.tif
23  Florence/20180916b_RGB/20180916bC0795230w324630n.tif
24  Florence/20180916b_RGB/20180916bC0795400w324500n.tif
25  Florence/20180916b_RGB/20180916bC0795400w324630n.tif
DONE
Parsing out information about images from their paths ... 
Process finished with exit code -1073741571 (0xC00000FD)

Expected behavior
pstorm catalog should generate a catalog .csv

Additional context
The problem appears to be with the Cataloging._get_archive_from_path function. If this is disabled, the catalog gets generated:

psi-collect/psicollect/cataloging/make_catalog.py

Lines 419 to 437 in cb8043a

    
           def _get_archive_from_path(scope_path: Union[bytes, str] = None) -> str: 
        
               scope_path = h.validate_and_expand_path(scope_path) 
        
               path_head, path_tail = os.path.split(scope_path) 
        
               if path_head == os.path.splitdrive(scope_path)[1] and path_tail == '': 
        
                   # If the filesystem root directory is reached, a storm-specific catalog cannot be found 
        
                   raise PathParsingException(objective='the archive name') 
        
               if ('20' in path_tail and '_jpgs' in path_tail) is False or scope_path == s.DATA_PATH: 
        
                   # If the current directory does not look like an archive name or is the data path 
        
                   # Keep recursively checking each directory to match the pattern (traverse back through path) 
        
                   return Cataloging._get_archive_from_path(scope_path=os.path.split(scope_path)[0]) 
        
               else: 
        
                   return path_tail

I'm not entirely sure, but I think it might have to do with comparing paths when doing the recursion, because I'm on windows. If I query these variables, I get slightly different paths (note the slashes), due to how s.DATA_PATH is constructed. I think paths should be compared using os.path.normpath or os.path.realpath rather than just using the strings to avoid this issue.

In[2]: scope_path
Out[3]: 'C:\\Users\\z5189959\\psi\\collect\\data\\archives\\Florence\\20180916b_RGB\\20180916bC0793900w325700n.tif'
In[3]: s.DATA_PATH
Out[4]: 'C:\\Users\\z5189959/psi/collect\\data'

use `pathlib` vs `os.path`

I think a lot of the Path manipulation can be done with the pathlib library that was introduced in Python 3.4. This might help clean up the code, rather than using os.path.

from @chrisleaman 's JOSS review: openjournals/joss-reviews#2075 (comment)

New docs to describe v1 -> v2

explain requests -> request-html change

Docs

I don't have experience with docs, but the home page of the docs is not the same as the README.md, which it should be:

for example, the team order is different, the USGS data page is still listed, and many of the internal docs links on the home page are routed to the old documents.

follow dirs to package and distribute

in the docs:
https://psi-collect.readthedocs.io/en/master/distributing_pypi/

add more detail to paper.md

I would recommend adding a bit more motivation to the paper.md. Based on our conversation at AGU I know that there many qualities of this package that make collection and cataloging of images way easier, probably more appropriately described as actually possible. This doesn't come through clearly in the paper. I would recommend expanding the statement of need and/or adding a section that focuses on challenges to accessing this dataset that sets up the statement of need.

from:
openjournals/joss-reviews#2064 (comment)

@Matmorcat I will add text, but if you have any specific ideas, pls feel free to add or ping me.

Build is broken for py38

The Travis build is broken for py38.

there is a time out for the build.. all dependencies seem to be installed w/o a problem, so it will take a bit of sleuthing

Cataloging function not working for multiple storms?

Describe the bug
The cataloging function doesn't seem to work for a number of storms, possibly because these storms don't ship with .geom files? I have had issues with Sandy, Maria, Irma, and Irene. I notice that none of these seem to come with .geom files for their images (whereas storms such as Dorian and Gordon do).

To Reproduce

pstorm collect -s Sandy -a oct31ajpeg (or Irma or Maria etc.) -d
*Notice the lack of .geom files
pstorm catalog -p .../Sandy (or Irma or Maria etc.)

Expected behavior
The catalog csv file should be produced.

Screenshots
I get this error message when trying to catalog images for Irma

Additional context
This seems similar to #74 . Am I missing something here, e.g. downloading the images incorrectly so that the .geom files don't ship?

Note: Submitting this issue as part of the JOSS review process (openjournals/joss-reviews#2075)

reduce repo size

generally we might just reduce the repo size (i.e., remove catalogs, since the module can build catalogs)...

Chris even mentioned it in his review, which i have turned into an issue: #86

I know our the JOSS bot had troubles building the paper because the repo was big (see the review issue).. originally I wanted the catalog to be in the repo for search features, but that enhancement is a bit downstream.

@Matmorcat , are you ok if the catalogs are removed?

More tests and coverage

self explanatory, would like to get test coverage higher

Add use case w/ figshare data

Is your feature request related to a problem? Please describe.
No, not a problem, just an enhancement/ documentation idea

Describe the solution you'd like
It would be nice if we showed a use case w/ the figshare data (https://doi.org/10.6084/m9.figshare.11604192.v1) — i.e., users could grab the labeler images from figshare and download the correct repository (20180917a_jpgs) via
pstorm collect -s Florence -a 20180917a_jpgs -d

(recommendation after discussion w/ @MARDAScience)

Build failing

e.g., from Travis logs for TOXENV=py36:

py36 run-test: commands[1] | py.test --basetemp=/home/travis/build/UNCG-DAISY/psi-collect --cov=/home/travis/build/UNCG-DAISY/psi-collect/psicollect --cov-report=xml
ERROR: usage: py.test [options] [file_or_dir] [file_or_dir] [...]
py.test: error: argument --basetemp: basetemp must not be empty, the current working directory or any parent directory of it

ERROR: InvocationError for command /home/travis/build/UNCG-DAISY/psi-collect/.tox/py36/bin/py.test --basetemp=/home/travis/build/UNCG-DAISY/psi-collect --cov=/home/travis/build/UNCG-DAISY/psi-collect/psicollect --cov-report=xml (exited with code 4)

Codecov does not detect the config, and as a result does not show accurate coverage

Finish JOSS Draft

Paper

@Matmorcat — check out the paper... and please add any cool functionality that i missed (i.e., byte wise download?, etc), or LMK if there is anything else you want me to add...

I need to add references, but once that is done, and you are cool with it, i can send it around to the rest of the coauthors to get their approval... LMK...

reduce image size in test folder

Try reducing the size data in the tests folder. Do you really need to include the full resolution images to test on? Or is it possible to use a reduced size image or even mock them out completely. This might improve compatibility with build systems (which can have issues if the repo is too big) and might just generally be easier to work with.

from @chrisleaman 's JOSS review: openjournals/joss-reviews#2075 (comment)

Quick improvement to docs for clarity

I have a small suggestion to help make the main docs page more clear:

There is a brief "Usage" section on the main page as well as a more detailed "Usage" section that can be navigated to. This is true for "Development" as well. Because of this, I found myself assuming that clicking anything in the section list on the left side would just take me to different sections of the main page, rather than to more detailed descriptions of things on different pages. Perhaps I am the only one that might have this issue, but in my opinion this could be easily fixed by simply adding a line at the end of the "Usage" section on the main page such as "For a more detailed description and examples see the 'Usage section' " where a click on that would take the reader to the more detailed page. This would have helped my understanding of the layout a lot. You sort of do this already in the "Development" section which is actually how I discovered my confusion.

I know this may be a bit knit-picky, but after all this is the time to make this great tool as user friendly as possible.

Note: I am submitting this issue as part of the JOSS review process (openjournals/joss-reviews#2075)

Version for JOSS submission is listed incorrectly

The version for JOSS submission labelled as v0.6, when the latest software release is currently v1.0.0 (pypi and otherwise).

Delta and Zeta imagery not found

after pstorm collect -s 2020 Delta and Zeta imagery not found

Docs

update docs for this repo to exclude slides, reports, and dashboard (i.e., stuff not relevant for psi-collect)

Bad references result in data mismatch on attempt to download (copy from P-Sick/psi-collect)

When the collector finds a link for an archive that is bad and returns 0 content length, the program doesn't skip it, and instead halts with an error message.

Add feedback when no storm search term identified

Is your feature request related to a problem? Please describe.
Add short print statement when a storm search term or pattern is not found

Describe the solution you'd like
example response:

> pstorm collect -s katherine
Download Status Report (February 19, 2020 at 08:04 AM) <-s katherine -t .* -p C:\Users\<USER>\psi\collect\data\archives>

<No storms matching search term detected>

This is consistent with the feedback given when no archive files are detected for a specific search term or pattern.

Catalogs are not being included in distribution on PyPI

The distribution listed on PyPI does not include the catalogs as intended. Most likely due to ignoring non-python files, where data files to include need to be adjusted

tests fail on windows

Describe the bug
test_cataloging.py fails on Win32 Python 3.7.3

To Reproduce
run pytest in root directory produces this:

Additional context
I noticed that Travis was only set up to run in a linux env, so feel free to close this issue if you aren't worried about testing on Windows, just wanted you all to be aware that I had tried to run them and what the result was.

Note: Submitting this issue as part of the JOSS review process (openjournals/joss-reviews#2075)

	def _get_archive_from_path(scope_path: Union[bytes, str] = None) -> str:

	scope_path = h.validate_and_expand_path(scope_path)

	path_head, path_tail = os.path.split(scope_path)

	if path_head == os.path.splitdrive(scope_path)[1] and path_tail == '':
	# If the filesystem root directory is reached, a storm-specific catalog cannot be found

	raise PathParsingException(objective='the archive name')

	if ('20' in path_tail and '_jpgs' in path_tail) is False or scope_path == s.DATA_PATH:
	# If the current directory does not look like an archive name or is the data path

	# Keep recursively checking each directory to match the pattern (traverse back through path)
	return Cataloging._get_archive_from_path(scope_path=os.path.split(scope_path)[0])

	else:
	return path_tail

uncg-daisy / psi-collect Goto Github PK

psi-collect's Introduction

Post-Storm Imagery Collection

Quick Start

from PyPI:

from GitHub Repository

Usage Guide

An Example Use of psi-collect: Labeling Imagery

Contributing / Developing

Code of Conduct

Code Conventions

Project Pre-Requisites

Installing Dependencies for Development

Testing the Collect Script

NOAA Data 💾

Team Members and contributors

psi-collect's People

Contributors

Stargazers

Watchers

Forkers

psi-collect's Issues

Recommend Projects

Recommend Topics

Recommend Org

An Example Use of `psi-collect`: Labeling Imagery