yanirs / rls-data Goto Github PK
View Code? Open in Web Editor NEWReef Life Survey data processing
License: MIT License
Reef Life Survey data processing
License: MIT License
Useful snippet:
# TODO: new links:
# - "https://geoserver-portal.aodn.org.au/geoserver/ows?typeName=imos:ep_m2_inverts_public_data&SERVICE=WFS&outputFormat=csv&REQUEST=GetFeature&VERSION=1.0.0"
# - "https://geoserver-portal.aodn.org.au/geoserver/ows?typeName=imos:ep_m0_off_transect_sighting_public_data&SERVICE=WFS&outputFormat=csv&REQUEST=GetFeature&VERSION=1.0.0"
# - "https://geoserver-portal.aodn.org.au/geoserver/ows?typeName=imos:ep_m2_cryptic_fish_public_data&SERVICE=WFS&outputFormat=csv&REQUEST=GetFeature&VERSION=1.0.0"
# - "https://geoserver-portal.aodn.org.au/geoserver/ows?typeName=imos:ep_m1_public_data&SERVICE=WFS&outputFormat=csv&REQUEST=GetFeature&VERSION=1.0.0"
# TODO: survey_list and site_list needed?
# TODO: limit to only the RLS program? `&CQL_FILTER=(program%20LIKE%20'RLS')`
# TODO: parse the new files
for data_type in ('m0_off_transect_sighting', 'm1', 'm2_cryptic_fish', 'm2_inverts'):
url = ('https://geoserver-portal.aodn.org.au/geoserver/ows?SERVICE=WFS&outputFormat=csv&REQUEST=GetFeature&'
f'VERSION=1.0.0&typeName=imos:ep_{data_type}_public_data')
out_path = survey_data_dir / f"{data_type.lower()}.csv"
_logger.info(f'Downloading {url} to {out_path}')
with open(out_path, 'w') as fp:
fp.write(requests.get(url).text)
Then this line and the mapping can be removed:
Line 62 in 8c48b6f
It's possible to also download m0_off_transect_sighting
as a data type here:
Line 24 in 8c48b6f
This would require parsing an additional file and inferring the correct method. It might not be worth it as it's only 5M of additional data.
Having multiple sources of truth for survey methods is brittle. It'd probably be better to drop the static mappings from this repo in favour of getting the methods from species.json
, or to completely retire the legacy api-*.json
files.
In some cases, the generated map doesn't look good since there's a single site right on the edge of the map. Such cases may be addressed by reducing the checked area when checking if all the sites fit or by adding padding in addition to the hardcoded extents (e.g., a couple of degrees in each direction, or something like 2% to 5% to maintain a constant aspect ratio).
The diff can be too noisy when species are added/removed, as all the IDs change.
Minification can be done when the JSONs are fetched ahead of serving. We get nicer diffs and simpler code without the minification.
For data backward compatibility, AODN downloading is limited only to RLS surveys:
Line 21 in 8c48b6f
It's possible to remove the CQL_FILTER
condition, which would increase the overall download size by over 100M.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.