yanirs / rls-data Goto Github PK

Useful snippet:

# TODO: new links:
# - "https://geoserver-portal.aodn.org.au/geoserver/ows?typeName=imos:ep_m2_inverts_public_data&SERVICE=WFS&outputFormat=csv&REQUEST=GetFeature&VERSION=1.0.0"
# - "https://geoserver-portal.aodn.org.au/geoserver/ows?typeName=imos:ep_m0_off_transect_sighting_public_data&SERVICE=WFS&outputFormat=csv&REQUEST=GetFeature&VERSION=1.0.0"
# - "https://geoserver-portal.aodn.org.au/geoserver/ows?typeName=imos:ep_m2_cryptic_fish_public_data&SERVICE=WFS&outputFormat=csv&REQUEST=GetFeature&VERSION=1.0.0"
# - "https://geoserver-portal.aodn.org.au/geoserver/ows?typeName=imos:ep_m1_public_data&SERVICE=WFS&outputFormat=csv&REQUEST=GetFeature&VERSION=1.0.0"
# TODO: survey_list and site_list needed?
# TODO: limit to only the RLS program? `&CQL_FILTER=(program%20LIKE%20'RLS')`
# TODO: parse the new files
for data_type in ('m0_off_transect_sighting', 'm1', 'm2_cryptic_fish', 'm2_inverts'):
    url = ('https://geoserver-portal.aodn.org.au/geoserver/ows?SERVICE=WFS&outputFormat=csv&REQUEST=GetFeature&'
           f'VERSION=1.0.0&typeName=imos:ep_{data_type}_public_data')
    out_path = survey_data_dir / f"{data_type.lower()}.csv"
    _logger.info(f'Downloading {url} to {out_path}')
    with open(out_path, 'w') as fp:
        fp.write(requests.get(url).text)

• Also update expectations based on new server files

Then this line and the mapping can be removed:

It's possible to also download m0_off_transect_sighting as a data type here:

This would require parsing an additional file and inferring the correct method. It might not be worth it as it's only 5M of additional data.

Having multiple sources of truth for survey methods is brittle. It'd probably be better to drop the static mappings from this repo in favour of getting the methods from species.json, or to completely retire the legacy api-*.json files.

In some cases, the generated map doesn't look good since there's a single site right on the edge of the map. Such cases may be addressed by reducing the checked area when checking if all the sites fit or by adding padding in addition to the hardcoded extents (e.g., a couple of degrees in each direction, or something like 2% to 5% to maintain a constant aspect ratio).

The diff can be too noisy when species are added/removed, as all the IDs change.

Minification can be done when the JSONs are fetched ahead of serving. We get nicer diffs and simpler code without the minification.

• Species can be obtained from the new API
• It'd probably be a good idea to remove unlisted species from the survey JSON
• Scraping can be done separately to check the site (might be unnecessary)

For data backward compatibility, AODN downloading is limited only to RLS surveys:

It's possible to remove the CQL_FILTER condition, which would increase the overall download size by over 100M.