Code Monkey home page Code Monkey logo

meetnetten-occurrences's Introduction

Meetnetten.be - Monitoring of priority species in Flanders, Belgium

Rationale

This repository contains the functionality to standardize datasets of Meetnetten.be to Darwin Core Occurrence datasets that can be harvested by GBIF.

Datasets

Title (and GitHub directory) IPT GBIF
Meetnetten.be - Chorus counts for Amphibia in Flanders, Belgium meetnetten-amphibia-roepkoren-occurrences https://doi.org/10.15468/d4bu8j
Meetnetten.be - Egg counts for butterflies in Flanders, Belgium meetnetten-butterflies-egg-occurrences https://doi.org/10.15468/hsfq2u
Meetnetten.be - Exuviae counts for dragonflies in Flanders, Belgium meetnetten-libellen-larvenhuidjes-occurrences https://doi.org/10.15468/ue87ux
Meetnetten.be - Fyke counts for Amphibia in Flanders, Belgium meetnetten-amfibieen-fuiken-occurrences https://doi.org/10.15468/zeaq2t
Meetnetten.be - Larvae and metamorph counts for Amphibia in Flanders, Belgium meetnetten-amfibieen-larven-metamorfen-occurrences https://doi.org/10.15468/swgure
Meetnetten.be - Population counts for dragonflies in Flanders, Belgium meetnetten-libellen-populatietelling-occurrences https://doi.org/10.15468/crbudg
Meetnetten.be - Sightings for Natterjack toad in Flanders, Belgium meetnetten-rugstreeppad-zichtwaarneming-occurrences https://doi.org/10.15468/2xfw8y
Meetnetten.be - Site counts for butterflies in Flanders, Belgium meetnetten-butterflies-area-occurrences https://doi.org/10.15468/hvgkh4
Meetnetten.be - Transects for butterflies in Flanders, Belgium meetnetten-butterflies-occurrences https://doi.org/10.15468/kfhvy4
Meetnetten.be - Transects for dragonflies in Flanders, Belgium meetnetten-libellen-transect-occurrences https://doi.org/10.15468/y8u6e9
Meetnetten.be - Transects for fire salamanders in Flanders, Belgium meetnetten-vuursalamander-transect-occurrences https://doi.org/10.15468/nbsk9h
Butterfly monitoring scheme in Flanders, Belgium bms-occurrences https://doi.org/10.15468/wwrgcd

Repo structure

The structure for each dataset in datasets is based on Cookiecutter Data Science and the Checklist recipe. Files and directories indicated with GENERATED should not be edited manually.

├── sql                      : Darwin Core SQL queries
│
└── specs                    : Whip specifications for validation

Contributors

List of contributors

License

MIT License for the code and documentation in this repository. The included data is released under another license.

meetnetten-occurrences's People

Contributors

dimevil avatar peterdesmet avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

meetnetten-occurrences's Issues

Remove/update fields in SQL queries

@DimEvil about a month ago I checked behavior in all SQL queries from the IPT, and recorded what fields should be renamed/completed or could be removed. Can you have a look at this? Note: since I checked this a while ago, not all might still be the case.

meetnetten-amfibieen-larven-metamorfen-occurrences

  • event:datasetID: to complete
  • event:parentLocality0: remove
  • event:recordedBy: remove
  • mof:parentEventID: remove

meetnetten-libellen-populatietelling-occurrences

  • event:datasetID: to be completed
  • event:parentLocality0: remove
  • event:recordedBy: remove
  • occ:behaviour: rename

meetnetten-amfibieen-fuiken-occurrences

  • event:datasetID: to complete
  • event:parentLocality0: remove
  • event:recordedBy: remove

meetnetten-butterflies-egg-occurrences

  • event:datasetID: ok
  • event:parentLocality0: remove
  • event:locality1: remove
  • event:recordedBy: remove

meetnetten-butterflies-area-occurrences

  • event:datasetID: ok
  • event:parentLocality0: remove
  • event:locality: remove
  • event:recordedBy: remove
  • mof:parentEventID: remove
  • mof:eventDate: remove

meetnetten-butterflies-occurrences

  • event:datasetID: ok
  • event:parentLocality0: remove
  • event:locality: remove
  • event:recordedBy: remove
  • mof:parentEventID: remove
  • mof:eventDate: remove
  • occ:protocol: remove

meetnetten-libellen-larvenhuidjes-occurrences

  • event:datasetID: to complete
  • event:parentLocality0: remove
  • event:recordedBy: remove
  • mof:parentEventID: remove
  • mof:eventDate: remove
  • occ:behaviour: rename

meetnetten-libellen-transect-occurrences

  • event:datasetID: ok
  • event:parentLocality0: remove
  • event:recordedBy: remove
  • mof:parentEventID: remove
  • occ:behaviour: rename

meetnetten-vuursalamander-transect-occurrences

  • event:datasetID: to complete
  • event:parentLocality0: remove
  • event:recordedBy: remove
  • occ:behaviour: rename

meetnetten-amphibia-roepkoren-occurrences

  • event:datasetID: to complete
  • event:parentLocality0: remove
  • event:recordedBy: remove
  • occ:behaviour: rename

meetnetten-rugstreeppad-zichtwaarneming-occurrences

  • event:datasetID: to complete
  • event:parentLocality0: remove
  • event:recordedBy: remove
  • occ:behaviour: rename

Dataset names

Current title  New title
Meetnetten.be - Chorus counts for Amphibia in Flanders, Belgium Meetnetten.be - Chorus counts for priority Amphibia in Flanders, Belgium
Meetnetten.be - Egg counts for butterflies in Flanders, Belgium Meetnetten.be - Egg counts for priority butterflies in Flanders, Belgium
Meetnetten.be - Exuviae counts for dragonflies in Flanders, Belgium Meetnetten.be - Exuviae counts for priority dragonflies in Flanders, Belgium
Meetnetten.be - Fyke counts for Amphibia in Flanders, Belgium Meetnetten.be - Fyke counts for priority Amphibia in Flanders, Belgium
Meetnetten.be - Larvae and metamorph counts for Amphibia in Flanders, Belgium Meetnetten.be - Larvae and metamorph counts for priority Amphibia in Flanders, Belgium
Meetnetten.be - Population counts for dragonflies in Flanders, Belgium Meetnetten.be - Population counts for priority dragonflies in Flanders, Belgium
Meetnetten.be - Sightings for Natterjack toad in Flanders, Belgium Meetnetten.be - Sightings for Natterjack toad in Flanders, Belgium
Meetnetten.be - Site counts for butterflies in Flanders, Belgium Meetnetten.be - Site counts for priority butterflies in Flanders, Belgium
Meetnetten.be - Transects for butterflies in Flanders, Belgium Meetnetten.be - Transects for priority butterflies in Flanders, Belgium
Meetnetten.be - Transects for dragonflies in Flanders, Belgium Meetnetten.be - Transects for priority dragonflies in Flanders, Belgium
Meetnetten.be - Transects for fire salamanders in Flanders, Belgium Meetnetten.be - Transects for fire salamanders in Flanders, Belgium
Transects for non-target butterfly species in Flanders, Belgium Transects for butterflies in Flanders, Belgium

Whip dwca-meetnetten-vaatplanten-oppervlakte-occurrences-v1.5

  • event.coordinateUncertaintyInMeters is correctly 707 for "coordinates are generalized from a POLYGON to a UTM 1km grid", but 68 records have 3536. To be corrected
  • Some occurrence have an occ.eventID that cannot be found in events
INBO:MEETNET:EVENT:193850
INBO:MEETNET:EVENT:193847
INBO:MEETNET:EVENT:193848
INBO:MEETNET:EVENT:000973

Lifestage for libellen transects

adult

ok

imago

ok

imago (not fully colored)

168 rows
I would just make these "imago"

exuvium

152 rows

This is not a lifeStage. In the waarnemingen.be datasets, we indicate this info in occurrenceRemarks as found as exuviae. I would do the same here.

Not sure if we can say something about the lifeStage for an exuviae. Dragonflies moult multiple times?

Review of meetnetten-butterflies-egg-occurrences

Event

  • coordinateUncertaintyInMeters: 3536 is the data generalized to UTM? If so, we should have:
    • verbatimCoordinates: 31UFS18B and alike
    • verbatimCoordinateSystem: UTM 5km
    • verbatimSRS: WGS84
    • georeferenceRemarks: coordinates are centroid of used grid square
    • dataGeneralizations: coordinates are generalized to a 5x5km UTM grid`
  • informationWithheld: High resolution data available on request => it might be better to write something in dataGeneralizations instead, see remark above
  • parentEventID: INBO:MEETNET:VISITID :000487: drop extra space
  • parentEventID: can we have those even if they are not present in the dataset?
  • samplingProtocol: butterflies egg counts -> egg count
  • eventDate: why do times have decimals 2016-08-16 10:00:00.0?
  • eventDate: what is the time zone of the dates?
  • locationID: why the long padding with 0: INBO:MEETNET:LOCATION:0000017359?
  • georeferenceRemarks: original geomerty is a: POLYGON typo and I'm not sure I understand this, see first remark

Occurrence

  • occurrenceRemarks: data collected in the Gentiaanblauwtje monitoring scheme = very similar to eventRemarks: data collected in the Gentiaanblauwtje project. I would only keep eventRemarks
  • recordedBy: Meetnetten. This is a bit odd. If you want to use something generic, then I would use https://meetnetten.be
  • samplingProtocol: Vlinders - Eitellingen this is going to conflict with the samplingProtocol from Events. Drop field.
  • class: this field is empty (expecting Insecta). Is class a reserved name?
  • order: add this field with Lepidoptera?
  • scientificNameAuthorship: maybe add this field?
  • taxonRank: add this field with species

Occurrences and measurements without event in site count

The following eventIDs occur in both the occurrence and measurement extension, but do not exist in the event core:

INBO:MEETNET:EVENT:189369
INBO:MEETNET:EVENT:189505
INBO:MEETNET:EVENT:189949
INBO:MEETNET:EVENT:189997
INBO:MEETNET:EVENT:190418
INBO:MEETNET:EVENT:190439
INBO:MEETNET:EVENT:190440
INBO:MEETNET:EVENT:194065
INBO:MEETNET:EVENT:194078
INBO:MEETNET:EVENT:194085
INBO:MEETNET:EVENT:194086
INBO:MEETNET:EVENT:194087
INBO:MEETNET:EVENT:194088
INBO:MEETNET:EVENT:195861
INBO:MEETNET:EVENT:195863
INBO:MEETNET:EVENT:195947

Review of ABV post 2016

Event

  • datasetName: "Meetnetten - Algemene broedvogelmonitoring, Belgium, post 2016"
      1. I don't think the title should be in Dutch
      1. We should somehow link it with the old dataset: ABV - Common breeding birds in Flanders, Belgium
      1. The scope is probably Flanders, not Belgium?
      1. The post part should be in parenthesis cf. other datasets.
    • So maybe: ABV - Common breeding birds in Flanders, Belgium (post 2016)? Especially since you also used ABV as collection code rather than meetnetten
  • informationWithheld is typically "original locations available upon request", why not here?
    information is not withheld here, similar as original ABV
  • dataGeneralizations is typically "coordinates are generalized from a ...", why not here?
    information is not withheld here, similar as original ABV
  • eventID: is of format INBO:MEETNET:AB:EVENT:005450 rather than INBO:MEETNET:EVENT:005450. I would keep it consistent with other datasets.
  • samplingEffort: typo, correct to 6 times 5 minutes count per UTM 1km
  • eventDate is currently of format 2017-03-01 07:30:00.0. Since it is only precise to the minute, it should have format 2017-03-01T07:30 (notice the T as well) yyyy-mm-dd now
  • locationID is of format INBO:MEETNET:LOCATION:0000006265 rather than INBO:MEETNET:LOCATION:006265 (6 digits). I would keep it consistent with other datasets.
  • What is the difference between locality (Dijleland) and verbatimLocality (ABV_FS2132)? It seems the verbatim one is the precise one that is always populated), while locality does not provide much. I would keep the values of verbatimLocality, but move it to locality.
    verbatimLocality is the ABV_CODE_UTM_CODE , locality is the region name It's given when provided in the database. As abv is in the database these are not verbatimCoordinates

Occurrence

  • taxonRank: missing field, good to add it
  • class: has AVES rather than Aves
  • eventID: same format issue
  • occurrenceID has format INBO:MEETNETTEN:AB:OCC:0013078 rather than INBO:MEETNETTEN:OCC:0013078. I would keep it consistent with other datasets.
  • recordedBy is meetnetten while other datasets use https://meetnetten.be
  • taxonID: is this a new field? If these are EURING codes, I would express them as euring:id, cf. watervogels + broedvogelatlas: https://github.com/inbo/data-publication/search?q=euring

Uncontrolled measurementorfact values in transect dataset

Some values in the measurementorfact extension of the transect dataset are not according the controlled values. I've also indicate if we "should" or "could" fix it

  • cloudiness (should: 244 rows): partially clouded -> partly cloudy (3/8 - 5/8)
  • cloudiness (should: 174 rows): unclouded -> ?
  • cloudiness (should: 60 rows): half clouded -> ?
  • cloudiness (should: 20 rows): heavily clouded -> ?
  • wind force (should: 641 rows): windstil (0 Bft) ) -> 0
  • temperature (should: 1893 rows): unknown: I think it is better to use an empty value here, as it allows number as datatype + easier checking of values. People will still know it it unknown, because a meas record is available
  • temperature (could): almost all values use xx.x (one decimal), but some don't

Coordinates not in centre of UTM

I noticed coordinates for 25+32 meetnetten-amfibieen-larven-metamorfen-occurrences being far West (2.50 rather than the maximum 2.53). Plotting it in QGIS, I don't understand how the coordinates are obtained. They don't seem to be the centroid of the UTM10 grid:

Screenshot 2020-07-31 at 15 26 39

Review of 4 meetnetten-vuursalamander-transect-occurrences

event

  • datasetName: should be Meetnetten.be - Transects for fire salamanders in Flanders, Belgium (currently lowercase in singular salamander)

occ

  • lifeStage: all values are post-metamorf (what is the English translation for this? adult?) -> post-metamorph
  • sex: all values are unknown: if it is never recorded, than we can drop this field (is already the case in other datasets) nofix

Review of 25 meetnetten-amfibieen-larven-occurrences

IPT dataset: meetnetten-amfibieen-larven-occurrences

event

  • datasetName: Update title (see #9)
  • samplingProtocol: use "larvae" (see #9)

occ

  • lifeStage: lowercase larva. Are there also metamorph indicated as such? If not, do we really need it in the dataset title?

mof

  • ph valueph
  • present fishfish present (first one implies a list of fish, second works better for boolean)
  • surface pondpond surface
  • waterqualitywater quality
  • maximum depth: would change one value so it is a range: <0.5, 0.5-1, 1-1.5 (rather than >1), >1.5
  • ph value: has unknown, but I would use an empty value instead, as it is a number field (cf. number of sweeps)?
  • waterquality: map slecht (verwaarloosde poel me...) to bad

Change amphibia titles

To be consistent with:

Meetnetten.be - Egg counts for butterflies in Flanders, Belgium
Meetnetten.be - Transects for butterflies in Flanders, Belgium

We should change the titles for:

  • Meetnetten.be - Amphibia in fykes in Flanders, Belgium → Meetnetten.be - Fyke counts for Amphibia in Flanders, Belgium
  • Meetnetten.be - Amphibia larval & metamorph count in Flanders, Belgium → Meetnetten.be - Larvae and metamorph counts for Amphibia in Flanders, Belgium
  • Meetnetten.be - Amphibia choir count in Flanders, Belgium → Meetnetten.be - Choir counts for Amphibia in Flanders, Belgium

Review of meetnetten-butterflies-occurrences

Event core

  • datasetID: use full URL https://doi.org/10.15468/kfhvy4
  • datasetName: Meetnetten.be - Transects for butterflies in Flanders, Belgium: currently missing the -
  • datasetName: additional question. Should we use "Transect counts for butterflies"? No

occurrence core

  • occurrenceStatus: add this field
  • taxonRank: technically, Pieris spec. should have rank genus?

MOF extension

  • measurementType: wind force instead of wind-force
  • measurementValue (wind force): 4 has a )
  • measurementValue: I would write cloudiness values as in #2 (comment)

Whip butterflies-algemene-occurrences

Event

  • Many occ records link to an eventID that does not exist, e.g. INBO:MEETNET:EVENT:213498
  • datasetName was updated: [Butterfly monitoring scheme in Flanders, Belgium]

Occ

  • Remove samplingProtocol: it is part of the event core

MOF

  • Are there exactly 1000 mof records for this dataset?
  • There are records with empty cloudiness (not allowed in previous datasets): removed
  • There are records with empty wind force (not allowed in previous datasets): removed
  • Use empty temperature rather than unknown (it is otherwise not possible to validate numeric values): removed

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.