terraref / brapi Goto Github PK

It looks like, although the observationunits endpoint itself no longer returns observation values, the observationUnitDbId query parameter (e.g., https://brapi.workbench.terraref.org/brapi/v1/observationunits?observationUnitDbId=MAC Field Scanner Season 4 Range 48 Column 11 E) does return them.

I'm hoping it's possible to get the studyDbId (e.g., https://brapi.workbench.terraref.org/brapi/v1/observationunits?studyDbId=6000000034) and observationVariableDbId (https://brapi.workbench.terraref.org/brapi/v1/observationunits?observationVariableDbId=stand_count) query parameters to be able to return observation values also? This is because we're looking to get values for only certain variables and for particular studies.

I don't have much experience with APIs and their development, but it seems like maybe a query that's similar to the one set up for observationUnitDbId (lines 45-81 of observationunits script) could be done for those other two parameters?

The /crops endpoint has been deprecated, and replaced by /commoncropnames

See https://brapi.docs.apiary.io/#reference/crops

per plantbreeding/BrAPI#274 endpoint names are lowercase

observationunits does not provide accurate count for total pages. pagination does not work since no matter how many results exist in the db, the total page count is always 1.

These are the responseswithin observationvariables

• traits: unique variables.name where variables.id in traits.variable_id
• methods: unique methods.name where method.id in traits.method_id
• scales: unique variables.units where variables.id in traits.variable_id
  • "scaleName" : "Centimeter"
  • (and?/) or including units explicitly as proposed in: plantbreeding/BrAPI#441
• ontologies: (optional? there is a dictionary of ontology term mappings in contrib/variables_annotations.csv)
• Here is an example of a valid endpoint for genomes2fields data https://imagebreed.org/brapi/v1/studies/241/observationvariables

note:

I would like to query all data of a particular variable and filter by a particular study

Specifically, this query currently returns an error:
https://brapi.workbench.terraref.org/brapi/v1/observationunits?studyDbId=6000000010&observationVariableDbId=6000000196

But I get this:

{
  "detail": "The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.",
  "status": 500,
  "title": "Internal Server Error",
  "type": "about:blank"
}

But I would expect it to return something similar to https://brapi.workbench.terraref.org/brapi/v1/observationunits?observationUnitDbId=MAC Field Scanner Season 4 Range 13 Column 10&observationVariableDbId=6000000196

Suspected problem

It appears that this might be caused by the lines

https://github.com/terraref/brapi/blob/master/api/observationunits.py#L45

What was the rationale for this (@max-zilla ?) Perhaps to avoid returning too much data? When we look for a particular variable, this is less of an issue.

I was just trying to download our traits via brapi … the observationsunits endpoint, e.g. https://terraref.org/brapi/v1/observationunits?studyDbId=6000000010, is very different from the example in the api documentation https://terraref.org/brapi/v1/ui/#/ , e.g. it doesnt have the observation elements etc. I think these used to be there … do you know what happened or am I hitting the wrong endpoint?

e.g. this is the example from the documentation:

"result": {
    "data": [
      {
        "X": "1",
        "Y": "1",
        "blockNumber": "1",
        "entryNumber": "1",
        "entryType": "TEST",
        "germplasmDbId": "1",
        "germplasmName": "Name001",
        "observationLevel": "plot",
        "observationLevels": "block:1;plot:1",
        "observationUnitDbId": "1",
        "observationUnitName": "Plot 1",
        "observationUnitXref": [
          {
            "id": "SAMEA179865230",
            "source": "ebi.biosample"
          },
          {
            "id": "INRA:CoeSt6 _SMH03",
            "source": "gnpis.lot"
          },
          {
            "id": "239865",
            "source": "kernelDB"
          }
        ],
        "observations": [
          {
            "collector": "A. Technician",
            "observationDbId": "1",
            "observationTimeStamp": "2013-06-15T02:03:51Z",
            "observationVariableDbId": "MO_123:100002",
            "observationVariableName": "Plant height",
            "season": "fall 2011",
            "value": "1.2"
          },
          {

and this is the result that I am getting:

 "result": {
    "data": [
      {
        "observationUnitName": "Earth",
        "seasonDbId": "a8317fa6b59bfa8cfe29004423f5613b",
        "studyDbId": "6000000010",
        "treatmentDbId": 6000000020,
        "treatments": [
          {
            "factor": "MAC Sorghum Season 4",
            "modality": "Pre-plant Nitrogen fertilizer incorporated in the field"
          }
        ]
      },
      {
        "observationUnitName": "MAC Field Scanner Season 4 Range 51 Column 7 E",
        "seasonDbId": "a8317fa6b59bfa8cfe29004423f5613b",
        "studyDbId": "6000000010",
        "treatmentDbId": 6000000020,
        "treatments": [
          {
            "factor": "MAC Sorghum Season 4",
            "modality": "Pre-plant Nitrogen fertilizer incorporated in the field"
          }
        ]
      },

cc @KristinaRiemer @robkooper @Chris-Schnaufer @max-zilla

Create a preliminary implementation of the phenotypes search, document any issues related to mapping BETY concepts to BRAPI in preparation for September hackathon.

curl --insecure -o observations_plot1311.json 'https://brapi.workbench.terraref.org/brapi/v1/observationunits?observationVariableDbId=6000000196&observationUnitDbId=MAC%20Field%20Scanner%20Season%204%20Range%2013%20Column%2011'

Appears to return duplicate records (same value, same observationDbId):

       "observations": [
          {
            "observationDbId": "6001965924",
            "observationTimeStamp": "2017-09-15T05:00:00Z",
            "observationVariableDbId": "6000000196",
            "observationVariableName": "aboveground_dry_biomass",
            "operator": "Newcomb, Maria",
            "quality": "unchecked",
            "replicate": 6000031146,
            "value": "25900"
          },
          {
            "observationDbId": "6001965924",
            "observationTimeStamp": "2017-09-15T05:00:00Z",
            "observationVariableDbId": "6000000196",
            "observationVariableName": "aboveground_dry_biomass",
            "operator": "Newcomb, Maria",
            "quality": "unchecked",
            "replicate": 6000031146,
            "value": "25900"
          }

Need to correct this

• are you able to access and understand our germplasm responses?

• is the information accurate and useful (i.e. can you check out the cross refs and additional meta-data that we looked up? e.g. the taxon ids and references to genysis?

• Is there enough metadata or additional metadata that we should provide?

• any other suggestions and feedback welcome!

https://terraref.org/brapi/v1/germplasm
https://terraref.org/brapi/v1/studies/6000000034/germplasm

@ajo2995

Add information to https://github.com/plantbreeding/API/blob/master/brapi-resources.json in format

      {
        "name": "TERRA REF",
        "description": "
        "logo": "*.png",
        "resource": [
          {
            "name": "????",
            "base-url": "",
            "logo": "",
            "description": ""
          }
        ]
},

The BRAPI germplasm calls support a number of attributes that are not currently in BETY, but are in a Gist (https://gist.github.com/dlebauer/6b7b0e181cc5ae5034b992f725712ba4). There is a proposal to add a generic table in BETY to support extended attributes, that we'd like to consider as an implementation.

An alternative is to create a separate genotypes table.

The task is to define how to handle these extended attributes in BETY and decide whether to use the above proposed model. If so, implement it.

• reduce the openapi.yml to only provide the endpoints that we support
• regenerate stubs
• update README.md
  • document how to add new endpoints from the yml
• get the API working again

1. test against http://webapps.ipk-gatersleben.de/brapivalidator/
2. set up continuous integration to
   • ensure that endpoints are complaint
   • require PRs to pass compliance (and other!) tests

the files contrib/germplasm/season_1_lines.csv has pedigree information. Need to figure out how this information can be stored in the BrAPI germplasm model and then add it to germplasm.json.

Can also consider putting some of the categorical phenotypes (e.g. color, photoperiod sensitivity) into an appropriate endpoint; perhaps easiest would be in traits, except that traits isn't designed to store categorical data and therefore perhaps could go in a separate observations.json file (or attributes table in BETYdb).

Common name information in commercial_hybrids can also used to replace the commonname field in germplasm.json for these lines (currently this is the same as the accession name).

1. replace /phenotypes-search with /phenotypes

• /phenotypes-search is v1.2 and /phenotypes v1.3
• could maintain both endpoints

2. Changes to make

• rename treatment_definition --> observationtreatment
• rename treatment_factor --> season

pass compressed responses, a standard approach is using gzip

this should not be difficult, e.g. see https://stackoverflow.com/a/62331275/199217

The traits table has a quality flag.

• any data marked -1 should not be returned
• data marked "0" should indicate that it has not been checked, perhaps "quality":"unchecked"
• data marked "1" should indicate that it has been reviewed and approved perhaps "quality":"unchecked"

The values should follow a standard vocabulary ... perhaps ISO 8000 has some clues?

Should / could plot info, e.g. from https://brapi.workbench.terraref.org/brapi/v1/studies/6000000034/layouts, contain geojson (?)

Or could update locations: https://github.com/terraref/brapi/blob/master/api/locations.py based on v2.x https://app.swaggerhub.com/apis/PlantBreedingAPI/BrAPI-Core/2.0#/Location

• do not return eventParameters if they are null or empty string
• convert planting (plants / m2) --> planting w/ level and units
• convert fertilizer_X --> fertilization + type = 'X' (where X might be, e.g., N
• Only return observationunits for events that were not applied to the entire field (e.g. those that are related to a specific treatment)

plantbreeding/BrAPI#364 Provides required information for ICASA / AgMIP

Although not currently in the spec, we should go ahead and implement b/c this is the appropriate way to export our managements information

From @robkooper:

• no need to generate events.py, just create, magic happens to link it from the openapi spec to the actual implementation. Each path in the yaml file is a file in the api folder, so in your case for path: events you create a file called api/events.py. In this file you can implement the functions:

• def search(parameters) : the GET operator on /brapi/v1/events?arg1
• def get(eventId) : the GET operator on /brapi/v1/events/{eventId}
• def post(parameters) : POST
• if you don't implement a function the default is for it to return a 501 error

• breeding program
  • one or more trials
    • one or more studies per trial, each study corresponds to a single location

• Program A Program can contain multiple Trials. A Trial can contain multiple Studies.
• Trial Corresponds to the "investigation" concept in MIAPPE (Minimal Information about a Plant Phenotyping Experiment).
• Study a phenotyping experiment conducted at a single geographic location. One Trial can have multiple studies conducted (e.g. multi location international trials).

What breeding programs do we have?

1. Sorghum Biomass Association Panel
2. Durum Wheat
3. Sorghum recombinant inbred lines

Here are the responses for one of our sets of germplasm -

germplasm.txt

How should I get this into the database?

I've also generated a set of insert statements that look like:

"insert into attributes (table,  table_id, json), values (6000000231'cultivars'{\"accessionNumber\":[\"PI564163\"],\"acquisitionDate\":[\"1977----\"],\"biologicalStatusOfAccessionCode\":[500],\"breedingMethodDbId\":[\"\"],\"commonCropName\":[\"sorghum\"],\"defaultDisplayName\":[\"PI564163\"],\"donors\":{\"donorAccessionNumber\":[\"\"],\"donorInstituteCode\":[\"\"],\"germplasmPUI\":[\"\"]},\"genus\":[\"Sorghum\"],\"germplasmDbId\":[\"0ba28636-e634-428a-aa58-4346a20de326\"],\"germplasmName\":[\"PI564163\"],\"germplasmPUI\":[\"https://purl.org/germplasm/id/0ba28636-e634-428a-aa58-4346a20de326\"],\"instituteCode\":[\"USA016\"],\"instituteName\":[\"\"],\"pedigree\":{},\"seedSource\":{},\"species\":[\"bicolor\"],\"speciesAuthority\":[\"(L.) Moench\"],\"subtaxa\":[\"subsp. bicolor\"],\"subtaxaAuthority\":[\"(L.) Moench\"],\"synonyms\":[\"BTx623\",\"BTx623\",\"SAP-32\",\"SAP-32\"],\"taxonIds\":[{\"sourceName\":[\"ncbiTaxon\"],\"taxonId\":[\"http://purl.obolibrary.org/obo/NCBITaxon_4558\"]},{\"sourceName\":[\"USDA Plants\"],\"taxonId\":[\"https://plants.usda.gov/core/profile?symbol=SOBIB\"]}],\"typeOfGermplasmStorageCode\":[10],\"xref\":[\"PI564163\",\"https://terraref.ncsa.illinois.edu/bety/cultivars/6000000231\"]});"

Automate #17

add logging to brapi. brapi currently does not have logging

As requested by ETL in this PR elixir-europe/plant-brapi-etl-faidare#22

We should add docs.terraref.org as the documentationURL for each study.

Currently the brapi observationunits returns a field 'location_abbrevation' which should be 'observationUnitName'. The field 'observationLevel' also needs to be added, with plot for plot and if the plot ends with E or W, subplot.