terraref / brapi Goto Github PK
View Code? Open in Web Editor NEWBreeder's API implementation for TERRA-REF
Home Page: https://terraref.org/brapi/v1/calls
License: BSD 3-Clause "New" or "Revised" License
Breeder's API implementation for TERRA-REF
Home Page: https://terraref.org/brapi/v1/calls
License: BSD 3-Clause "New" or "Revised" License
It looks like, although the observationunits endpoint itself no longer returns observation values, the observationUnitDbId query parameter (e.g., https://brapi.workbench.terraref.org/brapi/v1/observationunits?observationUnitDbId=MAC Field Scanner Season 4 Range 48 Column 11 E) does return them.
I'm hoping it's possible to get the studyDbId (e.g., https://brapi.workbench.terraref.org/brapi/v1/observationunits?studyDbId=6000000034) and observationVariableDbId (https://brapi.workbench.terraref.org/brapi/v1/observationunits?observationVariableDbId=stand_count) query parameters to be able to return observation values also? This is because we're looking to get values for only certain variables and for particular studies.
I don't have much experience with APIs and their development, but it seems like maybe a query that's similar to the one set up for observationUnitDbId (lines 45-81 of observationunits script) could be done for those other two parameters?
The /crops endpoint has been deprecated, and replaced by /commoncropnames
See https://brapi.docs.apiary.io/#reference/crops
per plantbreeding/BrAPI#274 endpoint names are lowercase
observationunits does not provide accurate count for total pages. pagination does not work since no matter how many results exist in the db, the total page count is always 1.
These are the responseswithin observationvariables
note:
BrAPI Term | BETYdb term | notes |
---|---|---|
traitName | variables.name | |
ObservationVariable | variable+method+units | in BrAPI a variable is defined as variable+method+scale |
ObservationUnit | sites | these are individual plots |
observationDbId | traits | these are individual observations |
method | method | yay something matches |
scaleName | units | |
traitClass | variables.type |
I would like to query all data of a particular variable and filter by a particular study
Specifically, this query currently returns an error:
https://brapi.workbench.terraref.org/brapi/v1/observationunits?studyDbId=6000000010&observationVariableDbId=6000000196
But I get this:
{
"detail": "The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.",
"status": 500,
"title": "Internal Server Error",
"type": "about:blank"
}
But I would expect it to return something similar to https://brapi.workbench.terraref.org/brapi/v1/observationunits?observationUnitDbId=MAC Field Scanner Season 4 Range 13 Column 10&observationVariableDbId=6000000196
It appears that this might be caused by the lines
https://github.com/terraref/brapi/blob/master/api/observationunits.py#L45
What was the rationale for this (@max-zilla ?) Perhaps to avoid returning too much data? When we look for a particular variable, this is less of an issue.
I was just trying to download our traits via brapi … the observationsunits endpoint, e.g. https://terraref.org/brapi/v1/observationunits?studyDbId=6000000010, is very different from the example in the api documentation https://terraref.org/brapi/v1/ui/#/ , e.g. it doesnt have the observation elements etc. I think these used to be there … do you know what happened or am I hitting the wrong endpoint?
e.g. this is the example from the documentation:
"result": {
"data": [
{
"X": "1",
"Y": "1",
"blockNumber": "1",
"entryNumber": "1",
"entryType": "TEST",
"germplasmDbId": "1",
"germplasmName": "Name001",
"observationLevel": "plot",
"observationLevels": "block:1;plot:1",
"observationUnitDbId": "1",
"observationUnitName": "Plot 1",
"observationUnitXref": [
{
"id": "SAMEA179865230",
"source": "ebi.biosample"
},
{
"id": "INRA:CoeSt6 _SMH03",
"source": "gnpis.lot"
},
{
"id": "239865",
"source": "kernelDB"
}
],
"observations": [
{
"collector": "A. Technician",
"observationDbId": "1",
"observationTimeStamp": "2013-06-15T02:03:51Z",
"observationVariableDbId": "MO_123:100002",
"observationVariableName": "Plant height",
"season": "fall 2011",
"value": "1.2"
},
{
and this is the result that I am getting:
"result": {
"data": [
{
"observationUnitName": "Earth",
"seasonDbId": "a8317fa6b59bfa8cfe29004423f5613b",
"studyDbId": "6000000010",
"treatmentDbId": 6000000020,
"treatments": [
{
"factor": "MAC Sorghum Season 4",
"modality": "Pre-plant Nitrogen fertilizer incorporated in the field"
}
]
},
{
"observationUnitName": "MAC Field Scanner Season 4 Range 51 Column 7 E",
"seasonDbId": "a8317fa6b59bfa8cfe29004423f5613b",
"studyDbId": "6000000010",
"treatmentDbId": 6000000020,
"treatments": [
{
"factor": "MAC Sorghum Season 4",
"modality": "Pre-plant Nitrogen fertilizer incorporated in the field"
}
]
},
Create a preliminary implementation of the phenotypes search, document any issues related to mapping BETY concepts to BRAPI in preparation for September hackathon.
curl --insecure -o observations_plot1311.json 'https://brapi.workbench.terraref.org/brapi/v1/observationunits?observationVariableDbId=6000000196&observationUnitDbId=MAC%20Field%20Scanner%20Season%204%20Range%2013%20Column%2011'
Appears to return duplicate records (same value, same observationDbId):
"observations": [
{
"observationDbId": "6001965924",
"observationTimeStamp": "2017-09-15T05:00:00Z",
"observationVariableDbId": "6000000196",
"observationVariableName": "aboveground_dry_biomass",
"operator": "Newcomb, Maria",
"quality": "unchecked",
"replicate": 6000031146,
"value": "25900"
},
{
"observationDbId": "6001965924",
"observationTimeStamp": "2017-09-15T05:00:00Z",
"observationVariableDbId": "6000000196",
"observationVariableName": "aboveground_dry_biomass",
"operator": "Newcomb, Maria",
"quality": "unchecked",
"replicate": 6000031146,
"value": "25900"
}
Need to correct this
are you able to access and understand our germplasm responses?
is the information accurate and useful (i.e. can you check out the cross refs and additional meta-data that we looked up? e.g. the taxon ids and references to genysis?
Is there enough metadata or additional metadata that we should provide?
any other suggestions and feedback welcome!
https://terraref.org/brapi/v1/germplasm
https://terraref.org/brapi/v1/studies/6000000034/germplasm
Add information to https://github.com/plantbreeding/API/blob/master/brapi-resources.json in format
{
"name": "TERRA REF",
"description": "
"logo": "*.png",
"resource": [
{
"name": "????",
"base-url": "",
"logo": "",
"description": ""
}
]
},
The BRAPI germplasm calls support a number of attributes that are not currently in BETY, but are in a Gist (https://gist.github.com/dlebauer/6b7b0e181cc5ae5034b992f725712ba4). There is a proposal to add a generic table in BETY to support extended attributes, that we'd like to consider as an implementation.
An alternative is to create a separate genotypes table.
The task is to define how to handle these extended attributes in BETY and decide whether to use the above proposed model. If so, implement it.
the files contrib/germplasm/season_1_lines.csv has pedigree information. Need to figure out how this information can be stored in the BrAPI germplasm model and then add it to germplasm.json.
Can also consider putting some of the categorical phenotypes (e.g. color, photoperiod sensitivity) into an appropriate endpoint; perhaps easiest would be in traits, except that traits isn't designed to store categorical data and therefore perhaps could go in a separate observations.json file (or attributes table in BETYdb).
Common name information in commercial_hybrids can also used to replace the commonname field in germplasm.json for these lines (currently this is the same as the accession name).
pass compressed responses, a standard approach is using gzip
this should not be difficult, e.g. see https://stackoverflow.com/a/62331275/199217
The traits table has a quality flag.
"quality":"unchecked"
"quality":"unchecked"
The values should follow a standard vocabulary ... perhaps ISO 8000 has some clues?
Should / could plot info, e.g. from https://brapi.workbench.terraref.org/brapi/v1/studies/6000000034/layouts, contain geojson (?)
Or could update locations: https://github.com/terraref/brapi/blob/master/api/locations.py based on v2.x https://app.swaggerhub.com/apis/PlantBreedingAPI/BrAPI-Core/2.0#/Location
planting (plants / m2)
--> planting w/ level and unitsfertilizer_X
--> fertilization
+ type = 'X' (where X might be, e.g., Nplantbreeding/BrAPI#364 Provides required information for ICASA / AgMIP
Although not currently in the spec, we should go ahead and implement b/c this is the appropriate way to export our managements information
From @robkooper:
What breeding programs do we have?
Here are the responses for one of our sets of germplasm -
How should I get this into the database?
I've also generated a set of insert statements that look like:
"insert into attributes (table, table_id, json), values (6000000231'cultivars'{\"accessionNumber\":[\"PI564163\"],\"acquisitionDate\":[\"1977----\"],\"biologicalStatusOfAccessionCode\":[500],\"breedingMethodDbId\":[\"\"],\"commonCropName\":[\"sorghum\"],\"defaultDisplayName\":[\"PI564163\"],\"donors\":{\"donorAccessionNumber\":[\"\"],\"donorInstituteCode\":[\"\"],\"germplasmPUI\":[\"\"]},\"genus\":[\"Sorghum\"],\"germplasmDbId\":[\"0ba28636-e634-428a-aa58-4346a20de326\"],\"germplasmName\":[\"PI564163\"],\"germplasmPUI\":[\"https://purl.org/germplasm/id/0ba28636-e634-428a-aa58-4346a20de326\"],\"instituteCode\":[\"USA016\"],\"instituteName\":[\"\"],\"pedigree\":{},\"seedSource\":{},\"species\":[\"bicolor\"],\"speciesAuthority\":[\"(L.) Moench\"],\"subtaxa\":[\"subsp. bicolor\"],\"subtaxaAuthority\":[\"(L.) Moench\"],\"synonyms\":[\"BTx623\",\"BTx623\",\"SAP-32\",\"SAP-32\"],\"taxonIds\":[{\"sourceName\":[\"ncbiTaxon\"],\"taxonId\":[\"http://purl.obolibrary.org/obo/NCBITaxon_4558\"]},{\"sourceName\":[\"USDA Plants\"],\"taxonId\":[\"https://plants.usda.gov/core/profile?symbol=SOBIB\"]}],\"typeOfGermplasmStorageCode\":[10],\"xref\":[\"PI564163\",\"https://terraref.ncsa.illinois.edu/bety/cultivars/6000000231\"]});"
Automate #17
add logging to brapi. brapi currently does not have logging
As requested by ETL in this PR elixir-europe/plant-brapi-etl-faidare#22
We should add docs.terraref.org as the documentationURL for each study.
Currently the brapi observationunits returns a field 'location_abbrevation' which should be 'observationUnitName'. The field 'observationLevel' also needs to be added, with plot for plot and if the plot ends with E or W, subplot.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.