jingli201802 / taxonomic Goto Github PK

Make it could show the updated outputs which mapping to the schema.
Fix PDF parts.

draft uploaded on google drive.

Totally 4 sheets.
TNC_TaxonomicName is almost finished. To do:

• The fullNameWithAuthorship needs to import article author (if not mention taxonomic author) from Tarasom functions.
• Find subspecies, common names.

TNC_TaxonomicNameUsage and TNC_BibliographicReference are in process..

Instead of describing whether the name is scientific or common, it should instead describe whether the name is an instance of sp. nov., comb. nov, gen. nov. etc.

Currently typification.csv records too much information, accidentally capturing some type descriptions. Each type should be described in around 1 sentence and this sentence should contain gender, location and identifier.

The client has specified an output format which uses four classes. In order to fit this format the PDF output needs to be in the form of four CSVs.

Definition of done:
Executing the PDF extraction code will produce a folder with four CSV's- each CSV containing the list of instances of a particular class in the TNU schema.

scr includes more docs; configuration ...

From the response generated by citethisforme, grab the appropriate fields from the page source and translate these into a dictionary output which can be used to populate bibliography.csv

Currently our name detection is not reliable enough to use in the final release.

By aggregating our extraction result with the result of an existing web application (https://github.com/GlobalNamesArchitecture/gnrd) we will be able to obtain a more accurate and reliable list of scientific names.

use flask with ui, pdf input, xml input. output accordingly.

Deal with instances of comb. nov. as detailed by Haylee,

see handover issus in google doc
https://docs.google.com/document/d/19cB0DGzzpEuWftartd-TqGzBXE9oQOSB/edit#

getting contact with shadow team to see how they build server on ANU. If not applicable, research other server platforms (Herok)

Making imporvements on accuracy

sending email to clients asking for specific requirements of input and output preview, if needed.

Current TNU ID's are generated in the TNU csv, this is incorrect, they should be generated in TaxonomicName.csv and then passed on to TNU csv and typification.csv respectively

editable output, output display

If either citethisforme or gnparser cannot be accessed, it is important that the client can be made aware of this:

A) So they can tell that the issue is not an error with our program
B) If the issue persists they can consult our documentation to understand the nature of the problem and possible solutions. (It is possible that instead of replacing the webservice our interface with that service might just have to be adapted to changes they have made.)

update risk register

Access citethisforme's free auto-citation page and request a citation of the article in question's DOI

When multiple entries for the same name are recorded, delete the one which contains less information.

Haylee has requested a another column in the typification csv which contains the coordinates of the type.

Currently even though the authorship value in GNParser's JSON output is always a list, only the first value is used (usually there is only one value).

Definition of done:
The authorship list in GNParser's output is concatenated into a single string and this string is inserted into the final CSV output of the program.

Name detection has two significant ways it can be improved

• By accounting for punctuation in the name which should tell the program certain words don't belong. Eg. capital letters where there shouldn't be, unclosed brackets and misplaced full stops.

• By checking if GNParser returns an "unparsed tail" string which contains words adjacent to the sp./gen./comb. Because these words can be assumed to be in the name, this error means that too many words are included in front of the name (so remove the front word and try again)
  Since GNParser is a webservice it is important to do these checks in rounds, instead of 1 by 1 as that would increase the latency immensely and may also cause gnparser to flag the ip of the program's machine as malicious.

Include coordinates which are described using decimals instead of DMS, and include more room for variation of spacing, punctuation etc. in the regex.

The output need to have the preview output function.

Currently PYPDF2 (the library used to convert PDFs to strings) adds many unnecessary and unpredictable line breaks which make it difficult to parse important information (eg. references)

Possible solutions include:

• Replacing PYPDF2 with more a more suitable tool
• Attempting to correct the line breaks manually

Finishing codes.
Start real test.