investigate http://opencitations.net/ corpus for citation lookups when deduping or filling out sparse citation metadata

right now there is no easy way to change author order in a publication

https://github.com/sdispater/poetry

Currently date published information is stored in the database as text and parsed at run time to use in elasticsearch indices as dates. We cannot store the data as a date field because quite often month and day information is absent. With date information across three fields it will be much simpler to filter by publication year from the database.

log the orphans that were deleted and counts

A few guidelines:

1. avoid issuing Platform and Sponsor all queries for every publication
2. couple header + row values better - currently they are in two distinct data structures that can very easily get out of sync and are difficult to maintain
3. consider creating some kind of stateful class & improve the API between the header / row value generation
4. use values_list(flat=True) instead of generating a tuple that you index
5. replace get_attribute_values and generate_boolean_list with list comprehensions

https://developer.clarivate.com/apis

Possible Use Cases:

• Importing user requested publications
• Importing keyword searches

store the entire object in the audit log before any destructive operation

• perform diffs between objects
• undo / redo is easier, just swap in the persisted object

Come up with a taxonomy of terms:

journal
personal / project URL
Version Control Repository: github / bitbucket
Digital Repository: osf, figshare, zenodo, comses

add schema support + automatic classifier of Publications that assigns the field if it's not already set

extract data_export.py from the management command using Calvin's example data file as the target schema

@alee

add D3 visualizations for real-time interactive stats and scheduled static image generation

potential design: scheduled processes write from primary data to redis or other cache, D3 + VueJS frontend on the wagtail comses.net site dashboard to interact with the redis cache

• graphs of the number of publications over the year
• the percentage of model code available per year
• most popular journals (plus % code availability)
• most popular authors (plus % code availability)
• bar charts models are described and code is archived.

We can migrate the data from the date_published_text field of the publication to the year_published field

remove bibtexparser spam from console logs

for a given publication, have some external lookup for number of citations referencing that pub

https://stackoverflow.com/questions/37528286/how-to-make-migrations-for-a-reusable-django-app

Perform code review with Dhruvil and identify areas to improve.

The main goal is to design a coherent API + data structures + algorithms for generating and accessing the visualization data.

Things to note:

• this should be independent of SearchQuerySet implementation details, which will go away when we move from django-haystack to elasticsearch
• classes and methods are jumbled at the moment

Data inconsistencies should be noted on a publication so that a entry can be reviewed

(right now data import problems are just saved in a pickle file to inspect later)