This requires changing the self.baseurl in FigshareInstituteAdmin to api.figsh.com

An easy keyword input of stage with it set to False should fix the issue.

Since this is an issue fix, I'll incorporate via hotfix/v0.10.1

The application would include:

1. Figshare administrative information page (e.g., user account, groups, projects, collections)
2. Information page on existing datasets under curatorial review
3. Feature to retrieve data and place is on the data curation storage array
4. Management tool of data curation workflows (e.g., widget buttons to move data through different stages)
5. Integration with Trello API

Because it's an open port, security restrictions are needed (e.g., limited access to only a few users).

This is related to #77 but to focus on having all README_template codes within ReadmeClass instead of having some of it in curation.retrieve.

This feature is in feature/packaging

This includes:

1. setup.py script [363d994]
2. requirements.txt [4bc918a]
3. config file specifier in __init__ (see iss #29) [666c719, 451cdf0]
4. Absolute imports after setup.py is working [1a52e43]
5. Re-organize figshare_api module to be in curation.api [33c4d04]
6. Update README.md to describe installation steps [42efa74]

Currently the configuration file is specified through the main_config_file variable given in __init__.py.

Ideally this will be the default location. However for many scripts, it is beneficial to provide a different configuration file. This allows for flexibility.

It would best to work as a dictionary so settings can be easily passed around. We should switch from individual variables in config/__init__.py toward a dictionary.

Came across this stack overflow that illustrate how to do this with existing libraries: config.ConfigParser. Note that this is python 2, but most of it will work:

dictionary = {}
for section in config.sections():
    dictionary[section] = {}
    for option in config.options(section):
        dictionary[section][option] = config.get(section, option)

One trick is handling boolean. We might require retrieving it and checking if it is 'True' or 'False' and then doing the config.getboolean approach.

• We will therefore have a function that imports the config INI file and -> dict.
• We will also have the default config file as a dictionary and that is used if say the optional input is not provided:
  config_dict=default_config_dict

This should be part of the admin sub-package.

Relocation should happen between:

• 1.ToDo
• 2.UnderReview
• 3.Reviewed
• 4.Published
• 5.Rejected

1. A general main() function should exists to move in any given direction [2f012f4]
2. A get_source_stage() function to retrieve curation stage [f258bd2]
3. A move_to_next() function that identify current path and moves to the next stage [ece3c33]
4. A reject() function to move to reject stage [f45d533]

It currently reads as:

---------------------------------------------
## Contributor Roles

The roles are defined by the CRediT taxonomy https://casrai.org/credit

This should change to:

---------------------------------------------
## Contributor Roles

The roles are defined by the CRediT taxonomy http://credit.niso.org/

This will prevent verbose messaging when scripts use DepositorName

Default is True

These settings include:

1. Paths for data curation storage array (both local and remote) [72a87cc]
2. Paths for different staging folders of data curation (e.g., 1.ToDo, 2.UnderReview) [72a87cc]
3. Paths to organize curatorial review (e.g. DATA and UAL_RDM) to separate data from curation review documents [fdc9f51]
4. Trello API

Need to change ownership if path points to a specific file.

Use isfile(path), similar to admin.permissions.curation

The following error was encountered.

~/codes/UALibraries/DataRepository_curation/DataRepository_curation/curation/depositor_name.py in get_name_dict(self)
     99         # curation IDs, email, and title
    100         name_dict['article_id'] = self.article_id
--> 101         name_dict['curation_id'] = self.cur_loc_dict['id']
    102         name_dict['depositor_email'] = temp_dict['email']
    103         name_dict['title'] = self.curation_dict['item']['title']

AttributeError: 'DepositorName' object has no attribute 'cur_loc_dict'

Due to some refactoring, self.cur_loc_dict is deprecated. Switching to self.curation_id

Fix will be implemented in hotfix/v0.8.1 along with a bug in #37

This would be within the prereq_script script.

• Input for article_id could be a comma separated (e.g., 12345678,99999999,55555555) to split out and a for loop over each article is done. Note that a space would NOT be allowed

• To check if a deposit exists, this requires getting the folder and see if it exists. We might need an option to create retrieve data if it does not exists.

Calls to admin.permissions may be setting things for DATA and UAL_RDM folder. However the parent folder (one level up) permissions is not properly set.

During the curatorial process, sometimes a DOI is not minted. To fix this, we should add a feature to mint a DOI if the DOI dictionary key is not available. This would be more streamline for the README file as it does not contain such information.

Note that if a DOI is missing, the content is an empty string:

In [14]: curation_dict = fs_admin.get_curation_details(curation_id)
In [16]: curation_dict['item']['doi']                                                                                                                              
Out[16]: ''

This should be implemented within the FigshareInstituteAdmin class as a new function and implemented in PrerequisiteWorkflow class. This would avoid issues in ReadmeClass.retrieve_article_metadata

At the moment depositor_name consists of [FirstName] [LastName] and the get_depositor_name function returns such a string without any middle name.

For more generalized use, it should be a dictionary that contains:

1. Full First Name: firstName
2. Full Last Name surName
3. Simplified First Name (no middle name) simplify_firstName
4. Simplified Last Name (no suffix) simplify_surName
5. Combination of the full possibility and simplified possibility: fullName and simplify_fullName

This would allow for better checking.

Decided to create a new module (depositor_name) within curation package [32956fb, 2197ec8, 2e46c06]. Instead of a function, it is an object class.

I implemented this in the curation.main workflow [02cb0ac] and in Qualtrics object [953c41c]

Retrieve:

1. Data Curation Review Report (Word document)

If possible, retrieve signed deposit agreement from Qualtrics

As I determined from testing, the creation of folder is not implemented correctly.

Requires a separate function (make_folders) and calling it to ensure that it happens.

API documentation is available here.

This is within feature/qualtrics_api and feature/qualtrics2 branches.

This feature will include:

1. A Qualtrics class to retrieve basic information [4a4c219, 99d7920, ef80421, a2c9431, 830b18c, 2ca475e, bb99a63, bdaabf1, 75e110b, d0793cb, d29eaf1, e641a74, ec9ba7d, 35616de, 53b092f]
   1. Ability to check if a depositor has completed the deposit agreement [297c2f2, 63a9d39, d834d69, 47f0b77]
   2. Ability to retrieve the report in HTML or PDF (See below for a quick hack to make this possible) [4946c3d, ceb7970, 82d2932, bdaabf1, 4389c09]
2. Configuration settings in config/default.ini [d3b6f34, 402f632]

The ability to use existing administrative API token to create an article/item for a user, and upload data.

Based on API documentation, this appears to be possible:

When creating folders and files, the permissions for these files should be appropriately set.

This might easily be done using the built-in os.chmod module. Examples are available here.

In addition, changes to ownership may be a feature that is needed. This is through os.chown module.

Some of the GET REST API requests have pagination. At the moment, I fixed this by requiring a maximum of 1000 (page_size=1000), but this limit could be hit (e.g., many publications, many users) within the FigshareAdmin class. This not critical (hence the wontfix label), but could be resolved later when we encounter this problem.

LD-Cool-P = "Library Data Curation Tool in Python"

This includes updating:

1. README.md
   1. Adding graphics for logo and full name [f3308c5]
   2. Update for package name that will appear in conda list [fdf2442]
   3. Updating installation instructions (this will happen right before the GitHub repository name change) [f647932]
2. setup.py and DataRepository_curation folder [46124ea]
3. All absolute imports involving DataRepository_curation [46124ea]
4. Renaming the software on GitHub
5. Address merge conflicts against v0.8.1 [d62efcb]
6. Bump version from v0.8.1 to v0.9.0 [fe1c238]

This particularly a problem for large files. The urlopen option that was originally used reads the content in memory.

There are a number of ways to fix this:

1. requests: psf/requests#885
2. urlretrieve with authentication information: https://stackoverflow.com/questions/45247983/urllib-urlretrieve-with-custom-header

I decided to go with the latter. It was quickly tested and appears to work.

Describe the bug (REQUIRED)
Figshare API does a curation review retrieval by article_id. For deposits with multiple versions (i.e., multiple curation items), we are only using the most recent one (this is done via df_to_dict_single()). This poses a problem if we want to retrieve a specific version and not the most recent.

Reproduction Steps (REQUIRED)

from ldcoolp.curation.api import figshare

In [1]: from ldcoolp.curation.api import figshare
In [2]: fs_admin = figshare.FigshareInstituteAdmin(token='API TOKEN')
In [3]: curation_df = fs_admin.get_curation_list(article_id=12712856)
In [4]: print(curation_df.shape)  # five entries found
(5, 13)
In [5]: from ldcoolp.curation import df_to_dict_single
In [6]: df_to_dict_single(curation_df)  # Only first entry selected by default
Out[6]: 
{'index': 0,
 'status': 'approved',
 'modified_date': '2020-08-14T17:16:03Z',
 'resolution_comment': '',
 'review_date': None,
 'account_id': 2408075,
 'assigned_to': 2283558,
 'request_number': 5,
 'comments_count': 1,
 'version': 4,
 'created_date': '2020-08-14T17:14:55Z',
 'group_id': 26136,
 'article_id': 12712856,
 'id': 529559}

Outputted Messages (REQUIRED)
No actual error.

Version information (please complete the following, default in brackets): (REQUIRED)

• LD_Cool-P version: 0.13.0
• ReQUIAM branch info: release/v0.13.0
• pandas version: 1.0.2
• requests version: 2.22.0
• numpy version: 1.17.4
• jinja2 version: 2.11.2

Recommendations:

• Enable option for curation_id input into df_to_single_dict()
• Enable option for curation_id input into DepositorName

This is to ensure that the webbrowser is not launched on a VM or cloud instance that is purely command-line.

A feature needed is to update the README.txt.

This will also require a script to re-run to update the content.

This is implemented in feature/readme_update:

• Add an update method in ReadmeClass. While I could modify other methods, this might be the cleanest for MVP without creating bugs. More testing later for better integration [adb3885, b5424f7]
  • Find current path using MoveClass.get_source_stage [340466a]
• Create an update_readme script to enable easy execution [18dec85]
• Update README.md to describe using update_readme [0602d4c]
• Update README.md changelog [2c24eba]
• Update README.md to provide execution instructions [75ea692]

This should will include:

1. Create curation folder with proper name [28f7c6f, 10ced5a, 7442a15, f5cd0a0, 4429878]
   • depositor_name is obtained from curation metadata with cross-matching based on account_id. Takes into account whether the depositor is an author.
2. Retrieving data [dca9021, 4429878]
3. Setting permissions and ownership [dca9021, 4429878]
4. Downloading curatorial review report [dca9021, 4429878]
5. Use Qualtrics API (this is desired, but not required) [2dea169, 63a9d39, d834d69, 117172e]
   1. Check if deposit agreement exists
   2. Retrieve Qualtrics form via the Qualtrics API
6. Check for README.txt file (this will happen in a second release when inspect_readme is in order)
   1. If such file does not exist, download README.txt template
   2. Update with based information from metadata

Previous curation had old naming convention for folders. For the purpose of completeness, we need a script that translates the old names to the new ones.

This is in hotfix/v0.13.1 branch

This bug has to do with the default settings. As it turns out, the params passed into the issue_request call is not correct.

This page indicates that limit and offset should be used instead of page_size and page, respectively.

As it currently stands, the self.folderName is determined in DepositorName based the the depositor information and the corresponding author (usually the first author). As such, if multiple deposits are under curation, this will break the curation.main.workflow. admin.move.move_to_next will get confused.

A simple solution is to include either the article_id or the curation_id in the self.folderName. This can be implemented as a simple hotfix.

Previous folders may need to be updated for compatibility.

At the moment, I'm specifying the default.ini file in almost every modules. It would be better to have a common variable that handles the filename. This would ensure that the same config file is read in

This would allow for a faster query and would not require pagination if the curation list is too large.
This is possible through a params input for article_id.
See: https://docs.figshare.com/#account_institution_curations

When the default.ini contains ***override***, it prompts by default.

However, because this code is getting used more often, it may be best to have the ability to pass in the figshare class that contains the API token, which is defined in other codes, rather than redefine it within curation.retrieve sub-module.

One of Qualtrics features is Embedded Data using Q_PopulateResponse

Information about Embedded Data is available here with more specifics on Q_PopulateResponse here.

The responses to questions can be provided in JSON format.

Here's an example:
https://uarizona.co1.qualtrics.com/jfe/form/SV_39rs7lHGLUYFK7P?Q_PopulateResponse={"QID4":{"1":"Depositor Name", "2":"<depositor>@email.arizona.edu"},"QID7":"Title of deposit"}

As it turns out, it is not necessary to provide Embedded Data in the Survey Flow. Providing the responses in the URL will suffice.

For questions with single field, such as QID7, it is fairly straightforward. For multi-field responses, passing another JSON for the question with specification of "1", "2", "3", etc. for the response fields will suffice.

1. This will be implemented with the Qualtrics API object with a function called generate_url() within feature/qualtrics2 branch

2. Suggested by @zoidy: We can use hidden embedded data fields to provide an article_id or curation_id. This would be a more stable way to identify deposit agreement for each deposit as information provided in the fields could be different from metadata information that is provided in Figshare (e.g., Name spelling, email address, etc). - Implemented in 44f4061

3. After embedded article_id, we will need to first search via this before looking at DepositorName info. - implemented in e7dcaef

At the moment, each code read in the config file and configure accordingly. It would be more elegant and better managed to have a config module (could be a main.py or __init__.py within config module)

This will ensure consistent global variable names across the board and makes it easier to troubleshoot with a simple from ldcoolp.config import <variable> command.

Action items:

• Create config module with __init__.py [96b3e2f, e484280]
• Update api.qualtrics with config settings [f0bf983]
• Update curation.inspection.readme with config settings [873c508, fea3892]
• Update curation.inspection.readme.populate with config settings (this is actually obsolete code that we will delete later) [f0c8ea1, 3ff8506]
• Update curation.main with config settings [435a2aa, 208a245]
• Update curation.reports with config settings [be6840d]
• Update curation.retrieve with config settings [ccb7ba2]
• Update admin.move with config settings [f73d6b6]

There may be an issue with ORIGINAL_DATA. It's currently set as read only. Fernando reported an issue with moving files. However, move.move_to_next() seem to be working.

I have not tested this fully to see if the issue is specific to drag/drop or if os commands will permit.

Currently main.workflow performs a move after the contents are retrieved and folder structure is in place. However, it would be best to move contents over when we are ready to being our review. This is particularly helpful to retrieve multiple datasets but to only interact with one deposit.

• Solution: An interactive option via input() is needed for the curator to perform the move.

Currently when no README file is uploaded, it will use the default README_template.md. This is saved as README.txt in the DATA folder. This poses a problem as re-generation is not as straightforward and having a template in the folder will enable easier updates to Figshare metadata. This follows the same workflow of editing the template for had the user uploaded such a file.

Note that some improvements is needed since it appears that README-related codes are in a number of locations.

As part of the prerequisite workflow, it would be nice to generate mark-up text for an acknowledgement email. This should include:

1. A list of To address: e.g., depositor and first author (if an email can be retrieved)
2. A proper salutation with the names
3. The custom Qualtrics URL for the deposit agreement

This should include:

1. A config file (default.ini) to handle private information (e.g., API token) [8d6accb, d8ed925]

2. Create a FigshareInstituteAdmin class to: [29f42d1, 33814ff, b0d1947, 35577b6, 5947e5e, eb15074, eb179ab]
   a. Retrieve all articles [1de1608, 35e4f0b, b289464]
   b. Retrieve account information [d18bc78, 7c1818c, 3d39be2, e22e72e, 282439a, 2dc4e4b, 0a348bd], groups [1c6226c, fd79fbf], group roles [33fa790], user's articles, projects and collections [10e2d6b, 0d0e9a6, 2c9c1e4, db961d7]
   c. Retrieve Figshare curation information [3ba9895, b56c357, 13fd6b1, 85da22f, cc98636]

3. Use previous figshare code to:
   a. Retrieve information about the upload [11e195b]
   b. Retrieve data that is in curatorial process [4be9307, f379228, 5e8a068, 3742c1e, 3ba9895, b56c357, 96b2f15]

4. Create a separate issue_request that enable passing in additional parameters [e4a42f7, e53a9f5, 19d8aba, fc3b249]

5. Update to documentation [2b81dd7, 84bbff0, bd23d8d, 7dee063]

Since README construction is now part of the prerequisite, it makes more sense to not perform a move.move_to_next prior to constructing the README. Thus, the README will be constructed in the 1.ToDo step.

• Update paths in ReadmeClass
• Move the call to move_to_next later and see #82

We currently have two folders for data curation, DATA and UAL_RDM. It has been decided unanimously to have a separate folder for ORIGINAL_DATA.

Here are the action items:

1. We will need to update the default.ini and other config files [7515c3e]
2. ORIGINAL_DATA should be read-only [ad71b4c]
3. If the README.txt file is provided, we should have two copies, one in ORIGINAL_DATA and another in DATA [f199108]
4. We will need to change curation.main to work with the new folder architecture [addc0d7]

Note: The inspection_readme feature (iss #15, PR #16) will generate a README.txt file if one does not exists. We will need to incorporate a duplicate copy following similar logic as above.

This will be in hotfix v0.9.2. The bug is actually in curation.main.workflow as we are calling find_deposit_agreement and this side-steps retrieve_deposit_agreement when ResponseId is ill-defined.

For full documentation purpose, logging through python logger library will be used to preserve records of changes conducted.

This will be in the feature/logging branch and will ultimately be part of the v0.14.0 release.

• Construct LogClass class and log_stdout function [22ad217, 005f213, da2b06c, ead168a, b30a7d4, d9bcc5c]
• Update prereq_script script to use LogClass [4b95be8]
• Update get_curation_list script to use LogClass [d1484e2]
• Update generate_qualtrics_link script to use LogClass [ad5ea2d, 927678a]
• Update default.ini with logging settings [e4025ad]
• Update curation.reports module to enable logging [9226682]
• Update admin.move module to enable logging [da19265]
• Update Qualtrics class to enable logging [c421f46]
  • Improve with f-strings [7485cde]
• Update ReadmeClass class to enable logging [72b2db6]
• Update DepositorName class to enable logging [18825c0]
  • Improve with f-strings [8ee4778]
• Update curation.main module to enable logging [520d29e]
• Update curation.retrieve module to enable logging [e6a2bbb]
  • Improve with f-strings [4e7a862]
• Update figshare.FigshareInstituteAdmin class to enable logging [c8b6367, a9c50cd]
  • Improve with f-strings [1998a80]
• Retrieve and log commit hash and commit message [4e0262d, b5ea612]
• Add logs about user and server details (name, IP, OS installation) [04edea0, ac16d4c, 67a17d8]
• Write df.to_markdown() content to log [808a448, 52840b9]
• For logs, needs to change permission mode for +rw (0o666) [a2728da, 3557de2]

Describe the bug (REQUIRED)

On occasion, the scripts do not run if an incorrect config file path is provided for --config:

Reproduction Steps (REQUIRED)
Steps to reproduce the behavior:

1. python ldcoolp/scripts/get_curation_list --config config/blah.ini

Outputted Messages (REQUIRED)

Traceback (most recent call last):
  File "ldcoolp/scripts/get_curation_list", line 54, in <module>
    config.get('curation', 'log_parent_dir'))
  File "/Applications/anaconda3/envs/figshare/lib/python3.7/configparser.py", line 780, in get
    d = self._unify_values(section, vars)
  File "/Applications/anaconda3/envs/figshare/lib/python3.7/configparser.py", line 1146, in _unify_values
    raise NoSectionError(section) from None
configparser.NoSectionError: No section: 'curation'

Version information (please complete the following, default in brackets): (REQUIRED)

• LD_Cool-P version: [e.g. 0.13.1]
• ReQUIAM branch info: [e.g., feature/logging]
• pandas version: 1.0.2
• requests version: 2.22.0
• numpy version: 1.17.4
• jinja2 version: 2.11.2

Expected behavior (Optional)
If file is not found, return a FileNotFoundError

The curation.retrieve.download_files is used in main.workflow() to retrieve the dataset during the curatorial process. However, in the case that a dataset is public, I believe the private links are not working. As it gave me a HTTPError: HTTP Error 400: Bad Request. This happened for a number of datasets that were made public.

I'll need to come up with a different solution then.

A set of scripts:

• A simple script (will need a scripts folder) that would allow for command-line execution. This would minimize the need for a number of things via iPython. Script is called: prereq_script [278861b]
• Another script to retrieve the curation list so that the article ID is retrievable [8fb06a0]
• Another script to get the deposit agreement custom URL to provide [96deb8e]

Additional changes:

• Provide version number in banner message for each script [bd338f8]
• Provide current active git branch in banner message for each script [bd338f8]

For README.txt file, this includes:

1. Check that a README.txt file exists or if it exists elsewhere [9620dd9, 98094f8, f4512b9, 04a8b81, 9995d15, 040b2e4]
2. Function to retrieve README.txt file and deposit in correct curation path if such file does not exist [f57e92a, 00513ec, f73c2ec, 244a377, 4cd6fc5, 3d4e13a, c95392c, 3b16a4d, 3bf26cd]
   1. Call the strip() function to remove extraneous information [9bfc4a8]
   2. Also, avoid saving an original copy and avoid overriding that file [17c1b55]
3. Check that README file format is correct
   1. Remove <!-- to --> content and white space before [cb8fa3d, 2d667c2, 64e2a56, a940805, 156f8fa, 31f6f35, edcea61]
   2. Check for Title (No longer needed if we ask depositors to not fill out this section)
   3. Check for preferred citation (No longer needed if we ask depositors to not fill out this section)
   4. Check for Corresponding Author (No longer needed if we ask depositors to not fill out this section)
   5. Check for License (No longer needed if we ask depositors to not fill out this section)
   6. Check for DOI (No longer needed if we ask depositors to not fill out this section)
4. README.txt template file with jinja2 templating [94b7fac, 520bf59, abcf216, fcbf788, d5910fa]
5. ReadmeClass to enable the construction of this README file and use jinja2 template [94ef05e]
6. When README.txt is created, retrieve information from figshare API, and populate it with the following information
   1. Title [retrieve: a4a8edb, populate: 9b0020c]
   2. Preferred figshare citation [retrieve: a4a8edb, populate: 7186da1]
   3. Corresponding Author (assume first author) [retrieve: a4a8edb] - this requires separating out first and surname and getting the email address [3bb87ae]
   4. License [retrieve: a4a8edb, populate: 94b7fac]
   5. DOI [retrieve: a4a8edb, populate: c85d902, 9b0020c]
   6. Description - requires parsing HTML text [d4ffbba, 06d47c2]
   7. References [d4ffbba]
7. After much though on the workflow, I realized that we need the ability to handle different README template inputs. Ultimately we aim to provide the jinja2-based template to users and inform them not to fill in title information, etc. If we do this, then we should use this template as it contains custom information that we can re-use easily. If one is not provided, than the default one that we have is used. [cad5ed6, 3aa378f, 715fc2e]
8. For data retrieval, the README.txt copy in the DATA folder should be a template, so we should name it as README_template.md. This requires modification on the release branch [3e4d962, a62c67b]
9. Use README walkthrough for iss #38 where a duplicate copy of README files is saved. (this is low priority, so saving for later)

I came across the following that might be helpful with handling markdown content:
https://stackoverflow.com/questions/761824/python-how-to-convert-markdown-formatted-text-to-text
https://daringfireball.net/projects/markdown/dingus

Jinja2 might be our best bet as a template engine.
See:

• https://gist.github.com/glombard/7554134
• https://gist.github.com/wrunk/1317933/f5da8e19abee44ea576acc2404836afa0619ce5a

I'm proceeding forward with jinja2. Handling of HTML is still needed for the Figshare description metadata. One possible solution is html2text

Suggestion 1:
The link to the README.txt template is outdated. Currently, it redirects to:

https://osf.io/sj8xv/download

Maybe (?) it should redirect to:

https://github.com/ualibraries/LD_Cool_P/blob/master/ldcoolp/curation/inspection/readme/README_template.md

Suggestion 2:
Can we automate the version update through GitHub Actions for things like:

You should see that the version is `0.13.2`.

Suggestion 3:
Below ### Installation Instructions, add also a HowTo for updating the ldcoolp package:

If you previously installed an old version of ldcoolp, you can update the package to the latest version:

(curation) $ cd /path/to/parent/folder/LD_Cool_P
(curation) $ (sudo) python setup.py develop