ual-re / requiam_csv Goto Github PK
View Code? Open in Web Editor NEWConstruct list of research themes and organization mapping to work with figshare patron management
License: MIT License
Construct list of research themes and organization mapping to work with figshare patron management
License: MIT License
The original UA Org Codes sheet is missing any leading 0's for 3-digit code (e.g., 0410
shows up as 410
). This breaks things on the patron side as an EDS query of 410
returns nothing.
Summary
logging
module is not refactored into redata-commons
. Let's migrate to using it as a dependency.
Objectives
Proposal
Testing notes
Additional notes
Implemented in: TBD
This is the main issue thread for Travis CI integration.
Steps include:
This is implemented in the feature/travis_ci
branch
feature/organize
requirements.txt
fileDescribe the hotfix
README.md contains duplicate sections from our Read the Docs documentation. The task is to simplify the maintenance of README.md by one of three methods:
Using the external library m2r2 to generate ../../README.md from source/. Drawback: it hasn't been maintained for a few months now.
Using the external library MyST parser as in (1). This library seems more up-to-date.
Simply stripping sections that need constant maintenance, such as code and Python versions. Then, update the TOC with links to our RTDs, for instance:
- [Getting Started](https://requiam_csv.readthedocs.io/en/latest/gettinnstarted.html)
Issue: if the links to RTDs are implemented before merging #66 into master, we will not be able to properly test them because they use the "/latest/" path.
Version information
Plan of action
- Getting Started
- Requirements
- Installation Instructions
- Configuration Settings
- Testing Installation
- Execution
- Workflow
- Versioning
https://requiam_csv.readthedocs.io/en/latest/GettingStarted.html
https://requiam_csv.readthedocs.io/en/latest/GettingStarted.html#requirements
https://requiam_csv.readthedocs.io/en/latest/GettingStarted.html#installation-instructions
https://requiam_csv.readthedocs.io/en/latest/GettingStarted.html#configuration-settings
https://requiam_csv.readthedocs.io/en/latest/GettingStarted.html#testing-installation
https://requiam_csv.readthedocs.io/en/latest/Execution.html
https://requiam_csv.readthedocs.io/en/latest/Execution.html#workflow
...etc
## Developers' note:
Full documentation for this code is available on
[Read the Docs](https://requiam_csv.readthedocs.io/en/latest/). The table of contents below links to the appropriate sections within our Read the Docs documentation.
Additional context
@astrochun: This can be implemented as a simple PR that will be merged but without any bumping
Implemented in: >= v0.12.4
Main function (create_csv
) will:
The file should be exported to a path within this repository so it can be made public on GitHub.
Configure for easy installation through pypi
.
This requires:
setup.py
After documentation with RTDs #66 is finished, add checkmark reminders to update documentation on:
Implement as hotfix
Summary
It's been decided to migrate from ualibraries to UAL-ODIS.
Objectives
Full transfer with branches, issues, PR and project boards. Settings may need to be manually transfer over
Proposal
Testing notes
Already tested transfer here
No branch deployment here.
Summary
After completing redata-commons UAL-RE/redata-commons#23 and ReQUIAM Python version upgrades UAL-RE/ReQUIAM#170, this repository shall be updated to be consistent with other ReDATA software.
Objectives
Migrate to Python 3.9
Proposal
Testing notes
Need to verify that the current version of python, numpy, and pandas work as usual. Testing shall go through "script_run" and create_csv.py and Inpspect_csv.py under "/requiam_csv"
Additional notes
redata >=0.4.2 shall work. However, for the maintainability, we try to use the same sets of dependencies (redata, numpy, pandas, sphinx)
Implemented in: TBD
When script_run
is executed, conduct a comparison to identify differences and use that output to generate a git commit
containing details of the changes. The commit message will consists of individual lines with each entry
Summary
For future enhancements including typing checks, we should include CI for Python 3.9.
Objectives
Eventually take advantage of Python 3.9 typing features
Proposal
Modify python-package.yml to include Python 3.9.
Implemented in: TBD
Change use of logger from create_csv
to script_run
Add type hints to later integrate into our documentation.
Summary
The University Libraries portal was created for purchased datasets. Through discussion, we decided to have all University of Libraries employees under this portal. Below summarizes the steps needed to create a main portal or any portal (e.g., sub-portal).
Objectives
Grouper group membership updated for UAL members
Proposal
Here are the steps to implement the change:
Figshare/ReDATA:
libraries
as the UserAssociationCriteria (done some time ago). I might have done ual
but since this is set, this should be fine.Google Sheet:
libraries
ReQUIAM:
add_grouper_group
script with --main_portal
and --add
to create the new group on both figstest and figshare stems
portal:libraries
ReQUIAM_csv:
develop
and run script (dry run) to check the result of the outputDeployment:
run_script
using the bash alias with --portal
and --sync
. This should trigger on the new group to include.Testing notes
Additional notes
Implemented in: TBD
The config file (default.ini
) should include:
outfile
nameA primary script called script_run
to execute via the command line:
python script_run
The script should:
default.ini
configuration filecreate_csv
For step no. 2, setup.py
is needed to work with the absolute package, DataRepository_research_themes
An arg option to generate the CSV file but not overwrite the default existing CSV file ('data/research_themes.csv')
Changes are implemented in feature/dry_run
branch.
Update README.md to describe dry run and full execution.
Update default.ini to use a 'data/' path for output
Note: Fast-forward version to 0.7.0
Mirror LD_Cool-Ps .github
folder structure here:
A logging
-based object that handles the messaging to stdout and a logfile
Logs should be placed in a logs/
folder relative to path of execution. The logfile
filename should include an ISO-formatted date.
The logfile prefix should be specified in default.ini
Change will be from "DataRepository_research_themes" to "ReQUIAM_csv"
Updates needed:
Summary
Upgrade RTD and Sphinx dependencies as these of redata-commons and ReQUIAM
Objectives
Proposal
Testing notes
Additional notes
redata-commons UAL-RE/redata-commons#32
ReQUIAM UAL-RE/ReQUIAM#180
Implemented in: TBD
A function is needed to
General issue for best practices. This ticket will remain open indefinitely.
Currently, scripts are using .format()
for strings.
While this ok, the pythonic way prefers f-strings for their readability
and several other reasons. (See Item 4 in Effective Python)
Example:
log.info(" {} : {}".format(dept.loc[bb], bb + off))
## Change to:
log.info(f" {dept.loc[bb]} : {bb + off}")
This can be a good first issue implemented as a feature for the next release.
A nice feature would be a code that allows one to compare two files that provide the University's organization codes.
The idea would be to identify differences for maintainers to understand.
Ultimately it would be nice to update the Google Sheet with just the changes through Google Sheet API.
Summary (REQUIRED)
With CI build tests already in place with Travis CI, it's relatively straightforward to migrate to GitHub Actions.
Motivation (REQUIRED)
Because of the change in pricing model by Travis CI for open-source software, it becomes cost prohibitive to use Travis CI moving forward when credits run out.
Objectives (REQUIRED)
CI build passes much like how it has occurred for Travis CI.
Proposal (REQUIRED)
This doc page and the GitHub Action YAML is a starting point:
https://docs.github.com/en/free-pro-team@latest/actions/guides/building-and-testing-python
We can start with this but not include python 2.7.
Will need to include other dependencies such as pytest-cov.
Best effort to resemble the .travis.yml should be considered
We should require a skip if "ci skip" or "skip ci"
We should also look at excluding certain files, such as README
Testing notes (Optional)
This might be useful for testing purposes before committing the github actions:
https://github.com/nektos/act
Action items:
python-package
GitHub Actions workflowImplemented in: feature/gh_actions_build_test
Deploy actions/create-release
action to create a new release on any tagged version that is pushed.
This issue will remain indefinitely. It is intended to track all changes to research_themes.csv
After cleaning up the Google Sheet so it would run effectively, the following error came up:
Traceback (most recent call last):
File "DataRepository_research_themes/script_run", line 29, in <module>
create_csv(url, outfile, log_dir, logfile)
File "/Users/cly/codes/UALibraries/DataRepository_research_themes/DataRepository_research_themes/create_csv.py", line 91, in create_csv
no_org_code = no_org_code_index(df_new)
File "/Users/cly/codes/UALibraries/DataRepository_research_themes/DataRepository_research_themes/commons.py", line 15, in no_org_code_index
df['Overall Themes'].isna())[0]
The issue has to do with df_new
not containing the 'Overall Themes' column as that was dropped.
This should be an an easy hotfix.
We are ready to provide documentation through ReadTheDocs.
This tutorial is a good starter. A couple of key points:
docs/
folder relative to the repo parentsphinx_rtd_theme
HTML theme. You will need to include to include the sphinx_rtd_theme
extensionAction item:
docs/
for Sphinx compilationA declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.