ual-re / requiam Goto Github PK
View Code? Open in Web Editor NEWIdentity and Access Management backend software for UA Research Data Repository (ReDATA)
Home Page: https://requiam.readthedocs.io
License: MIT License
Identity and Access Management backend software for UA Research Data Repository (ReDATA)
Home Page: https://requiam.readthedocs.io
License: MIT License
requiam
folder. Let's have a scripts
folder in the parent pathgit
branch (requires a get_branch_name
module)I have performed tests of LDAP queries, Grouper queries, and delta synchronization in the tests folder.
Next step is to package it together into a working module(s).
This is in the feature/ldap_grouper_delta
branch
script_run
) to run the query [777ba4f, 306b1c8, 3303169, 4a6a103 8882d20, 1385487, aa296b8, bdb6cb3, ec10d9e, 01a5d18, 8f5dfe2, 3b636da, c6317b0, 6223407, 452aeed, d9f60df, ba1e8cb, fa48fc0, 6413065, 89af40a, 3469d3a, a64c668, fe1f397, bfe373c, e2099ad, 8d9788a, 3e1a5b0, 47e45e3, 7aa387f]script_run
[66821d8, 0093af9, 6413065, 433364b]In the user_update
script, multiple calls to grouper_delta_user
is made and a set of sync/nosync commands are included.
Now that grouper_delta_user can receive a LogClass
object, let's refactor into grouper_delta_user
. This will make the code more easy to read.
The manual_override
script enable the ability to add users in Grouper portal
and quota
groups. However, it does not have the capability to add users to the active
Grouper group. This ensure IAM access for the ReDATA platform. Note that this is a temporary solution for large group of users before the service is entirely live. This provides an alternative solution from APPS-IAM or the Grouper interface.
Some feature to include:
Note that this has been well-developed and available here.
Code is available in feature/delta
The following error was received.
11:26:36 - WARNING: portal not found on Grouper!
Traceback (most recent call last):
File "scripts/user_update", line 167, in <module>
raise SystemError
SystemError
This is simply due to check_grouper_exists()
which does not find a portal group name root
.
Solution:
if vargs['portal'] != 'root'
exception needed hereA Grouper portal group can have association from multiple organizations. Thus a direct comparison between LDAP and Grouper to identify additions and deletions requires that multiple LDAP queries be conducted and merging those membership.
Describe the bug (REQUIRED)
The follow error message was returned:
22:17:50 - INFO: Removing ___ from current quota: 2147483648
Traceback (most recent call last):
File "scripts/user_update", line 397, in
production=grouper_production)
File "/home/mozart/codes/ReQUIAM/requiam/grouper_query.py", line 135, in grouper_delta_user
member_set = update_entries(member_set, netid, uaid, action, log=log)
File "/home/mozart/codes/ReQUIAM/requiam/manual_override.py", line 210, in update_entries
remove_netid = [netid[i] for i in range(len(netid)) if
File "/home/mozart/codes/ReQUIAM/requiam/manual_override.py", line 211, in
uaid[i] in remove_uaid]
TypeError: 'set' object is not subscriptable
We need to add a column that allows for additional comments. This could include when the addition was done or changes that was made. The ManualOverride.update_dataframe
function would mark changes done through user_update
In a number of scripts (e.g., add_grouper_groups
), we need the list of quotas to execute all possible cases.
For this to work, we need:
pandas
DataFrameadd_grouper_groups
The above list can be implemented within feature/grouper_api_admin
Not necessarily a bug, but the path for the grouper_base_path
is specific for the tool to add members: grouper-ws/servicesRest/json/v2_2_001/groups
As such, it will not work with some components of the Grouper API, specifically permission. There's a simple hack to strip the trailing group
, but a more ideal situation is to keep it generic and set the proper endpoint as needed.
This should be implemented in release/v0.12.0
Notes:
feature/grouper_api_admin
for GrouperAPI
and GrouperQuery
feature/multi_user
uses GrouperQuery
, it should be fine when we everything is merged into release/v0.12.0
Undergrads will by default have a quota of zero. Only those with privileges are manually added using the user_update
script. Thus an option should be added to disable running it the ugrad
quota.
Options are:
I will implement this in feature/limited_set
The following error message was encountered when both --portal
and --quota
were not specified:
Traceback (most recent call last):
File "scripts/user_update", line 164, in <module>
if not portal_check or not quota_check:
NameError: name 'portal_check' is not defined
Simple solution is to pre-defined portal_check
and quota_check
. This avoids the SystemError
when either of them do not exists on Grouper
A separate warning should be made to indicate the neither of them are set.
For script_run
, only the pandas
DataFrame is available via stdout
. It would be easy to save this similar to what we did in LD-Cool-P
Note that this requires updating pandas
to >= 1.0.2 for .to_markdown()
. I will use 1.1.0 since this will have index=False
option. Also, tabulate is needed. I suggest using tabulate=0.8.3
since it worked for LD-Cool-P
This issue has split off from #43.
Primary work will be in user_update
script. At the moment, this script can only handle a single patron.
Minimum: a comma-separated list of netid
Desired: Minimum + ability to use simple txt file with netid's (one per row).
user_update
:user_uaid
)get_current_groups
needs to handle multi-user. Best to use dict within dictroot
option which implies no association with a quotaManualOverride
to handle multi-user add to tables, etc.Currently we have two grouper modules, grouper_query
and grouper_admin
. It was done in a modular manner to avoid any conflicts while implementing admin tools. Now that both modules are stable and well-tested, it's time to merge the two so there's only one grouper
module.
Action Items:
grouper_admin
module to grouper
GrouperAPI
class to Grouper
grouper_query.GrouperQuery
into a query
method for grouper_admin.GrouperAPI
figshare_group
function to be part of commons
modulegrouper_delta_user
function into grouper
module. Note that this is heavily used in user_update
scriptImplemented in feature/87_grouper_refactor
(PR #117)
This is in feature/service_architecture
.
Currently, user_update
directly add members to figshare:active
.
Moving forward, figshare:active
should only had indirect membership. The direct membership makes it complicated. With new groups being added, it's perhaps better to create a new group that has those members directly and to indirectly add that group in figshare:active
. This is best as an interactive prompt option that adds for the names, etc.
Re-purpose add_groups
function to create_groups
and have it outside of add_grouper_groups
script [3def1a5, e3ca412]
Create a function called create_active_group
to enable creating such a group in figtest:group_active
stem [d501e26]
Add option in user_update
to use create_active_group
and add members [bd937dc]
Note that this is related to #65 as the limited use should also consider adding active members by at least creating the group and adding them there.
There are two parts of this feature request:
First, this most likely will be used with quota increases, but can also be used with figshare groups / "portals".
When a quota request is approved, there needs to be a query to retrieve the specific user and perform the synchronization based on the change in a file that provides manual changes.
The simple solution is:
uid
and old figshare Grouper attributes based on the EDSuaid
(NetID)delta
delta
The removal/addition can be done by creating a set from the LDAP query and stripping/adding the uid
and passing to delta
with the Grouper query. delta
will identify the drops and additions.
There are a number of steps:
user_update
script that accepts inputs, such as the NetID, and the specified changes to quota or/and portal [5361b6f, 14de970, cccc483, 1492ea1, 33603a7, 3594a5e, d3bce32]netid
[d3bce32], changing to the same portal [6699787], checking for whether the portal/quota exists first [65ea345, 225e580].root
option such that it simply removes current portal settings and defaults to the main/root level. This would require that we also remove it from the CSV files. Unsure if I should do something similar for quota [fd446c7 e4a2be0]ismemberof
attributes [0e560ac, 7d63c2b, 078385c, 7dd4195]ismemberof
attributes [078385c, 14de970]manual_override
and grouper_query
to avoid circular imports [17fae2c, 70ca119, f899bd0]Second, is the automation of handling changes to quota and/or portal based on an input database that is a CSV file. This ensures that when someone has a portal/quota change, it stays when the automated script is run. This requires:
config/figshare.ini
[531ca48]manual_override
module with a ManualOverride
class [1102aa1, 7af93a4]script_run
[b031f3e]The following scripts should be made executable (chmod +x
):
add_grouper_groups
get_org_code_numbers
grouper_permission_checks
script_run
user_update
This can be implemented as hotfix/v0.13.5
Also, updates to README.md for simpler execution.
Although outside this issue, for deployment aliases on patron server needs to be modified. This is not crucial but helps to keep things clean.
Describe the bug (REQUIRED)
When running script_run
the list of remove
and adds
are noted. For the adds
, this is generally consistent but the removal often includes accounts that don't apply due to the conditional requirement. This can be improved by stripping down to only the list of users that is in the group where it applies.
Version information (please complete the following, default in brackets): (REQUIRED)
0.12.2
feature/limited_set
Reproduction Steps (REQUIRED)
Steps to reproduce the behavior:
script_run
Expected behavior (Optional)
A less verbose message containing applicableA clear and concise description of what you expected to happen.
Screenshots (Optional)
If applicable, add screenshots to help explain your problem.
Note: This will be implemented as part of the feature/limited_set
branch
Retrieves Grouper membership information
Rather than run the full script, an option to limit it to certain organization codes, sub-portal names.
This should first be working with:
--groups
option) [3b1b713]--org_codes
option) [3b1b713]portals
. add_only
option in Delta
figshare:active
. Related to #72 [d055ecb, ef30dd3, dcb0a94]
create_active_group
functiongrouper_query
module to import [05d5fd0]The user_update
script automatically add portal
and quota
settings to the ManualOverride
dataframe regardless. We use this for individual handling; however, we should have an option to disable this with the default of adding it.
This is a quick fix and can be implemented in feature/limited_set
We had renamed DataRepository_research_themes
to ReQUIAM_csv
. As such, the current URL will return a 404 error.
The config file should include:
In addition, __init__
function is needed to run the script via a single command line
Describe the bug (REQUIRED)
The problem seems to stem from set vs list. The former is not in the same order as the latter. This creates some discrepancies.
By default, running the user_update
script with --portal root
will strip current portal association from the CSV. However, we now came across the case when a user is by default assigned to a given sub-portal but belongs at the top university level (i.e., 'root'). In this case, we need a simple fix that prompts whether to add 'root' to the portal file.
This will ensure that the override is in place and that the user is not returned to the sub-portal based on organization classification
Quota tiers are:
These queries will retrieve the full list and later be modified to exclude individuals with different tiers.
By default, script_run
executes on the production stems. An option should be added to disable this and perform updates on the stage/test.
Received this error:
File "/Users/cly/codes/UALibraries/DataRepository_patrons/tests/ldap_connection.py", line 89
return ldap_query
^
SyntaxError: invalid syntax
The use of .format()
did not have a complete set of parentheses.
read_manual_file
is a separate function and can easily be refactored into ManualOverride
as it is not used elsewhere. This would simplify input calls with a simple portal
or quota
to refer to which CSV files to read in.
Deploy actions/create-release action to create a new release on any tagged version.
Describe the bug (REQUIRED)
No production
option is set when calling GrouperAPI
, thus default is False
Version information (please complete the following, default in brackets): (REQUIRED)
0.12.1
master
pandas
version: 1.1.0requests
version: 2.22.0numpy
version: 1.18.0Reproduction Steps (REQUIRED)
Steps to reproduce the behavior:
user_update
scriptExpected behavior (Optional)
production
unless an option in the script grouper_figtest
indicate otherwise [8f0b7b7]manual_override.get_current_groups
needs a production
option [937aa85]manual_override.grouper_delta_user
will need to specify production
option. Default is True
[8f0b7b7]Screenshots (Optional)
If applicable, add screenshots to help explain your problem.
Additional context (Optional)
Add any other context about the problem here.
From DataRepository_patrons to ReQUIAM (ReDATA EDS Query and Update for Identity and Access Management)
Logs are currently stored locally. As such, this poses a problem for ephemeral design.
Persistent storage was provisioned. To store logs, we need:
This can be implemented as hotfix/v0.11.3
We need some statistics on the number of individuals in an organization code with Library privileges. We should retrieve:
ual-pgrp
Grouper groups
Since this touches on many codes/scripts, it's best to have this in release/v0.12.0
rather than a feature as that will include feature/grouper_api_admin
and feature/multi_user
The following actions are needed:
LD-Cool-P
for examples [d9d49fa]requiam
version, branches, and commit hash. The first two are already included. [9264182]Delta
GrouperAPI
grouper_query.grouper_delta_user
ldap_query.LDAPConnection
ManualOverride
, read_manual_file
, update_entries
, get_current_groups
GrouperQuery
add_grouper_groups
script_run
user_update
This was seen in release/v0.11.0
.
netid uaid portal
1 figshare01-test T440413500927388672 arts_design
Traceback (most recent call last):
File "/Applications/anaconda3/envs/ual-patron-groups/lib/python3.7/site-packages/pandas/core/ops/__init__.py", line 968, in na_op
result = expressions.evaluate(op, str_rep, x, y, **eval_kwargs)
File "/Applications/anaconda3/envs/ual-patron-groups/lib/python3.7/site-packages/pandas/core/computation/expressions.py", line 221, in evaluate
return _evaluate(op, op_str, a, b, **eval_kwargs)
File "/Applications/anaconda3/envs/ual-patron-groups/lib/python3.7/site-packages/pandas/core/computation/expressions.py", line 70, in _evaluate_standard
return op(a, b)
File "/Applications/anaconda3/envs/ual-patron-groups/lib/python3.7/site-packages/pandas/core/ops/roperator.py", line 13, in rsub
return right - left
TypeError: unsupported operand type(s) for -: 'set' and 'str'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "requiam/script_run", line 164, in <module>
ldap_members = mo.identify_changes(ldap_members, portal, 'portal')
File "/Users/cly/codes/UALibraries/ReQUIAM/requiam/manual_override.py", line 81, in identify_changes
outside_df['uaid'], 'remove', self.log)
File "/Users/cly/codes/UALibraries/ReQUIAM/requiam/manual_override.py", line 199, in update_entries
new_ldap_set = ldap_set - uaid
File "/Applications/anaconda3/envs/ual-patron-groups/lib/python3.7/site-packages/pandas/core/ops/__init__.py", line 1048, in wrapper
result = na_op(lvalues, rvalues)
File "/Applications/anaconda3/envs/ual-patron-groups/lib/python3.7/site-packages/pandas/core/ops/__init__.py", line 970, in na_op
result = masked_arith_op(x, y, op)
File "/Applications/anaconda3/envs/ual-patron-groups/lib/python3.7/site-packages/pandas/core/ops/__init__.py", line 450, in masked_arith_op
assert is_scalar(y), type(y)
AssertionError: <class 'set'>
Currently script_run
queries Grouper for membership in various Grouper groups. However, there's no failsafe check to make sure those group exists. We have within GrouperAPI the check_grouper_exists
method that can be easily used.
This is straightforward and should be a hotfix
Describe the bug (REQUIRED)
The quota.ual_ldap_quota_query
ldap3
query strings will return faculty
, grad
and ugrad
members. However, there is overlaps between these groups (e.g., a grad
member can also be staff). To fix this issue, it's best to mutually exclusive with the highest rank determining the quota level.
Version information (please complete the following, default in brackets): (REQUIRED)
feature/limited_set
Reproduction Steps (REQUIRED)
from requiam import quota
from requiam import ldap_query
# Get query string
grad = quota.ual_ldap_quota_query('grad')
faculty = quota.ual_ldap_quota_query('faculty')
ugrad = quota.ual_ldap_quota_query('ugrad')
ldc = ldap_query.LDAPConnection(ldap_host='eds.iam.arizona.edu',
ldap_base_dn='dc=eds,dc=arizona,dc=edu',
ldap_user='**override**', ldap_password='**override**')
# Query for members
faculty_uaid = ldap_query.ldap_search(ldc, faculty)
grad_uaid = ldap_query.ldap_search(ldc, grad)
ugrad_uaid = ldap_query.ldap_search(ldc, ugrad)
# Info
In [10]: len(faculty_uaid)
Out[10]: 23558
In [11]: len(grad_uaid)
Out[11]: 15450
In [19]: len(ugrad_uaid)
Out[19]: 46777
In [13]: len(faculty_uaid & grad_uaid)
Out[13]: 860
In [17]: len(faculty_uaid & ugrad_uaid)
Out[17]: 435
In [18]: len(grad_uaid & ugrad_uaid)
Out[18]: 4217
Expected behavior (Optional)
Exclusive list for grad
and ugrad
General issue for best practices. This ticket will remain open indefinitely.
The script could simply be executed after the main script script_run
to show summary data.
This would help to identify any sync issue.
This would be beneficial to also have it when user_update
script is executed. There may be a few minute delayed needed.
When performing portal and quota updates, please add the option for additional verbosity which not only includes summary information about what will change, but include the actual changes as well. This increases our capacity to troubleshoot issues should they arise.
For example, current log info when performing a portal update on the tech_eng portal (similarly for quota updates)
11:33:07 - INFO: Working on Technology & Engineering (tech_eng) portal
<snip>
11:33:11 - INFO: EDS size 1250
11:33:11 - INFO: Grouper size 1
11:33:11 - INFO: ldap and grouper have 0 members in common
11:33:11 - INFO: synchronization will drop 1 entries from grouper group
11:33:11 - INFO: synchronization will add 1250 entries to grouper group
11:33:11 - INFO: dry run, not performing synchronization
Desired logging info
11:33:07 - INFO: Working on Technology & Engineering (tech_eng) portal
<snip>
11:33:11 - INFO: EDS size 1250
11:33:11 - INFO: Grouper size 1
11:33:11 - INFO: ldap and grouper have 0 members in common
11:33:11 - INFO: synchronization will drop 1 entries from grouper group
11:33:11 - DEBUG: Dropping (name : employeeprimarydept )
Jane R. Smith : 0121
11:33:11 - INFO: synchronization will add 1250 entries to grouper group
11:33:11 - DEBUG: Adding (name : employeeprimarydept )
Mary A. Researcher : 2308
Mary B. Researcher : 2308
Mary C. Researcher : 2308
<snip>
11:33:11 - INFO: dry run, not performing synchronization
passwd -> password
Currently Grouper use is limited to addition and deletion of users in a Grouper group. We should extend the API use to include:
add_grouper_groups
script [793fd16]grouper_dict
feature/manual_override
as a MVP)The Grouper API can be found here.
A Good demonstration of the Grouper API is available in this github repository
Add a simple test that uses a test NetID account and a Grouper group to synchronize and then undo. This will confirm that the code is functioning.
This was listed in #16, but will be a separate issue.
Check list:
if args.test
Did a dry run which updated portals and quotas. Portals ran without errors but quotas failed. These are the last few lines of the log
21:58:36 - INFO: Working on grad quota : 536870912 bytes
22:00:38 - INFO: Removing : ['figshare01-test', 'kalcock']
22:00:38 - INFO: EDS size 16082
Traceback (most recent call last):
File "scripts/script_run", line 238, in
grouper_group=grouper_quota)
File "/home/mozart/codes/ReQUIAM/requiam/grouper_query.py", line 42, in init
if 'wsSubjects' in rsp.json()['WsGetMembersLiteResult']:
File "/opt/anaconda3/envs/figshare_patrons/lib/python3.7/site-packages/requests-2.22.0-py3.7.egg/requests/models.py", line 897, in json
return complexjson.loads(self.text, **kwargs)
File "/opt/anaconda3/envs/figshare_patrons/lib/python3.7/json/init.py", line 348, in loads
return _default_decoder.decode(s)
File "/opt/anaconda3/envs/figshare_patrons/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/opt/anaconda3/envs/figshare_patrons/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I also have a question about why 'figshare01-test', 'kalcock' and sometimes 'chunly' were going to be removed from every group. Presumably they should only be removed from groups they're actually in? It could be a symptom of me not fully understanding the log output. I can make a separate issue for this if needed
This is the main issue thread for Travis CI integration.
Steps include:
ualibraries/ReQUIAM
This is implemented in the feature/travis_ci
branch
The body message is not properly captured from the tag message.
Change:
# Use tag subject and message body for release description
body: |
${{ env.TAG_SUBJECT }}
${{ env.TAG_BODY }}
To:
# No body: defaults to tag message and body
Also, need to strip the "Get tag info" step as that is no longer needed
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.