philloooo / github-tests Goto Github PK

• enable reqtimeout for slowloris vulnerability
• turn on secure for cookie
  partially solve uc-cdis/cloud-automation#219

Just read through this post
The end of that post has a good run through in java.

We are missing step 1, verifying the origin url. And should be able to gleam the information from one/multiple of these

local_settings is not installed as a module under fence in prod

AS an Actor I want to do this so that this business value is acheieved

• Allow linking of a user's personal Google account to their User identity
• GET /link/google will run user through oauth with google to link their Google account
• redirect and other needed information is saved in the session before the oauth flow with Google
• after "linking", the endpoint will also add the Google account to their Proxy Group so it has access to the same things as the user. This addition is temporary, by default the expiration is 24 hours
• Once authN and adding to proxy group is done, fence will redirect back to client (with errors if anything bad happened)
• error handling is done similar to oauth2 where there exist query params in the url (error and error_description)
• PATCH /link/google will extend the expiration of a user's linked Google account to 24 hours from current time
• DELETE /link/google will remove the account linkage permanently

NOTE:

• needs uc-cdis/cirrus#10

resolves fencePXD-28

Hi,

I'm trying to set up Fence to use with my project, and I'm having trouble understanding some of the process. The installation instructions simply tell you to fill out local_settings.py. Some of them are obvious e.g. database credentials, google oauth, but others are not, for example:

STORAGE_CREDENTIALS = {
    "cleversafe-server-a": {
        'backend': 'cleversafe',
        'aws_access_key_id': '',
        'aws_secret_access_key': '',
        'host': 'somemanager.osdc.io',
        'public_host': 'someobjstore.datacommons.io',
        'port': 443,
        'is_secure': True,
        'username': 'someone',
        'password': 'somepass',
        "calling_format": OrdinaryCallingFormat(),
        "is_mocked": True
    }
}

I'm not sure what a "cleversafe-server" is, or how I would go about setting one up if it is required. I would appreciate any help you can render in settings this up.

Allow redirect query param on data download endpoint

Resolves PXD-112.

Hard-code attempt to load /var/www/local_settings.py if the module was not found in the first attempts. This way the script in the kubernetes pod is always able to locate the settings (since the file is mounted to that location by the deployment settings).

Creating of Google Buckets for STaging. Alex - Please feel free to delete this if its duplicate.

Resolves PXD-66.

Notable changes:

• Removes the original get_current_user() usage; instead, uses current_user proxy which will do at most 1 lookup for user per request and store the user info from the database as a dictionary
• Changes endpoints to use require_auth (instead of login_required) and use the current_token proxy where possible to get necessary information from the token instead of database
• Changes MOCK_AUTH to be an entire fake token instead of a boolean, to support "stateless" mocked auth

Changes left:

• Change storage manager setup to use token (place that only needs username)
• Appease codacy

To resolve PXD-165.

AS an Actor I want to do this so that this business value is acheieved

no role hierachy, just allow CRUD roles and attach/detach roles to users

we need to support generating presigned url for buckets that's owned by another aws account. This third-party aws account will create a role that has read access to data, and allow our aws account to assume the role.

test description

/.well-known/jwks endpoint was returning n and e as base64url-encoded, not base64urlUInt-encoded. This changes Keypair to use python-jose for creating the required fields in the JWK response, which fixes the problem.

• find user by access_token cookie
• just clear all the cookies including access_token , session cookie, and others(eg: jupyter cookie)

Changes:

• Added OAuth2TestClient class to consolidate auth asserts (notably the POST -> 200, GET -> 302 for authorization endpoint) and and redid tests using this
• Setting the authorization endpoint parameter prompt=login will now clear cookies and redirect to the authorization endpoint again to redo authentication.
• Removed test_prompt_login_no_authn; now redundant since prompt=login redirects to authorization endpoint so the normal tests for auth cover this.

Add all TTL settings into local_setting
r [~rudyardrichter] @philloooo

WIP

feature request from SBG:

Regarding the dbGap data we need `email, dbGap userId, first name, last name, phone, role

These fields are included in dbgap csvs from their ftp site but we aren't parsing them before

• modify fence-create script and dbgap syncing script to allow pass these information (email, first_name, last_name, phone_number, dbgap_role)
• These information will also need to be included in both user endpoint and the jwt id token

Alex to create more issues, if required based on ISB Phased implementation doc.

https://docs.google.com/document/d/1NcgQU9aRkpEpye9jFRhKXDFoUmNJ80gF8wddTe8_4RU/edit#heading=h.veednt7vc9lc

• change /cdis to /api for API keys
• refactor endpoints into separate resources with get, post, delete for the access keys
• add all creds endpoints to swagger doc, show /cdis as deprecated

NOTE: this contains changes from uc-cdis/fence#158 as well, hoping that gets merged first

…d endpoint, change POST to GET

AS an Actor I want to do this so that this business value is acheieved

• Modify upload/download endpoints to generate signed urls for Google documents (e.g. gs://blahblahblah)
• Refactor upload/download and signed url endpoints to more clearly support different protocols in an OO model
• Modify previous test suite to run the same tests against s3 and gs locations
• additionally, test when acl information is in index document as acl field AND when acl is in metadata["acls"] field

Details on implementation can be found here: https://docs.google.com/presentation/d/1lZjKcKqxsjXAceOzM-vj0rPs2dm3fT4V2EaApXjMb_M/edit?usp=sharing

Details on expiration times for various resources in fence here:
https://docs.google.com/spreadsheets/d/1fyOos1WFjddjPVVBA-i6wrCmUjAB4NipaKq4Re9rgCY/edit?usp=sharing

Depends on uc-cdis/cirrus#11

https://github.com/lepture/authlib/releases

Now supports OIDC server, some breaking change in 0.6 (follow guides/info to update our code)

coupled with
uc-cdis/indexd#109

The Form submission that happens when a user clicks 'yes' to confirm to grant the oauth2 client access to their resources, send csrftoken via form data.
Right now our reverse proxy only does csrf verification check by making sure the csrftoken in headers matches what's in cookie. so it doesn't look for the csrftoken in the body.
We wouldn't want the reverse proxy to look into the request body because it then needs to cache the whole body in memory or disk(body can be huge).
And you can't modify the header for submiting form, unless you make POST via ajax call.
Making ajax call doesn't trigger browser reload, but browser also doesn't follow redirect to another domain via ajax call. So our oauth2 flow breaks here(fail to redirect to client with a code)

one way to fix:
make /oauth2/authorize endpoint for POST confirm return 200 instead of 302, and have the location in the body instead of header. Then browser javascript handles redirect

Newest version, breaking changes: https://github.com/OAI/OpenAPI-Specification

A converter that may be useful to check out:
https://github.com/Mermade/swagger2openapi

Blog post about differences
https://dev.to/mikeralphson/comparing-openapiswagger-20-and-300-rc1

pyca/bcrypt#105
happens in current master fence
but works if I uninstall and install the same version of bcrypt==3.1.4

it's hardcoded to key-01 now
we should generate random kid like: gen3-<randomid> during the fence setup process in cloud-automation and use that in fence

bump for uc-cdis/userdatamodel#36

in https://dev.planx-pla.net:

• after logout, it redirects to /user endpoint
• there are other cookies in the same domain, eg: jupyterhub token cookie, we should just clear all the cookies on logout
• add regression tests

resolve PXD-146

For uc-cdis/authutilsPXD-4.

Use the authutils text fixtures from uc-cdis/authutilsPXD-8 (instead of loading keys from files), so the tests/resources/keys files are no longer necessary for testing.

Depends on uc-cdis/authutilsPXD-16.

1. Create Google domain names - Trevar
   QA domain name, Production domain name

• don't try to resolve scheme that's not supported yet (eg: gs)
• if the bucket is configured to be '*' (without creds) in settings, return the http url
• latest pip is broken in docker build, pin to 9
• remove deadcode (http conf)

Needs to use InCommons, which uses eduGAIN (eduGAIN for GenoMEL)

Allow login Shibboleth endpoint to pass an alternative IdP URL as a URL parameter

Create a new endpoint that returns shibboleth endpoint (would be BASE_URL + "Shibboleth.sso/DiscoFeed")

• Add POST /link/google
• Add details to /link/google
• More restful interpretation of service account registration
• /google/service_accounts and /google/service_account/{id}
• GET (list service accounts for user)
• POST (new service account)
• Then for specific service accounts: GET, PATCH (change dataset access), DELETE

P.S. I've been using https://editor.swagger.io/ to visualize

• reject invalid identity provider settings
• don't require jwt for login session
• add regression tests

Resolves PXD-178