cryptonomic / nautilus-cloud Goto Github PK
View Code? Open in Web Editor NEWCloud interface for Nautilus infrastructure
Cloud interface for Nautilus infrastructure
GET /github-callback
path with POST /users/github-init
.code
parameter can be passed in a body request/users/me
(without the current redirect)In favor of timed tokens.
Users must be able to see a list of changes that have occurred to their account. The following field should be captured.
A few changes are required as to how roles are applied to routes. These are:
In addition, given the current implementation, the create user route should be disabled as we have no immediate use case to allow users to register outside of an authentication provider mechanism.
Look at ForgingBlock and propose a design for integration with NC.
Embedded PostgreSQL needs ~20 seconds to run. We have 4 tests using that database which means that we spend ~80 seconds waiting for the database to run when the test run in 10 seconds. We can optimize it by running the embedded database only once and reusing it for further tests which should reduce the time of running all tests 3 times (from 90 to 30 seconds).
Introduce a new administrator only endpoint to search for users given a partial or full apikey and an optionally defined environment parameter.
Users should be assignable to multiple tiers according to their needs. The API should therefore support creating and modifying tiers and assigning / reassigning users to these tiers.
We can start with the following tiers and sub-tiers:
Please keep in mind that in the future switching tiers might require payment by users.
Add the necessary API routes / endpoints to allow a user to refresh his keys for a given environment i.e dev or prod. Note that keys are never deleted from the database.
Additionally routes returning key information for a user should also output resource information with the key (which service, platform, network, description etc) as opposed to just the resourceId.
Finally, add a simple HTML page demonstrating this functionality. The mock ups may be referred to for this exercise.
Add a route to permit users to delete their account. Upon successful delete operation, they should be logged out immediately and the back end should ensure that all associated keys are suspended.
For this specific issue, the delete operation will be limited in scope to removing the specific row from the Users
table. This has been decided in the presence of @vishakh and @anonymoussprocket .
Two keys must be created on first login for the user.
For this 4 static resource entries may be added to the resources table, one for each service (conseil/tezos) and its environment/platform (prod/dev).
Keys maybe generated by taking random chars and seeds as input and using a fixed length hash functioning to transform them - or - any other suitable mechanism that will ensure no collision of keys between users.
Start the project by initializing a new Scala SBT project with a basic API and database tables.
If there are no objections, we can start with the same stack as Conseil, i.e. Akka HTTP with Slick and Postgres. We are, however, completely open to any new choices presented here.
This is the initial API spec:
Route | Method | Description |
---|---|---|
apiKeys | GET | Gets all API keys |
apiKeys/{apiKey} | GET | Validates given API key |
users | PUT | Add new user |
users/{user} | GET | Fetches user info |
users/{user}/apiKeys | GET | Get all API keys for given user |
users/{user}/usage | GET | Gets the number of queries used by the given user |
The tables should be created accordingly.
This issue covers the discussion on how to bootstrap administrators on the system given that none exist at the time of deployment.
Retail users will be sub-accounts of some house userid. PK on this table would be userid+managerid.
Column | Data Type | Description |
---|---|---|
userid | numeric | FK into users table |
managerid | numeric | FK into users table (userid) |
effectivedate | timestamp | record date |
Column | Data Type | Description |
---|---|---|
tierid | numeric, primary key | sequence auto-increment pk |
tier | text, not null | tier name |
description | text, not null | long-form service description |
monthlyhits | numeric, not null | cumulative number of requests per calendar month - static window |
dailyhits | numeric, not null | 24-hour window request allowance - sliding window |
effectivedate | timestamp, not null | validity period start |
enddate | timestamp, not null | validity period end |
We would like to record high level usage statistics within the Nautilus Cloud database for non real time uses such as billing etc. This data should be stored in a separate table.
A proposed schema for this table is:
The above schema may be enhanced as required.
A proposed algorithm for collecting this data is as follows:
Given:
A
, the time interval after which the algorithm runs.B
, the time interval within which we are attempting to gather statistics.Then:
A
wake up and gather data.period_start
, period_end
).period_start
to period_end
and b) period_end + B
.A few notes :
A
and B
may not necessarily align i.e its possible A >> B.The CORS support should be configurable
This is a follow up to #2.
Add support for roles, starting simply with 'user' and 'administrator' roles. Roles should determine access to route / HTTP method pairs. The 'user' role can be the default for now.
Add additional routes for role management.
Area's to cover :
Column | Data Type | Description |
---|---|---|
resourceid | numeric, primary key | sequence auto-increment pk |
resourcename | text, not null | |
description | text, not null | |
platform | text, not null | 'Conseil', 'Tezos', etc |
network | text, not null | 'prod', 'mainnet', etc |
The module should be able to do the following:
Column | Data Type | Description |
---|---|---|
keyid | numeric, primary key | sequence auto-increment pk |
key | text, not null | 32-char API key |
resourceid | numeric | FK reference to resources table |
userid | numeric | FK reference to users table |
tierid | numeric | FK reference to tiers table |
dateissued | timestamp | date from which the key is active |
datesuspended | timestamp | a means of terminating a key |
We can leave the code in the repo as it may be easier for local testing, however in which case some mechanism should be available to disabling it.
The back end user registration flow should work as such:
The following attributes are required and will necessitate a schema change:
Other requirements
userRole
should be removed from this endpoint.Reference: Discussion at Cryptonomic/Nautilus-Cloud-Ui#5
Metering functionality (#11) will need to be instantaneous. To accomplish this we should keep the necessary at a in memory. This can likely be done with Postgres configuration to force contents of the related tables to reside in memory
Column | Data Type | Description |
---|---|---|
userid | numeric, primary key | sequence auto-increment pk |
username | text, not null | login name |
useremail | text, not null | |
userrole | text, not null, default('user') | role, probably enum, other values might be acctadmin (corp customers may want control over their sub-accounts), infraadmin (require 2FA) |
registrationdate, not null | timestamp | validity period start |
accountsource | text, nullable | might be enum, values could be web, campaign id, "internal", "manual" (when we ourselves might make accounts for people) |
accountdescription | text, nullable | to go with account source, something we may enter |
Incoming requests need to be gated to confirm service availability for the provided key, the service type and tier it is assigned, and compared against total monthly, and windowed utilization numbers.
The docker build process currently copies all jar files available in the build directory.
This is undesirable and the process should only take the assembled uber jar for NC.
Additionally, a update to documentation is required for handling of code changes and local docker images.
This ticket is the first in many towards tier based services support in NC.
The first step is to establish plumbing from InfluxDB to NC. The following flow is envisioned to achieve this goal:
Notes:
relates to #23
While running tests we can experience OutOfMemoryError: Metaspace
. It looks that we have some memory leak because Metaspace usage seems to grow incessantly on each test run inside of sbt console. When you're running them with sbt test
it's ok because a JVM is shut down after the job is done so it works now but it will stop working when we'll have more tests.
Possible actions:
embedded-postgres
, we should consider replacing it with https://www.testcontainers.org/modules/databases/postgres/ which is a nice thing to do anyway because embedded-postgres
is not maintained anymore,Following is the list of tooling / runbooks required to improve the QoL of folks engaging in devops.
api key
similar to what NC and conseil use to exchange the key list.API Keys when being generated should be validated against the DB to ensure that a previously generated key is not accidentally reused.
Introduce new / modify existing endpoints to support the following functionality required for administration:
An endpoint for fetching the complete list of users. The data returned for now can contain the userid, registration date and email id at the minimum.
Additionally, a search functionality should be present which takes in complete or partial userId or email and returns the same information as mentioned above. This can be a separate endpoint or baked into the one above.
An endpoint that permits accounts to be deleted . The semantics of such an operation is the same as what is present in code at this point in time.
All of the above actions must be limited to users with admin roles. Pagination support is desirable.
Column | Data Type | Description |
---|---|---|
userid | numeric, primary key | FK into users table |
oauthprovider | text, not null | OAuth service (OpenID, Github) |
oauthtoken | text, not null | OAuth token |
effectivedate | datetime, not null | |
duration | numeric, not null | seconds til oauth token expiration |
The ToS / PP may change over time, we need functionality to be able to re accept the terms.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.