subutai-io / cdn Goto Github PK

We need to re-upload all templates on repository to make it:

• signed
• owned by "subutai" user

After this we can remove "public" user to correct logic and change level of "Not signed template" from warning to critical error

userid = 7e6a231303b69374c6c0e0a4f3df2ffcfc298d3a
Quota: 2039.98 Mb used

We need to allow users upload private artifacts, that will be accessible only from their account.
User should be able to choose private or public scope for his artifacts.
Private artifacts should be accessible only using token.

From https://cdn.subut.ai:8338/kurjun/rest/raw/list i got the following files

{"id":"raw.59e7a306b4f4e6980e4145b03f378254","name":"subutai_4.0.0_amd64.snap","size":18590328,"md5Sum":"59e7a306b4f4e6980e4145b03f378254","version":"1462638155","fingerprint":"public"},

{"id":"raw.c027fb316d76fea2d550d3708f0f31c0","name":"subutai_4.0.0_amd64.snap","size":18224582,"md5Sum":"c027fb316d76fea2d550d3708f0f31c0","version":"","fingerprint":"public"}

and when requesting info for each of them i'm getting same result

After recent "info" endpoint changes added to make it return latest management template, this endpoint now returns single random template if none of name nor id was passed as argument:
https://eu0.cdn.subut.ai:8338/kurjun/rest/template/info

Need to fix it and make it return either full list of template or empty page

Replace paging logic where page number passed as page number, page size to index, row count as Abdysamat suggested:

But if we switch to index, row count approach, we can request row count + 1 and detect that there are more rows exists

info about if there exist more rows is needed when showing Next page or Load more button in table with paging

index,row count approach is more flexible than dozen, dozen length

For usage of Kurjun by clients we need to be able to supply either permanent or long lived tokens that are precreated by user and managed via Hub UI.
Analogous to amazon keys that are used by automated tools.

Function that provides latest file by name, returns result from all repositories. That brings some inconsistency problem.
We need to change this function to work well for all requests.

Here are the REST endpoints which are used by the Hub.
Lines marked with "+" at the line beginning, those works.
Lines marked with "-" not working, the reason is written at the same line.

-POST: rest/identity/user/add       no url
reponse:    [text]


-GET: rest/identity/user/get-active     no url
response:   [text]


-POST: rest/identity/user/auth      no url
response:   [text]


+GET: rest/template/list
response:   [json]


-POST: rest/template/upload         can't test because can't authorize (401 Unauthorized)
request: [file stream & text]
response:   [text]


+GET: rest/template/get
response: [file stream]


+GET: rest/template/info
response:   [json]


-DELETE: rest/template/delete       can't test because can't authorize (401 Unauthorized, Empty token)
response:   [http status]


-GET: rest/deb/list                 don't return following entries: "id", "extra" (from extra file size was being taken)
response:   [json]


-POST: rest/deb/upload              can't test because can't authorize
request: [file stream & text]
response:   [text]


-GET: rest/deb/info                 without ID is not working
response:   [json]


-GET: rest/deb/get                  without ID is not working
response:   [file stream]


-DELETE: rest/deb/DELETE            without ID is not working, can't test because can't authorize
response:   [http status]


+GET: rest/file/list
response:   [json]


-POST: rest/file/upload             can't test because can't authorize
response:   [text]


+GET: rest/file/get
response:   [file stream]


-DELETE: rest/file/delete           can't test because can't authorize
response:   [http status]


-GET: rest/repository/list          no url
response:   [json]


-GET: rest/relations/list           no url
response:   [json]


-GET: rest/relations/object/{id}        no url
response:   [json]


-PUT: rest/relations/trust          no url, can't test because can't authorize
response:   [http status]


-POST: rest/relations/{id}          no url, can't test because can't authorize
response:   [http status]


-DELETE: rest/relations/{id}        no url, can't test because can't authorize
response:   [http status]

https://cdn.subut.ai:8338/kurjun/rest/raw/info?SubutaiTray this request will return JSON OBJECT if there is one SubutaiTray file on gorjun... And will return JSON ARRAY if there are more then one SubutaiTray files on gorjun.
It causes many problems. For example we need always check if we received array or single object. It's harder to maintain and write some scripts.

It's better to return array with one element if there is only one file on gorjun.

Following lines should be removed from Gorjun when SS will be ready to work without it:

w.Header().Set("Access-Control-Allow-Origin", "*")
w.Header().Set("Access-Control-Allow-Credentials", "true")

Now any user can get full list of artifacts with single public API URI.
With big number of artifacts it will require much CPU time.
We need to replace getting full list of templates with search module. If number of results is too big we can use paging for responses.
Public artifacts owned by Subutai can be showed as default value if no request parameters was specified.

Now we have mechanism where token is obtained once by clients and supplied to env creation workflow.
This workflow uses token in 2 steps: import templates and clone containers.
Import template can take significant amount of time base on connection speed.
Thus, token might get expired till the workflow reaches the clone containers step.
We need to be able to periodically refresh a token somehow so that its TTL is prolonged.

If file name contains spaces, when download it in Mozilla Firefox, it doesn't show full name with extension, only part of name until first space.

In result, file will be saved with incorrect name and without extension.

In Google Chrome it works ok:

Current implementation includes only hardcoded configuration options.
We need to have ability to change service configuration through config file.

All CDN artifacts should be digitally signed by their owners.
This also implies checking the digital signature in addition to the checksum to confirm authenticity.

Issue moved from subutai-attic/subos#492

Uploading non-deb file to gorjun's apt repository causes daemon panic:

Need to add initial check before trying to unpack and parse uploaded file

Latest changes in gorjun broke downloading files by name. It should be fixed, requesting file by name will provide latest file with particular name

Currently any registered user have unlimited storage size and this is definitely bad.
We must add user quota to protect Kurjun from running out of disk space. Also, we should count storage usage "on the go" to get rid of possible problem with uploading huge amount of data as single file

Now we are allowing to user add only single GPG key. If user loosing his key there is no way to restore it.
User should be able to add several keys to his account.
All keys should have equivalent access.
User should have ability to manipulate his keys.

Empty upload request causes panic at gorjun

Need to make additional arguments sanity check

Now we have duplicates of some fields that confusing people.
Artifact should be identified like user/filename or just hash.
public or private scope in ID is useless. If user is not authorized he should see only public artifacts. If user using token for request he should see all artifacts allowed to him.

{
    "md5Sum": "5d72cdcaa2092e4ff013a0eb06039291",
    "id": "public.5d72cdcaa2092e4ff013a0eb06039291",
    "ownerFprint": "public",
    "owner": ["public"],
    "parent": "openjre7",
    "name": "hadoop",
    "size": 46300982,
    "version": "4.0.0"
}

Since one file can be stored in different repositories, we should add option to supplement file's metadata with repo-specific information. For example, if deb file was uploaded to "raw" and "apt" repositories, metadata should include both sets of information (currently file contains only first repo metadata). Not critical, but in some cases may cause unexpected behavior of related projects.

Now we have some URI in gorjun API that duplicates each other.
For example:
/kurjun/rest/template/download
/kurjun/rest/template/get

/kurjun/rest/raw/download
/kurjun/rest/raw/get
/kurjun/rest/file/get

We need to remove useless duplicated to make API simple.

We will need to show to SS user detailed information about a template in UI such as version, architecture etc. This way she can choose appropriate template to use.

When I am getting a file from gorjun and specify it's name and owner - it will return file even if owner was not correct:

https://eu0.cdn.subut.ai:8338/kurjun/rest/raw/get?id=<ID>&owner=<SOME_RANDOM_USER>
https://eu0.cdn.subut.ai:8338/kurjun/rest/raw/get?name=<NAME>&owner=<SOME_RANDOM_USER>

This doesn't happens when I use the following form:

https://eu0.cdn.subut.ai:8338/kurjun/rest/raw/<SOME_RANDOM_USER>/<NAME>

Right now embedded Kurjun downloads CDN artifacts from from nearest CDN nodes. After implementing #22 we should enable peer-to-peer downloads of large public artifacts from all CDN nodes and other embedded Gorjun instances. Implementing digital signatures is a requirement as well for this feature: #21.

Issue moved from subutai-attic/subos#493

Gorjun REST API endpoint info should support filtering by "verified" owners.
"Verified" owners should be a list of system or well knows users.
With "verified" argument user can request templates by name only, without specifying owner.

A lot of properties in artifacts info is not used. Some of them just duplicates each other.
We need to make list of properties simple and effective.

{
    "architecture": "AMD64",
    "configContents": "",
    "id": "public.5d72cdcaa2092e4ff013a0eb06039291",
    "md5Sum": "5d72cdcaa2092e4ff013a0eb06039291",
    "name": "hadoop",
    "ownerFprint": "public",
    "package": "",
    "packagesContents": "",
    "parent": "openjre7",
    "size": 46300982,
    "version": "4.0.0",
    "owner": ["public"]
}

If user uploading file to one of the repositories, this file (with same hash sum) cannot be added to another repo.
We need to fix this problem. User should be able to upload same file to any repo in any time.

To improve CDN caching functionality and reduce transferred data, we need to add special HTTP headers to responses of file download.
If-Modified-Since
Last-Modified

We need to start integrating embedded Gorjun to replace Kurjun.
Also we need to add a swtich to subutai import/close command to be able to import/clone template by its id.

Issue moved from subutai-attic/subos#486

Marat Bediev
it will be good, if i can calculate version on build stage,
and after deploy, builder will check calculated version with version returned from cdn node

Timur Zununbekov
build short commit id into binary using makefile and then print it on some rest endpoint?

Marat Bediev
for builder it will be enough

We can add custom tags to the LXC configuration file like this:

subutai.tags = bigdata, apache, database

Kurjun should expose this information as JSON array and support filtering by this tags.

On updating information about template tag signature getting removed.
We need to fix this problem and keep both fields in artifact information.

It would be convenient to have info about uploaded file in auth/sign output, like:
File (fileid) has been signed by userid

Gorjun's function which should return latest template works with "search" bucket which works as simple index table. Before 4.0.10 release everything was ok - we took last entry from index and it was latest version, but now it is not (4.0.10 goes before 4.0.9):

[management-subutai-template_4.0.0-master_amd64.tar.gz] 
[management-subutai-template_4.0.1-master_amd64.tar.gz] 
[management-subutai-template_4.0.10-master_amd64.tar.gz] 
[management-subutai-template_4.0.2-master_amd64.tar.gz] 
[management-subutai-template_4.0.3-master_amd64.tar.gz] 
[management-subutai-template_4.0.4-master_amd64.tar.gz] 
[management-subutai-template_4.0.5-master_amd64.tar.gz] 
[management-subutai-template_4.0.6-master_amd64.tar.gz] 
[management-subutai-template_4.0.7-master_amd64.tar.gz] 
[management-subutai-template_4.0.8-master_amd64.tar.gz] 
[management-subutai-template_4.0.9-master_amd64.tar.gz] 

Need to handle this situation somehow

When searching of filtering files by their name, letter case should be ignored.

Kurjun server for all 3 environment should be updated after each commit to particular branch.

• dev branch should be deployed to the devcdn.subut.ai;
• master branch should be deployed to the mastercdn.subut.ai;
• latest release version should be deployed to the cdn.subut.ai.

On starting gorjun checking dir for db - /opt/gorjun/data/db,
and if db is not exits return FATAL error.

but no errors if files dir does not exist
service starting successfully, and return 500 on upload

We have following REST endpoint for checking if user is registered or not in Kurjun:
https://cdn.subut.ai:8338/kurjun/rest/auth/key?user=C8BC87C879151C7F3FABE1C85ECE9EE17445322D

Currently it accepts user parameter only in lower case.
I think better to make it case insensitive.

Looks like case with multiple owner for the same file is not working properly - deletion of the file by the first owner also removes this file for other owners too. Need to check and fix it.

We need to be able to clone container by ID (even private one) without supplying token for the case when template is already imported to RH. See original discussion here #94

If you run gorjun, when other process already running
2nd process will run without any error messages
but will not listen any port

We need to be able to search files by their name in Kurjun. Searching should find files by any occurrence of search string in file name, with ignoring letter case.

Also may be would be useful to make search endpoint to accept filtering parameters.
Kurjun already has filter by owner field, following fields also candidates for filtering:

• in templates:

1. architecture
2. parent template
3. version

• in raw files:

1. version

Since Kurjun identifies files by their md5 hash, it doesn't allow uploading same file twice, it adds new owner to existing file, as I understand. But there is case, when two users may upload same file with different name, and second user will not find his file by second name, because it is showed in list with first name.
Files should be displayed with respective name for each uploaded user.

There is a possibility to make raw file and template private or shared with some people.
If template of raw file was made private or shared, it shouldn't be listed in public list of templates of raw files. Currently all files are listed in public list. And if user, who is not owner or not in shared list tries to download private file, Kurjun responses with "File not found" message.