lacmus-foundation / lacmus-web-api Goto Github PK

View Code? Open in Web Editor NEW

11.0 3.0 2.0 281 KB

API to the lacmus project

License: GNU General Public License v3.0

Dockerfile 3.78% Python 94.97% Shell 1.25%

python dotnet-core backend web-app

lacmus-web-api's Introduction

lacmus-web-api

FTP API (server) to the lacmus project

lacmus-web-api's People

Contributors

Stargazers

Watchers

Forkers

sharov-am ipovalyaev

lacmus-web-api's Issues

create dev stage

create dev server
configure OS & install packages:
- docker + docker compose
- open stack client + zerotile
give ssh access

Krakend enhancements

We need some enhancements:

Add the ability to proxy fields of the jwt token to the backend (luraproject/lura#244)
Add proxy error status codes from the backend
Add the ability to execute a request sequence with an unsafe request (post / put) at the end (luraproject/lura#194 (comment), luraproject/lura#359 (comment))

api planning [epic]

draw the general architecture of the application, divide the application into services
describe data model
describe the functionality of api
split existing services into clusters for using applications with openstack and kubernetes
consider each service separately and describe its api (using openapi)
write wiki documentation

split existing services into clusters

split existing services into clusters for using applications with openstack and kubernetes
usefull links

https://opendatascience.slack.com/archives/CE8869WEQ/p1594313091006700 - ods message in #ml_piepline

describe the functionality of api

[EPIC] create Predict API service

Using Postgres db (or MySql) and ef .core create compression microservice. Service shoud compree image, store it and return id to user. Add support for VOC format for image annotations and store it in json(xml) format.

Move code from web-storage

Move code from https://github.com/lacmus-foundation/lacmus-web-storage or add as submodule

ci/cd

make a CI using git actions or something else

write wiki documentation

add jwt

add user authorization

Allow ftp users (admins) to clean result folders

During production use was found users usually upload photos by batch, wait for result and then upload another. It would be useful to let then clean also processing results in found and empty folders of their project

FTP API crushes and not processing more files under some curcumstances

Seems under unstable connection it could happen 2 messages IN_CLOSE_WRITE coming from inotify for the same file, which causes race conditions and crush of monitoring thread, so processing of files for this project stops
Also it would be nice, if ftp server after restart automatically picked up of files in the project dirs (currently files need to be reuploaded)
also would be reasonable to have some simple monitoring, checking files in project dirs and DB are beeing processed

2021-03-28 12:47:13,493 Events received in stream: ['IN_CLOSE_WRITE']
2021-03-28 12:47:13,493 File DJI_0681.JPG noticed by notify thread for project 7 in /etc/ftp_projects/project-7
2021-03-28 12:47:13,493 verifying file DJI_0681.JPG in /etc/ftp_projects/project-7
2021-03-28 12:47:13,494 tag: ImageDescription (270) - type: string (2) Tag Location: 22 - Data Location: 1012 - value: b'DCIM\\100MEDIA\\DJI_0681.JPG\x00'
2021-03-28 12:47:13,494 tag: Make (271) - type: string (2) Tag Location: 34 - Data Location: 1039 - value: b'DJI\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
2021-03-28 12:47:13,494 tag: Model (272) - type: string (2) Tag Location: 46 - Data Location: 1069 - value: b'FC220\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

2021-03-28 12:47:13,503 uploading file DJI_0681.JPG in /etc/ftp_projects/project-7
2021-03-28 12:47:13,526 Starting new HTTP connection (1): localhost:9000
2021-03-28 12:47:13,528 http://localhost:9000 "GET /project7?location= HTTP/1.1" 200 128
2021-03-28 12:47:13,893 http://localhost:9000 "PUT /project7/DJI_0681.JPG HTTP/1.1" 200 0
2021-03-28 12:47:13,894 Collected user_id **** for file DJI_0681.JPG. Creating record in DB
2021-03-28 12:47:13,961 Removing /etc/ftp_projects/project-7/DJI_0681.JPG

....
2021-03-28 12:47:14,084 Events received in stream: ['IN_CLOSE_WRITE']
2021-03-28 12:47:14,085 File DJI_0681.JPG noticed by notify thread for project 7 in /etc/ftp_projects/project-7
2021-03-28 12:47:14,085 verifying file DJI_0681.JPG in /etc/ftp_projects/project-7
2021-03-28 12:47:14,085 Exception while verifying file DJI_0681.JPG in /etc/ftp_projects/project-7
Traceback (most recent call last):
  File "./core/processing.py", line 18, in process_incoming_file
    im = Image.open(os.path.join(path, file_name))
  File "/usr/local/lib/python3.8/dist-packages/PIL/Image.py", line 2891, in open
    fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/etc/ftp_projects/project-7/DJI_0681.JPG'
2021-03-28 12:47:14,103 moving file to error
2021-03-28 12:47:14,103 moving from /etc/ftp_projects/project-7/DJI_0681.JPG to /etc/ftp_projects/project-7/error/DJI_0681.JPG 
2021-03-28 12:47:14,226 Cleaning-up inotify.

learn the technology
run it on dev server
build configuration management via ansible

add open-api

minIO

learn technology
make Dockerfile with minIO

Krackend as gateway

remove C# ocelot code and add krackend as api-gateway

[EPIC] make and configure dev stage

create dev server
- configure OS & install packages
configure
- PostgreSQL DB
- Ansible
- Docker + Docker-compose
- MinIO
- ML worker
- Open stack client