Code Monkey home page Code Monkey logo

immuni-backend-exposure-ingestion's Introduction

Immuni Backend Ingestion Service


Table of contents

Context

This repository contains the source code of Immuni's Exposure Ingestion Service. More detailed information about Immuni can be found in the following documents:

Please take the time to read and consider the other repositories in full before digging into the source code or opening an Issue. They contain a lot of details that are fundamental to understanding the source code and this repository's documentation.

Installation

This backend service comes with an out-of-the-box installation, leveraging Docker containers. Please ensure that Docker is installed on your computer (docker-compose version >= 1.25.5).

git clone --recurse-submodules [email protected]:immuni-app/immuni-backend-exposure-ingestion.git
cd immuni-backend-exposure-ingestion/docker
docker-compose build \
    --build-arg GIT_BRANCH=$(git rev-parse --abbrev-ref HEAD) \
    --build-arg GIT_SHA=$(git rev-parse --verify HEAD) \
    --build-arg GIT_TAG=$(git tag --points-at HEAD | cat) \
    --build-arg BUILD_DATE=$(date -u +"%Y-%m-%dT%H:%M:%SZ")

The backend service can then be launched.

docker-compose up

At this point, the service is available at http://0.0.0.0:5000/swagger.

For more information about how the project is generated and structured, please refer to the Contributing section below.

Contributing

Contributions are most welcome. Before proceeding, please read the Code of Conduct for guidance on how to approach the community and create a positive environment. Additionally, please read our CONTRIBUTING file, which contains guidance on ensuring a smooth contribution process.

The Immuni project is composed of different repositories—one for each component or service. Please use this repository for contributions strictly relevant to Immuni's backend services. To propose a feature request, please open an issue in the Documentation repository. This lets everyone involved see it, consider it, and participate in the discussion. Opening an issue or pull request in this repository may slow down the overall process.

Contributors

Here is a list of Immuni's contributors. Thank you to everyone involved for improving Immuni, day by day.

Licence

Authors / Copyright

Copyright 2020 (c) Presidenza del Consiglio dei Ministri.

Please check the AUTHORS file for extended reference.

Third-party component licences

Please see the Technology Description’s Backend Services Technologies section, which also lists the corresponding licences.

Licence details

The licence for this repository is a GNU Affero General Public Licence version 3 (SPDX: AGPL-3.0). Please see the LICENSE file for full reference.

immuni-backend-exposure-ingestion's People

Contributors

alessandroberlati avatar alfo1995 avatar astagi avatar earien avatar gnfrisicaro avatar immuniopensource avatar immunistaff avatar immuniteam avatar ocardia avatar sebaker88 avatar xelhark avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

immuni-backend-exposure-ingestion's Issues

[GENERAL] Error "Need service name for --build-arg option" after docker-compose

Issue content

After running the docker compose command as described in the documentation I get the following error: Need service name for --build-arg option, command below:

test@test-VirtualBox:~/immuni/immuni-backend-exposure-ingestion/docker$ docker-compose build     --build-arg GIT_BRANCH=$(git rev-parse --abbrev-ref HEAD)     --build-arg GIT_SHA=$(git rev-parse --verify HEAD)     --build-arg GIT_TAG=$(git tag --points-at HEAD | cat)     --build-arg BUILD_DATE=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
ERROR: Need service name for --build-arg option

Docker version:

test@test-VirtualBox:~/immuni/immuni-backend-exposure-ingestion/docker$ sudo docker version
Client:
 Version:           19.03.6
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        369ce74a3c
 Built:             Fri Feb 28 23:45:43 2020
 OS/Arch:           linux/amd64
 Experimental:      false

[GENERAL] CONTRIBUTING.md

There is a typo in CONTRIBUTING.md

security and quality standards in the development tools described in should be security and quality standards using the development tools described in

[GENERAL] CODEOWNERS is wrongly structured

The CODEOWNERS file should be structured as follows:

# These owners will be the default owners for everything in
# the repo. Unless a later match takes precedence,
# @global-owner1 and @global-owner2 will be requested for
# review when someone opens a pull request.
*       @global-owner1 @global-owner2

[BUG] Endpoint v1/ingestion/upload returns status code 500

Describe the bug

Endpoint v1/ingestion/upload returns status code 500 if the teks parameter is an empty list.

To Reproduce

  1. cd docker
  2. docker-compose build
  3. docker-compose up
curl --location --request POST 'localhost:5000v1/ingestion/upload' \
--header 'Immuni-Dummy-Data: 0' \
--header 'Immuni-Client-Clock: 1589903340' \
--header 'Content-Type: application/json; charset=utf-8' \
--header 'Authorization: Bearer 4ec50b1f75ef4ad97521f5b3610cee605595266b7e2c42e6ce72eadff067c108' \
--data-raw '{
  "exposure_detection_summaries": [
    {
      "attenuation_durations": [
        300,
        0,
        0
      ],
      "date": "2020-05-27",
      "days_since_last_exposure": 1,
      "exposure_info": [
        {
          "attenuation_durations": [
            300,
            0,
            0
          ],
          "attenuation_value": 45,
          "date": "2020-05-16",
          "duration": 5,
          "total_risk_score": 4,
          "transmission_risk_level": 1
        }
      ],
      "matched_key_count": 2,
      "maximum_risk_score": 4
    }
  ],
  "padding": "S0meP4Dd1nG",
  "province": "AG",
  "teks": []
}'

Expected behaviour

The server should return a more explanatory error with status code 400.

[GENERAL] Concurrency model concerns

First of all, thank you for all the effort you put in this project, your care for sensitive topics has convinced even the paranoid people (like myself) to use the app. Kudos.

I’m a bit concerned about the concurrency model, you have chosen a really fast async framework (Sanic) running with the AsyncIO event loop but you are using incompatible network libraries, most notably MongoEngine, which is a mongoDB ODM on top of pymongo. Pymongo does not support AsyncIO

Let’s take a look at this chunk of code

upload_model = Upload(keys=teks)
otp = await validate_otp_token(request.token, delete=True)
upload_model.symptoms_started_on = otp.symptoms_started_on
upload_model.save()

upload_model.save() is a blocking operation, you are not yielding control to another coroutine during the write operation, blocking the entire event loop - the most visible effect is that you can't serve other requests during the operation.
I would suggest to use an async mongoDB driver instead of pymongo (Motor) or to run the blocking code in a concurrent.future.ThreadPoolExecutor using asyncio.loop.run_in_executor

Moreover, here

def process_uploads() -> None:
"""
Celery does not support async functions, so we wrap it around asyncio.run.
"""
asyncio.run(_process_uploads())
you are using Celery to schedule a periodic task, which instantiate an event loop purposely-built for an async function call. But it turns out that the function is not meant to be async at all:

Why bother with all the concurrency "inception" (Celery -> AsyncIO evt loop -> coroutine) then, when what you want achieve is a serial execution? There is no need of concurrency here. What about a celery task that runs synchronous code?
One could also argue that even Celery is an overkill when the need is to launch a cron job, there are a bunch of tools that allows to keep control of task scheduling on the application layer, without all the celery machinery, APScheduler for instance.

Cheers

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.