felipevolpone / freeipa-pr-ci Goto Github PK

The PR CI as it was designed do not support the needs of lots of runners (60+ runners), with that in mind, I propose a new architecture for the project.

This is a diagram of how I imagine it:

This is the workflow described in details:

1. GitHub (GH) trigger the web server when a PR is created or when a PR it's modified (using webhooks).

2. With the PR information, the web server will read the .freeipa-pr-ci.yaml spec file and will create a Task to each test described on it. A Task is basically an object with info of the test, the priority, if that Task depends on another one, etc.

   1. An important detail here. When a PR is created, we first need to run the build process, then the other tests can run. At the first time, the web server would read the PR info, see that it doesn't have any tasks created (in its task list), and it would create only the build Task and submit it to the RabbitMQ.
      When the build process is done, the Publisher will update GH saying that the build succeeded. With this GH webhook (it's triggered when a status of a commit changes), the web server can check that the build is done and then create the other Tasks.

3. The web server will add the Tasks in the TASKS_QUEUE on the RabbitMQ. This queue is only for tasks that the runners will consume and run. Important to say that we can have different kinds of tests to run, some of then consume less resource, but others demand bigger runners (VMs with more RAM and vCPU). So, we will need different queues for each kind of tests to run. Tests that consume less resource, will be added to the "medium tasks queue" and the bigger ones to "big tasks queue". In practice, we will not have only the BITASKS_QUEUE, but BIG_TASKS_QUEUE, SMALL_TASKS_QUEUE and so on.

4. The runners consume the TASKS_QUEUE and each runner will get a task to run. Important to say that RabbitMQ has its own mechanisms to prevent that a message in a queue could be consumed by two consumers. With the Task on hands, the runner knows what do to.

5. After running the tests, the runner will create a Result object, that will be added to the RESULTS_QUEUE on the RabbitMQ.

6. A Publisher (this is could be a service running on some machine) that consumes the RESULTS_QUEUE queue will get the Result object and publish the logs on FedoraPeople and change the GH task list, showing if the tests succeed or not (as we already do today)

Advantages:

• This way we will never hit the API rate limit; given that we will use webhooks
• It's totally uncoupled from GH. In the new arch. the tasks will be created by the Web Server based on the info received by GH, however, if sometime in the future we move from GH, we would just need to change the code that talks with the PR list.

Disadvantages:

• It's more complex and it has more point of failures
• It's harder to deploy than it is today. Today we just need to deploy the runners. With this new proposal, we would need to deploy: the RabbitMQ, the web server, the runners and the publisher. All this can be automated.
• It's harder to debug, given that it has more pieces on the game.

Challenges:

• Serialization of the object (Task and Result). It's not obvious to serialize and deserialize big objects like classes and so on. So a default "protocol" could/should be used, like only transfer JSON files.

PS:

• The items II, III, IV and VI are in blue color to show that they belong to our infrastructure.
• The items II and IV could be deployed on the same machine.
• The item IV could run as a service