mozilla-releng / buildhub2 Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 13.0 2.44 MB

A database index of buildhub.json files built by Mozilla

Home Page: https://buildhub2.readthedocs.io/

License: Mozilla Public License 2.0

Python 84.25% HTML 0.70% JavaScript 8.05% CSS 0.35% Makefile 1.78% Shell 3.92% Dockerfile 0.94%

buildhub2's People

Contributors

Stargazers

Watchers

Forkers

peterbe ctsanders1 mozilla-github-standards bhearsum acmiyaguchi smarnach ahal gbrownmozilla jcristau

buildhub2's Issues

Use django-configurations

That makes it easier to organize settings and rock solid solution to make sure we use settings specifically for tests.

Unbreak version sort in SearchKit

When we upgraded SearchKit we lost the ability to custom sort the versions.

--skip-invalid option in kinto-database-migration

We've discovered that if you run ./manage.py kinto-database-migration --skip-validation there are some really really broken records going into the JSONField. They're so broken that it breaks running ./manage.py reindex-elasticsearch because it assumes that every JSON blob has the keys build, target, source, and download. If any of them are missing, reindex-elasticsearch will crash.

Add bors to this project

Most important; a note in README or dev.rst about how to use it :)

Not able to run db migration to bootstrap the database

Background

Resources for -stage environment (database, elasticsearch, application stack and etc.) are set up and the application is able to start up.

Trying to run the migrate command for the first time to bootstrap the database in this environment.

Issue

Here is the command we ran:

$ docker run --rm --net host --env-file /etc/dockerflow/buildhub2.txt --log-driver=syslog hub.prod.mozaws.net/pipelines/buildhub2/buildhub2:latest migrate

And here is the error:

Not waiting for any services
Traceback (most recent call last):
  File "manage.py", line 8, in <module>
    from configurations.management import execute_from_command_line
  File "/usr/local/lib/python3.6/site-packages/configurations/management.py", line 3, in <module>
    importer.install(check_options=True)
  File "/usr/local/lib/python3.6/site-packages/configurations/importer.py", line 49, in install
    importer = ConfigurationImporter(check_options=check_options)
  File "/usr/local/lib/python3.6/site-packages/configurations/importer.py", line 67, in __init__
    self.check_options()
  File "/usr/local/lib/python3.6/site-packages/configurations/importer.py", line 87, in check_options
    add_help=False)
TypeError: __init__() takes 1 positional argument but 2 were given

Please note that the user scenario is the same as #187, but the error this time is different.

Move security headers to config

We should set all good-for-security headers with config. I.e. nginx through cloudops-deployment.

When we have this set up we don't need to set these things in Django additionally.

Kinto migration via a Postgres connection

At the moment, kinto-migration relies on HTTP. We need to make it be able to draw straight from a kinto Postgres database.

refresh_interval gets stuck after reindexing

If you run DJANGO_ES_REFRESH_INTERVAL=-1 ./manage.py reindex-elasticsearch
the refresh_interval gets "stuck". Meaning, the next time you start Elasticsearch it's still set to -1.

See http://localhost:9400/buildhub2 (look at settings.refresh_interval). Need a way to custom change this when running reindex-elasticsearch.

Make sure static assets through Whitenoise have far future cache headers

Title says it all.

Require "build" in JSON Schema

At the moment, "build" is not a required key.

This is rather odd. All valid records on Buildhub Prod (the legacy one) have a "build" key.

Dockerize

If this prototype is ever going to run in CloudOps it needs to have a Dockerfile. And we need to make sure it works as a docker file locally so we need to write a docker-compose.yaml file.

buildhub2 app failed to start on -stage env

Error shows

Jul 12 03:22:07 ip-172-31-27-217 systemd[1]: Started Daemon for buildhub2.
Jul 12 03:22:08 ip-172-31-27-217 docker-buildhub2[6141]: Not waiting for any services
Jul 12 03:22:08 ip-172-31-27-217 docker-buildhub2[6141]: python: can't open file 'gunicorn': [Errno 2] No such file or directory

gunicorn doesn't seem to be installed in the docker image.

$ sudo docker run -it --rm hub.prod.mozaws.net/pipelines/buildhub2/buildhub2 pip list |egrep -c gunicorn
0

Look for S3 keys ending with 'buildhub.json'

Because of the way releases are all put into the same directory, it doesn't make sense to look for a file called exactly buildhub.json.

For example, a directory might contain: Firefox Setup 61.0.exe, Firefox Setup 60.0.exe and Firefox Setup 62.0.exe. In that case the buildhub.json file is going to be called 61.buildhub.json (or was it 61-buildhub.json).

We decided that the minimum is that the file will always end on buildhub.json.
Therefore this or this won't work.

Whitenoise to serve static assets

URLs like / and /static/main.css should be served from disk with Whitenoise.

Typo in document

I think there is a typo on this line. It probably should be DJANGO_ES_URLS.

Make sure Crash-Stats buildid links continue to work

For example, a crash like this links to https://mozilla-services.github.io/buildhub/?q=20180807170231 which is going to have to change in Crash-Stats but it should still make sure the right buildid search works and highlights.

Custom check that checks Elasticsearch connection health

Need to be aware that the check might fail when you start docker-compose or honcho.

black all the code

https://pypi.org/project/black/ is awesome but still alpha. Are we ready to adopt?

Forward searches to Elasticsearch

The UI uses the /search API endpoint but it doesn't pass any of the request.POST to Elasticsearch.

Help Socorro migrate

https://github.com/mozilla-services/socorro/blob/8adaee728c9d960fb6613260c053720bcdfb799f/socorro/processor/processor_2015.py#L120

Elasticsearch 6.3.0 in docker-compose

Because that's what we're going to deploy in Stage and Prod.

Set up pyup

Once moved to mozilla set up Pyup.

Warn about _forcemerge

If you use the ./manage.py reindex-elasticsearch it warns about setting ES_REFRESH_INTERVAL

If you do that you should run _forcemerge afterwards. The command is this:

'curl -XPOST http://localhost:9200/{}/_forcemerge?max_num_segments=5'.format(index_name)

buildhub2 -stage web site fails to load the second time

STR

The issue can be reproduced in both Firefox and Chrome browser. I describe the steps to reproduce it in Chrome browser first, and in Firefox browser later.

open a chrome incognito window (this way it is a clean slate without any local caches).
open the Developer Tools pannel in the window and click on the network tab so you can watch the networking activities.
visit https://buildhub2.stage.mozaws.net
note the website is loaded correctly (even though there are 4 javascript errors in the console tab)
click the reload button to reload the page.
And see that the website is not reloading and shows a "Welcome to nginx" message.

The issue can be re-produced in Firefox browser as well. The only difference is that if using a Firefox browser, between the above step 4 and 5, you need to wait for more than 24 hours (presumably, that is the amount of time specified in the response cache-control header, where it demands max-age=86400, public for the root page.

Other notes

In the step 5 above, when clicking the reload button, the following two requests were received at the server end.
```
"GET /favicon.ico HTTP/1.1" 200 34885 ...
"GET /service-worker.js HTTP/1.1" 200 1383 ...
```
In both browsers, running the following code in the javascript console will allow the web site to reload again.
```
navigator.serviceWorker.ready.then(registration => { registration.unregister(); });
```

Document (cookbook) some good Elasticsearch queries

In Buildhub1 there used to a section of example queries
Some of them are Kinto specific and some of them are Elasticsearch.

We should have list like this too somewhere for Buildhub2.

Help pushlog-addon migrate

This link (permlink) might rot but it looks like this:

async function getBuildInfo(buildid) {
  const url = "https://buildhub.prod.mozaws.net/v1/buckets/build-hub/collections/releases/search";
const query = {
   ...

Help crash-stop-addon migrate

See https://github.com/mozilla/crash-stop-addon/blob/a83d0440cad85caed481ffaa23559f71f57deb33/crashstop/buildhub.py#L13

App returns 400 for health check endpoint

Background

DB migration is done. App is able to start up successfully.

Issue

The app instances can't register with the ELB because the app returns 400 for /__lbheartbeat__ endpoint.

Here is the app log:

Aug 10 19:18:28 ip-172-31-21-40.ec2.internal systemd[1]: Started Daemon for buildhub2.
Aug 10 19:18:29 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: Not waiting for any services
Aug 10 19:18:30 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: [2018-08-10 19:18:30 +0000] [6] [INFO] Starting gunicorn 19.9.0
Aug 10 19:18:30 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: [2018-08-10 19:18:30 +0000] [6] [INFO] Listening at: http://0.0.0.0:8000 (6)
Aug 10 19:18:30 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: [2018-08-10 19:18:30 +0000] [6] [INFO] Using worker: sync
Aug 10 19:18:30 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: [2018-08-10 19:18:30 +0000] [9] [INFO] Booting worker with pid: 9
Aug 10 19:18:30 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: [2018-08-10 19:18:30 +0000] [10] [INFO] Booting worker with pid: 10
Aug 10 19:18:30 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: [2018-08-10 19:18:30 +0000] [11] [INFO] Booting worker with pid: 11
Aug 10 19:18:30 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: [2018-08-10 19:18:30 +0000] [12] [INFO] Booting worker with pid: 12
Aug 10 19:18:30 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: [2018-08-10 19:18:30 +0000] [13] [INFO] Booting worker with pid: 13
Aug 10 19:18:33 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: {"Timestamp": 1533928713849477376, "Type": "markus.backends.datadog", "Logger": "buildhub", "Hostname": "ip-172-31-21-40.ec2.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 10, "Fields": {"msg": "DatadogMetrics configured: localhost:8125 "}}
Aug 10 19:18:33 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: {"Timestamp": 1533928713855258112, "Type": "markus.backends.datadog", "Logger": "buildhub", "Hostname": "ip-172-31-21-40.ec2.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 11, "Fields": {"msg": "DatadogMetrics configured: localhost:8125 "}}
Aug 10 19:18:33 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: {"Timestamp": 1533928713858804992, "Type": "markus.backends.datadog", "Logger": "buildhub", "Hostname": "ip-172-31-21-40.ec2.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 12, "Fields": {"msg": "DatadogMetrics configured: localhost:8125 "}}
Aug 10 19:18:33 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: {"Timestamp": 1533928713859929856, "Type": "markus.backends.datadog", "Logger": "buildhub", "Hostname": "ip-172-31-21-40.ec2.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 13, "Fields": {"msg": "DatadogMetrics configured: localhost:8125 "}}
Aug 10 19:18:33 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: {"Timestamp": 1533928713860281344, "Type": "markus.backends.datadog", "Logger": "buildhub", "Hostname": "ip-172-31-21-40.ec2.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 9, "Fields": {"msg": "DatadogMetrics configured: localhost:8125 "}}
Aug 10 19:19:40 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:19:40 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"
Aug 10 19:19:54 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:19:54 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"
Aug 10 19:20:09 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:20:09 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"
Aug 10 19:20:24 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:20:24 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"
Aug 10 19:20:39 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:20:39 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"
Aug 10 19:20:47 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:20:47 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"
Aug 10 19:20:54 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:20:54 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"
Aug 10 19:21:02 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:21:02 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"
Aug 10 19:21:09 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:21:09 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"

There is no further info in the logs to tell why it returns 400, unfortunately.

The app also returns 400 for other standard endpoints, such as /__version__, /__heartbeat__, /__lbheartbeat__, /v1/__version__, /v1/__heartbeat__ and etc.

Help PollBot migrate

https://github.com/mozilla/PollBot/blob/1c9748ebadd130efea398b79ebef8c7142359305/pollbot/tasks/buildhub.py#L11 and https://github.com/mozilla/PollBot/blob/1c9748ebadd130efea398b79ebef8c7142359305/pollbot/tasks/buildhub.py#L71 amongst some.

Move contribute.json into root

Serve it like we do in Tecken.

Help missioncontrol migrate

Here's now missioncontrol queries buildhub
(and settings)

Once we have a new URL for Buildhub2 (and we're confident we have all old and new builds) we should help them migrate.

Tests for ingestion

Need to mock boto like crazy

Help crash-clouseau migrate

See https://github.com/mozilla/crash-clouseau/blob/d29096f4448f639642cd23f9d993baa040e7398b/crashclouseau/buildhub.py#L16

CI build docs separately

See this PR for a good example

Backfill cron job script

We're sure to make mistakes with SQS (code or operational) so a cron job backfill script would be nice to suck up all buildhub.json files that exist.

Document how to bootstrap from an empty database

When you start from absolutely scratch, you have nothing the database nor in Elasticsearch. If you try to use the website from that point you'll get an error like this from ES:

{
  "status": 404,
  "error": {
    "index_uuid": "_na_",
    "index": "buildhub2",
    "resource.type": "index_or_alias",
    "root_cause": [
      {
        "index_uuid": "_na_",
        "index": "buildhub2",
        "resource.type": "index_or_alias",
        "resource.id": "buildhub2",
        "reason": "no such index",
        "type": "index_not_found_exception"
      }
    ],
    "reason": "no such index",
    "type": "index_not_found_exception",
    "resource.id": "buildhub2"
  }
}

But that error isn't important arguably because you shouldn't be hitting the web API URLs until you have bootstrapped.

The question is; how do you bootstrap. It's not documented. Documentation on this should help both local dev coders and fresh deployments.

Script to backfill from Kinto

To be run once to fill up the database with legacy builds that weren't made as buildhub.json files.

Test in CircleCI

After we have a Dockerfile we should test in Circle with

Use docker_layer_caching in CircleCI

docker_layer_caching: true only works if it's enabled on the account.

Unit tests use Elasticsearch

Our current unit tests actively use Postgres. We should equally use Elasticsearch but we should not pollute the index we use for running the dev server.

Not able to run db migration to bootstrap the database

Background

Resources for -stage environment (database, elasticsearch, application stack and etc.) are set up and the application is able to start up.

Trying to run the migrate command for the first time to bootstrap the database in this environment.

Issue

Here is the command we ran:

$ docker run --rm --net host --env-file /etc/dockerflow/buildhub2.txt --log-driver=syslog hub.prod.mozaws.net/pipelines/buildhub2/buildhub2:latest migrate

And here is the error:

Not waiting for any services
{"Timestamp": 1533331050280736768, "Type": "markus.backends.datadog", "Logger": "buildhub", "Hostname": "ip-172-31-41-251.ec2.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 6, "Fields": {"msg": "DatadogMetrics configured: localhost:8125 "}}
Traceback (most recent call last):
  File "manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 365, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 332, in execute
    self.check()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 364, in check
    include_deployment_checks=include_deployment_checks,
  File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 58, in _run_checks
    issues.extend(super()._run_checks(**kwargs))
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 351, in _run_checks
    return checks.run_checks(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/core/checks/registry.py", line 73, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/app/buildhub/dockerflow_extra.py", line 41, in check_elasticsearch
    health = fetch(url)["status"]
  File "/usr/local/lib/python3.6/site-packages/backoff/_sync.py", line 99, in retry
    ret = target(*args, **kwargs)
  File "/app/buildhub/dockerflow_extra.py", line 33, in fetch
    response.raise_for_status()
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 939, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 408 Client Error: Request Timeout for url: http://internal-buildhub2-es-s-elb-1qf0a53abi35c-1034878185.us-east-1.elb.amazonaws.com:9200/_cluster/health/buildhub2

So it looks like there is some kinda of chicken-egg problem. Because it tries to check the health of a non-existing index before running the database migration.

Is it possible to document the steps of how to bootstrap a brand new environment? That would be ideal if we can follow those steps exactly to set this up.

A less ideal solution would be let us know how to manually work around it. But that means we'll have to play the same "manual tricks" when we deploy -prod environment.

$ ./manage.py reindex-elasticsearch
{"Timestamp": 1536193993444640512, "Type": "markus.backends.datadog", "Logger": "buildhub", "Hostname": "ip-172-31-2-154", "EnvVersion": "2.0", "Severity": 6, "Pid": 8, "Fields": {"msg": "DatadogMetrics configured: localhost:8125 "}}
System check identified some issues:

WARNINGS:
main.Build.metadata: (postgres.E003) JSONField default should be a callable instead of an instance so that it's not shared between all field instances.
	HINT: Use a callable instead, e.g., use `dict` instead of `{}`.
Force merged Elasticsearch index after bulk insert

Fwiw, despite of the above warning, the buildhub2 index was created successfully (it seems like).

Not sure how serious the warning message is or if you want to fix it or not.

mozilla-releng / buildhub2 Goto Github PK

buildhub2's People

Contributors

Stargazers

Watchers

Forkers

buildhub2's Issues

Background

Issue

STR

Other notes

Background

Issue

Background

Issue

Recommend Projects

Recommend Topics

Recommend Org