odoo / runbot Goto Github PK

Currently, delegate reviewers can't configure the merge method (mostly on multi-commit PRs): the idea of delegation was that it would mostly be used on/with third-parties (e.g. partners) for testing / functional review, and the delegator would have handled the more technical review and set up the merge method themselves.

However we're sometimes using delegation internally to hand off final say to e.g. a colleague with good technical knowledge of the subsystem but who may not have full review rights themselves, in which case the delegator may not come back on the PR or notice the lack of configuration, and the delegatee can't do it.

Possibilities:

• let delegate reviewers configure the merge method, it can make for ugly merges but it doesn't actually break anything
• add one more flag / trust level to note that a reviewer can configure the merge method (and possibly other configurable items like priority?)
• other?

/cc @Elkasitu @odony @rco-odoo

_validate is invoked when a commit is created or written to, but if a PR is updated to an existing commit (either via synchronize or the UI) it seems possible that the PR's state will not be updated to reflect this.

Given a project P over two repos A and B, if a PR(target=A.foo) fails while building B.foo the PR will be failed with the error message "unknown reason".

The reason should be a CI failure on B.foo.head

Just in case he broke the branches again and is currently failing all the stagings.

Currently, pushing & retrying a PR in error will automatically mark it as "ready" (and re-stage it). This may or may not be the correct behaviour, I'm conflicted: it means the retry will skip pre-staging runbot despite the code has changed, OTOH one could expect the changes to be minor and the PR could be explicitly r-'d then r+'d to perform the entire cycle?

Alternatively, push + retry on a PR could move it to approved (would still have to pass runbot but would be immediately re-staged afterwards) or even opened (the latter is what happened when updating a PR which is neither closed nor in an error state)

/cc @odony @rco-odoo

gh calls regularly fail in various ways. Often this is a spurious issue, but there's no easy way to know after the fact.

On failure, the content of the response should be logged much more extensively than it currently is (which is pretty much not at all) such that we can get e.g. X-GitHub-Request-Id to pass along to github's support.

https://github.com/odoo/enterprise/pull/4994#issuecomment-517691394 has a space before the @, and thus didn't match as a command.

If a staging fails and gets split, a PR being de-readied doesn't seem to be propagated to the split, so a PR might be mistakely merged at that point.

Cf odoo/odoo#32842: author realised the PR was mis-targeted, and closed the PR, then re-opened and retargeted it (after inquiring whether it would work).

However on opening the PR was reset to "open", it got re-approved but got stuck waiting for CI validation which would never come as CI had already run on the commit.

A PR will appear in the "blocked" list despite not being blocked if:

• its own priority is 0, being linked to a p=0 PR is taken in account but being p=0 itself apparently isn't handled
• there exist merged PRs with the same label, it only checks if the possibly-linked PRs are !ready

selected todos

• clarify dashboard / staging status

  staging state is just the runbot notification, however things get a bit odd when a staging is cancelled (usually because one of the PRs is closed/updated while being staged): the staging is active=False so it's never marked as Success or Failure because it's not found when updating CI state. This is an opportunity: a computed status field can have all the states of state + "cancelled" when state=='pending' and not active

• Allow @ prefix to bot

  People are used to @-call others, they might do that for the mergebot user, but the bot currently expect command lines to start with its own name (and that only). It should allow being @-ed for wetware compatiblity

• flag stagings when fast-forward failed, currently the build is disabled but looks successful (since the statuses passed) fixed with #41

ideas (to decide)

• Ask for multiple reviews, currently a review by a reviewer is "binding". Delegating reviews does handle some cases of multi-review, but more flexibility might be useful?

• Automatically remove branch when merged (when stable enough, though) [NIM]

  [xmo]: should probably be a separate tool instead, deletion conditions TBD.

• Better handle reopening: currently re-creates the PR in "opened" mode, even if the PR previously passed review & CI, maybe check for that? Alternatively: a ping/check signal for the mergebot to completely reload the PR?

• add github rate limit to dashboard

  github has a rate limit of 5000 requests/h. Assuming 1 staging/h/branch (1h is about the time for an enterprise staging to get tested ATM), 8 batches/staging, 10 branches and 2 repos gives us ~160PRs/h if the pipeline is always full, or a limit of ~31 queries/PR staged. Squash and merge don't come close (they're on the order of 3~6 queries I'd say) so we should be fine, but keeping an eye on it might still be useful (rebase & merge is 2 queries / commit + a baseline of 3~4 so rebase-merging PRs with many commits could easily be above 50, though that should be a very rare event).

• add CacheControl

  The risk of hitting rate limits is low (cf above) but when staging & restaging we might fetch the same PR info several times. Github does recommend using conditional requests. A wrinkle here is we might do these fetches on different runbot instances so would need do implement & use a shared cache in postgres for it to really be useful.

Currently checking CLA is done sequentially with other runbot tasks despite there being no real dependency on the runbot testing infrastructure & al, aside from testing the code. It might be useful to set up a separate status pipeline which would run on a different machine (runbot10 should have unused capacity, mergebot is pretty much idle), and would generate additional statuses (possibly even gh checks?) onto which mergebot could depend:

• CLA, which could be extended to check all commits rather than just the last
• linting commit messages
• code linting and formatting (JS and / or python, possibly moving the pylint pass out of the runbot's sequence)
• documentation CI
• ease of adding further tasks?

Furthermore, it would allow only performing these checks for PRs as we usually don't care much about them in WIP branches (in which some of them don't even make sense e.g. all contributors pushing to dev repos should pass CLA by definition).

This could be either bespoke or done through e.g. a gitlab ci/cd instance or something like that, these needs don't seem super complex and ought be pretty much solved by existing solutions.

Also should probably ignore comment edition, it's seriously weird when @mart-e does it.

e.g. mergebot believed odoo/odoo#30305 had multiple commits, likely because the PR was originally incorrectly targeted to 12 before being re-targeted to 11.

runbot_merge creates a staging branch for every repository, but if there is no PR for one of the repos that staging branch will just be a copy of the existing target. Which means 1. runbot_merge does not force a rebuild with the correct version of the other repos and 2. runbot_merge may already have a status for that commit. If the target is in failure (transient or not), the staging will fail immediately without even rebuilding the entire thing.

Solution: create a dummy commit at the head of every staging branch (not fast-forwarded into the target) to ensure all staging branches are always built in full.

Should not do anything but I can't think of any other hypothesis as to why odoo/odoo#30525 did not work properly: there are no errors in the log

Staging pr odoo/odoo:30492 for target 12.0; squash=True
Staged pr odoo/odoo:30492 to 12.0 by rebase-ff: 925421684311a2cd7314dd01a80c1b67523acba4 -> b42cd64ace5f859184b8cc47a0db0963c03ae73b
Staging pr odoo/enterprise:3501 for target 12.0; squash=True
Staged pr odoo/enterprise:3501 to 12.0 by rebase-ff: 2a08d3b259ac70104310e8b8e206dd3afce3f614 -> ed9feaad29934429bb6352ac3561eb3280085095
Staging pr odoo/odoo:30525 for target 12.0; squash=True
Staged pr odoo/odoo:30525 to 12.0 by rebase-ff: b42cd64ace5f859184b8cc47a0db0963c03ae73b -> d5ffd327c76263007085f3a3cdb43923c69f3a5d
Staging pr odoo/odoo:30529 for target 12.0; squash=True
Staged pr odoo/odoo:30529 to 12.0 by rebase-ff: d5ffd327c76263007085f3a3cdb43923c69f3a5d -> ce6e66cef930a30d9a62988a16f9f8319cb9f25c

However the specified pr commit (d5ffd327) simply doesn't show up in any of odoo/odoo's branches, and the purported following commit (ce6e66ce) does show up with b42cd64a as its parent rather than the expected d5ffd327.

edit: weirder, odoo/odoo#30525 had been merged, but combined with ce6e66cef930a30d9a62988a16f9f8319cb9f25c

Looking by PR number is by far the most common thing to do in that list, it's annoying to have to go through custom filters.

Branches can't be deleted if they have PRs / stagings linked, but that means "decommissioned" branches (older saas, or "dead" saas) remain forever on the mergebot dashboard.

While I try to run runbot, it cannot download odoo:runbot_tests
Unable to find image 'odoo:runbot_tests' locally
Trying to pull repository docker.io/library/odoo ...
Trying to pull repository registry.fedoraproject.org/odoo ...
Trying to pull repository quay.io/odoo ...
Trying to pull repository registry.access.redhat.com/odoo ...
Trying to pull repository registry.centos.org/odoo ...
Trying to pull repository docker.io/library/odoo ...
/usr/bin/docker-current: manifest for docker.io/odoo:runbot_tests not found.
See '/usr/bin/docker-current run --help'.

This is mostly for older PRs being revived (due to comments): if the repo is deleted, the PR ends up with no label whatsoever. The fetch loop currently doesn't like that at all, and will crash trying to create the PR object:

2019-02-27 07:37:48,577 16223 ERROR mergebot odoo.sql_db: bad query: b'INSERT INTO "runbot_merge_pull_requests" ("id", "priority", "merge_method", "method_warned", "state", "link_warned", "number", "label", "author", "target", "reposito
ry", "head", "squash", "message", "create_uid", "write_uid", "create_date", "write_date") VALUES(<…>) RETURNING id'
ERROR: null value in column "label" violates not-null constraint

2019-02-27 07:37:48,577 16223 ERROR mergebot odoo.addons.base.ir.ir_cron: Call from cron Check for PRs to fetch for server action #497 failed in Job #16
Traceback (most recent call last):
  File "/home/odoo/src/odoo/11.0/odoo/tools/safe_eval.py", line 350, in safe_eval
    return unsafe_eval(c, globals_dict, locals_dict)
  File "", line 1, in <module>
  File "runbot/runbot_merge/models/pull_requests.py", line 151, in _check_fetch
    f.repository._load_pr(f.number)
  File "runbot/runbot_merge/models/pull_requests.py", line 203, in _load_pr
    'pull_request': pr,
  File "runbot/runbot_merge/controllers/__init__.py", line 138, in handle_pr
    'message': message,
  File "runbot/runbot_merge/models/pull_requests.py", line 698, in create
    pr = super().create(vals)
  File "odoo/11.0/odoo/models.py", line 3377, in create
    record = self.browse(self._create(old_vals))
  File "odoo/11.0/odoo/models.py", line 3470, in _create
    cr.execute(query, tuple(u[2] for u in updates if len(u) > 2))
  File "odoo/11.0/odoo/sql_db.py", line 155, in wrapper
    return f(self, *args, **kwargs)
  File "odoo/11.0/odoo/sql_db.py", line 232, in execute
    res = self._obj.execute(query, params)
psycopg2.IntegrityError: null value in column "label" violates not-null constraint

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "odoo/11.0/odoo/addons/base/ir/ir_cron.py", line 102, in _callback
    self.env['ir.actions.server'].browse(server_action_id).run()
  File "odoo/11.0/odoo/addons/base/ir/ir_actions.py", line 554, in run
    res = func(action, eval_context=eval_context)
  File "odoo/11.0/addons/website/models/ir_actions.py", line 57, in run_action_code_multi
    res = super(ServerAction, self).run_action_code_multi(action, eval_context)
  File "odoo/11.0/odoo/addons/base/ir/ir_actions.py", line 430, in run_action_code_multi
    safe_eval(action.sudo().code.strip(), eval_context, mode="exec", nocopy=True)  # nocopy allows to return 'action'
  File "odoo/11.0/odoo/tools/safe_eval.py", line 373, in safe_eval
    pycompat.reraise(ValueError, ValueError('%s: "%s" while evaluating\n%r' % (ustr(type(e)), ustr(e), expr)), exc_info[2])
  File "odoo/11.0/odoo/tools/pycompat.py", line 86, in reraise
    raise value.with_traceback(tb)
  File "odoo/11.0/odoo/tools/safe_eval.py", line 350, in safe_eval
    return unsafe_eval(c, globals_dict, locals_dict)
  File "", line 1, in <module>
  File "runbot/runbot_merge/models/pull_requests.py", line 151, in _check_fetch
    f.repository._load_pr(f.number)
  File "runbot/runbot_merge/models/pull_requests.py", line 203, in _load_pr
    'pull_request': pr,
  File "runbot/runbot_merge/controllers/__init__.py", line 138, in handle_pr
    'message': message,
  File "runbot/runbot_merge/models/pull_requests.py", line 698, in create
    pr = super().create(vals)
  File "odoo/11.0/odoo/models.py", line 3377, in create
    record = self.browse(self._create(old_vals))
  File "odoo/11.0/odoo/models.py", line 3470, in _create
    cr.execute(query, tuple(u[2] for u in updates if len(u) > 2))
  File "odoo/11.0/odoo/sql_db.py", line 155, in wrapper
    return f(self, *args, **kwargs)
  File "odoo/11.0/odoo/sql_db.py", line 232, in execute
    res = self._obj.execute(query, params)
ValueError: <class 'psycopg2.IntegrityError'>: "null value in column "label" violates not-null constraint
" while evaluating
'model._check_fetch(True)'

would it be possible to tag commit 549452f so that community runbot code can refer to that for testing until we've adapted that to current changes? Thanks in advance!

Sometime Often, bad code is committed.

It should be great if robodoo can revert a PR.

Since username are not case sensitive making roboodo command case incensitive make sence, especially when writing Robodoo r+ from a smartphone.

@xmo-odoo

Currently, nothing is logged on a synchronize event unless the PR is staged (cancels the staging), closed or merged (logs an error and ignores). This is inconvenient for troubleshooting.

Scenario Odoo 11 module Runbot:

• Create branches
• Choose Repository
• Input Ref Name
• Save
  error:
  TypeError: create() missing 1 required positional argument: 'vals'

• don't show PRs which are not quite ready (e.g. with a non-ready linked PR or no merge method) in the "awaiting" list, this is confusing
• stagings without statuses should not look cancelled

edit: furthermore for long PRs people apparently miss the comment entirely, may need to spam them? Maybe make these checks only on "r+"?

In odoo/odoo#32261 the message appears twice with a delay of ~25mn between them. Unclear why it'd appear at all, and why twice.

More logging of that bit would probably be helpful, a quick check of the log didn't show anything though a more detailed review (looking up activity around the commit itself not just the PR) might reveal interesting info.

Or ignore PRs from @pimodoo and say they'll never be merged.

More seriously, with bots taking over the workflow and people not necessarily caring about the details (especially when things go well) they might have no reason to actively check for the progress of things, only to find out there was no progress, and because "no news = good news" the common assumption is that no update means everything's fine.

Proactively pushing concerns means lowered cognitive load and less channels to actively check. Some checks have already been implemented as "pushed" through PR comments, getting notified of a reviewed but failing branch has been requested by @ged-odoo and @pimodoo

Concerns:

• what about timeouts
• generic oneshot/repeat system, current notifications are bespoke with an ad-hoc todo/done flag, should this be genericised yet?
• notification spam: ability to disable proactive notifications? Should mergebot ping every single PR to explain what's what and various useful commands?

GH currently doesn't have split rights between the code and issues / PR part of a repository:

Improving permissions is something the team would like to implement in the future. There are some discussions happening within the Engineering and Product Teams and I believe one of those points is adding separate permissions for Issues.

We'll pass your feedback onto the team to consider.

This means external contributors or new employees can't be empowered with assigning folks or requesting reviews⁰ without also giving them write access to the repository, and we can't further reduce write access to the main repositories without creating more issues.

Commands could be added to @robodoo (or to an other bot drafted for the occasion e.g. @C3POdoo should already have the necessary rights and avoid the extended rights @robodoo needs for its own work) though this requires either that anyone can use the assigner bot¹ or that employees and contributors get opted in manually (similar to @robodoo but worse).

A relatively easy option would be to use highfive, which can auto-assign (based on directories & stuff) but is overridable (r? <username> will change the assignee to that). As far as I can tell we'd still have to deploy our own with whatever bot / user we want as it's not a GH application we can just install on the repo (and we might want our own customisations e.g. highfive seems to only handle assigning PRs, we may want reviews as well? ).

/cc @odony @JKE-be

[0]: which some teams use in their own workflows ↩

[1]: not necessarily a huge issue especially if we can blacklist abusers ↩

Hi together

I just discovered this repository and I would like to try runbot but I can't figure out how to configure it correctly.
Just to start: I would like to know how I can configure the standard odoo repository to start a new build. From there I can work my way through.
A sample config and the command how to trigger the build would be really helpful.

Thank you so much!

Given it's the staging head rather than the merge commit for the PR, people get confused. Since the relevant commit is back-referenced via the automatically added "closes" annotation, the link is also redundant.

Cf odoo/odoo#27596 as well as personal inquiries/testimonies of colleagues.

runbot_merge should annotate commits with who r+'d a given PR.

If a multi-PR staging fails CI mergebot is going to split the staging in order to try and find which PR is the source of the failure.

This is generally what we want, but in the case of false positives and especially when there's a PR backlog, we'd rather cancel the staging and create a fresh one which will hopefully avoid the tour's flakiness.

Currently, doing so generally requires unreviewing one of the staged PRs or somesuch.

• remove quotes, they're noisy and don't seem necessary (git commit -s doesn't quote)
• it's unclear that we're using it properly, according to the kernel documentation this looks more like sign-off is a CLA-equivalent? /cc @KangOl

Currently, mergebot errors / issues are checked once in a while through the log, might be worth investigating sending it to sentry for better short and long-term observability.

edit: according to @julienlegros there's a sentry integration thing / module somewhere pretty much ready to use

edit 2: said sentry integration thing is in internal & uses the old SDK (Raven). Integration seems pretty simple either way so might as well use the current SDK and its goodies (wsgi integration, …)

Apparently @gdgellatly blocked robodoo. Rather than just make robodoo's comments invisible, this results in the following error when mergebot tries to leave a comment on their PRs:

422 Unprocesssable Entity
{'message': 'Validation Failed', 'errors': [{'resource': 'IssueComment', 'code': 'unprocessable', 'field': 'data', 'message': 'User is blocked'}], 'documentation_url': 'https://developer.github.com/v3/issues/comments/#create-a-comme

This currently is not handled, and as a result locks up the feedback cron (every time the cron runs it tries to send the comment, that fails, which raises, which stops & rollbacks the cron).

Note: "change tags" has a somewhat similar issue, check that it skips tags it can't change (the tag endpoint often transiently returns a 502 Bad Gateway).

Currently, the mergebot has 3(.5) modes of integration with limited explicit control knobs:

• if only a single commit, squash it
• if 2+ commits, rebase-then-merge
• if rebase-, just merge (unless the head of the PR is a merge commit at which point it can do something more complex).

Some people have apparently expressed interest in the ability to explicitly ask for a rebase-and-ff, and possibly also explicitly squashing any number of commits?

/cc @qdp-odoo @tde-banana-odoo @rco-odoo @antonylesuisse @odony

When a branch or a repo is deleted, their corresponding builds are not killed.

This causes two problems:

• The running builds are running indefinetly
• The _find_port method can return a port that is used by a "phantom" build still running. Leading to a traceback 'Address already in use'

With the Big jQuery3 Branch (odoo/odoo#28457), @VincentSchippefilt inquired whether it was possible to pause/restrict staging on some branches.

Currently it's not, but it might be a useful feature e.g. pause stagings on a branch, or more likely / usefully restrict staging to some priority level (which would include "no staging") e.g. allow p=0 and p=1 but not "normal" p=2 for some time. Stagings could also be restricted by submitter or reviewer but that seems overly complex.

The staging process already goes over every commit of the PR so this should be possible.

When running remote tests, "repomaker" currently blows up if for some reason the previous test failed to properly cleanup a repository it created.

Might also be worth running a retry loop around all the setup/teardown gh bits (ideally using a session with a Retry: https://stackoverflow.com/a/35636367/8182118).

see current 12.0 branch of odoo repo:

odoo/odoo#27947

When the PR is reopened it's internally reset to the base state (~seen), but there is no way to tell that from the PR. runbot_merge should either clear all tags on reopen, or do so on close.

Cf MERGEBOT-A, a set_ref() failing with a 502 apparently forgets to report the various request infos we care about (possibly because we're performing the calls with error-reporting disabled and so it's up to the caller to do it)

Sometimes people screw up and don't put co-authored-by at the end of the message, which github really really wants for some reason.

Since we're already rewriting much of the message anyway we might as well extract them all, not just those at the end.

When for whatever reason job_end on the build has not been filled, the clean-up will not count it as active. Might be worth it to add a OR condition based on the create date of the build and not only on the last modification date of the log file.
As the log files are cleaned up, it's actually more difficult to read them when the logs are non existing :-)

Final FF when merging staging into branch:

• can't be rollbacked because the branch is protected

• has a pre-check (trying it out in tmp) to try and mitigate direct push issue

• however there is still an issue of a glitch (e.g. 502) when FF-ing github, as occurred on staging 2341:

  2019-02-19 16:15:57,869 24523 INFO mergebot odoo.addons.runbot_merge.models.pull_requests.cron: Checking active staging runbot_merge.stagings(2341,) (state=success)
  2019-02-19 16:15:58,271 24523 DEBUG mergebot odoo.addons.runbot_merge.github: head(odoo/odoo, master) -> f5c269db121323d6fcc6d58b0ef335ca3be24db1
  2019-02-19 16:15:59,032 24523 DEBUG mergebot odoo.addons.runbot_merge.github: set_ref(update, odoo/odoo, tmp.master, f5c269db121323d6fcc6d58b0ef335ca3be24db1 -> 200 (OK)
  2019-02-19 16:15:59,423 24523 DEBUG mergebot odoo.addons.runbot_merge.github: head(odoo/enterprise, master) -> 578c6cec84b3ab6ee11151306ae2807b760f98c2
  2019-02-19 16:16:00,292 24523 DEBUG mergebot odoo.addons.runbot_merge.github: set_ref(update, odoo/enterprise, tmp.master, 578c6cec84b3ab6ee11151306ae2807b760f98c2 -> 200 (OK)
  2019-02-19 16:16:01,017 24523 DEBUG mergebot odoo.addons.runbot_merge.github: fast_forward(odoo/odoo, tmp.master, 5062a26d18055abab76ab3ec97c8d93f90dbf38e) -> OK
  2019-02-19 16:16:01,781 24523 DEBUG mergebot odoo.addons.runbot_merge.github: fast_forward(odoo/enterprise, tmp.master, 35667baac526ea5e2109205a863e1ea587593062) -> OK
  2019-02-19 16:16:02,569 24523 DEBUG mergebot odoo.addons.runbot_merge.github: fast_forward(odoo/odoo, master, 5062a26d18055abab76ab3ec97c8d93f90dbf38e) -> OK
  2019-02-19 16:16:02,654 24523 DEBUG mergebot odoo.addons.runbot_merge.github: fast_forward(odoo/enterprise, master, 35667baac526ea5e2109205a863e1ea587593062) -> ERROR
  Traceback (most recent call last):
    File "/home/odoo/src/runbot/runbot_merge/github.py", line 79, in fast_forward
      self('patch', 'git/refs/heads/{}'.format(branch), json={'sha': sha})
    File "/home/odoo/src/runbot/runbot_merge/github.py", line 39, in __call__
      r.raise_for_status()
    File "/usr/lib/python3/dist-packages/requests/models.py", line 935, in raise_for_status
      raise HTTPError(http_error_msg, response=self)
  requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.github.com/repos/odoo/enterprise/git/refs/heads/master
  2019-02-19 16:16:02,655 24523 WARNING mergebot odoo.addons.runbot_merge.models.pull_requests.cron: Could not fast-forward successful staging on odoo/enterprise:master
  Traceback (most recent call last):
    File "/home/odoo/src/runbot/runbot_merge/github.py", line 79, in fast_forward
      self('patch', 'git/refs/heads/{}'.format(branch), json={'sha': sha})
    File "/home/odoo/src/runbot/runbot_merge/github.py", line 39, in __call__
      r.raise_for_status()
    File "/usr/lib/python3/dist-packages/requests/models.py", line 935, in raise_for_status
      raise HTTPError(http_error_msg, response=self)
  requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.github.com/repos/odoo/enterprise/git/refs/heads/master
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/home/odoo/src/runbot/runbot_merge/models/pull_requests.py", line 1228, in check_status
      staging_heads.get(repo_name + '^') or head
    File "/home/odoo/src/runbot/runbot_merge/github.py", line 83, in fast_forward
      raise exceptions.FastForwardError()
  odoo.addons.runbot_merge.exceptions.FastForwardError
  

At this point, already-ff'd branches are merged, non-ff'd branches are not, and the staging is considered failed and to retry. The staging will be retried, will hopefully succeed and no data is lost (unless people start messing with their PR, which they probably will…).

However, because the entire staging is failed every PR will be re-merged, leading to duplicate empty commits on the already merged branch e.g. odoo/odoo@7a80ebe was re-merged empty as odoo/odoo@3cbae36.

• Short term solution: if an FF is not the first in the series, retry it a few times with a few seconds' sleeping (at least if it's a 502, but possibly in other cases as well?). It's possible but very unlikely this will be an actual issue (somebody having pushed directly to the branch), so it's very likely to eventually succeed.
• Also possible: disable staging on the branch entirely until somebody comes around to re-enable it, possibly fixing things by hand? Could make for very strange situations.
• Probably not possible: un-protect the branch, as rules don't necessarily map trivially to branches (they're glob patterns)

Long-term possibility: stop interacting with the repo over the github API, use a local git instead. It's possible to use an API token as password.