mozilla-releng / beetmoverscript Goto Github PK

mtabara> aki: hm, question - beetmoverscript uploads to s3 via put from https://github.com/mozilla-releng/beetmoverscript/blob/master/beetmoverscript/script.py#L167
17:22:05 what happens if that artifact alreeady exists?
17:22:30 i'm not sure. i'd like it to be idempotent
17:22:39 it'll probably get overwritten
17:22:49 worth an experiment or s3 doc lookup
17:23:29 when we retrigger BM tasks just to make sure a new change doesn't affect the logic, don't we run that again, hence idempotent?
17:24:04 jlund: I think you've played with this in your staging bucket. any idea? ^
17:26:00 mtabara: beetmoverscript overwrites. beetmover (release promotion) checks for size and existence first
17:26:09 it would be better to get to relpro parity in that regard

In terms of:

• dephash
• docs badge + readthedocs
• more badge icons from https://pypi.python.org/pypi/coveralls ?
• add coveragerc config file + enable the branch coverarge, not just statement coverage

• Use Docker & Docker Compose for running a local dev environment, based on the python:3 image
• Use pip-compile-multi, and Pyup.io for dependency management
  • See https://github.com/mozilla-releng/releng-rfcs/blob/master/rfcs/0022-app-best-practices.md#dependency-management for requirements file details.
• Use tox as a test runner
• Use Taskcluster for CI
• Ensure production pushes happen manually, in response to bugs filed
• Ensure stage pushes happen in response to balrog_stage tags on Dockerhub
• Eliminate the development environment, if it exists

(Remove items that are not applicable to this project).

https://github.com/mozilla-releng/beetmoverscript/blob/master/beetmoverscript/test/fake_config.json#L16 should be pointing at real templates to have a minimum coverage there as well.

Fennec 60.0RC failed for some reason and we can't figure it out from the logs which platform did it use under https://taskcluster-artifacts.net/IieginJmQUiSW0d7Yxqmtg/0/public/logs/live_backing.log. We need to add more logging to make sure we understand what the values are and what the platform is. And not only that.

context.task['payload']['product'] doesn't exist outside of push-to-releases. In nightly/candidates, we get the product from the AppName in the balrog manifest. We could change that and reorg a bit the code to be uniformed. Another side effect is rewriting some of the functions to provide us the bucket name and nickname.

Need to add docstrings and docs to explain adjustments from Bug 1313154.

Related to #10 we should all investigate a potential merge between en-US and l10n templates for simplicity.

The TEMPLATE_KEY_PLATFORMS is solely used to toggle between the template files (fennec_nightly vs fennecx86_nightly). But looking at the two templates for the two different Android platforms, I noticed they only differ by "arm" vs "i386 in labels. I could easily merge them and just toggle that part via a template arg. That way we get rid both of this env dict, the fennecx86_* yml files and the logic from this function.

Looking at this buildhub.json I see:

"download": {
        "date": "2018-12-10T11:46:18.893852+00:00",
        "mimetype": "application/octet-stream",
        "size": 69585249,
        "url": "https://archive.mozilla.org/https%3A/archive.mozilla.org/2018/12/2018-12-10-09-55-04-mozilla-central/firefox-65.0a1.en-US.linux-x86_64.tar.bz2"
    },

Looks like something went wrong with mangling the URL.

I didn't even know this was a thing. TIL. Apparently we can teach Travis to ship the new release to Pypi (or whatever) whenever we add a new release tag in the repo. I need to investigate to see how that can be done.

https://github.com/mozilla-releng/beetmoverscript/blob/master/beetmoverscript/utils.py#L339 ends up looking like a path with a space in it, we should reformat it and check for other similar error messages

Tracked in bug 1316125

As @JohanLorenzo pointed, we don't use this in neither of the scriptworkers, yet somehow this variable ended up in some of the schemas. Let's retire it.

jlorenzo> mtabara: another quick beetmover question: how does it know when to use the en-US template or the l10n one?
16:40:30 — jlorenzo sees https://tools.taskcluster.net/groups/XGBsIkBURE6k7kZTY2UktQ/tasks/UPEtXJsJSi2tg5oR8O172Q/runs/0/logs/public%2Flogs%2Flive_backing.log#L86 used the en-US one, even though that's a l10n job
16:41:17 if there's any task.payload.locale set, it'll choose the l10n one
New messages
16:42:38 aaaaah
16:42:41 looking at your task, being a l10n beetmover job, it should have locale definied
16:42:44 something like https://tools.taskcluster.net/groups/XGBsIkBURE6k7kZTY2UktQ/tasks/R0XKOnutSIm_OkVyDyppDQ/details
16:42:50 I thought payload.upstreamArtifact.locale was enough
16:43:06 payload.upstreamArtifacts[i].locale
16:43:33 in the future we will simplify that indeed, but for now that's the way to go without adding more hacs
16:43:46 — mtabara adds a note on that in the issues, good point

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please reach out to [email protected].

(Message COC001)

More details here.

To prevent situations like this: https://public-artifacts.taskcluster.net/HrTLb_wUSzauU7so9BXYwQ/2/public/logs/live_backing.log

2017-03-17 23:18:29,390 - asyncio - DEBUG - Using selector: EpollSelector
2017-03-17 23:18:29,392 - asyncio - ERROR - Creating a client session outside of coroutine
client_session: <aiohttp.client.ClientSession object at 0x7f2cffe75940>
2017-03-17 23:18:29,395 - beetmoverscript.task - DEBUG - {'type': 'object', 'required': ['payload', 'dependencies'], 'title': 'Taskcluster beetmover task minimal schema', 'properties': {'payload': {'optional': ['build_number', 'version'], 'type': 'object', 'required': ['upload_date', 'upstreamArtifacts'], 'properties': {'upstreamArtifacts': {'items': {'type': 'object', 'required': ['taskId', 'taskType', 'paths', 'locale'], 'properties': {'taskType': {'type': 'string'}, 'locale': {'type': 'string'}, 'paths': {'items': {'type': 'string'}, 'minItems': 1, 'type': 'array', 'uniqueItems': True}, 'taskId': {'type': 'string'}}}, 'minItems': 1, 'type': 'array', 'uniqueItems': True}, 'version': {'type': 'string'}, 'build_number': {'type': 'number'}, 'upload_date': {'type': 'number'}}}, 'dependencies': {'items': {'type': 'string'}, 'minItems': 1, 'type': 'array', 'uniqueItems': True}}}
2017-03-17 23:18:29,398 - beetmoverscript.task - INFO - Buckets: ['nightly']
2017-03-17 23:18:29,399 - beetmoverscript.task - INFO - Action types: ['push-to-nightly']
2017-03-17 23:18:29,400 - beetmoverscript.utils - INFO - generating manifest from: /builds/scriptworker/lib/python3.5/site-packages/beetmoverscript/templates/fennec_nightly.yml
2017-03-17 23:18:29,401 - beetmoverscript.utils - INFO - /builds/scriptworker/lib/python3.5/site-packages/beetmoverscript/templates/fennec_nightly.yml
Traceback (most recent call last):
File "/builds/scriptworker/bin/beetmoverscript", line 9, in
load_entry_point('beetmoverscript==0.4.3', 'console_scripts', 'beetmoverscript')()
File "/builds/scriptworker/lib/python3.5/site-packages/beetmoverscript/script.py", line 272, in main
loop.run_until_complete(async_main(context))
File "/tools/python35/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
return future.result()
File "/tools/python35/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/tools/python35/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/builds/scriptworker/lib/python3.5/site-packages/beetmoverscript/script.py", line 104, in async_main
await action_mapcontext.action
File "/builds/scriptworker/lib/python3.5/site-packages/beetmoverscript/script.py", line 41, in push_to_nightly
mapping_manifest = generate_beetmover_manifest(context)
File "/builds/scriptworker/lib/python3.5/site-packages/beetmoverscript/utils.py", line 111, in generate_beetmover_manifest
manifest = yaml.safe_load(tmpl.render(**tmpl_args))
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/init.py", line 94, in safe_load
return load(stream, SafeLoader)
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/init.py", line 72, in load
return loader.get_single_data()
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/constructor.py", line 35, in get_single_data
node = self.get_single_node()
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/composer.py", line 36, in get_single_node
document = self.compose_document()
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/composer.py", line 55, in compose_document
node = self.compose_node(None, None)
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/composer.py", line 133, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/composer.py", line 133, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/composer.py", line 133, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/composer.py", line 133, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/composer.py", line 82, in compose_node
node = self.compose_sequence_node(anchor)
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/composer.py", line 110, in compose_sequence_node
while not self.check_event(SequenceEndEvent):
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/parser.py", line 98, in check_event
self.current_event = self.state()
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/parser.py", line 382, in parse_block_sequence_entry
if self.check_token(BlockEntryToken):
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/scanner.py", line 115, in check_token
while self.need_more_tokens():
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/scanner.py", line 149, in need_more_tokens
self.stale_possible_simple_keys()
File "/builds/scriptworker/lib/python3.5/site-packages/yaml/scanner.py", line 289, in stale_possible_simple_keys
"could not find expected ':'", self.get_mark())
yaml.scanner.ScannerError: while scanning a simple key
in "", line 81, column 5:
target.awsy.tests.zip
^
could not find expected ':'
in "", line 82, column 13:
s3_key: fennec-55.0a1.multi.android-ar ...
^
exit code: 1

See comments in https://github.com/mozilla-releng/beetmoverscript/pull/9/files/38f3056473f2507a6860974d67c693cd13264cec#r86997743

Specifically,

@catlee @lundjordan - please Archive this repo.

Because we don't have 100% code coverage, any tested code that's removed changes the percentage of covered code. I removed a function in #211 and had to reinsert it, with tests, because otherwise the code coverage had dropped by 0.005% and we weren't able to merge. This seems like the wrong approach.

Is there a different way coveralls can assess a decrease in coverage?

Currently we're reading all the payload_version with FirefoxVersion. That includes both classic cases (gecko) but also maven counterparts (mobile products). Since mozilla-version==0.3.1 was rolled-out, there's a more specialized class MavenVersion to handle the latter specific cases. However, there's an interesting caveat here.

In geckoview beetmover jobs, we have payload_version like XX.0bY which we validate and then parse to infer the major, minor, etc. However, MavenVersion no longer supports this format in its regexes. On the other hand, if we keep the old format, FirefoxVersion, we don't have the snapshot support.

Solution:
a) temporarily hack beetmoverscript to handle both for formats for now to unblock snapshots roll-out to production
b) enrich regex in MavenVersion to enable gecko beta / nightly versions? (Not sure we actually want to relax that regex just so that we could parse the values)
c) modify in-tree the way we transmit the payload_version in beetmover jobs.

We are adding a lot of things in task.payload that don't seem to be covered by the schema... let's add them, even if they're optional.

kudos to @lundjordan for noticing this.

checksums file has to be updated and uploaded by the signingscript task. The apk checksum is the unsigned apk one: http://bucketlister-delivery.stage.mozaws.net/pub/mobile/nightly/latest-date-android-api-15/fennec-52.0a1.multi.android.arm.checksums

I think this is because our logs are DEBUG, and boto debug logs are very chatty.
We should be able to specify specific modules to be DEBUG, and others to be INFO.

Github pointed out there's a security alert in requests 2.19.1 and lower - more details.

With beetmover reaching a certain level of maturity, maybe it's time to (re)assess some of its deps. Lots of the code can go away now that we have declarative artifacts in place and most of the pre-processing is useless.

To improve security, ideally we'd have 3 sets of creds for firefox beetmover:

• dep (can push to staging / test area only)
• nightly (can push to nightly bucket only)
• release (can push to release bucket only)

and we'd choose which set of creds to use based on scopes. project:releng:beetmover:nightly and project:releng:beetmover:release would be required to use either the nightly or release creds.

This would help us make sure we don't write to the wrong bucket from a low-security tree.

See https://github.com/mozilla-releng/beetmoverscript/pull/9/files/38f3056473f2507a6860974d67c693cd13264cec#r86997874

after https://github.com/mozilla-releng/beetmoverscript/pull/32/files lands we're going to have sha256 as hash algo added in the checksums file. right now it's being computed but it'd be worth grabbing it directly from ChainOfTrust.json artifacts from each of the depending tasks, saving us few cycles.

To investigate if it's worth adding some logic to serve this purpose.

Would be cool to automate the testing for this repo as we have for all others. Particularly useful for future PRs automatic testing.

With the Bug 1313154 new adjustments to cope with nightly graph, logic has been changed but no tests were added to correct this. I will do that.

In a bunch of tasks in the release automation it'd make a lot of sense if we had support for chunks. Once we have declarative artifacts and maven within the new beetmoverscript, I think it'd be neat to support this as well rather than always needing to add countless beetmover jobs into the graph.

See comments in https://github.com/mozilla-releng/beetmoverscript/pull/9/files/38f3056473f2507a6860974d67c693cd13264cec#r86996452