mozilla-releng / firefox-infra-changelog Goto Github PK
View Code? Open in Web Editor NEWAutomated tool which builds a changelog of commits happening on git and hg that could affect Firefox CI Infra.
Automated tool which builds a changelog of commits happening on git and hg that could affect Firefox CI Infra.
Repo in question is: https://github.com/mozilla-releng/scriptworker
Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.
As PullRequest #28 was already merged, and it has a few issues, we will need to come back to this and fix all the issues listed in the PR.
Please push the fixed in a stand alone PR, without other content, just the fixes.
Repo in question is: https://github.com/mozilla-releng/signingscript
Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.
After the PR #85 the json files format for git repositories has been changed and introduces new issues regarding the generation of the main markdown table.
The issues is introduced by the first json element that now stores the last checked date.
Steps to reproduce the behavior:
Run the script
Check the main markdown table file.
Actual Result
Only the changes from the mercurial files are updated.
Expected behavior
The main markdown table should be updated with the latest changes on the git files and hg files.
Additional context
This appeared after structural modification of the git json files.
Now that we have the list of repositories we should start by asking each of the developers/contributors (that maintain these repositories) which files/folders (if that's the case) affects our infrastructure and how bad in case of an issue.
As discussed in the meeting from November 5'th, 2018 we should start asking every shift with the contributors.
repositories.json
file)We want to use the last time when we run the script, and then to use it for getting the changes since that date-time.
We have to figure out how to save the date-time format into a file and then to use it to get the changes since that date.
I have tried to save it as a string but it does not work. The script returns an error at the following line:
.get_commits(since=lastRun):
Repo in question is: https://github.com/mozilla-releng/mozapkpublisher
Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.
Keeping the code clean and well document. This will allow us to easier read and understand the code, but also will help any new or returning people who are going to work on this to better understand what has been done thus far.
I'm suggesting to attach this issue to the milestone with the closest deadline so we always do a tidiness-check before each deadline.
Repo in question is: https://github.com/mozilla-releng/shipitscript
Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.
Short description
We can figure out if a change affects CI by checking if this change happened in one of the CI files.
First solution
After we have a list with all the CI files we can check if the affected file is in our list, if it is we can save those changes if not we can ignore them.
First step
We can start to create a list with all the CI files
Proposal
We will need to create a list of all mercurial repositories that are being used for firefox-ci and/or could influence the infra.
A good place to start would be our team's Google Drive Escalation Doc.
Please post in the following format:
<Name of repo> - <Link to mercurial repo>
Repo in question is: https://github.com/mozilla-releng/pushapkscript
Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.
Using tests/json_changelog.py create a test script that will read the data stored in the following two json files: git_changelog.json
, hg_changelog.json
and add the aggregated and formatted data to changelog.json
I have read this article JSON crash course that explains short words how json looks and how we can access the data inside using Python. I think this is a very good starting point, as it will give us a good basic understanding of basic concepts, which will help us come up with more implementation specific questions that Moz devs can help us with.
Using a copy/paste of an example from the article linked above:
{
"text key": "string",
"integer key": 123,
"boolean key": true,
"simple list": [
"value 1",
"value 2"
],
"dict": {
"element 1": "value 1",
"element 2": "value 2"
},
"dict list": [
{
"key 1.1": "value 1.1",
"key 1.2": "value 1.2"
},
{
"key 2.1": "value 2.1",
"key 2.2": "value 2.2"
}
]
}
The difficult part, as I see it, is that we need to be able to sort the data with expressions like:
While most of this will be done via filter logic, to be able to work with clean code, we need a good json structure that is intuitive to anyone, including devs that might not know all the fine details.
Please ask questions or post findings/ideas/theories/proofs of concept in this issue.
Repo in question is: https://github.com/mozilla-releng/build-puppet
Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.
Repo in question is: https://github.com/mozilla-releng/treescript
Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.
Describe the bug
https://github.com/Akhliskun/firefox-infra-changelog/blob/zfay_dev/hg_files/autoland.md
As shown in the link , at the 14 and 3 commits the table is broken, because the commit message contains the char " | "
Define a list of files from this Repository which affects our infrastructure.
Identify who are the contributors of the project
Get in touch with them ( Email, Irc, GitHub etc)
Explain to them what we are trying to accomplish
Take notes and make sure to update this issue
Using tests/hg_api.py create a test script that is able to pull changelog (commits) from any of the Mozilla HG repositories.
Talk to devs in #vcs about how we can grab the data that we need.
Don't forget to ask how we should handle private repos.
hg_changelog.json
We have a few branches that came as PRs that shouldn't.
After @mutterroland 's next shift, I will remove them, in preparetion of moving this repository under mozilla org.
Create separate GItHub and Mercurial handler definitions, outside the main function.
I would prefer a specific definition of each handler that has proper documentation.
The current definitions are buggy and are missing features.
Both functions should contain/do the following:
.md
and .json
files for each repository.This issue will track all the progress/bugs/requests and ToDos.
Fell free to use this Issue to discuses about steps, plans and ideas on what should go into the first release of FF-Infra
5th November 2018
Repo in question is: https://github.com/mozilla-releng/beetmoverscript
Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.
We will need to create a list of all github repositories that are being used for firefox-ci and/or could influence the infra.
A good place to start would be our team's Google Drive Escalation Doc.
Please post in the following format:
<Name of repo> - <Link to github repo>
Define a json file that stores all of the git and hg repositories in a single place
My solution to this problem would be a json file that would look something like this:
{
"HG": {
"Repo_HG1": "link1",
"Repo_HG2": "link2",
"Repo_HG3": "link3"
},
"Git": {
"Repo_GIT1": "link1",
"Repo_GIT2": "link2",
"Repo_GIT3": "link3"
}
}
Another option would be using a convertor, but most of them needs to be installed and kept up to date.
Also a big advantage would be the simplicity that would give in case we want to store another data in those repositories, e.g specific files to check.
{
"HG": {
"Repo_HG1": {
"link": "https://example.com",
"important_file": "https://example.com/important",
"file_to_ignore": "https://example.com/ignore"
},
"Repo_HG2": "link2",
"Repo_HG3": "link3"
},
"Git": {
"Repo_GIT1": "link1",
"Repo_GIT2": "link2",
"Repo_GIT3": "link3"
}
}
Repo in question is: https://github.com/mozilla-releng/pushsnapscript
Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.
Repo in question is: https://github.com/mozilla-releng/build-cloud-tools
Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.
As per Jordan's request, I will sync up with him on 6th and get the project moved.
Everyone will have to update the upstream link on your local repository clones.
Repo in question is: https://github.com/mozilla-releng/OpenCloudConfig
Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.
Make sure that you only have 1 main dictionary in which you have nested dictionaries containing all the information.
Bring the formatting of Roland's MD files to the master branch. These MD files should contain commit information such as: repo, commit message, ,link of the commit, and date.
Should we add the name/e-mail of the committer?
If a commit message contain a special character like '\n' for new line or '\r' carriage return or '\t' tab ...( #, """, , %, etc), it will affect our files (Json and MD)
We have two option to fix this:
Option 1: We have to encode the information using UTF-8 witch will interpret those characters as a string so our .json and .md will not be affected
Option 2: We have to remove those characters from text to make it more clean and easy to read
First option: we can encode each information using ".encode("utf-8")" after we get it from API
Option two: we can import 're' library and use "re.sub" function to remove the characters from the string
Any better idea will be welcome.
We have to create a function to generate a markdown table based on data that's written in changelog.json. I'm going to self assign for this one.
Idea to have a list with all the information we need to save from each changes/commits.
We can start to create a few templates which contain the information about changes.
We have to find out which information would be useful to save.
Using tests/git_api.py create a test script that is able to pull changelog (commits) from any of the Mozilla RelEng repositories.
https://developer.github.com/webhooks/
github_changelog.json
As Dustin suggested, we should distinguish between things that deploy automatically and things that are deployed via some separate action.
This would help avoid the confusion when looking for a particular change or when it took effect.
As the title suggest , I've added a new column named Commit Url. That column refers to hyperlink to each commit stored in "URL" string.
Repo in question is: https://github.com/mozilla-releng/funsize
Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.
Note: This repository has been archived so it now read-only. Its last activity was a year ago.
After the merge in PR be7b8fc was done, the code brakes with "NameError: name 'url' is not defined"
To Reproduce
Simply run the script and something like
Traceback (most recent call last): File "E:/bcrisan/repos/fic2/client.py", line 187, in <module> create_md_table(repository_name) File "E:/bcrisan/repos/fic2/client.py", line 41, in create_md_table "|" + date + '\n' NameError: name 'url' is not defined
should pop out.
Expected behavior
The script should run and create the md/json files in repositories.
Desktop (please complete the following information):
Additional context
Not required.
Repo in question is: https://github.com/mozilla-releng/addonscript
Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.
Describe the bug
While running the script the following error appears :
Traceback (most recent call last): File "C:/repo/firefox-infra-changelog/client.py", line 351, in <module> create_files_for_hg() File "C:/repo/firefox-infra-changelog/client.py", line 221, in create_files_for_hg create_hg_md_table(repository_name) File "C:/repo/firefox-infra-changelog/client.py", line 112, in create_hg_md_table md_file.write(value + "\n\n") File "C:\Users\adrian.pop\AppData\Local\Programs\Python\Python36-32\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\u0131' in position 2583: character maps to <undefined>
Repo in question is: https://github.com/mozilla-releng/bouncerscript
Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.
Added two repos, which have the highest potential to cause bustage:
taskcluster-auth : https://github.com/taskcluster/taskcluster-auth
taskcluster-queue : https://github.com/taskcluster/taskcluster-queue
Still to confirm if we should track all files/folders or not.
Opening this feature request as I would like to suggest creating a main markdown table containing information such as one listed below:
What's your opinion on this?
Repo | Link | Last Message | Data |
---|---|---|---|
ship-it | link | Nightly 65: Merge and deply when 65 builds are available for all locales (#238) | 2018-10-23T09:27:10Z |
release-services | link | staticanalysis/frontend: Load all tasks & support http errors (#1651) | 2018-10-26T09:44:46Z |
beetmoverscript | link | Bug 1408083 - Drop support for beta/releases of single locale repacks for Fennec (#187) | 2018-10-03T10:31:17Z |
addonscript | link | Merge pull request #96 from mozilla-releng/pyup-update-scriptworker-14.0.0-to-15.0.0 Update scriptworker to 15.0.0 | 2018-08-06T02:31:08Z |
shipit-v2 | link | RM ALL | 2018-09-06T19:59:08Z |
build-cloud-tools | link | Merge pull request #368 from garbas/remove-uplift-frontend terraform: remove uplift frontend domains | 2018-10-23T14:44:31Z |
build-puppet | link | Bug 1481203 - allow worker to use new thunderbird ship-it scopes (#278) The release-mark-as-shipped task for Thunderbird has new scopes assigned to it. This patch updates the scriptworker configuration to recognize those scopes, preventing the 'This worker is not configured to handle scope "project:comm:thunderbird:releng:ship-it:server:production"' error when run. | 2018-10-26T20:11:29Z |
shipitscript | link | Merge pull request #13 from escapewindow/py37 require py37 to be green | 2018-07-27T23:05:59Z |
bouncerscript | link | Merge pull request #41 from tp-tc/stub-path-platforms Check that paths exist per-platform. | 2018-09-04T22:05:15Z |
treescript | link | Merge pull request #111 from tp-tc/bump-robustcheckout.py Bump robustcheckout.py. | 2018-10-02T16:43:57Z |
mozapkpublisher | link | 0.9.0 | 2018-10-19T12:42:47Z |
Describe the bug
After adding all of the github repositories to repositories.json
, the script started throwing the following error
Traceback (most recent call last): File "E:/bcrisan/repos/fic/client.py", line 186, in <module> useful_data = filter_commit_data(commit_data) File "E:/bcrisan/repos/fic/client.py", line 151, in filter_commit_data commit_message = item['commit']['message'] TypeError: string indices must be integers
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The creation of multiple .md
that would contain the commits table for each repository newly added under github json.
Desktop (please complete the following information):
def filter_commit_data(commit) currently breaks generated markdown file.
Will modify the function for a workaround.
Example:
Commit Number | Commiter | Commit Message | Date |
---|---|---|---|
1 | Patrick Cloke | Bump Thunderbird to version 65. (#239) | 2018-10-29T22:40:41Z |
2 | Rob Lemley | Bug 1502445 - Remove obsolete locales from Thunderbird. (#241) |
The downloads page on thunderbird.net uses thunderbird_primary_builds.json
to generate the download list. There's some locales included there
that have been removed from Thunderbird in bug 1310221. This patch
brings thunderbirddetails.py (mostly) inline with all-locales in
comm/mail/locales.|2018-10-29T22:30:04Z
|3|Rail Aliiev|Use raw string for regexes (#242)|2018-10-29T22:22:50Z
|4|Pascal Chevrel|Nightly 65: Merge and deply when 65 builds are available for all locales (#238)|2018-10-23T09:27:10Z
|5|Nick Thomas|Use N most recent releases only (#237)|2018-10-17T20:10:15Z
|6|Pascal Chevrel|Update false Fennec instructions in template (#236)|2018-09-10T16:20:59Z
|7|Patrick Cloke|Bump Thunderbird nightly version to 64. (#234)|2018-09-05T14:40:36Z
|8|Pascal Chevrel|Update available firefox nightly version to 64. (#233)|2018-09-05T14:28:27Z
|9|Julien Cristau|Bug 1460752 - bump CURRENT_ESR to 60 (#235)|2018-09-05T13:27:57Z
|10|Rail Aliiev|Reformat, add more info (#232)|2018-07-13T20:49:08Z
|11|Rail Aliiev|Handle "legacy" releases properly, kill python2.6 (#231)
Repo in question is: https://github.com/mozilla-releng/balrogscript
Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.
Git and HG returns the date and time in different ways.
eg. Unix Timestamp from HG
To not be confusing we have to use a single date/time format.
We could start deciding what date/time format to use. For example the most popular date/time format is the following: YYYY-MM-DD Thh:mm:ss (eg 2018-11-02 T12:10:30) (follow the order of largest to smallest units)
-where:
YYYY = four-digit year
MM = two-digit month (01=January, etc.)
DD = two-digit day of month (01 through 31)
hh = two digits of hour (00 through 23) (am/pm NOT allowed)
mm = two digits of minute (00 through 59)
ss = two digits of second (00 through 59)
-also we can add:
TZD = time zone designator (Z or +hh:mm or -hh:mm)
Z = UTC (Coordinated Universal Time)
Repo in question is: https://github.com/mozilla-releng/signtool
Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.
Currently the date in changelog.json is displayed like : 2018-10-29T22:40:41Z and commit numbers are nothing but "upside-down". We want date to be displayed without those "T" and "Z".
Using tests/md_changelog.py create a test script that will read the data stored in the following two json files: git_changelog.json
, hg_changelog.json
and add the aggregated and formatted data to changelog.md
Time | SHA | Commit Msg | Committed by | Repository | Product | Component |
---|---|---|---|---|---|---|
07/13/2018 - 20:30 | 86fbbd2df71d | Danut Labici | The best commit that fixes all bugs | Release Warrior | Desktop | Cloud-Tools |
07/13/2018 - 20:30 | 86fbbd2df71d | Danut Labici | The best commit that fixes all bugs | Build Puppet | Desktop | Cloud-Tools |
07/13/2018 - 20:30 | 86fbbd2df71d | Danut Labici | The best commit that fixes all bugs | OpenCloudConfig | Desktop | Cloud-Tools |
07/13/2018 - 20:30 | 86fbbd2df71d | Danut Labici | The best commit that fixes all bugs | BB Configs | Desktop | Cloud-Tools |
Time | SHA | Commit Msg | Committed By | Tree | Product | Component |
---|---|---|---|---|---|---|
07/13/2018 - 20:30 | 86fbbd2df71d | Danut Labici | The best commit that fixes all bugs | Moz-Build | Desktop | Cloud-Tools |
07/13/2018 - 20:30 | 86fbbd2df71d | Danut Labici | The best commit that fixes all bugs | Moz-Build | Desktop | Cloud-Tools |
07/13/2018 - 20:30 | 86fbbd2df71d | Danut Labici | The best commit that fixes all bugs | Moz-Build | Desktop | Cloud-Tools |
07/13/2018 - 20:30 | 86fbbd2df71d | Danut Labici | The best commit that fixes all bugs | Moz-Build | Desktop | Cloud-Tools |
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.