Monitoring tooling I've looked at: https://github.com/MeasureOSS/Measure

from zalando.github.io.

This issue needs to be done first, in order to get started on evaluating monitoring tools. #26

from zalando.github.io.

Purpose

To define what we want to measure, how should the monitoring dashboard look like, and the type of reports and alerts to be generated from the monitoring infrastructure MVP.

Problem Statement

The Open Source team is a newly formed entity within Zalando. It is very important we gather metrics around Zalando's Open Source involvement so that we can measure the impact this new team is able to bring and be able to effectively drive the open source strategy that the team has in place.

Background information

CatWatch is a Zalando Incubator web application that fetches GitHub statistics about the total number of repositories, contributors, languages, forks, stars etc. for our GitHub accounts, saves this data in a database, then makes the data available via a REST API.

This data is limited and we will need more insights (discussed below). In order to achieve this, below is the proposed solution.

Proposed Solution

Points italicised are out of scope from the MVP.

Measure

Analytics, dashboard and reports: Be able to measure all the activity in Zalando's Github organisations. This would be presented in a customised dashboard view and reports generated periodically.

• New projects added to the incubator
  • No. of projects graduated from the incubator
• Total no. of PRs opened vs merged
  • Time taken to merge a PR
  • PRs from external contributors
• Total no. of issues created vs closed
  • Time taken to respond to an issue
  • Total no. of comments in an issue
  • Issues created from external contributors
  • Issue comments from external contributors
• Top contributors - weekly, monthly
  • Organisation wide
  • Project specific
  • External top contributors
• No. of new contributors
  • Organisation wide
  • Project specific
  • External new contributors
• No. of external contributors
  • Organisation wide
  • Project specific
  • Female contributors
• Comparison of all the above metrics with old projects vs new projects
• No. of cross-team collaborations (inside Zalando) - Issue + PR
  • No. of new collaborations - Issue + PR
  • No. of female participants - Issue + PR
  • No. of non-tech collaborations - Issue + PR
  • No. of PRs opened
  • Time taken to open vs merge cross-team PR
  • No. of issues created
  • Time take to open vs close issue
• No. of Zalandos contributing to external projects
  • No. of issues created to external projects
  • No. of PRs opened to external projects
  • Time taken to open vs merge Pr
  • Time taken to open vs create issue
  • No. of issue comments on external projects
  • No. of female contributions to external projects
    • Issues created by female contributors
    • PRs opened by female contributors
    • Issue comments by female contributors
• No. of projects with releases
  • No. of new releases
• Diversity(Over time, these numbers should improve) 👩
  • No. of female contributors
  • No. of female maintainers
  • No. of new female contributors
  • No. of top female contributors
  • No. of external vs internal female contributors
  • No. of PRs opened vs merged by female contributors
  • No. of issues opened vs merged by female contributors
  • No. of issue comments by female contributors
• No. of non-tech contributions
  • No. of PRs opened to update docs
  • Time taken to open vs merge
  • Issues opened to updated docs
  • Time take to open vs close
  • No. of external contributors
  • No. of female contributors
• Retention / Attrition
  • No. of external contributors hired
  • Time taken to hire
  • No. of internal contributors who no longer work for Zalando
  • Time spent contributing to open source by internal contributor before leaving
  • No. of external female contributors hired
    • No. of internal female contributors left
• Compliance
  • Number of compliant projects based on Rules Of Play
  • No. of new PRs which aren't compliant (Over time, these numbers should decrease)
• Social media
  • Twitter impressions on a tweet mentioning particular project
  • Twitter impressions from top contributors published
  • Increase in twitter followers due to the above 2 points

Alerts

• New project added to the incubator
• Project graduated from the incubator
• New PR opened
• New issue created
• Organisation stats reaching a milestone
  • Stars, contributors, forks == 100, 500, 1000 etc.
• Project stats reaching a milestone
  • Stars, contributors, forks == 100, 500, 1000 etc.
• PRs, issues reaching a milestone = 100, 200, 300 etc.
• PR, issue staying open for a long time, more than a month
• PR, issue unattended for more than two weeks
• External contributor hired
• Internal contributor left
• New PR, issue from a female contributor
• New tweet about a Zalando specific project

from zalando.github.io.

MVP

Measure

Alerts

• Organisation stats reaching a milestone
  • Stars, contributors, forks == 100, 500, 1000 etc.
• Project stats reaching a milestone
  • Stars, contributors, forks == 100, 500, 1000 etc.
• Total PRs, issues reaching a milestone == 100, 200, 300 etc.
• PR, issue staying open for a long time, more than a month
• PR, issue unattended for more than two weeks

from zalando.github.io.

As an additional KPI we should measure our teams ability to publish content on the team blog, social media and participating in events

• Number of blog posts on the tech blog
• Number of talks from a team member
• Twitter impressions on Zalando Github repo links

from zalando.github.io.