Proposal: Reduce the Entropy of TiKV Community Governance about community HOT 70 CLOSED

But please focus on current task.

I think we need to deal with this issue as soon as we change the permission model. They will immediately lose permissions of the release branch.

Make sense. I can create an inline proposal in #122 . Although I previously think of separation actions so that we can focus one by one.

from community.

How about leaving just one sig-tikv? (by @sykp241095 from gist)

There is a TiKV Project after this proposal applied. The difference is that we also have PD project and Raft project since they seem quite independent from TiKV projects. Hadoop, ZooKeeper, and Ratis are different project and I think it is similar to TiKV, PD, and Raft. BTW, this comes from the offline discussion with @BusyJay .

from community.

Are we going to break down SIGs? (by @breeswish from offline)

Nope. We define a community governance struct that project owns repositories, and you might regard it more as SIG combination. The details of projects and the mapping from repository to project is as above.

Rationale

We should decouple community governance from enterprise organizational structure where SIG is born. Because the community governance reflect the community itself and the evolution. SIG Engine exists because there is a component, not because there is a team inside a company working for it.

I do agree theoretically it is possible to maintain OWNERS files to keep community governance reflect the software structure in fine-grained. But it costs too much and I do not think TiKV is gonna need it. We can use a simpler and permissive model, that is, the project model, to avoid the mismatch problem. All TiKV reviewers and committers are equal to react to all events happen in the project. We do never separated ourselves unnecessarily, especially inside a repository.

from community.

cc @BusyJay @yiwu-arbug @innerr @zhangjinpeng1987 @nolouch

from community.

cc @hi-rustin

from community.

If there is no objection on the general proposal, I'll elaborate concrete actions and call for applying.

from community.

How existing SIG structure works under the new project-based governance? Does SIG specific maintainers and reviewers still being maintained, or transfer to become project maintainers and reviewers?

from community.

and you might regard it more as SIG combination.

If this is actually a combination of SIG, then I would prefer it to follow the concept of SIG from the point of view of infrastructure changes, so that there are minimal changes to the whole infrastructure.

If this part introduces the concept of project, then the community's website will need to reorganized, as both tikv and tidb SIGs are now displayed on the website. We may need to redesign the website to show the concept and architecture of project. The architecture of the bots also needed to be adapted, including ti-community-bot and tichi. This required a huge cost to do the migration.

from community.

How existing SIG structure works under the new project-based governance? Does SIG specific maintainers and reviewers still being maintained, or transfer to become project maintainers and reviewers?

short answer: "transferred"

@yiwu-arbug you can see the details of migration above for a proposal.

I think the major issue is that SIG concept should be barely special interest group, not associated with permission. As in the Apache Spark community there are expert groups (SIGs) for SparkR / Runtime / Streaming, but all permissions are flatten to Spark project.

With an offline discussion with @winkyao there is one more issue : a number of current maintainers and reviewers are already inactive, we should help them emeritus.

from community.

and you might regard it more as SIG combination.

If this is actually a combination of SIG, then I would prefer it to follow the concept of SIG from the point of view of infrastructure changes, so that there are minimal changes to the whole infrastructure.

If this part introduces the concept of project, then the community's website will need to reorganized, as both tikv and tidb SIGs are now displayed on the website. We may need to redesign the website to show the concept and architecture of project. The architecture of the bots also needed to be adapted, including ti-community-bot and tichi. This required a huge cost to do the migration.

@hi-rustin Name problem is significant. We can focus the permission migration at first but a subtask for name problem will be created and push to done.

1. permission migration
2. bot project name adaption
3. website updates

from community.

• permission migration

• bot project name adaption

Technically speaking, the two steps are inseparable. At least on tikv, they are inseparable.

from community.

Name problem is significant.

I remember you saying in the offline discussion that it didn't matter what you called it, so I prefer to follow the concept of SIG rather than introduce a new concept.

from community.

Name problem is significant.

I remember you saying in the offline discussion that it didn't matter what you called it, so I prefer to follow the concept of SIG rather than introduce a new concept.

Fine. I can see the overhead but still insist that permission rules changed and we'd better avoid override concepts on the same name.

I'd emphasize that project is different from sig.

• Member : Project = 1:n
• Project : Repository = 1:n

A reviewer / committer has permission to a PR of a repository because he / she is one of members of the project the repository belongs to.

So far, sig works on labels / default sig / default to all sigs. And permission works in addition with trust team / trust member. We don't need too much. They are legacy patches for the wrong permission model.

from community.

A reviewer / committer has permission to a PR of a repository because he / she is one of members of the project the repository belongs to.

Can't we just change the sig's permission logic to this?

from community.

It seems like we are just merging sigs and assigning permissions to those sigs according to the level of the repo?

from community.

A reviewer / committer has permission to a PR of a repository because he / she is one of members of the project the repository belongs to.

Can't we just change the sig's permission logic to this?

Yes we can. That is what I mean in

1. >permission migration
2. bot project name adaption
3. website updates

I'll remind that TiDB using sig model also and you cannot change at once. That means, base on "They are legacy patches for the wrong permission model.", you are adding new patches.

from community.

That means, base on "They are legacy patches for the wrong permission model.", you are adding new patches.

If it's essentially merging sigs and managing permissions by repo level, then I don't see it as adding a patch, but rather correcting our original faulty permission model according to your logic? Because currently in our community, sig is already tied to permissions.

from community.

It seems like we are just merging sigs and assigning permissions to those sigs according to the level of the repo?

Definitely, you can see also #118 (comment) .

In additionally we may enable emeritus but that is easy because we can add a file which does not participant permission model.

Said

- tikv project
  - member.json
  - emeritus.[ext]

or

// member.json
{
  "emeritus-committers" : [ "..." ],
}

from community.

I don't understand why we need to introduce new concepts if essentially what's going on is merging sigs and adjusting the permission model?

from community.

I don't understand why we need to introduce new concepts if essentially what's going on is merging sigs and adjusting the permission model?

I don't insist a new name but focus on the permission model and keep member permission configs done inside tikv org (project : repository could be done in ti-community-infra/config and I think it is how ASF does it).

from community.

If committers and reviewers of SIGs are transferred to TiKV, then there will be no SIGs anymore, is it correct?

If I have write permission in a repository, then I should also have write permission in other repositories within the same project, is it correct?

from community.

@BusyJay

If committers and reviewers of SIGs are transferred to TiKV, then there will be no SIGs anymore, is it correct?

See "Where do current things go?" section:

SIG itself means expert group, which always exists in fact, and serves for better reference to help. However, this concept should never be associated with permission.

Other community example, Apache Spark community has expert groups (SIGs) in fact for SparkR / Runtime / Streaming, but all permissions are flatten to Spark project.

If I have write permission in a repository, then I should also have write permission in other repositories within the same project, is it correct?

Yes and no. You can follow the term

"I am a committer of TiKV project, then I have write permission to repositories belongs to TiKV project, like tikv/tikv, tikv/client-rust, and so on."

"No" because we don't grant permission directly to a person in repository level.

from community.

...However, this concept should never be associated with permission.

Then what's point of becoming a committer/reviewer of TiKV that can't write/review any code in TiKV org?

"No" because we don't grant permission directly to a person in repository level.

For repository like rust-prometheus that is maintained by community, how to grant the write permissions for community members not in TiKV project (and any SIGs)?

from community.

Then what's point of becoming a committer/reviewer of TiKV that can't write/review any code in TiKV org?

Any misunderstanding? If you are a committer of TiKV "project", you have the write permission of all its repositories, including tikv/tikv, tikv/client-rust, and so on. As listed above in details of "TiKV project"

"No" because we don't grant permission directly to a person in repository level.

For repository like rust-prometheus that is maintained by community, how to grant the write permissions for community members not in TiKV project (and any SIGs)?

As listed above in details of "TiKV project", it belongs to TiKV project.

UPDATE: after an offline discussion with @BusyJay , we agree with that current SIGs will be transferred to projects. SIGs won't exist as part of community governance and thus there is no committer / reviewer of a SIG any more.

For "rust-prometheus" part, it might not belong to TiKV project and deserve its own project like Raft project for raft-rs. We will discuss in the subtasks.

from community.

I will then start the migration subtasks with confirmation from stakeholders. Let me know if you would like to participant in the migration committee and thus help on decision making. Or other stakeholders who should be included.

@BusyJay
@yiwu-arbug
@zhangjinpeng1987
@nolouch
@innerr
@hi-rustin
@skyzh
@breeswish
@andylokandy
@youjiali1995
@winkyao
@sunxiaoguang

from community.

Other community example, Apache Spark community has expert groups (SIGs) in fact for SparkR / Runtime / Streaming, but all permissions are flatten to Spark project.

To add, we originally referenced cncf's sig architecture, and their sig is tightly tied to permissions.
For example: https://github.com/kubernetes/community/tree/master/sig-network#subprojects

from community.

SIGs won't exist as part of community governance and thus there is no committer / reviewer of a SIG any more.

To clarify, we will not remove the concept of SIG directly, we will follow the concept of SIG in the implementation. The projects we are talking about now will become the new SIGs, e.g. sig-tikv, sig-raft, sig-pd.

The reasons for this I discussed with @tisonkun

1. what we are essentially doing is changing the permission model and merging some sigs, so it doesn't matter what the implementation is called. Our sig has been running for a long time and is bound with permissions, so we can stick with the sig concept. Instead of making huge changes.

2. because our current tidb and tikv communities are governed together, and introducing new concepts would lead to confusion in the community governance. Also the introduction of a new concept itself brings learning and promotion costs.

3. If new concepts are introduced, the whole community infrastructure will have to be changed, ,including bots, community website, and governance framework. (Introducing the new concept required a re-implementation of almost all permission-related tools and took at least 1-2 months to implement.)

from community.

Other community example, Apache Spark community has expert groups (SIGs) in fact for SparkR / Runtime / Streaming, but all permissions are flatten to Spark project.

To add, we originally referenced cncf's sig architecture, and their sig is tightly tied to permissions.
For example: https://github.com/kubernetes/community/tree/master/sig-network#subprojects

@hi-rustin you can always regard project as a new sig, and we should stop battle on names.

As you can see k8s's subprojects are well formed in repos and subdirs. The repos part is the same as what proposed here. The subdir parts are tightly coupled with how Golang project construct codetree. It is not quite easy to use for general code repositroy.

from community.

@hi-rustin Name problem is significant.

@hi-rustin you can always regard project as a new sig, and we should stop battle on names.

🤣 Okay I will stop battle on names.

from community.

LGTM

from community.

I very glad to see this proposal, the permission issues are also bothering me. sometimes one feature may cross the sig or the project, but permission delayed our review work. but I have some question about how to work with the new proposal:

• I more agree with @hi-rustin that tidb and tikv communities should be governed together. Such as, how do we do the cross-project works?
• And we may need more detailed instructions on how to obtain the corresponding permissions. Am I already a committer of the sig, that is the committer of this project? and How about the newcome?
• As the above discussion, will a project be divided into many sub-projects, It's may build some wheels for the project, such as the rust-prometheus above. Will it meet the sig problem again?

from community.

Generally looks good to me. This solves the long-lasting permission issues for us. And I hope that kvproto could later be part of the TiKV project.

But this proposal casts doubt on how new contributors engage with our community. Who should they ask for a review after submitting PR? Previously we have sig-copr, so they could directly connect to us. Now in the new model, all of the committers and reviewers belong to the TiKV project, and this makes it unclear to know who has prior experience on a component and who can review a PR.

from community.

@nolouch

• I more agree with @hi-rustin that tidb and tikv communities should be governed together. Such as, how do we do the cross-project works?

We first apply the governance rule on tikv org and popularize it to the tidb community for better collaboration. tidb community suffers from unclear sig-repo permission model also.

We finally reach a place that, for example, you @nolouch are both a committer of pd and tidb and thus has the permission to commit to tikv/pd and pingcap/tidb.

• And we may need more detailed instructions on how to obtain the corresponding permissions. Am I already a committer of the sig, that is the committer of this project? and How about the newcome?

Yes. We will elaborate details in subtasks when we reach consensus on the overall direction described here.

For you question, with an offline discussion with @BusyJay and you can see the sentence above, we agree with that we have checkpoint (mandatory process) on "majority decision by maintainers" to promote a reviewer / committer. And every project can has its guideline (non-mandatory principle) for how to become a committer.

Please be aware that sig-scheduling, sig-coprocessor has gone, at least not associated with permission anymore.

• As the above discussion, will a project be divided into many sub-projects, It's may build some wheels for the project, such as the rust-prometheus above. Will it meet the sig problem again?

Nope.

The problem of SIG on permission model is we use multiple rules on review / commit permission

1. sig-label
2. default sig
3. default to all sigs
4. trust member
5. trust team

After the migration we have only one rule that if and only if you are a member of projects this repository belongs to, you have the corresponding permission to the repository.

from community.

But this proposal casts doubt on how new contributors engage with our community. Who should they ask for a review after submitting PR? Previously we have sig-copr, so they could directly connect to us. Now in the new model, all of the committers and reviewers belong to the TiKV project, and this makes it unclear to know who has prior experience on a component and who can review a PR.

For auto assign or domain expert issue, we can make use of GitHub CODEOWNERS. And for open source community we always overcome this issue by over communication. We have a number of members and one of members notice a new PR he/she will help to dispatch.

... or if the project agrees on it, it can post an expert map as in scala community, but the map itself is a hint, not associated with permission.

@hi-rustin may have more insights on auto assign or dealing with stale pr.

from community.

The problem of SIG on permission model is we use multiple rules on review / commit permission

1. sig-label

2. default sig

3. default to all sigs

4. trust member

5. trust team

After the migration we have only one rule that if and only if you are a member of projects this repository belongs to, you have the corresponding permission to the repository.

Regarding the trust user and trust team part, I would like to know how the release team will be handled? Is it added directly to each project?

from community.

@hi-rustin may have more insights on auto assign or dealing with stale pr.

Technically we could use CODEOWNERS, but as mentioned above CODEOWNERS are very expensive to maintain.

Another possible solution would be to automatically assign them based on the history of changes to the code, but it may still not be accurate because some people just used to work on certain modules and are no longer responsible for and maintain that part of the code.

Overall, this problem gets worse when permissions are scaled up. Unless we maintain and update the owners file (either GitHub's own or k8s' solution).

But I understand the cost of maintaining CODEOWNERS and k8s owners is about the same.

from community.

@hi-rustin if someone helps a lot on the release of a project, he/she undoubtedly deserves as a committer.

For blocking merge on releasing branch, I think it is another mechanism and not the main stream of community affairs.

That is, the release manager coordinate the release process and cheery-pick-approved is a tool helps the release manager.

from community.

Overall, this problem gets worse when permissions are scaled up. Unless we maintain and update the owners file (either GitHub's own or k8s' solution).

No. For now you maintain the membership file or CODEOWNERS file in pingcap/tidb already. It doesn't get worse.

from community.

No. For now you maintain the membership file or CODEOWNERS file in pingcap/tidb already. It doesn't get worse.

I see a lot of this operation in pr, and it doesn't actually work.

from community.

@hi-rustin if someone helps a lot on the release of a project, he/she undoubtedly deserves as a committer.

Got it, so we'll add the core members of the release team to each project.

from community.

Got it, so we'll add the core members of the release team to each project.

I don't like this sentence. We promote people who helps a lot on the release of a project the project committer.

No "core members of the release team".

from community.

looks good to me!

from community.

I'm going to break down this proposal as several subtasks now...

from community.

I'm going to break down this proposal as several subtasks now...

Done as

• #121
• #123
• #124

Audiences of this issue please take a look.

from community.

@winkyao do we need review from CNCF side?

from community.

@BusyJay Governance review by maintainer is enough, I will look into these discussions later. thanks

from community.

LGTM

from community.

LGTM

from community.

LGTM

from community.

+1, the proposal LGTM.

Another problem that hasn't discussed is how to efficiently manage github issues and pull requests among multiple repos for a project. For example, tikv-project has the following repos as described in the proposal, it's too hard to maintain issues pull requests on these repos:

tikv
client-c
client-py
client-go
client-java
client-rust
client-cpp
agatedb
async-speed-limit
bytes
client-validator
copr-test
crc64fast
crossbeam
fail-rs
grpc-rs
importer
jemalloc
jemallocator
minitrace-go
minitrace-rust
minstant
mock-tikv
mur3
pprof-rs
procinfo-rs
protobuf-build
raft-engine
rocksdb
rusoto
rust-prometheus
rust-rocksdb
sig-transaction
sysinfo
terraform-tikv-bench
tikv-operator
tikv.github.io
titan
tlaplus-specs
website
yatp

from community.

For example, tikv-project has the following repos as described in the proposal, it's too hard to maintain issues pull requests on these repos:

@zz-jason

For pull request, it is not quite hard because pr is created where code changes. The difficult part would be how to coordinate related pull requests.

For issue part, yes. From my experience in Flink community, we use a single JIRA board to trace all issues and dispatch to repos. If we directly follow this pattern, we can close all issue function on satellite repositories and open on tikv/tikv as the issue board.

But let's say it is an issue tracking and code collaboration topic which could be eased by best practice sharing on that domain. It doesn't block this topic.

If you are troubled on this kind of problem, I'd suggest you create a topic on our discussion forum and I'll be glad to discuss deeper.

from community.

But let's say it is an issue tracking and code collaboration topic which could be eased by best practice sharing on that domain. It doesn't block this topic.

@tisonkun agree, let's discuss this problem in the forum.

from community.

As SIGs are merged into projects and SIG tech leaders becomes project maintainers, it's confusing when talking about TiKV maintainers. It can refer to current TiKV maintainers listed in https://github.com/tikv/tikv/blob/master/MAINTAINERS.md, it can also refer to project TiKV maintainers.

from community.

As SIGs are merged into projects and SIG tech leaders becomes project maintainers, it's confusing when talking about TiKV maintainers. It can refer to current TiKV maintainers listed in https://github.com/tikv/tikv/blob/master/MAINTAINERS.md, it can also refer to project TiKV maintainers.

Yes we override the term and it makes confusions...I think @winkyao could give inputs here.

At the very first time of tikv org there is almost tikv/tikv only, while later we have pd, raft-rs and rust-promethues join the community. However, we don't update the description of original "TiKV Maintainers" and actually they hardly work as described in the doc.

The role of original "TiKV Maintainers" are more like TOC (technical oversight committee) nowadays. In the description above I use the term "committees" to refer it. I will refresh the content to mention it consistent as TOC.

from community.

* tikv/community is for governance and only the community TOC members has rights.

There will be some infrastructure changes in the community repository, maybe we need to give sig-community-infra permissions?

Also, for the tikv project I currently have triage permissions to solve problems that people might encounter with bots, and I can also use sig-community-infra permissions to control the merging of PRs(in case of bot failure). How will this permission be handled when our permission model is modified ?

Also the permission of the release team needs to be taken into account this time?

cc: @Mini256 @zhouqiang-cl

from community.

@hi-rustin when you think of community infra (mechanism, not a concrete release), it is out of community governance scope. We might, due to the unstable state of infra, give infra team responding permission to help nurse the case. But it must be temporary and infra team as a support team resolve the underneath environment and request the project committer to take action (re-run / merge).

For the release team, I'm gonna create an issue propose promote them as committers as a followup. They deserve for the previous works. But please focus on current task.

from community.

But please focus on current task.

I think we need to deal with this issue as soon as we change the permission model. They will immediately lose permissions of the release branch.

from community.

it is out of community governance scope

The community's infrastructure team should be considered for community governance because it is clear that the team is responsible for the community's infrastructure and will be actively involved in the community's day-to-day collaboration. If the team does not belong to the tikv community, where should it belong? The support team of PingCAP or the support team of ti community?

from community.

The community's infrastructure team should be considered for community governance because it is clear that the team

Yes you are right. I should learn from other infra team description. But I insist the collaboration process and actor described above. #118 (comment)

from community.

I'd like to post a rename proposal here from #122 (review) by @hi-rustin

I still want to use the concept of teams rather than projects.
This will help people understand the new governance structure and associated permissions.

I think it is reasonable except we switch to a new name. But it is ok because we are just in a proposal phase. If there is no objection or we can reach a consensus in 72 hours, i.e., before 2021-06-11T19:35:00(UTC+8), I will update all description.

from community.

I'd like to post a rename proposal here from #122 (review) by @hi-rustin

I still want to use the concept of teams rather than projects.
This will help people understand the new governance structure and associated permissions.

I think it is reasonable except we switch to a new name. But it is ok because we are just in a proposal phase. If there is no objection or we can reach a consensus in 72 hours, i.e., before 2021-06-11T19:35:00(UTC+8), I will update all description.

I will update the description from "project" to "team" now.

from community.

I'm going to create teams based on the consensus on #122 ...

from community.

Could you consider sig-migration? There are some components in TiKV codebase that are written and maintained by sig-migration. Sig-migration needs review and commit permission too. 🙂

from community.

Could you consider sig-migration? There are some components in TiKV codebase that are written and maintained by sig-migration. Sig-migration needs review and commit permission too. 🙂

I think your proposal is about the TiKV team only and thus before we formalize our community governance description, I'd like to involve current TiKV team maintainers to give advice and setup a lazy consensus with at least 2 approvals and no vetos to make the proposal passed.

Proposal 1.

@overvenus proposed that according to the previous work of part of sig-migration members, we promote them as reviewers or committers in the TiKV team.

@overvenus could you list the members here? It would be better we only list those who involved but not list all members in sig-migration just for convention because some of them only focus on pingcap org repositories and contribute nothing to tikv repository. You might think of writing in form

committers:
  @github-id-3
  @github-id-4
reviewers:
  @github-id-1
  @github-id-2

Proposal 2.

@jebter proposes that according to the previous work made by qa team we promote them as reviewers in TiKV team. They are

• @jebter
• @cosven
• @zyguan (already TiKV reviewer)
• @lilinghai
• @aytrack
• @shuke987

PTAL. cc for all TiKV team maintainers @BusyJay @andylokandy @breeswish @hicqu @innerr @nrc @skyzh @yiwu-arbug

from community.

Could you consider sig-migration? There are some components in TiKV codebase that are written and maintained by sig-migration. Sig-migration needs review and commit permission too. 🙂

I think your proposal is about the TiKV team only and thus before we formalize our community governance description, I'd like to involve current TiKV team maintainers to give advice and setup a lazy consensus with at least 2 approvals and no vetos to make the proposal passed.

Proposal 1.

@overvenus proposed that according to the previous work of part of sig-migration members, we promote them as reviewers or committers in the TiKV team.

@overvenus could you list the members here? It would be better we only list those who involved but not list all members in sig-migration just for convention because some of them only focus on pingcap org repositories and contribute nothing to tikv repository. You might think of writing in form

committers:
  @github-id-3
  @github-id-4
reviewers:
  @github-id-1
  @github-id-2

Proposal 2.

@jebter proposes that according to the previous work made by qa team we promote them as reviewers in TiKV team. They are

• @jebter
• @cosven
• @zyguan (already TiKV reviewer)
• @lilinghai
• @aytrack
• @shuke987

PTAL. cc for all TiKV team maintainers @BusyJay @andylokandy @breeswish @hicqu @innerr @nrc @skyzh @yiwu-arbug

I create a subtask for this case #131 that we already have to write, about how to vote and how to retire. The content will be present in the next week and we can value this case as the very first input to polish our rules. Hopefully it is based on trust and still respect majority consensus by Meritocracy.

from community.

Thank you, here is the list.

committers:
  @lonng
  @kennytm
  @Little-Wallace
  @overvenus
reviewers:
  @3pointer
  @glorv
  @amyangfei
  @ben1009
  @liuzix
  @lichunzhu
  @gozssky

from community.

@overvenus Thanks for your information.

• lonng, overvenus, and Little-Wallace are already tikv-commiters.

So the proposal is

1. Promote kennytm as TiKV committers.
2. Promote 3pointer, glorv, amyangfei, ben1009, liuzix, lichunzhu, gozssky as TiKV reviewers.

I'm gonna deal with #131 to clarify the promotion and later we ask for TiKV maintainers to review this promotion proposal, i.e., whether or not they are eligible to that roles.

from community.

Hi audience of this thread! You can see the vote for governance rephrase in #133 which will finally close this proposal.

@overvenus you can take a look at the governance rephrase and see how to propose new committer / reviewer then.

from community.

Thanks for all of your participant! Close as resolved.

@yiwu-arbug you can nominate new committers follow the section voting. We do need an example to follow later.

@overvenus you can reach tikv team maintainers to nominate current data platform members as committers / reviewers and then we remove the legacy team.

from community.

@tisonkun Thank you. Will initiate a vote when we need to promote someone. For now I don't have a candidate in mind.

from community.