tikv / community Goto Github PK

Background

To evolve towards the next generation database engine kernel, we reimplemented RocksDB using rust. This is our project address: https://github.com/rust-lib-project/rocksdb-rs. Of course, we will not implement all the functions of RocksDB. Our purpose is to create a more general KV data engine for TiKV services, not a transactional data engine for MyRocks services. In this project, we will eliminate most of the features that TiKV in RocksDB does not use, including transactions and two-phase commit code, in order to simplify our code as much as possible and make the project easier to maintain.
For the cloud ecosystem, asynchronous IO is essential for high-latency cloud disks. Fortunately, rust provides an asynchronous framework that is easier to use than C++, allowing us to easily write asynchronous IO code. Therefore, most of the interfaces of this project are provided as asynchronous method. Thanks to all interfaces supporting asynchrony, we can merge the compaction thread pool with other background thread pools in tikv to reduce thread switching and control CPU resources more precisely.
At present, rocksdb-rs still maintains similar interface methods and functions as rocksdb, but it adopts a low-coupling architecture, which can be easily refactored into a more friendly interface for TiKV in the future.

Roadmap

At present, the project has implemented the basic functions of rocksdb, including read and write queries and background merging. But it still lacks a lot of tests, and some key functions (you can see more details in https://github.com/rust-lib-project/rocksdb-rs/issues), I hope he can get help from more developers under the tikv organization and play a bigger role.

Due to historical reasons, components like tidb_query_aggr, tidb_query_expr etc. are put in tikv repository as they are part of coprocessor. But it may not be a good choice as they are highly tidb specific, may not suitable for a general KV project. And we also lack of active maintainers/committers for these components. So I suggest to move these components to pingcap org.

The components to be moved include:

• tidb_query_expr
• tidb_query_codegen
• tidb_query_common
• tidb_query_datatype
• tidb_query_executors

They also depend on some of internal components including

• log_wrappers
• codec
• collections
• tikv_util
• file_system

They should be either published to crates.io or inlined all used functions/types.

After this change, these projects will not be part of tikv/tikv anymore. But they will still be part of release binaries.

Any thoughts?

Due to the fact that the governance document has been outdated for some time. I hereby propose a refresh of governance document according to the new affiliations of steering committee members and infra members.

The changes are:

• Clear steering committee member Jay Li's affiliation based on his own preference.
• Replace infra members, Zili Chen and Qiang Zhou, to Junchen Ye and Wuhui Zuo.

Please make comments if you have any concern, thanks

Subtasks:

• #121
• #123
• #124
• #134

What is the problem our community meet?

• Most of our repositories have no clear owners; for example, it is hard to say who has review or commit permission of mur3 by our governance and community information.
• Special Interest Group benefits domain-specific discussion, but sucks when coupled with governance; for example, it is hard to say a contributor contributing to TiKV security belongs to which SIG. It is to be blamed for that many of our committer level contributors (e.g., @innerr) don't become committers really.
• We have no active exit mechanism with which we take down review or commit permission of member abuses. And thus our community is a bit closed to involve new member, and break down members into SIGs even in one repository for limiting permission which sucks.

What is proposed to do?

• Break down the review and commit permission group into teams, one team could own one or more repositories.
• Associate every repository with one and one and only one team.
• Define the team governance to include explicit exit mechanism, and community governance to build new team.

Details about actions

Break down the review and commit permission group into teams.

A team has its reviewer, committer and maintainer, and owns one or more repositories.

The permission of reviewer and committer is similar with current state but indistinguishable among the team. It means a reviewer of the team can review all pull requests in repositories owned by the team.

The permission of maintainer, based on that of committer, is mainly for exit mechanism and other team level decision making, which will be elaborated later.

Within this proposal, there are 4 teams proposed.

• TiKV team
• PD team
• Prometheus team
• Raft team

Associate every repository with one and one and only one team.

Every repository of TiKV org will be associated with one and only one team. Reviewer and committer of that team has the review or commit permission to the repository.

As we build 4 teams above, the relation between team and repository as listed below. Comment if you think it is not the real case.

The exceptions are tikv/community and tikv/rfcs.

• tikv/rfcs is associated with all teams which means we put rfcs together among teams.
• tikv/community is for governance and only the maintainers has permission.

Define the team governance to include explicit exit mechanism, and community governance to build a new team.

A team has its reviewer, committer and maintainer.

• A reviewer is able to review pull request for quality and correctness of repositories associated with the team. His/Her LGTM is counted.
• A committer is able to both review and commit pull request.
• A maintainer, based on the rights of a committer, is able to promote or evict(emeritus member for reference) a reviewer, a committer or a maintainer by majority decision.

The maintainers is able to evict member by majority decision and thus we have an exit mechanism. It is suggested that a team defines its own bylaws to elaborate other team level decision making process.

Besides the teams, the whole TiKV community has its TOC members for evolving the community. The TOC members consist of part of maintainers of teams and are in charge of community level decision making process like build a new team, or arbitrate divergence between team by majority decision.

Where do current things go?

Here is the migration details from current SIG membership to team membership.

Where does SIG concept go?

SIG itself means expert group, which always exists in fact, and serves for better reference to help. However, this concept should never be associated with permission.

Shall we add a new concept / infra named "Team"?

Yes and no. We name it as "Team" for distinguished from "SIG", but the authentication #123 is quite similar as "SIG at repo granularity" or said GitHub teams. There is no much difference.

Maybe when he is not that tired, he'd like to contact someone who knows the community rule of the tikv to help him solve the problem.

Based on my knowledge, only tikv committers can push tags directly.

Possible solutions include as follows:

• Inviting one of the committers to take care of this project, maybe @andylokandy? They don't need to take part in the development, just need to push tags for release. I think it's not a big burden since pprof-rs is not heavy traffic.
• Ping one of tikv's maintainers to give you more rights on this project. Because it doesn't make sense that you are the maintainer of pprof-rs but don't have permission to push tags.
• Transfer this project out of tikv's umbrella if the tikv community doesn't have an interest in maintaining it.

Also, ping @tisonkun for more advice about the current situation.

Databend depends heavily on pprof-rs and I'm interested in maintaining pprof-rs, we can help maintain this project.

Originally posted by @Xuanwo in tikv/pprof-rs#139 (comment)

subtask of #118

Since our permissions are flatten into repository level. @hi-rustin and I tend to use GitHub team to maintain the permission and leave the bot simply respect GitHub permission.

ti-chi-bot can apply such rules according to this page

use_github_permission: Use GitHub permissions and have write and admin collaborators as reviewers and committers

And for repositories where bot is not configured (bot has a limit on events received), the model retains.

In voting-150, we passed the scheme of using commit message in the PR to associate the issue, so that the squash commit can also associate with the issue.

But in the process of practice, it will bring some troubles to code contributors, especially in the case where we have multiple release branches, we need to submit meaningless commits or modify the commit message many times to pass the check.

For now, we forked out a customized version of prow, which is the upstream code repository of the @ti-chi-bot .

Based on it, we add some functions to make @ti-chi-bot be able to automatically extract issue numbers from the PR body and automatically populate the issue number to the squash commit message when merging.

For example:

We just need to fill in a line starting with Issue Number: of text in the PR body. (For concrete rules, please refer to the description in the TiDB development guide.)

The robot will extract the issue number section in the squash commit message when merging.

I think that this schema replacing our original schema will make our work easier. If that makes sense to everyone, I will

1. Disable the format-checker on tikv and pd repo, no longer enforcement requires issue number in PR's commmit.

2. Call for a public vote on the new schema, continue to complete the issue #149 ( cc: @zhangyangyu )

As discussed in #152, we use alphabetical order in team member json files. Therefore, we may need a style checker for this enforcement.

subtask of #118

Here is the dedicated issue for making decision on member : team : repository mappings.

A team has its reviewer, committer and maintainer, and owns one or more repositories.

The permission of reviewer and committer is similar with current state but indistinguishable among the team. It means a reviewer of the team can review all pull requests in repositories owned by the teams.

The permission of maintainer, based on that of committer, is mainly for exit mechanism and other team level decision making, which will be elaborated later.

I create a pull request #122 with the proposed teams and their members as well as repositories associated with.

As one of the founders of project TiKV, Dongxu Huang has made significant contributions to TiKV. I hereby nominate Dongxu Huang as member of TOC.

Everyone is welcome to comment.

As one of the founders of TiKV project, Qi Liu has made significant contributions to TiKV. I hereby nominate Qi Liu as member of TOC.

Everyone is welcome to comment.

• make the proposal and pass the vote #162
• modify PR template tikv/tikv#11886 tikv/pd#4603
• update contribute guide tikv/tikv#11831
• apply new tide ti-community-infra/configs#505

match_template is a procedural macro that generates repeated match arms by pattern, which is almost the only way to dispatch code path to static types depending on a runtime value. Since it's general enough, I propose to move it to TiKV org and then publish it to crates.io so that the rust community could be benifit from it.

subtask of #118

With consensus in #118 we'd like to formalize the governance and make it no longer a draft. This issue is a subtask tracing all contents updates. So far, it seems containing

• TOC description
• Team description

Also we should remove all stale contents including committee/, sig/, and wg/.

cc @luzizhuo @gingerkidney

subtasks

• #131

• #151
• tikv/tikv#11016
• tikv/pd#4186
• ti-community-infra/tichi#764
• ti-community-infra/tichi#768
• tikv/tikv#11097

Proposal: TiKV Development Guide was posted at 2021-07-21, and during the last half year we finished several chapter of the guide at https://github.com/tisonkun/tikv-dev-guide.

However, there're still 36 outstanding issues and thus at least 36 sections to be written. After a discussion with @zz-jason, he proposed that we can transfer the dev guide to tikv org and proceed the work.

I'm glad to see any pull requests to fulfill new sections, and don't insist on what owner the repo belongs to. The important thing, from my side, is the content itself. If transferring the repo to tikv org can help moving forward the progress, I'm willing to do so.

So here is the issue to discuss about such a transfer. Looking forward to responses.

For achieving the transfer, I think I have all required permission to do it.

subtask of #124

• roles in TiKV community, user, contributor, reviewer, committer, maintainer (and TOC member later in other subtask)
• how a member goes to a role, i.e., the voting process
• how a member retire from a role, i.e., the exit mechanism