Code Monkey home page Code Monkey logo

kvproto's Introduction

kvproto

Protocol buffer files for TiKV

Usage

  • Write your own protocol file in proto folder.
  • If you need to update raft-rs, please download the proto file respectively and overwrite the one in include folder.
  • Run make to generate go and rust code. We generate all go codes in pkg folder and rust in src folder.
  • Update the dependent projects.

Dependencies

  • Rust
  • Go
  • Protoc 3.8.0

Docker image

The easiest way to compile the protobufs is to use docker.

./scripts/docker-build.sh  # build the docker image once
./scripts/docker-run.sh make go

Build locally

There are many dependencies to be installed, please refer to the relevant scripts in workflow to set them up.

kvproto's People

Contributors

3pointer avatar 5kbpers avatar andremouche avatar andylokandy avatar breezewish avatar bufferflies avatar busyjay avatar connor1996 avatar coocood avatar disksing avatar hicqu avatar huachaohuang avatar husharp avatar iosmanthus avatar jayson-huang avatar jmpotato avatar joccau avatar leavrth avatar morefreeze avatar myonkeminta avatar ngaut avatar nolouch avatar nrc avatar overvenus avatar qiuyesuifeng avatar rleungx avatar siddontang avatar sticnarf avatar tiancaiamao avatar zhangjinpeng87 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

kvproto's Issues

BR: new interface for AWS EBS snapshot based Backup & Restore

Parent Feature pingcap/tidb#33849
Acceptance Criteria:
Define the interface between TiKV and BR

Task:

  • interface for BR to read region meta from a TiKV
  • interface for BR to send recover command to a TiKV
    phase 1: mainly on assign a leader to tikv
    phase 2: set tombstone (optional) it looks PD also has the capability to shutdown the peer when it is invalid.
  • interface for BR to send resolved data to a TiKV
    delete data from tikv by backup global resolved_ts

Support enum output string when json serialization of Message in go

Related issue: pd#3372

Since there is no MarshalJSON in the currently generated go code, when using the default json library of go to serialize the Message, the enum type will be displayed as a number, like {"role":3}.

We want it to be displayed as a string to get a better reading experience, like {"role":"DemotingVoter"}.

But in go, it is very difficult to change the behavior of json outside the package, so we need to generate relevant code in kvproto. There is a draft here. We need to discuss this.

Generate Wrappers for Prost protocols

Generate wrapper code which will wrap the code generated by Prost and emulate the API generated by rust-protobuf. The goal is that minimal changes are required by clients to use Prost rather than rust-protobuf.

Debug pretty printing

cc #377, #378

Printing using protobuf-rust has regressed. We should see what the printing with Prost is like.

Notes: should not print hidden fields (cached length, etc) or unset fields,

README instructions incorrect.

In attempting to follow the instructions on the README.md I'm unable to build the updated protos:

hoverbear@nomad:git/kvproto โ€นmasterโ€บ$ make
mkdir -p /home/hoverbear/git/kvproto/bin
# Standalone GOPATH
./generate_go.sh
+ check_protoc_version
++ protoc --version
+ version='libprotoc 3.6.1'
++ echo libprotoc 3.6.1
++ sed -n -e 's/.*\([0-9]\{1,\}\)\.[0-9]\{1,\}\.[0-9]\{1,\}.*/\1/p'
+ major=3
++ echo libprotoc 3.6.1
++ sed -n -e 's/.*[0-9]\{1,\}\.\([0-9]\{1,\}\)\.[0-9]\{1,\}.*/\1/p'
+ minor=6
+ '[' 3 -gt 3 ']'
+ '[' 3 -eq 3 ']'
+ '[' 6 -ge 1 ']'
+ return 0
++ basename ./generate_go.sh
+ PROGRAM=generate_go.sh
+ '[' -z /home/hoverbear/go ']'
+ GO_PREFIX_PATH=github.com/pingcap/kvproto/pkg
+ gogo_protobuf_url=github.com/gogo/protobuf
++ pwd
+ CURRENT_DIR=/home/hoverbear/git/kvproto
+ GOGO_ROOT=/home/hoverbear/git/kvproto/vendor/github.com/gogo/protobuf
+ GO_OUT_M=
+ GO_INSTALL='go install'
+ echo 'install gogoproto code/generator ...'
install gogoproto code/generator ...
+ cd /home/hoverbear/git/kvproto/vendor/github.com/gogo/protobuf/protoc-gen-gofast
+ go build -o /home/hoverbear/git/kvproto/bin/protoc-gen-gofast
main.go:32:2: cannot find package "github.com/gogo/protobuf/vanity" in any of:
        /usr/lib/go/src/github.com/gogo/protobuf/vanity (from $GOROOT)
        /home/hoverbear/go/src/github.com/gogo/protobuf/vanity (from $GOPATH)
main.go:33:2: cannot find package "github.com/gogo/protobuf/vanity/command" in any of:
        /usr/lib/go/src/github.com/gogo/protobuf/vanity/command (from $GOROOT)
        /home/hoverbear/go/src/github.com/gogo/protobuf/vanity/command (from $GOPATH)
make: *** [Makefile:13: go] Error 1

I think this is because the repo is not in GOPATH? Since it is a requirement it should be detailed.

Failed to make

Following instructions in README, and failed with the following error messages:

++ which grpc_cpp_plugin
+ protoc -I.:/Users/breezewish/Work/PingCAP/src/github.com/pingcap/kvproto/_vendor/src/github.com/gogo/protobuf:/Users/breezewish/Work/PingCAP/src/github.com/pingcap/kvproto/_vendor/src/github.com/gogo/protobuf/protobuf:../include --grpc_out ../cpp/kvproto --plugin=protoc-gen-grpc= coprocessor.proto deadlock.proto debugpb.proto enginepb.proto errorpb.proto import_kvpb.proto import_sstpb.proto kvrpcpb.proto metapb.proto pdpb.proto raft_cmdpb.proto raft_serverpb.proto tikvpb.proto
/Users/breezewish/Work/PingCAP/src/github.com/pingcap/kvproto/_vendor/src/github.com/gogo/protobuf: warning: directory does not exist.
/Users/breezewish/Work/PingCAP/src/github.com/pingcap/kvproto/_vendor/src/github.com/gogo/protobuf/protobuf: warning: directory does not exist.
: program not found or is not executable
--grpc_out: protoc-gen-grpc: Plugin failed with status code 1.
+ exit 1
make: *** [c++] Error 1

# breezewish @ ~/Work/PingCAP/src/github.com/pingcap/kvproto on git:master x [21:34:35] C:1
$ PATH="$(pwd)/protoc/bin:$PATH" which grpc_cpp_plugin
grpc_cpp_plugin not found

# breezewish @ ~/Work/PingCAP/src/github.com/pingcap/kvproto on git:master x [21:34:39] C:1
$ PATH="$(pwd)/protoc/bin:$PATH" protoc --version     
libprotoc 3.1.0

What step am I missing?

[QUESTION] ImportSST::Upload and Ingest rpc method can be used or not ?

i notice that the upload and ingest methods:
I wonder if it can be used to import SST files into tikv(used with txn-client) ?

rpc Upload(stream UploadRequest) returns (UploadResponse) {}
// Ingest an uploaded SST file to a region.
rpc Ingest(IngestRequest) returns (IngestResponse) {}

I checked the source code of some import tools of tidb like tikv-br, tidb-lightning and found that they did not use these two interfaces (upload and ingest), but called importsst:: write / rawwrite. Why?

rpc Write(stream WriteRequest) returns (WriteResponse) {}
rpc RawWrite(stream RawWriteRequest) returns (RawWriteResponse) {}

Failed to make because protoc version doesn't match library version

The make will fail even if I follow the README guide as follows.

โžœ /Users/zhl/go/src/tikv/kvproto โ˜ž git:(master) โœ— PATH="$(pwd)/protoc/bin:$PATH" make
mkdir -p /Users/zhl/go/src/tikv/kvproto/bin
cargo check && \
	cargo check --no-default-features --features prost-codec
    Finished dev [unoptimized + debuginfo] target(s) in 0.06s
    Finished dev [unoptimized + debuginfo] target(s) in 0.06s
/Users/zhl/go/src/tikv/kvproto/scripts/check.sh
mkdir -p kvprotobuild && cd kvprotobuild && cmake ../cpp -DCMAKE_PREFIX_PATH=$GRPC_INSTALL_PATH && make
CMake Warning at /opt/homebrew/Cellar/cmake/3.20.4/share/cmake/Modules/FindProtobuf.cmake:520 (message):
  Protobuf compiler version 3.8.0 doesn't match library version 3.17.3
Call Stack (most recent call first):
  CMakeLists.txt:17 (find_package)


-- Using protobuf: 3.17.3 : /usr/local/include, /usr/local/lib/libprotobuf.dylib
CMake Error at CMakeLists.txt:22 (find_package):
  Could not find a package configuration file provided by "gRPC" with any of
  the following names:

    gRPCConfig.cmake
    grpc-config.cmake

  Add the installation prefix of "gRPC" to CMAKE_PREFIX_PATH or set
  "gRPC_DIR" to a directory containing one of the above files.  If "gRPC"
  provides a separate development package or SDK, be sure it has been
  installed.


-- Configuring incomplete, errors occurred!
See also "/Users/zhl/go/src/tikv/kvproto/kvprotobuild/CMakeFiles/CMakeOutput.log".
See also "/Users/zhl/go/src/tikv/kvproto/kvprotobuild/CMakeFiles/CMakeError.log".
make: *** [c++] Error 1

make go and make rust seems broken in master

make rust:

+ protoc -I.:/home/neil/repo/gopath/src/github.com/pingcap/kvproto/vendor/github.com/gogo/protobuf:../include --rust_out ../src coprocessor.proto debugpb.proto errorpb.proto import_kvpb.proto import_sstpb.proto kvrpcpb.proto metapb.proto pdpb.proto raft_cmdpb.proto raft_serverpb.proto tikvpb.proto
google/protobuf/descriptor.proto: File not found.
gogoproto/gogo.proto: Import "google/protobuf/descriptor.proto" was not found or had errors.
gogoproto/gogo.proto:38:8: "google.protobuf.EnumOptions" is not defined.
gogoproto/gogo.proto: "google.protobuf.EnumOptions" is not defined.

make_rust.log

Check formatting in CI

Currently not tested so it regresses often. Unfortunately it is not as simple as rustfmt --check because the generated code is included and rustfmt can't trace it.

failed to make

mkdir -p /Users/patrick/projects/kvproto/bin
/Users/patrick/projects/kvproto/scripts/check.sh
# Standalone GOPATH
/Users/patrick/projects/kvproto/scripts/generate_go.sh
install tools...
generate go code...
GO111MODULE=on go mod tidy
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/anyhow-35627e4137388996/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/bindgen-87efd08c47447744/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/bitflags-c51cc174a34ea53d/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/clang-sys-a035b042a9bc7674/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/getrandom-a3891d1d29d9690c/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/grpcio-sys-5c63d71b2fe67458/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/indexmap-47721b6a1e7b2e13/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/kvproto-24b21c3190016d0f/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/kvproto-d75b303944c01ce3/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/libc-72b750060d85203c/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/libloading-a6b5b0d61e4abf44/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/libz-sys-9b642e5dc68b539f/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/log-e6ae193143335f61/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/memchr-9401237bd391b45c/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/nom-bbbf1ad522f4af88/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/pin-project-internal-006ceb83a9c5feb5/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/proc-macro-nested-5846ef5f81e807e7/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/proc-macro2-e0365d3b3b157ae7/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/prost-build-4e261873400ff31c/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/protobuf-74d98f5643646d2c/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/raft-proto-034c872e400f70c1/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/raft-proto-08a245442e30156d/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/syn-683952dcdb3d9cec/build-script-build.dSYM
warning: ignoring symlink /Users/patrick/projects/kvproto/target/debug/build/syn-8b1fc97de8e67b42/build-script-build.dSYM
go: finding module for package github.com/pingcap/kvproto/pkg/rustproto
github.com/pingcap/kvproto/pkg/backup imports
        github.com/pingcap/kvproto/pkg/rustproto: no matching versions for query "latest"
make: *** [go] Error 1

naming convention

Does here need pascal case?
Like:

use protobuf::Message as MessageImportedForFunctions;
use protobuf::ProtobufEnum as ProtobufEnumImportedForFunctions;

Fail to make due to sed's error

When I run make, I got the following error:

...
sed๏ผšๆ— ๆณ•่ฏปๅ– s/import _ \"gogoproto\"//g๏ผšNo such file or directory
sed๏ผšๆ— ๆณ•่ฏปๅ– s/import fmt \"fmt\"//g๏ผšNo such file or directory
sed๏ผšๆ— ๆณ•่ฏปๅ– s/import io \"io\"//g๏ผšNo such file or directory
sed๏ผšๆ— ๆณ•่ฏปๅ– s/import math \"math\"//g๏ผšNo such file or directory
sed๏ผšๆ— ๆณ•่ฏปๅ– s/import _ \".*rustproto\"//๏ผšNo such file or directory
WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
	A future release of golang/protobuf will delete this package,
	which has long been excluded from the compatibility promise.

sed๏ผšๆ— ๆณ•่ฏปๅ– s/import _ \"gogoproto\"//g๏ผšNo such file or directory
sed๏ผšๆ— ๆณ•่ฏปๅ– s/import fmt \"fmt\"//g๏ผšNo such file or directory
sed๏ผšๆ— ๆณ•่ฏปๅ– s/import io \"io\"//g๏ผšNo such file or directory
sed๏ผšๆ— ๆณ•่ฏปๅ– s/import math \"math\"//g๏ผšNo such file or directory
sed๏ผšๆ— ๆณ•่ฏปๅ– s/import _ \".*rustproto\"//๏ผšNo such file or directory
WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
	A future release of golang/protobuf will delete this package,
	which has long been excluded from the compatibility promise.

sed๏ผšๆ— ๆณ•่ฏปๅ– s/import _ \"gogoproto\"//g๏ผšNo such file or directory
sed๏ผšๆ— ๆณ•่ฏปๅ– s/import fmt \"fmt\"//g๏ผšNo such file or directory
sed๏ผšๆ— ๆณ•่ฏปๅ– s/import io \"io\"//g๏ผšNo such file or directory
sed๏ผšๆ— ๆณ•่ฏปๅ– s/import math \"math\"//g๏ผšNo such file or directory
sed๏ผšๆ— ๆณ•่ฏปๅ– s/import _ \".*rustproto\"//๏ผšNo such file or directory
WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
	A future release of golang/protobuf will delete this package,
	which has long been excluded from the compatibility promise.

sed๏ผšๆ— ๆณ•่ฏปๅ– s/import _ \"gogoproto\"//g๏ผšNo such file or directory
sed๏ผšๆ— ๆณ•่ฏปๅ– s/import fmt \"fmt\"//g๏ผšNo such file or directory
sed๏ผšๆ— ๆณ•่ฏปๅ– s/import io \"io\"//g๏ผšNo such file or directory
sed๏ผšๆ— ๆณ•่ฏปๅ– s/import math \"math\"//g๏ผšNo such file or directory
sed๏ผšๆ— ๆณ•่ฏปๅ– s/import _ \".*rustproto\"//๏ผšNo such file or directory
GO111MODULE=on go mod tidy
go: finding module for package github.com/pingcap/kvproto/pkg/rustproto
github.com/pingcap/kvproto/pkg/brpb imports
	github.com/pingcap/kvproto/pkg/rustproto: no matching versions for query "latest"
make: *** [go] Error 1

The sed version:

sed (GNU sed) 4.8
Copyright (C) 2020 Free Software Foundation, Inc.

Have someone met this before?

Publishing?

I'm interested in publishing this crate so that it can be used as part of the upcoming TiKV client in Rust (which will be published on crates.io).

This already builds cleanly on nightly and has no illegal dependencies, so there should be nothing blocking publishing this.

I would recommend renaming this crate to tikv_proto though, so in the future tikv_client and tikv_server will be consistent.

warning when make

WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
	A future release of golang/protobuf will delete this package,
	which has long been excluded from the compatibility promise.

Cannot build rust when nested message contains repeated field

In this commit of my PR: e23c6bb

And CI failed on building rust:

error[E0433]: failed to resolve: `super` in paths can only be used in start position
   --> /home/circleci/project/target/debug/build/kvproto-dc3c32c98dc79712/out/protos/wrapper_cdcpb.rs:385:65
    |
385 | #[inline] pub fn set_locks(&mut self, v: ::std::vec::Vec<event::super :: super :: kvrpcpb :: LockInfo>) { self.locks = v; }
    |                                                                 ^^^^^ `super` in paths can only be used in start position

error[E0433]: failed to resolve: `super` in paths can only be used in start position
   --> /home/circleci/project/target/debug/build/kvproto-dc3c32c98dc79712/out/protos/wrapper_cdcpb.rs:386:47
    |
386 | #[inline] pub fn get_locks(&self) -> &[event::super :: super :: kvrpcpb :: LockInfo] { &self.locks }
    |                                               ^^^^^ `super` in paths can only be used in start position

error[E0433]: failed to resolve: `super` in paths can only be used in start position
   --> /home/circleci/project/target/debug/build/kvproto-dc3c32c98dc79712/out/protos/wrapper_cdcpb.rs:387:70
    |
387 | #[inline] pub fn mut_locks(&mut self) -> &mut ::std::vec::Vec<event::super :: super :: kvrpcpb :: LockInfo> { &mut self.locks }
    |                                                                      ^^^^^ `super` in paths can only be used in start position

error[E0433]: failed to resolve: `super` in paths can only be used in start position
   --> /home/circleci/project/target/debug/build/kvproto-dc3c32c98dc79712/out/protos/wrapper_cdcpb.rs:388:66
    |
388 | #[inline] pub fn take_locks(&mut self) -> ::std::vec::Vec<event::super :: super :: kvrpcpb :: LockInfo> { ::std::mem::replace(&mut self.locks, ::std::vec::Vec::new()) }
    |                                                                  ^^^^^ `super` in paths can only be used in start position

error[E0412]: cannot find type `u64` in module `change_data_request`
   --> /home/circleci/project/target/debug/build/kvproto-dc3c32c98dc79712/out/protos/wrapper_cdcpb.rs:595:86
    |
595 | #[inline] pub fn set_txn_start_ts(&mut self, v: ::std::vec::Vec<change_data_request::u64>) { self.txn_start_ts = v; }
    |                                                                                      ^^^ not found in `change_data_request`
    |
help: possible candidates are found in other modules, you can import them into scope
    |
2   | use core::primitive::u64;
    |
2   | use std::primitive::u64;
    |

error[E0412]: cannot find type `u64` in module `change_data_request`
   --> /home/circleci/project/target/debug/build/kvproto-dc3c32c98dc79712/out/protos/wrapper_cdcpb.rs:596:68
    |
596 | #[inline] pub fn get_txn_start_ts(&self) -> &[change_data_request::u64] { &self.txn_start_ts }
    |                                                                    ^^^ not found in `change_data_request`
    |
help: possible candidates are found in other modules, you can import them into scope
    |
2   | use core::primitive::u64;
    |
2   | use std::primitive::u64;
    |

error[E0412]: cannot find type `u64` in module `change_data_request`
   --> /home/circleci/project/target/debug/build/kvproto-dc3c32c98dc79712/out/protos/wrapper_cdcpb.rs:597:91
    |
597 | #[inline] pub fn mut_txn_start_ts(&mut self) -> &mut ::std::vec::Vec<change_data_request::u64> { &mut self.txn_start_ts }
    |                                                                                           ^^^ not found in `change_data_request`
    |
help: possible candidates are found in other modules, you can import them into scope
    |
2   | use core::primitive::u64;
    |
2   | use std::primitive::u64;
    |

error[E0412]: cannot find type `u64` in module `change_data_request`
   --> /home/circleci/project/target/debug/build/kvproto-dc3c32c98dc79712/out/protos/wrapper_cdcpb.rs:598:87
    |
598 | #[inline] pub fn take_txn_start_ts(&mut self) -> ::std::vec::Vec<change_data_request::u64> { ::std::mem::replace(&mut self.txn_start_ts, ::std::vec::Vec::new()) }
    |                                                                                       ^^^ not found in `change_data_request`
    |
help: possible candidates are found in other modules, you can import them into scope
    |
2   | use core::primitive::u64;
    |
2   | use std::primitive::u64;
    |

error: aborting due to 8 previous errors

Use specific gogo proto version

Now, small changes in .proto files may lead huge changes across all generated .go files, and some of them are not even related. I think it is caused by an ambiguous gogo/protobuf generator.

We can use glide to resolved this problem, just like what we've done in tidb/Makefile.

cc @siddontang @shenli @ngaut

Build requires `bash` to be present.

In check_protoc_version() we use Command::new("Bash") with some arguments. Notably systems such as Windows and minimal Linux Distros do not commonly include bash.

Resolving this issue would require that we no longer depend on bash (perhaps by using pure rust), or provide a windows compatible function.

Remove TiFlash APIs

As a Protobuf of TiKV, it should contain no knowledge about TiFlash.

A list of APIs needs to be moved out:

  • TiFlash Compaction API
  • MPP

Generate cpp code

Currently we support generating Golang and Rust code. We should also consider to generate Cpp code, because there are many projects written in Cpp. It would be convenient for them to integrate the TiKV project, if kvproto provides Cpp code.

Streaming Scan RPC

Is there a reason the Scan RPC isn't streaming?

I'm interested in doing large range scans in TiKV over gRPC, and currently it looks like the only way of doing this is via a repeated scan reusing the end key as the start key and preforming batches of size N.

cannot run make rust correctly if I didn't install rust-protobuf in advance

If I didn't install rust-protobuf and run "make rust" directly, I will get such infomation:

./generate_rust.sh
generate rust code...
protoc-gen-rust: program not found or is not executable
--rust_out: protoc-gen-rust: Plugin failed with status code 1.
make: *** [rust] Error 1

I think we could add some codes to generate_rust.sh.

if ! cmd_exists protoc-gen-rust; then
    echo "missing rust-protobuf, try to download/install it"
    cargo install protobuf
fi

Installation requires cmake, protoc, golang, perl...

Prior to #347 I do not think cmake, protoc, golang, perl were required to pull kvproto into a Rust dependency.

Having these extra dependencies is rather problematic. protoc in particular is not a commonly installed tool.

Resolving this issue would involve ensuring downstream dependencies can use kvproto without:

  • cmake
  • protoc
  • golang
  • perl

Labelling this as a bug as I think it's a regression. cc @nrc .

Does not complete build

kvproto does not compile.. Unfortunately I do not know many specifics about this.. but the below error states that it seems to be an issue with permissions..

error: failed to run custom build command for `kvproto v0.0.2 (https://github.com/pingcap/kvproto.git#b8bc94dd)`

Caused by:
  process didn't exit successfully: `/home/procyclinsur/test/kvif/target/debug/build/kvproto-af7c9bf33012dd14/build-script-build` (exit code: 101)
--- stderr
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }', /home/procyclinsur/.cargo/registry/src/github.com-1ecc6299db9ec823/protobuf-build-0.11.1/src/prost_impl.rs:8:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

warning: build failed, waiting for other jobs to finish...
error: build failed

any insight on this would be appriciated

Documentation is lacking

Here's an example of a placement driver service definition:

kvproto/proto/pdpb.proto

Lines 18 to 78 in f7ca9b5

service PD {
// GetMembers get the member list of this cluster. It does not require
// the cluster_id in request matchs the id of this cluster.
rpc GetMembers(GetMembersRequest) returns (GetMembersResponse) {}
rpc Tso(stream TsoRequest) returns (stream TsoResponse) {}
rpc Bootstrap(BootstrapRequest) returns (BootstrapResponse) {}
rpc IsBootstrapped(IsBootstrappedRequest) returns (IsBootstrappedResponse) {}
rpc AllocID(AllocIDRequest) returns (AllocIDResponse) {}
rpc GetStore(GetStoreRequest) returns (GetStoreResponse) {}
rpc PutStore(PutStoreRequest) returns (PutStoreResponse) {}
rpc GetAllStores(GetAllStoresRequest) returns (GetAllStoresResponse) {}
rpc StoreHeartbeat(StoreHeartbeatRequest) returns (StoreHeartbeatResponse) {}
rpc RegionHeartbeat(stream RegionHeartbeatRequest) returns (stream RegionHeartbeatResponse) {}
rpc GetRegion(GetRegionRequest) returns (GetRegionResponse) {}
rpc GetPrevRegion(GetRegionRequest) returns (GetRegionResponse) {}
rpc GetRegionByID(GetRegionByIDRequest) returns (GetRegionResponse) {}
rpc ScanRegions(ScanRegionsRequest) returns (ScanRegionsResponse) {}
rpc AskSplit(AskSplitRequest) returns (AskSplitResponse) {
// Use AskBatchSplit instead.
option deprecated = true;
}
rpc ReportSplit(ReportSplitRequest) returns (ReportSplitResponse) {
// Use ResportBatchSplit instead.
option deprecated = true;
}
rpc AskBatchSplit(AskBatchSplitRequest) returns (AskBatchSplitResponse) {}
rpc ReportBatchSplit(ReportBatchSplitRequest) returns (ReportBatchSplitResponse) {}
rpc GetClusterConfig(GetClusterConfigRequest) returns (GetClusterConfigResponse) {}
rpc PutClusterConfig(PutClusterConfigRequest) returns (PutClusterConfigResponse) {}
rpc ScatterRegion(ScatterRegionRequest) returns (ScatterRegionResponse) {}
rpc GetGCSafePoint(GetGCSafePointRequest) returns (GetGCSafePointResponse) {}
rpc UpdateGCSafePoint(UpdateGCSafePointRequest) returns (UpdateGCSafePointResponse) {}
rpc UpdateServiceGCSafePoint(UpdateServiceGCSafePointRequest) returns (UpdateServiceGCSafePointResponse) {}
rpc SyncRegions(stream SyncRegionRequest) returns (stream SyncRegionResponse) {}
rpc GetOperator(GetOperatorRequest) returns (GetOperatorResponse) {}
}

PD is used by both peers and clients, yet there's not a single comment that gives the reader a clue about which methods are for peers only, which are for clients, nor what each of the methods actually supposed to do.

It would be nice to have Protocol Buffers files documented, especially those that are supposed to be consumed by clients.

Move to tikv/proto

As a protobuf of TiKV, it should be placed with other TiKV ecosystems, e.g. located in github.com/tikv/proto.

Related: #912

`make rust` is failing because `default_ref()` is missing from `raft-proto` structs

Repro steps:

  1. Clone the repository, everything clean
  2. make rust
  3. Got the error at the end:
error[E0599]: no function or associated item named `default_ref` found for type `raft_proto::prost::eraftpb::Entry` in the current scope
   --> src/prost/wrapper_debugpb.rs:348:44
    |
348 |             None => super::eraftpb::Entry::default_ref(),
    |                                            ^^^^^^^^^^^ function or associated item not found in `raft_proto::prost::eraftpb::Entry`

error[E0599]: no function or associated item named `default_ref` found for type `raft_proto::prost::eraftpb::Message` in the current scope
   --> src/prost/wrapper_raft_serverpb.rs:103:46
    |
103 |             None => super::eraftpb::Message::default_ref(),
    |                                              ^^^^^^^^^^^ function or associated item not found in `raft_proto::prost::eraftpb::Message`

error[E0599]: no function or associated item named `default_ref` found for type `raft_proto::prost::eraftpb::HardState` in the current scope
    --> src/prost/wrapper_raft_serverpb.rs:1234:48
     |
1234 |             None => super::eraftpb::HardState::default_ref(),
     |                                                ^^^^^^^^^^^ function or associated item not found in `raft_proto::prost::eraftpb::HardState`

error: aborting due to 3 previous errors

For more information about this error, try `rustc --explain E0599`.
error: Could not compile `kvproto`.

cc @nrc. I think the Rust code in both raft-proto and kvproto need to be regenerated using the same (exact) version of protobuf-build

Remove grpc-rs dep?

We depend on grpc-rs which causes a big dent in our compile time, in particular due to grpc-sys. However, the only reason we depend on grpc-rs is for some high-level types. With a bit of refactoring, we could move those into a separate crate (grpc-types) and greatly reduce our compile time.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.