Code Monkey home page Code Monkey logo

rk-grpc's Introduction

rk-grpc

Inject middlewares & server configuration of gRPC and grpc-gateway from YAML file.

This belongs to rk-boot family. We suggest use this lib with rk-boot.

Architecture

image

Quick Start

In the bellow example, we will start microservice with bellow functionality and middlewares enabled via YAML.

  • gRPC and grpc-gateway server
  • gRPC server reflection
  • Swagger UI
  • CommonService
  • Docs
  • Prometheus Metrics (middleware)
  • Logging (middleware)
  • Meta (middleware)

Please refer example at example/boot/simple.

Installation

go get github.com/rookie-ninja/rk-grpc/v2

1.Prepare .proto files

show
  • api/v1/greeter.proto
syntax = "proto3";

package api.v1;

option go_package = "api/v1/greeter";

service Greeter {
  rpc Greeter (GreeterRequest) returns (GreeterResponse) {}
}

message GreeterRequest {
  bytes msg = 1;
}

message GreeterResponse {}
  • api/v1/gw_mapping.yaml
type: google.api.Service
config_version: 3

# Please refer google.api.Http in https://github.com/googleapis/googleapis/blob/master/google/api/http.proto file for details.
http:
  rules:
    - selector: api.v1.Greeter.Greeter
      get: /v1/greeter
  • buf.yaml
version: v1beta1
name: github.com/rk-dev/rk-boot
build:
  roots:
    - api
  • buf.gen.yaml
version: v1beta1
plugins:
  # protoc-gen-go needs to be installed, generate go files based on proto files
  - name: go
    out: api/gen
    opt:
     - paths=source_relative
  # protoc-gen-go-grpc needs to be installed, generate grpc go files based on proto files
  - name: go-grpc
    out: api/gen
    opt:
      - paths=source_relative
      - require_unimplemented_servers=false
  # protoc-gen-grpc-gateway needs to be installed, generate grpc-gateway go files based on proto files
  - name: grpc-gateway
    out: api/gen
    opt:
      - paths=source_relative
      - grpc_api_configuration=api/v1/gw_mapping.yaml
  # protoc-gen-openapiv2 needs to be installed, generate swagger config files based on proto files
  - name: openapiv2
    out: api/gen
    opt:
      - grpc_api_configuration=api/v1/gw_mapping.yaml

2.Generate .pb.go files with buf

show
$ buf generate --path api/v1
.
├── api
│   ├── gen
│   │   └── v1
│   │       ├── greeter.pb.go
│   │       ├── greeter.pb.gw.go
│   │       ├── greeter.swagger.json
│   │       └── greeter_grpc.pb.go
│   └── v1
│       ├── greeter.proto
│       └── gw_mapping.yaml
├── boot.yaml
├── buf.gen.yaml
├── buf.yaml
├── go.mod
├── go.sum
└── main.go

3.Create boot.yaml

show
---
grpc:
  - name: greeter                     # Required
    port: 8080                        # Required
    enabled: true                     # Required
    enableReflection: true            # Optional, default: false
    enableRkGwOption: true            # Optional, default: false
    commonService:
      enabled: true                   # Optional, default: false
    docs:
      enabled: true                   # Optional, default: false
    sw:
      enabled: true                   # Optional, default: false
    prom:
      enabled: true                   # Optional, default: false
    middleware:
      logging:
        enabled: true                 # Optional, default: false
      prom:
        enabled: true                 # Optional, default: false
      meta:
        enabled: true                 # Optional, default: false

4.Create main.go

show
// Copyright (c) 2021 rookie-ninja
//
// Use of this source code is governed by an Apache-style
// license that can be found in the LICENSE file.
package main

import (
  "context"
  "embed"
  _ "embed"
  "github.com/rookie-ninja/rk-entry/v2/entry"
  "github.com/rookie-ninja/rk-grpc/v2/boot"
  proto "github.com/rookie-ninja/rk-grpc/v2/example/boot/simple/api/gen/v1"
  "google.golang.org/grpc"
)

//go:embed boot.yaml
var boot []byte

//go:embed api/gen/v1
var docsFS embed.FS

//go:embed api/gen/v1
var staticFS embed.FS

func init() {
  rkentry.GlobalAppCtx.AddEmbedFS(rkentry.DocsEntryType, "greeter", &docsFS)
  rkentry.GlobalAppCtx.AddEmbedFS(rkentry.SWEntryType, "greeter", &docsFS)
  rkentry.GlobalAppCtx.AddEmbedFS(rkentry.StaticFileHandlerEntryType, "greeter", &staticFS)
}

func main() {
  // Bootstrap basic entries from boot config.
  rkentry.BootstrapPreloadEntryYAML(boot)

  // Bootstrap grpc entry from boot config
  res := rkgrpc.RegisterGrpcEntryYAML(boot)

  // Get GrpcEntry
  grpcEntry := res["greeter"].(*rkgrpc.GrpcEntry)
  // Register gRPC server
  grpcEntry.AddRegFuncGrpc(func(server *grpc.Server) {
    proto.RegisterGreeterServer(server, &GreeterServer{})
  })
  // Register grpc-gateway func
  grpcEntry.AddRegFuncGw(proto.RegisterGreeterHandlerFromEndpoint)

  // Bootstrap grpc entry
  grpcEntry.Bootstrap(context.Background())

  // Wait for shutdown signal
  rkentry.GlobalAppCtx.WaitForShutdownSig()

  // Interrupt gin entry
  grpcEntry.Interrupt(context.Background())
}

// GreeterServer Implementation of GreeterServer.
type GreeterServer struct{}

// Greeter Handle Greeter method.
func (server *GreeterServer) Greeter(context.Context, *proto.GreeterRequest) (*proto.GreeterResponse, error) {
  return &proto.GreeterResponse{}, nil
}

5.Start server

$ go run main.go

6.Validation

show

6.1 gRPC & grpc-gateway server

Try to test gRPC & grpc-gateway Service with curl & grpcurl

# Curl to common service
$ curl localhost:8080/rk/v1/ready
{"ready":true}

6.2 Swagger UI

Please refer sw section at Full YAML.

By default, we could access swagger UI at http://localhost:8080/sw

sw

6.3 Docs UI

Please refer docs section at Full YAML.

By default, we could access docs UI at http://localhost:8080/docs

docs

6.4 Prometheus Metrics

Please refer middleware.prom section at Full YAML.

By default, we could access prometheus client at http://localhost:8080/metrics

prom

6.5 Logging

Please refer middleware.logging section at Full YAML.

By default, we enable zap logger and event logger with encoding type of [console]. Encoding type of [json] and [flatten] is also supported.

2021-12-28T05:36:21.561+0800    INFO    boot/grpc_entry.go:1515 Bootstrap grpcEntry     {"eventId": "db2c977c-e0ff-4b21-bc0d-5966f1cad093", "entryName": "greeter"}
------------------------------------------------------------------------
endTime=2021-12-28T05:36:21.563575+08:00
startTime=2021-12-28T05:36:21.561362+08:00
elapsedNano=2213846
timezone=CST
ids={"eventId":"db2c977c-e0ff-4b21-bc0d-5966f1cad093"}
app={"appName":"rk","appVersion":"","entryName":"greeter","entryType":"GrpcEntry"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.2","os":"darwin","realm":"*","region":"*"}
payloads={"commonServiceEnabled":true,"commonServicePathPrefix":"/rk/v1/","grpcPort":8080,"gwPort":8080,"promEnabled":true,"promPath":"/metrics","promPort":8080,"swEnabled":true,"swPath":"/sw/","tvEnabled":true,"tvPath":"/rk/v1/tv/"}
error={}
counters={}
pairs={}
timing={}
remoteAddr=localhost
operation=Bootstrap
resCode=OK
eventStatus=Ended
EOE

6.6 Meta

Please refer meta section at Full YAML.

By default, we will send back some metadata to client with headers.

$ curl -vs localhost:8080/rk/v1/ready
...
< HTTP/1.1 200 OK
< Content-Type: application/json
< X-Request-Id: 7e4f5ac5-3369-485f-89f7-55551cc4a9a1
< X-Rk-App-Name: rk
< X-Rk-App-Unix-Time: 2021-12-28T05:39:50.508328+08:00
< X-Rk-App-Version: 
< X-Rk-Received-Time: 2021-12-28T05:39:50.508328+08:00
< Date: Mon, 27 Dec 2021 21:39:50 GMT
...

6.7 Send request

We registered /v1/greeter API in grpc-gateway server and let's validate it!

$ curl -vs localhost:8080/v1/greeter             
*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 8080 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /v1/greeter HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< X-Request-Id: 07b0fbf6-cebf-40ac-84a2-533bbd4b8958
< X-Rk-App-Name: rk
< X-Rk-App-Unix-Time: 2021-12-28T05:41:04.653652+08:00
< X-Rk-App-Version: 
< X-Rk-Received-Time: 2021-12-28T05:41:04.653652+08:00
< Date: Mon, 27 Dec 2021 21:41:04 GMT
< Content-Length: 2
< 
* Connection #0 to host localhost left intact
{}

We registered api.v1.Greeter.Greeter API in gRPC server and let's validate it!

$ grpcurl -plaintext localhost:8080 api.v1.Greeter.Greeter 
{}

6.8 RPC logs

Bellow logs would be printed in stdout.

The first block of log is from grpc-gateway request.

The second block of log is from gRPC request.

------------------------------------------------------------------------
endTime=2021-12-28T05:45:52.986041+08:00
startTime=2021-12-28T05:45:52.985956+08:00
elapsedNano=85065
timezone=CST
ids={"eventId":"88362f69-7eda-4f03-bdbe-7ef667d06bac","requestId":"88362f69-7eda-4f03-bdbe-7ef667d06bac"}
app={"appName":"rk","appVersion":"","entryName":"greeter","entryType":"GrpcEntry"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.2","os":"darwin","realm":"*","region":"*"}
payloads={"grpcMethod":"Greeter","grpcService":"api.v1.Greeter","grpcType":"unaryServer","gwMethod":"GET","gwPath":"/v1/greeter","gwScheme":"http","gwUserAgent":"curl/7.64.1"}
error={}
counters={}
pairs={}
timing={}
remoteAddr=127.0.0.1:61520
operation=/api.v1.Greeter/Greeter
resCode=OK
eventStatus=Ended
EOE
------------------------------------------------------------------------
endTime=2021-12-28T05:44:45.686734+08:00
startTime=2021-12-28T05:44:45.686592+08:00
elapsedNano=141716
timezone=CST
ids={"eventId":"7765862c-9e83-443a-a6e5-bb28f17f8ea0","requestId":"7765862c-9e83-443a-a6e5-bb28f17f8ea0"}
app={"appName":"rk","appVersion":"","entryName":"greeter","entryType":"GrpcEntry"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"10.8.0.2","os":"darwin","realm":"*","region":"*"}
payloads={"grpcMethod":"Greeter","grpcService":"api.v1.Greeter","grpcType":"unaryServer","gwMethod":"","gwPath":"","gwScheme":"","gwUserAgent":""}
error={}
counters={}
pairs={}
timing={}
remoteAddr=127.0.0.1:57149
operation=/api.v1.Greeter/Greeter
resCode=OK
eventStatus=Ended
EOE

6.9 RPC prometheus metrics

Prometheus client will automatically register into grpc-gateway instance at /metrics.

Access http://localhost:8080/metrics

image

Supported features

User can enable anyone of those as needed! No mandatory binding!

Instance Description
gRPC gRPC defined with protocol buffer.
gRPC proxy Proxy gRPC request to another gRPC server.
grpc-gateway grpc-gateway service with same port.
grpc-gateway options Well defined grpc-gateway options.
Config Configure spf13/viper as config instance and reference it from YAML
Logger Configure uber-go/zap logger configuration and reference it from YAML
Event Configure logging of RPC with rk-query and reference it from YAML
Cert Fetch TLS/SSL certificates from remote datastore like ETCD and start microservice.
Prometheus Start prometheus client at client side and push metrics to pushgateway as needed.
Swagger Builtin swagger UI handler.
Docs Builtin RapiDoc instance which can be used to replace swagger and RK TV.
CommonService List of common APIs.
StaticFileHandler A Web UI shows files could be downloaded from server, currently support source of local and embed.FS.
PProf PProf web UI.
gRPC Web gRPC Web

Supported middlewares

All middlewares could be configured via YAML or Code.

User can enable anyone of those as needed! No mandatory binding!

Middleware Description
Metrics Collect RPC metrics and export to prometheus client.
Log Log every RPC requests as event with rk-query.
Trace Collect RPC trace and export it to stdout, file or jaeger with open-telemetry/opentelemetry-go.
Panic Recover from panic for RPC requests and log it.
Meta Send micsroservice metadata as header to client.
Auth Support [Basic Auth] and [API Key] authorization types.
RateLimit Limiting RPC rate globally or per path.
Timeout Timing out request by configuration.
CORS Server side CORS validation.
JWT Server side JWT validation.
Secure Server side secure validation.
CSRF Server side csrf validation.

YAML options

User can start multiple gRPC and grpc-gateway instances at the same time. Please make sure use different port and name.

show
---
#app:
#  name: my-app                                            # Optional, default: "rk-app"
#  version: "v1.0.0"                                       # Optional, default: "v0.0.0"
#  description: "this is description"                      # Optional, default: ""
#  keywords: ["rk", "golang"]                              # Optional, default: []
#  homeUrl: "http://example.com"                           # Optional, default: ""
#  docsUrl: ["http://example.com"]                         # Optional, default: []
#  maintainers: ["rk-dev"]                                 # Optional, default: []
#logger:
#  - name: my-logger                                       # Required
#    description: "Description of entry"                   # Optional
#    domain: "*"                                           # Optional, default: "*"
#    default: false                                        # Optional, default: false, use as default logger entry
#    zap:                                                  # Optional
#      level: info                                         # Optional, default: info
#      development: true                                   # Optional, default: true
#      disableCaller: false                                # Optional, default: false
#      disableStacktrace: true                             # Optional, default: true
#      encoding: console                                   # Optional, default: console
#      outputPaths: ["stdout"]                             # Optional, default: [stdout]
#      errorOutputPaths: ["stderr"]                        # Optional, default: [stderr]
#      encoderConfig:                                      # Optional
#        timeKey: "ts"                                     # Optional, default: ts
#        levelKey: "level"                                 # Optional, default: level
#        nameKey: "logger"                                 # Optional, default: logger
#        callerKey: "caller"                               # Optional, default: caller
#        messageKey: "msg"                                 # Optional, default: msg
#        stacktraceKey: "stacktrace"                       # Optional, default: stacktrace
#        skipLineEnding: false                             # Optional, default: false
#        lineEnding: "\n"                                  # Optional, default: \n
#        consoleSeparator: "\t"                            # Optional, default: \t
#      sampling:                                           # Optional, default: nil
#        initial: 0                                        # Optional, default: 0
#        thereafter: 0                                     # Optional, default: 0
#      initialFields:                                      # Optional, default: empty map
#        key: value
#    lumberjack:                                           # Optional, default: nil
#      filename:
#      maxsize: 1024                                       # Optional, suggested: 1024 (MB)
#      maxage: 7                                           # Optional, suggested: 7 (day)
#      maxbackups: 3                                       # Optional, suggested: 3 (day)
#      localtime: true                                     # Optional, suggested: true
#      compress: true                                      # Optional, suggested: true
#    loki:
#      enabled: true                                       # Optional, default: false
#      addr: localhost:3100                                # Optional, default: localhost:3100
#      path: /loki/api/v1/push                             # Optional, default: /loki/api/v1/push
#      username: ""                                        # Optional, default: ""
#      password: ""                                        # Optional, default: ""
#      maxBatchWaitMs: 3000                                # Optional, default: 3000
#      maxBatchSize: 1000                                  # Optional, default: 1000
#      insecureSkipVerify: false                           # Optional, default: false
#      labels:                                             # Optional, default: empty map
#        my_label_key: my_label_value
#event:
#  - name: my-event                                        # Required
#    description: "Description of entry"                   # Optional
#    domain: "*"                                           # Optional, default: "*"
#    encoding: console                                     # Optional, default: console
#    default: false                                        # Optional, default: false, use as default event entry
#    outputPaths: ["stdout"]                               # Optional, default: [stdout]
#    lumberjack:                                           # Optional, default: nil
#      filename:
#      maxsize: 1024                                       # Optional, suggested: 1024 (MB)
#      maxage: 7                                           # Optional, suggested: 7 (day)
#      maxbackups: 3                                       # Optional, suggested: 3 (day)
#      localtime: true                                     # Optional, suggested: true
#      compress: true                                      # Optional, suggested: true
#    loki:
#      enabled: true                                       # Optional, default: false
#      addr: localhost:3100                                # Optional, default: localhost:3100
#      path: /loki/api/v1/push                             # Optional, default: /loki/api/v1/push
#      username: ""                                        # Optional, default: ""
#      password: ""                                        # Optional, default: ""
#      maxBatchWaitMs: 3000                                # Optional, default: 3000
#      maxBatchSize: 1000                                  # Optional, default: 1000
#      insecureSkipVerify: false                           # Optional, default: false
#      labels:                                             # Optional, default: empty map
#        my_label_key: my_label_value
#cert:
#  - name: my-cert                                         # Required
#    description: "Description of entry"                   # Optional, default: ""
#    domain: "*"                                           # Optional, default: "*"
#    caPath: "certs/ca.pem"                                # Optional, default: ""
#    certPemPath: "certs/server-cert.pem"                  # Optional, default: ""
#    keyPemPath: "certs/server-key.pem"                    # Optional, default: ""
#config:
#  - name: my-config                                       # Required
#    description: "Description of entry"                   # Optional, default: ""
#    domain: "*"                                           # Optional, default: "*"
##    path: "config/config.yaml"                            # Optional
#    envPrefix: ""                                         # Optional, default: ""
#    content:                                              # Optional, defualt: empty map
#      key: value
grpc:
  - name: greeter                                          # Required
    enabled: true                                          # Required
    port: 8080                                             # Required
#    gwPort: 8081                                          # Optional, default: gateway port will be the same as grpc port if not provided 
#    description: "greeter server"                         # Optional, default: ""
#    enableReflection: true                                # Optional, default: false
#    enableRkGwOption: true                                # Optional, default: false
#    grpcWeb:
#      enabled: true
#      cors:
#        allowOrigins: []                                  # Optional, default: [*]
#      websocket:
#        enabled: true                                     # Optional, default: disable websocket
#        pingIntervalMs: 10                                # Optional, default: disable ping
#        messageReadLimitBytes: 32769                      # Optional, default: 32769
#    gwOption:                                             # Optional, default: nil
#      marshal:                                            # Optional, default: nil
#        multiline: false                                  # Optional, default: false
#        emitUnpopulated: false                            # Optional, default: false
#        indent: ""                                        # Optional, default: false
#        allowPartial: false                               # Optional, default: false
#        useProtoNames: false                              # Optional, default: false
#        useEnumNumbers: false                             # Optional, default: false
#      unmarshal:                                          # Optional, default: nil
#        allowPartial: false                               # Optional, default: false
#        discardUnknown: false                             # Optional, default: false
#    noRecvMsgSizeLimit: true                              # Optional, default: false
#    certEntry: my-cert                                    # Optional, default: "", reference of cert entry declared above
#    loggerEntry: my-logger                                # Optional, default: "", reference of cert entry declared above, STDOUT will be used if missing
#    eventEntry: my-event                                  # Optional, default: "", reference of cert entry declared above, STDOUT will be used if missing
#    sw:
#      enabled: true                                       # Optional, default: false
#      path: "sw"                                          # Optional, default: "sw"
#      jsonPath: [""]                                      # Optional
#      headers: ["sw:rk"]                                  # Optional, default: []
#    docs:
#      enabled: true                                       # Optional, default: false
#      path: "docs"                                        # Optional, default: "docs"
#      specPath: ""                                        # Optional
#      headers: ["sw:rk"]                                  # Optional, default: []
#      style:                                              # Optional
#        theme: "light"                                    # Optional, default: "light"
#      debug: false                                        # Optional, default: false
#    commonService:
#      enabled: true                                       # Optional, default: false
#    static:
#      enabled: true                                       # Optional, default: false
#      path: "/static"                                     # Optional, default: /static
#      sourceType: local                                   # Required, options: pkger, local
#      sourcePath: "."                                     # Required, full path of source directory
#    pprof:
#      enabled: true                                       # Optional, default: false
#      path: "/pprof"                                      # Optional, default: /pprof
#    prom:
#      enabled: true                                       # Optional, default: false
#      path: ""                                            # Optional, default: "metrics"
#      pusher:
#        enabled: false                                    # Optional, default: false
#        jobName: "greeter-pusher"                         # Required
#        remoteAddress: "localhost:9091"                   # Required
#        basicAuth: "user:pass"                            # Optional, default: ""
#        intervalMs: 10000                                 # Optional, default: 1000
#        certEntry: my-cert                                # Optional, default: "", reference of cert entry declared above
#    middleware:
#      ignore: [""]                                        # Optional, default: []
#      errorModel: google                                  # Optional, default: google, [amazon, google] are supported options
#      logging:
#        enabled: true                                     # Optional, default: false
#        ignore: [""]                                      # Optional, default: []
#        loggerEncoding: "console"                         # Optional, default: "console"
#        loggerOutputPaths: ["logs/app.log"]               # Optional, default: ["stdout"]
#        eventEncoding: "console"                          # Optional, default: "console"
#        eventOutputPaths: ["logs/event.log"]              # Optional, default: ["stdout"]
#      prom:
#        enabled: true                                     # Optional, default: false
#        ignore: [""]                                      # Optional, default: []
#      auth:
#        enabled: true                                     # Optional, default: false
#        ignore: [""]                                      # Optional, default: []
#        basic:
#          - "user:pass"                                   # Optional, default: []
#        apiKey:
#          - "keys"                                        # Optional, default: []
#      meta:
#        enabled: true                                     # Optional, default: false
#        ignore: [""]                                      # Optional, default: []
#        prefix: "rk"                                      # Optional, default: "rk"
#      trace:
#        enabled: true                                     # Optional, default: false
#        ignore: [""]                                      # Optional, default: []
#        exporter:                                         # Optional, default will create a stdout exporter
#          file:
#            enabled: true                                 # Optional, default: false
#            outputPath: "logs/trace.log"                  # Optional, default: stdout
#          jaeger:
#            agent:
#              enabled: false                              # Optional, default: false
#              host: ""                                    # Optional, default: localhost
#              port: 0                                     # Optional, default: 6831
#            collector:
#              enabled: true                               # Optional, default: false
#              endpoint: ""                                # Optional, default: http://localhost:14268/api/traces
#              username: ""                                # Optional, default: ""
#              password: ""                                # Optional, default: ""
#      rateLimit:
#        enabled: false                                    # Optional, default: false
#        ignore: [""]                                      # Optional, default: []
#        algorithm: "leakyBucket"                          # Optional, default: "tokenBucket"
#        reqPerSec: 100                                    # Optional, default: 1000000
#        paths:
#          - path: "/rk/v1/healthy"                        # Optional, default: ""
#            reqPerSec: 0                                  # Optional, default: 1000000
#      timeout:
#        enabled: false                                    # Optional, default: false
#        ignore: [""]                                      # Optional, default: []
#        timeoutMs: 5000                                   # Optional, default: 5000
#        paths:
#          - path: "/rk/v1/healthy"                        # Optional, default: ""
#            timeoutMs: 1000                               # Optional, default: 5000
#      jwt:
#        enabled: true                                     # Optional, default: false
#        ignore: [ "" ]                                    # Optional, default: []
#        signerEntry: ""                                   # Optional, default: ""
#        skipVerify: false                                 # Optional, default: false
#        symmetric:                                        # Optional
#          algorithm: ""                                   # Required, default: ""
#          token: ""                                       # Optional, default: ""
#          tokenPath: ""                                   # Optional, default: ""
#        asymmetric:                                       # Optional
#          algorithm: ""                                   # Required, default: ""
#          privateKey: ""                                  # Optional, default: ""
#          privateKeyPath: ""                              # Optional, default: ""
#          publicKey: ""                                   # Optional, default: ""
#          publicKeyPath: ""                               # Optional, default: ""
#        tokenLookup: "header:<name>"                      # Optional, default: "header:Authorization"
#        authScheme: "Bearer"                              # Optional, default: "Bearer"
#      secure:
#        enabled: true                                     # Optional, default: false
#        ignore: [""]                                      # Optional, default: []
#        xssProtection: ""                                 # Optional, default: "1; mode=block"
#        contentTypeNosniff: ""                            # Optional, default: nosniff
#        xFrameOptions: ""                                 # Optional, default: SAMEORIGIN
#        hstsMaxAge: 0                                     # Optional, default: 0
#        hstsExcludeSubdomains: false                      # Optional, default: false
#        hstsPreloadEnabled: false                         # Optional, default: false
#        contentSecurityPolicy: ""                         # Optional, default: ""
#        cspReportOnly: false                              # Optional, default: false
#        referrerPolicy: ""                                # Optional, default: ""
#      csrf:
#        enabled: true                                     # Optional, default: false
#        ignore: [""]                                      # Optional, default: []
#        tokenLength: 32                                   # Optional, default: 32
#        tokenLookup: "header:X-CSRF-Token"                # Optional, default: "header:X-CSRF-Token"
#        cookieName: "_csrf"                               # Optional, default: _csrf
#        cookieDomain: ""                                  # Optional, default: ""
#        cookiePath: ""                                    # Optional, default: ""
#        cookieMaxAge: 86400                               # Optional, default: 86400
#        cookieHttpOnly: false                             # Optional, default: false
#        cookieSameSite: "default"                         # Optional, default: "default", options: lax, strict, none, default
#      gzip:
#        enabled: true                                     # Optional, default: false
#        ignore: [""]                                      # Optional, default: []
#        level: bestSpeed                                  # Optional, options: [noCompression, bestSpeed, bestCompression, defaultCompression, huffmanOnly]
#      cors:
#        enabled: true                                     # Optional, default: false
#        ignore: [""]                                      # Optional, default: []
#        allowOrigins:                                     # Optional, default: []
#          - "http://localhost:*"                          # Optional, default: *
#        allowCredentials: false                           # Optional, default: false
#        allowHeaders: []                                  # Optional, default: []
#        allowMethods: []                                  # Optional, default: []
#        exposeHeaders: []                                 # Optional, default: []
#        maxAge: 0                                         # Optional, default: 0

Development Status: Stable

Build instruction

Simply run make all to validate your changes. Or run codes in example/ folder.

  • make all

Run unit-test, golangci-lint, doctoc and gofmt.

  • make buf

Test instruction

Run unit test with make test command.

Github workflow will automatically run unit test and golangci-lint for testing and lint validation.

Contributing

We encourage and support an active, healthy community of contributors; including you! Details are in the contribution guide and the code of conduct. The rk maintainers keep an eye on issues and pull requests, but you can also report any negative conduct to [email protected].

Released under the Apache 2.0 License.

rk-grpc's People

Contributors

agungcandra avatar donghun221 avatar dongxuny avatar shengyu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

rk-grpc's Issues

Allow to listen to localhost instead

I would like making my rkboot server listen to localhost rather than the default setting 0.0.0.0. It seems that the boot.yaml does not support such change for now.

About the error i got.

Hello, I got an error msg when i purchase payment orders via alipay.
It says "Balance not enough".
Please can u fix these problems for me?Thank you.

panic: proto: file "error.proto" is already registered

Hi guys, I have an issue while working rk-grpc/v2 with google-pubsub

I got panic: proto: file "error.proto" is already registered when starting my services, below is the complete error

panic: proto: file "error.proto" is already registered


        previously from: "github.com/rookie-ninja/rk-grpc/v2/boot/error/gen"
        currently from:  "github.com/googleapis/gax-go/v2/apierror/internal/proto"
See https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict

After investigating, I found the issue is because the both rk-grpc and gax-go file descriptor is used error.proto as proto name, unlike the suggestion from google that we should also add the namespace before the filename

Below I also attach my go.mod file

module gitlab.com/tools/2sample

go 1.17

require (
	cloud.google.com/go/pubsub v1.17.0
	github.com/alicebob/miniredis/v2 v2.20.0
	github.com/go-redis/redis/v8 v8.11.5
	github.com/jackc/pgx/v4 v4.15.0
	github.com/joeshaw/envdecode v0.0.0-20200121155833-099f1fc765bd
	github.com/joho/godotenv v1.4.0
	github.com/pkg/errors v0.9.1
	github.com/rookie-ninja/rk-entry/v2 v2.1.4
	github.com/rookie-ninja/rk-grpc/v2 v2.1.2
	github.com/stretchr/testify v1.7.1
	google.golang.org/grpc v1.45.0
)

require (
	cloud.google.com/go v0.100.2 // indirect
	cloud.google.com/go/compute v1.5.0 // indirect
	cloud.google.com/go/iam v0.3.0 // indirect
	github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a // indirect
	github.com/andres-erbsen/clock v0.0.0-20160526145045-9e14626cd129 // indirect
	github.com/beorn7/perks v1.0.1 // indirect
	github.com/cespare/xxhash/v2 v2.1.2 // indirect
	github.com/davecgh/go-spew v1.1.1 // indirect
	github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
	github.com/envoyproxy/protoc-gen-validate v0.6.2 // indirect
	github.com/fsnotify/fsnotify v1.5.1 // indirect
	github.com/go-logr/logr v1.2.2 // indirect
	github.com/go-logr/stdr v1.2.2 // indirect
	github.com/golang-jwt/jwt/v4 v4.3.0 // indirect
	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
	github.com/golang/protobuf v1.5.2 // indirect
	github.com/google/go-cmp v0.5.7 // indirect
	github.com/google/uuid v1.3.0 // indirect
	github.com/googleapis/gax-go/v2 v2.2.0 // indirect
	github.com/grpc-ecosystem/grpc-gateway/v2 v2.8.0 // indirect
	github.com/hashicorp/hcl v1.0.0 // indirect
	github.com/jackc/chunkreader/v2 v2.0.1 // indirect
	github.com/jackc/pgconn v1.11.0 // indirect
	github.com/jackc/pgio v1.0.0 // indirect
	github.com/jackc/pgpassfile v1.0.0 // indirect
	github.com/jackc/pgproto3/v2 v2.2.0 // indirect
	github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b // indirect
	github.com/jackc/pgtype v1.10.0 // indirect
	github.com/jackc/puddle v1.2.1 // indirect
	github.com/magiconair/properties v1.8.5 // indirect
	github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
	github.com/mitchellh/mapstructure v1.4.3 // indirect
	github.com/pelletier/go-toml v1.9.4 // indirect
	github.com/pmezard/go-difflib v1.0.0 // indirect
	github.com/prometheus/client_golang v1.12.1 // indirect
	github.com/prometheus/client_model v0.2.0 // indirect
	github.com/prometheus/common v0.32.1 // indirect
	github.com/prometheus/procfs v0.7.3 // indirect
	github.com/rookie-ninja/rk-logger v1.2.11 // indirect
	github.com/rookie-ninja/rk-query v1.2.14 // indirect
	github.com/soheilhy/cmux v0.1.5 // indirect
	github.com/spf13/afero v1.6.0 // indirect
	github.com/spf13/cast v1.4.1 // indirect
	github.com/spf13/jwalterweatherman v1.1.0 // indirect
	github.com/spf13/pflag v1.0.5 // indirect
	github.com/spf13/viper v1.10.1 // indirect
	github.com/subosito/gotenv v1.2.0 // indirect
	github.com/yuin/gopher-lua v0.0.0-20210529063254-f4c35e4016d9 // indirect
	go.opencensus.io v0.23.0 // indirect
	go.opentelemetry.io/contrib v1.4.0 // indirect
	go.opentelemetry.io/otel v1.5.0 // indirect
	go.opentelemetry.io/otel/exporters/jaeger v1.5.0 // indirect
	go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.5.0 // indirect
	go.opentelemetry.io/otel/sdk v1.5.0 // indirect
	go.opentelemetry.io/otel/trace v1.5.0 // indirect
	go.uber.org/atomic v1.7.0 // indirect
	go.uber.org/multierr v1.6.0 // indirect
	go.uber.org/ratelimit v0.2.0 // indirect
	go.uber.org/zap v1.21.0 // indirect
	golang.org/x/crypto v0.0.0-20210817164053-32db794688a5 // indirect
	golang.org/x/net v0.0.0-20220225172249-27dd8689420f // indirect
	golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b // indirect
	golang.org/x/sync v0.0.0-20210220032951-036812b2e83c // indirect
	golang.org/x/sys v0.0.0-20220227234510-4e6760a101f9 // indirect
	golang.org/x/text v0.3.7 // indirect
	google.golang.org/api v0.71.0 // indirect
	google.golang.org/appengine v1.6.7 // indirect
	google.golang.org/genproto v0.0.0-20220310185008-1973136f34c6 // indirect
	google.golang.org/protobuf v1.28.0 // indirect
	gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
	gopkg.in/ini.v1 v1.66.2 // indirect
	gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect
	gopkg.in/yaml.v2 v2.4.0 // indirect
	gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect
)

Allow excluding label for prometheus middleware

Current prometheus middleware has restPath as one of its label. When the content is highly varied (e.g. /resource/id, where id can be anyting), it causes significant load on the prometheus due to high cardinality.

It would be nice if there's an option to remove labels on the configuration.

PATCH request to endpoints served by grpc-gateway getting connection reset by peer

I am getting connection reset by peer when requesting to PATCH endpoints served by rk-grpc (with enableRkGwOption enabled)

 curl -vvv --location --request PATCH 'localhost:3000/v1/experiment/7' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data-raw '{
  "name": "Experiment model 123"
}'
*   Trying 127.0.0.1:3000...
* Connected to localhost (127.0.0.1) port 3000 (#0)
> PATCH /v1/experiment/7 HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/7.81.0
> Content-Type: application/json
> Accept: application/json
> Content-Length: 36
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

By default, enabling enableRkGwOption in your project will result in grpc endpoints and rest endpoints served in the same port using cmux as multiplexer. This is where the bug comes because cmux does not recognize PATCH HTTP methods in their default HTTP method. (https://github.com/soheilhy/cmux/blob/v0.1.5/matchers.go#L46-L55)

There are two possible solutions:

  1. Serving different port for gRPC and REST API (configurable in yml file)
  2. Adding PATCH method in boot/grpc_entry.go file so cmux recognize PATCH method:
httpL := tcpL.Match(cmux.HTTP1Fast("PATCH"))

Unable to set rateLimit per path

Hello team,

I tried to set rate limit per path but I always get 429 even though I have put reqPerSec > 0.

How to find the issue:

  1. Use example in https://github.com/rookie-ninja/rk-grpc
  2. Add rate limit per path in boot.yaml:
grpc:
  - name: greeter
    port: 8080
    enabled: true
    enableRkGwOption: true
    middleware:
      rateLimit:
        enabled: true
        paths:
          - path: "/v1/greeter"
            reqPerSec: 100
  1. go run main.go
  2. curl -v "localhost:8080/v1/greeter"

error":{"code":429,"status":"Too Many Requests","message":"slow down your request","details":[{"code":8,"status":"ResourceExhausted","message":"slow down your request"}]}}

Should I be able to call the endpoint right as I have set the reqPerSec to 100?

content type of http response is wrong when return grpc error through grpc gateway

How to find this issue:

  1. use example in rk-boot: https://github.com/rookie-ninja/rk-boot/tree/main/example/web/grpc
  2. enable grpc.enableRkGwOption in boot.yaml
  3. return an error in Hello function in main.go
  4. go run main.go
  5. curl -v "http://0.0.0.0:8080/v1/hello?hi="
*   Trying 0.0.0.0:8080...
* Connected to 0.0.0.0 (127.0.0.1) port 8080 (#0)
> GET /v1/hello?hi= HTTP/1.1
> Host: 0.0.0.0:8080
> User-Agent: curl/7.77.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Content-Type: application/json
< Content-Type: application/grpc   <---  here
< X-Demo-App-Domain: *
< X-Demo-App-Name: 
< X-Demo-App-Unix-Time: 2022-04-04T20:33:49.565908+08:00
< X-Demo-App-Version: 
< X-Demo-Received-Time: 2022-04-04T20:33:49.565908+08:00
< X-Request-Id: 83731756-78c3-4723-9c9b-99f894b69408
< Date: Mon, 04 Apr 2022 12:33:49 GMT
< Content-Length: 74
< 
* Connection #0 to host 0.0.0.0 left intact
{"error":{"code":400,"status":"Bad Request","message":"空","details":[]}}%   

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.