What is the issue about:

Methods support in Framework.

What happened:

nirvana doesn't support following methods right now:

• 405 Method Not Allowed
• 406 Not Acceptable
• 415 Unsupported Media Type

What you expected to happen:

All commonly used method should be supported

Anything else we need to know:

@caicloud/review-framework who wanna take this?

What is the issue about:
Improvement needed for StringFlag

What happened:
There are simply too many fields to be filled for StringFlag.

What you expected to happen:
Should be a builder or some other helper functions to fulfill default values.

Anything else we need to know:

What is the issue about:
GitHub pages should be set up along with two example projects:

• hello world
• todo app

What happened:
Nothing happened, I just figure that a more out-reaching stance this repo shall take.

What you expected to happen:
More open source usage, more tractions, etc.

Anything else we need to know:

• skeleton #40
• add status badge and add fork me at github button
• choose a better theme
• create CNAME and apply domain

We need to add validator support in openapi gen. It is treated as advanced topic in origin issue, but is quite important in most cases.

Warm up

Where we are last time and why we are here. Welcome @yejiayu.

I expect the whole project to be ready around end of Q4.

Topics

Review Topics (5~10 min)

• Prometheus
• OpenAPI Gen

Remaining Topics (20 min)

• Tracing
• Configuration Management
• API Validation

PR Discussion (40 min)

• Router implementation: Concept and design introduction

Action Items

We MUST decide the following items to end the meeting

• Division of tasks
  • LGTM on router implementation @walktall @caitong93 @supereagle @ddysher @xiaoq17 @jimexist (11/10/2017)
  • Configuration on staging @zoumo @caitong93 @walktall (11/17/2017)
  • Tracing catch up @yejiayu
  • README (>500lines) + PPT @liubog2008 (11/14/2017)
  • Framework @kdada (11/17/2017)
• Next Meeting Time
  • 11/10/2017

list all routes, methods, and handlers upon startup

for example http://www.dropwizard.io/1.2.2/docs/manual/core.html#snake-case

What is the issue about:

CLI package is not output full help information.

What happened:

$ ./cli -h       
this is an cli example

Usage:
  example [flags]

Flags:
  -h, --help   help for example

What you expected to happen:

$ ./cli -h       
this is an cli example

Usage:
  example [flags]

Flags:
  -h, --help   help for example
  -l, --log      blabla
  -d, --dev    blabla

Anything else we need to know:

This is built from cli example. @zoumo

回顾过去

prev meeting

PR Updates

• Error package @walktall #53
• Validation @walktall #70
• Tracing @yejiayu #56
• Configuration @zoumo #55
• OpenAPI Gen @liubog2008 #57
• Examples @kdada #71
• Examples @jimexist #49

Milestone

https://github.com/caicloud/nirvana/milestone/1

把握现在

• Logo #69
• More plan on Swapi example project @jimexist #85
• Code structure @kdada #83

PRs going over

• move /website #90
• add golint #88

Issue assignment

• #73 @kdada 12/22/2017
• #75 @kdada 12/22/2017
• #72 @liubog2008 01/01/2018
• #67 @kdada 12/22/2017
• #63 @kdada 12/22/2017

展望未来

• Next Meeting Time:
  • 12/29/2017

We need a logo, something like this --> 🐦

Plan A:

Plan B

下面图片加上 A 的翅膀

Introduction

api framework, or http web framework, is a long standing engineering backlog across engineering teams. The proposal aims to shed some light on how we want to move forward. This is by no means a thorough design, we hope to gather enough feedback to get going. Feel free to comment.

Background

No guideline is provided at this moment to start up a new apiserver from scratch, results in a divergent approaches on setting up new web projects. No conventions enforced at framework level also results in bugs due to breaking apis, validation errors, etc. On the other hand, engineering effort is wasted on solving the same range of problems which should ideally be solved in an API framework.

Goal

• Reduce api level errors and inconsistency
• Improve engineering productivity via removing repeated work, adding code generation, etc
• Adding new resource type should only require defining struct definition
• Adding validation should only require declaring validation method as part of struct definition
• Consistent behavior, structure and layout across all golang server projects
• Out-of-box server instrumentation, e.g. metrics, tracing, profiling, etc

Proposal

Survey

Below is a short survey on existing projects:

• kubernetes-admin style (go-restful)
  • routing wrapper around go-restful
  • a well-defined layout
  • count for 70% of all related projects
• loadbalancer-admin style (gin)
  • use gin directly
  • no well-defined project layout
  • count for 20% of all related projects
• liubo style (go-restful)
  • mirrors closely with kubernetes style
  • count for 10% of all related projects

Requirements

Following is all the required functionalities for our framework.

Routing, Request & Response

• Routes mapping from request to function
  • support for path parameter (e.g. /api/v1/animals/{id})
  • support for query parameter (e.g. /api/v1/animals?type=dog)
  • support regex
• Grouping Routes
  • routes can be grouped for better arragement, but doesn't have to be first class concept
• Request/Response API object
  • to/from structs in json
  • easy access to path,query,header parameters
• Middleware support
  • general middleware support is required for developer to add custom middleware
  • required default middleware: request logging, recovery, tracing
• Contextual process chain and parameter injection
  • all handlers accepts a context value representing a request context
  • requset parameters can be validated and injected into handler, see below
• API error convention
  • define canonical error code and type based on api convention
• multipart/urlencoded form support (low priority)
• file upload support (low priority)
• grpc support (low priority)

Instrumentation

• Provide default metrics at well-known endpoints for prometheus to scrape
  • need to define set of default metrics
• Tracing should be provided by default to allow better troubleshooting
  • follow opentracing, which is industry standard
  • use jaeger as tracing system
• Profiling can be enabled in debug mode for troubleshooting
  • use go profile

Validation

• Provide default validation on api types with json, e.g. required, within a range, etc
• Support custom validations defined by developers on api types
• support validation on all parameters (path, query, etc)

Usability

• A working project should be brought up with a few lines of code using the framework
• Framework must follow engineering conventions to help developers focus on business logic
• OpenAPI (swagger 2.0) specification can be generated automatically with no extra work
• Provides a well-established layout conforming to golang project layout
• Easy and standard configuration
• A reasonable support for websocket

Performance

• Performance is not a hard requirement for initial stage; but it should not introduce observable latency

None-requirements

A list of functionalities commonly seen in other frameworks that are not in our scope

• https support. While it's not hard to serve https in go, https is out of scope since all services are expected to be running behind API gateway
• html rendering, orm, etc. Let's stick it to a simple restful framework
• testing. testing should be a different library

Design and Implementation

Project Timeline

• Due 09.15: launch the project (@ddysher)
• Due 09.23: draft feature set (@ddysher)
• Due 09.29: finalize feature set and roadmap (@ddysher)
• Due 12.01: check in most features (framework, cli, validation, metrics)
• Due 12.20: check in all features, with reasonable test coverage (openapi, tracing)
• Due 01.31: documentation, examples, website should be ready

Feature Design and Planning

Following is a list of planned features, their respective design and planning.

• Basic framework
• API Validation
• Configuration Management
• OpenAPI Generation
• Instrumentation and Profiling
• Tracing

Meetings

Regular meetings will be held to discuss above topics.

• 1st meeting
• 2nd meeting
• 3rd meeting
• 4th meeting
• 5th meeting
• 6th meeting

Assignees

• Project tracking and proposal maintenance (@ddysher)
• Overall structure, project layout, etc (@ddysher)
• Routing, multiplexing, middleware, API error package, etc (@kdada)
• Configuration management (@zuomo)
• API validation (@walktall)
• OpenAPI generation (@liubog2008 @jimexist )
• Prometheus instrumentation (@caitong93)
• tracing (@yejiayu)

References

• go-restful
• gin
• chi
• validator
• code-generator
• prometheus
• jaeger

prev meeting

PR Updates

• Refactor nirvana @kdada #94

Milestone

https://github.com/caicloud/nirvana/milestone/1

Issues

• #15 @kdada

• #28 @kdada

• #31 @kdada

• #39 ``

• #41 ``

• #43 @Jimexist

• #51 @kdada

• #52 @walktall

• #59 @kdada

• #62 ``

• #63 @kdada

• #64 ``

• #66 ``

• #67 ``

• #68 ``

• #69 ``

• #72 @kdada

• #73 @kdada

• #75 @kdada

• #83 ``

• #84 ``

• #85 ``

• #92 ``

• Next Meeting Time:

  • 01/05/2017

From @walktall, current (11/10/2017) test coverage is 75%. There is still room for improvement.

Deadline

Dec 31 2017

Checklist

• cli setup
• list example
• validation example
• logging example
• errors handling example
• documentation & write up

Notice that there's no tracing example

Topics

Status Updates (30 min)

• OpenAPI Update
  • init pr in 12/15
• Validation Update
  • not much work
  • init pr after basic framework (within 2 days)
• Configuration Update
  • PR cli pkg and README(#37)
  • flag-gen
  • remaining work: viper integration
  • a struct definition for all flags (from @liubog2008)
• Instrumentation Update
  • need rewrite, pending issue
• Tracing Update
  • what do we want to trace?
  • groups/123/configs -> groups/{gname}/configs
  • next Friday, tracing demo

PR Discussion (20 min)

• Basic framework
  • more detail and status
  • suggestion on moving forward
• Task assignment
  • log @zoumo
  • errors @walktall

Action Items

• Next Meeting Time
  • 12/01/2017

Relentless automation!

One shall see that given limited time and efforts, programmers tend to skip writing tests, and code quality plummets when a whole team joins such action. Tests should be automated, if possible.

There should be an investigation into how it can be done with limited efforts from application developers.

There's gopter that was inspired by Scalacheck (which was in turn inspired by Quickcheck). Without a strong type system, Golang automated testing can be done via contracts written in the shape of metadata, etc. and extracted and enforced by reflections.

Let's see where it leads us to ...

Umbrella issue for tracking prometheus instrumentation tasks, scopes:

• Research on required metrics
• Research on profiling
• Proposal on metrics and profiling
• Integration with basic framework
• Documentation, e.g. meaning of metrics, etc
• Testing

@caitong93

What is the issue about:

Validator should have a refined error format.

What happened:

Currently, error returned from validator.v9 is not clearly enough.

What you expected to happen:

A well formatted error for nirvana.

Anything else we need to know:

Currently the errors package is a bit cumbersome:

errors.NotFound.NewFactory("not found", "").New("")

in order to make a simple not found error.

Suggest to use shortcuts or simply API structure.

What is the issue about:

Router regexp

What happened:

Given:

"/article/{id}/{opts}"
"/article/{iffd}/edit"

"/files/{id}"
"/files/{id}:delete""

Call

"/article/111/edit"

"/files/789:delete"

What you expected to happen:

Want iffd:111

But Got opts:edit id:111

Want id:789

But Got id:789:delete

Anything else we need to know:

Tests trigger these can be found in #20

Cases ported from here

@kdada @ddysher

Topics

PR Updates

• Error package
  • @walktall #53
• Validation
  • @walktall #70
• Tracing
  • @yejiayu #56
• Configuration
  • @zoumo #55
• OpenAPI Gen
  • @liubog2008 #57
• Examples
  • @kdada #71
  • @jimexist #49

TODOs

• Issue assignment
• Deadline check
  • From proposal
  • Agree on v0.1.0 release timeline + items to fix
    • Create P1 issues @caicloud/review-framework
• More examples (and assignment)
• Proposal on project refinement @kdada []
  • Project organization
  • Package clean up
  • Definition for codegen

Action Items

• Next Meeting Time

What is the issue about:
need better error messages for unsupported mime types.

What happened:
got error message: no producer for content type text/plain

What you expected to happen:
show a list of supported mime types

Anything else we need to know:

What is the issue about:
Adding more linter coverage.

What happened:
Golint and gocyclo could not be added because there are some pending fixes to be made.

What you expected to happen:
See lint errors.

Anything else we need to know:
@caicloud/review-framework esp. the router implementations, there are some complex functions that need to be refactored.

What is the issue about:
early throw in case of wrong parameters

What happened:
if I defined a handler function of type:

func(ctx context.Context, first int, after string)

while using definitions as

Parameters: []definition.Parameter{
					{
						Source: definition.Query,
						Name:   "after",
						Type:   reflect.TypeOf(""),
					},
					{
						Source:  definition.Query,
						Name:    "first",
						Type:    reflect.TypeOf(0),
						Default: 10,
					},
				},

then there's only an error thrown during request time.

But in fact, this check can be done at startup time.

What you expected to happen:

Anything else we need to know:

request for some built-in routes:

e.g.

• a helper to expose a /version route that can contain
  • version info e.g. git tag or sha digest
  • application name
  • other useful, public, non-security related info that'll help to identify the version
• a helper to define metrics (e.g. hook up with Promethus) at /metrics
• a ping method for load balance health check at /ping?

Umbrella issue for tracking basic framework tasks, scope:

• Research on existing solutions, pick one or create one
• Proposal on basic framework implementation
• Implementation: Routing, Multiplexing, Middleware, API error package, etc
• Documentation
• Testing

@kdada

Related to #5, we'll add more details later.

What is the issue about:

Open source license

What happened:

Missed

What you expected to happen:

Add the license to the repository and code.

Anything else we need to know:

What is the issue about:
4xx error should be in json format if Accept: application/json header is specified

What happened:
For now only text/plain is returned

What you expected to happen:

Anything else we need to know:

What is the issue about:

What happened:

What you expected to happen:

Anything else we need to know:

What is the issue about:
While using validator, the error message is unreadable

What happened:

While using validator:

Operators: []definition.Operator{
							validator.Var("gte=0"),
						},

the error message is:

[query]first: Key: '' Error:Field validation for '' failed on the 'gte' tag

What you expected to happen:

Anything else we need to know:

What is the issue about:

A request for more unit test for OpenAPI generation

Umbrella issue for tracking configuration management tasks, scopes:

• Research on cobra and viper
• Research on how oss create wrapper around cobra (kubernetes, docker, etc)
• Proposal on server start up options and configuration management
• Standalone implementation (if necessary and possible) (initial PR @zoumo 11/14)
• Integration with basic framework
• Testing

@zoumo

Setting up CI, and add badge to README.

@jimexist Please comment below about the tools you plan to use :)

Need better error message than:

panic: function parameters number does not adapt to definition

Topics

Previous action items

• Configuration
  • @zoumo #45
• Validation
  • @walktall #46
• Instrumentation
  • @caitong93 #48
• Project [folder] re-arrangement
  • @kdada
• Tracing
  • @yejiayu #56
• OpenAPI Update
  • @liubog2008 #57
• Start adding more documentation?
  • @walktall
  • @zoumo #54
• Example project
  • @jimexist #49

To-dos

• Walk-through issues
• Repository setup?
  • labels, milestones, etc
• Others
  • errors? @walktall
  • log? @zoumo
  • framework done? if yes, next items? @kdada
    • router change 405, xxx
    • tracing matching routes? Fill issue
    • re-org

Action Items

• Do we need a logo?
• Next Meeting Time ??

Umbrella issue for OpenAPI generation tasks, scopes:

• Research on kubernetes codegen
• Introduction on codegen, background and how it works, etc (11/07 @liubog2008 )
• Proposal on nirvana codegen
• Integration with basic framework
• Testing
• Detailed documentation on how to generate API docs

@liubog2008

Topics

Updates (5 min)

• OpenAPI Generation
  • thoughts after 11.10 seminar
  • @liubo is regularly busy, can we find another backup? @jimexist ?
• Tracing
  • @yejiayu catch up status
• Random thought
  • should we also start planning testing framework?
• Question from Slack
  • License
  • 框架暂时不支持 405 Method Not Allowed 406 Not Acceptable 415 Unsupported Media Type

PR Discussion (45 min)

• Briefing about CI badge @jimexist (5min)
  • what badges have you added
  • what needs to be taken care of whiling we add more code
  • ref: #17, #18, #19
• Check-in Test Coverage @walktall (10min)
  • what tests have you added and why do you think it's important
  • speak loud about the problems/caveats you found, and how team can help (and avoid making the same mistake in the future)
  • merge 🚀
• Bug fix PR @kdada (0.1min)
  • please add tests after we merge test coverage PR
• CLI package PR @zoumo (25min)
  • @caitong93 where is ur initial review?
  • First of all, tell us the story behind the name Mamba
  • PR Introduction and discussion
• Basic framework example @kdada (25min)
  • PR Introduction and discussion

Next Step

Need to decide our next move.

• Start OpenAPI? Yes
• Start Validation? Yes
• Start Tracing? Yes
• Start Instrumentation? Yes
• Next step for configuration? Fix and more docs

As a reminder, we need to have a working framework by end of Q4.

Action Items

• Next Meeting Time
  • 11/24/2017

@caicloud/review-framework <-- Our new team. Use this when you want to ping folks.

see dropwizard's section for example

Now a lot of methods are exported in router package, which should instead be private. Keeping them private prevent library users from abusing internal states.

@kdada This issue needs triage.

Umbrella issue for tracking API validation tasks, scopes:

• Research on requirements, e.g. what kind of validation do we need
• Research on existing solutions
• Proposal or demonstration on how validation should work in nirvana
• Integration with basic framework
• Testing

@walktall

this package should check to make sure name and usage are populated and not empty.

Topics

Updates (10 min)

• Proposal
• Generator
• Configuration
• API Framework

PR Check in (40 min)

Checkin Router implementation and create separate issues and assign to individuals (e.g. add minor features, documentation, test, etc). Also, other tasks like adding CI, etc.

Action Items

• Tasks
  • Due 11.14: introduction to generator (README + PPT)
  • Due 11.17: example configuration management
  • Due 11.17: send out basic API framework
• Next Meeting Time
  • 11/17/2017

see helmet for example

request for better documentation for cli structs.

Current docs for e.g. persistent bears little information. I think there's need for better explanation for what persistent means for a cli argument.

• better comment
• a section in the docs

Why to refactor the framework

1. We don't have a consistent design goal.

• A framework package should not expose methods/functions which is from other packages.
• A framework package provides interfaces to plug other packages rather than depend on those packages derectly.
• Main packages of nirvana should reduce dependencies with other packages (promote interface).

2. We don't have a consistent code style.

• Replace SOME_CONST with SomeConst.
• Hide unused vars and functions. #15

3. Package paths is chaotic.
4. Some packages contain limitations/bugs.

• Match mehtod in router should support returning error. #28 #72
• Helper functions for /version, /metrics, /ping. #51
• Use nirvana/errors instead of go errors and refine error messages. #52 #59 #75
• More validations. #60
• More middlewares. #62
• More debug info. #63
• Make definition.Operator to be an interface. #68

The most important Goal: Users only need to import packages which they used.

Target directory tree

.
├── cli
├── cmd
│   ├── openapi-gen
│   ├── client-gen
│   └── nirvana
├── definition
├── docs
│   └── gitbook
├── errors
├── examples
│   ├── api-basic
│   ├── cli
│   └── stream
├── hack
│   ├── flag-gen
│   ├── license
│   └── boilerplate
├── log
├── middlewares
│   └── tracing
├── operators
│   └── validator
├── service
│   └── router
├── utils
│   ├── metrics
│   ├── probe
│   ├── profiling
│   └── version
│   └── openapi
└── nirvana.go

Framework Core Packages

• definition: The package provides a self-described structure to define RESTful APIs.
• errors: The package defines a structured error for users to create reasonable errors.
• log: The package provides a glog style interface to output runtime info.
• service: The package transforms definitions to http routers and executors (from web).
  • router: The package provides a general-purpose router to match executors for paths.

Dependency Graph

                       -----------
                       | service | 
                       -----------
       ---------------------|--------------------
       |                |           |           |
--------------    ----------    -------    ----------
| definition |    | router |    | log |    | errors |
--------------    ----------    -------    ----------

Framework Core Tools

• cli: The package contains many utils to create commands.
• cmd: The package contains commands to generate project-related codes.
  • openapi-gen: The command generates openapi document from definitions.
  • client-gen: The command generates go client by definitions.
  • nirvana: The command generates standard project structure (openapi-gen and client-gen could be merged into this command).
• middlewares: The package contains all middlewares.
  • tracing: The package implements a middleware for opentracing.
• operators: The package contains operators for handling API fields.
  • validator: The package provides a bundle of validators to validate request fields.
• utils: The package contains many utils to help users to implement some functionalities quickly.
  • metrics: The util provides prometheus support.
  • probe: The util provides health check.
  • profiling: The util provides golang profiling.
  • version: The util provides http handler for application version.
  • openapi: The util provides functionalities for serving openapi pages.
• nirvana: The package is the root package of nirvana. It provides configurations to install utils and can run a server for application.

Dependency Graph

                          -------
                          | cli | 
                          -------
       ----------------------|-------------------
       |                |                       |
-----------    --------------------    ---------------------
| spf13/* |    | stretchr/testify |    | fsnotify/fsnotify |
-----------    --------------------    ---------------------

                               -----------------
                               | middlewares/* | 
                               -----------------
       --------------------------------|-------------------------
       |                               |                        |
----------------------    ----------------------------    -------------
| nirvana/definition |    | prometheus/client_golang |    | others... |
----------------------    ----------------------------    -------------

                                ---------------
                                | operators/* | 
                                ---------------
       --------------------------------|------------------------
       |                               |                       |
----------------------    ---------------------------    -------------
| nirvana/definition |    | go-playground/validator |    | others... |
----------------------    ---------------------------    -------------

          -----------
          | utils/* | 
          -----------
       --------|-------
       |              |        
-----------    -------------
| nirvana |    | others... |
-----------    -------------

Framework Chores

• docs: The package contains document for nirvana.
• examples: The package provides many examples to explain how to use nirvana.
• hack: The package contains some tools to maintain nirvana.

Plan

1. Adjust directory structure.
2. Refactor nirvana/log and nirvana/errors.

• Implement a simple logger in nirvana/log.
• Refine nirvana/errors.

👆 第一波

3. Refactor nirvana/service/router.

• Add errors in nirvana/service/router.

4. Refactor nirvana/definition and nirvana/service.

• Enrich nirvana/definition.
• Optimize nirvana/service.

5. Refator middlewares/* and operators/*.

• Implement nirvana/definition.Middlewares.
• Implement nirvana/definition.Operators.

6. Implement nirvana with plugin mode.

• Implement nirvana configurations.
• Implement nirvana server.

7. Refactor and implement utils.
8. Refactor cli.

• Remove functions which expose from cobra and viper directly.

9. Refactor cmd.
10. Refine docs, examples, hack.

What is the issue about:

travis is failing due to gometalinter timeout

What happened:

This makes every PR red

there is need for specification for streaming API and generator based handler

Topics

Updates

• OpenAPI Update
  • skip @liubog2008, interview
• Tracing Demo (15 min)
  • @yejiayu
  • Proposal or PR (next meeting)
• Configuration Update (10 min)
  • #45
  • @zoumo
• Validation Update (10 min)
  • #46
  • @walktall
• Instrumentation Update (10min)
  • #48
  • @caitong93
• Others
  • errors? @walktall
  • log? @zoumo
  • framework done? if yes, next items? @kdada
    • router change 405, xxx
    • tracing matching routes? Fill issue
    • re-org

Next Move

• Project [folder] re-arrangement?
  • @kdada
• Merge/New PRs (Before next Friday)
  • @walktall
  • @zoumo
  • @caitong93
  • New @yejiayu
• Repository setup?
  • labels, milestones, etc
• Can we start an example project?
  • @jimexist before next Saturday **
• Start adding more documentation?
  • @walktall
  • @zoumo
• OpenAPI
  • @liubog2008

Action Items

• Next Meeting Time
  • around 12/08/2017

Umbrella issue for tracking tracing tasks, scopes:

• Research on tracing tools and client
• Proposal on tracing, how it works in the framework
• Integration with basic framework
• Documentation
• Testing

@ddysher @xiaoq17