Due to https://github.com/openresty/lua-nginx-module#cosockets-not-available-everywhere, etcd v3 could not get the version in the context of init_worker_by_lua.

To better control time-to-live of keys, one could bind multiple keys to a single lease.

here is a temporary patch:
apache/apisix@4b349dc

** attempt to call method 'watch' (a nil value)**

The length of keyspace is not limited to 1000 anymore in v3. To avoid performance degradation and eventual storage space exhaustion, operation compact could drop all information about keys superseded prior to a given keyspace revision. And the compacted ones could not be accessed by get or watch. Normally, there would be auto compaction when etcd is running.
Detail could be found here.

Background

1. Currently, lua-resty-etcd supports cluster mode, but the way of implementation is too simple: each connection changes to the next endpoint.

2. Using this mechanism, it can work well under normal circumstances, but once an api or an instance has a problem, the consequences will be unpredictable.

Issues with the current solution

1. In cluster mode, when an instance is down, there is no way to skip the down instance, and it will still be polled every time.

2. When all instances of a certain api fail (such as auth api), it will cause crazy retries, which may eventually overwhelm the ETCD cluster.

Suggested changes

1. Implement a health check mechanism, no active check is required, only passive check is required, that is, it is recorded when the connection fails.

2. There is no need to poll all instances, and only switch instances when the connection fails.

3. In a certain period of time, if there are n consecutive failures, the instance is considered unhealthy, and the instance will not be connected for a certain period of time in the future (the duration and times can be configured).

related issue: apache/apisix#2899

what do you think ?

Thanks.

Add lint-check in makefile and travis CI for better formatting.

Use the same linting strategy as APISIX.

If value is large, v3:watch will cause json decode error.

http chunk proctocol like this:
[chunk size][\r\n][chunk data][\r\n][chunk size][\r\n][chunk data][\r\n][chunk size = 0][\r\n][\r\n]

Raw http chunk package is like:

`
32
{some etcd data}

1 <------(here will cause bug)

64
{some etcd data}

0

`

if body is "\n", decode_json will caused error

https://github.com/api7/lua-resty-etcd/blob/master/lib/resty/etcd/v3.lua#L523

I will be fix this bug soon, and get a pull request

this feature maybe depend on: #5

Hi there, when I use watch feature with etcd v3, unexpected error occurred when setting timeout to 0, which I need to keep watching the peers, any idea what I was missing?

my code
local func, err = _M.etcd:watch(key, { timeout = 0 -- forever watch })

log info
2021/02/04 12:08:50 [info] 10188#12103689: *5 [lua] v3.lua:518: request_chunk(): http request meth od: POST path: /v3/watch body: {"create_request":{"key":"c2VydmVy"}} query: nil, context: ngx.time r 2021/02/04 12:09:00 [error] 10188#12103689: *5 [lua] monitor.lua:28: watch_server(): timeout, context: ngx.timer 2021/02/04 12:09:00 [error] 10188#12103689: *5 lua coroutine: runtime error: .../local/Cellar/openresty/1.15.8.3_1/lualib/resty/http.lua:409: bad argument #1 to 'tonumber' (string expected, got nil) stack traceback: coroutine 0: [C]: in function 'tonumber' .../local/Cellar/openresty/1.15.8.3_1/lualib/resty/http.lua:409: in function <.../local/Cellar/openresty/1.15.8.3_1/lualib/resty/http.lua:383> coroutine 1: [C]: in function 'co_resume' .../local/Cellar/openresty/1.15.8.3_1/lualib/resty/http.lua:80: in function 'body_reader' ...cuments/projects/openresty//lua/vendor/resty/etcd/v3.lua:533: in function 'func' ...ents/projects/openresty//lua/goatkeeper/etcd/monitor.lua:25: in function 'watch_server' ...ents/projects/openresty//lua/goatkeeper/etcd/monitor.lua:84: in function 'watch_config' ...ments/projects/openresty//lua/goatkeeper/etcd/syncer.lua:66: in function <...ments/projects/openresty//lua/goatkeeper/etcd/syncer.lua:56>, context: ngx.timer 2021/02/04 12:09:00 [error] 10188#12103689: *5 [lua] monitor.lua:28: watch_server(): failed to dec ode json body: Expected value but found invalid token at character 1, context: ngx.timer 2021/02/04 12:09:00 [error] 10188#12103689: *5 [lua] monitor.lua:28: watch_server(): can't resume a dead coroutine, context: ngx.timer 2021/02/04 12:09:00 [error] 10188#12103689: *5 [lua] monitor.lua:28: watch_server(): can't resume a dead coroutine, context: ngx.timer 2021/02/04 12:09:00 [error] 10188#12103689: *5 [lua] monitor.lua:28: watch_server(): can't resume a dead coroutine, context: ngx.timer

auth is supported with v3 ?

reproduce step

1. set empty string

etcdctl set /t/empty ''

2. use etcd v2 client get and get body as below

{
    "node":{
        "nodes":[
            {
                "createdIndex":9,
                "key":"/t/empty",
                "modifiedIndex":9
            }
        ],
        "modifiedIndex":9,
        "createdIndex":9,
        "key":"/t",
        "dir":true
    },
    "action":"get"
}

also an error log from v2.lua

2020/08/11 22:15:44 [error] 8951#0: *2 [lua] v2.lua:274: decode_dir_value(): failed to decode json: Expected value but found T_END at character 1, context: ngx.timer

It seems that decode_dir_value didn't hand empty string properly.

After dive in the implement of v2.lua, does this lib only support for json string value?

I'm trying to write a simple timer function to watch a specified key in etcd.
It seems that this funtion can work: When I change the value of key in etcd, it will get the changes as expected. But when there is no changes, "lua tcp socket read time out" keeps flushing in error.log.

After I execute openresty -s reload, the previous worker process will not exit and keep the status "worker process is shutting down" for a very long time (I even do not know if it will keep this status forever)

Here is part of the code (Sorry that I cannot handle the style well...):
watch.lua:
local function watch(premature, self)
if premature then
return
end
local opts = {}
opts["protocol"] = config["etcd"][env]["protocol"]
opts["http_host"] = config["etcd"][env]["http_host"]

local cli, callback, err
cli, err = etcd.new(opts)
if err then
    ngx.log(ngx.ERR, "new etcd err: " .. err)
    ngx.timer.at(5, watch, self)
    return
end
local attr = {}
attr["timeout"] = 3
callback, err = cli:watchdir(config["etcd"][env]["watch_prefix"], attr)
if err then
    ngx.log(ngx.ERR, "get key error")
    ngx.timer.at(5, watch, self)
    return
end
local created_true = callback()
local watch_res = callback()
if watch_res then
    ngx.log(ngx.INFO, serializeTable(watch_res))
end
ngx.sleep(0.5)
ngx.timer.at(0, watch, self)

end

nginx.conf:
init_worker_by_lua_block{
local w = require("watch");
w:init();
}

#109 fixed the situation when some endpoints got taints it could be jumped over. But when all endpoints are unhealthy, it keeps the old method, so the problem of #101 still not fixed and "it will cause crazy retries". See discussion

• version: 0.8

• v3.watchdir

opts: progress_notify, prev_kv, fragment type error and etcd api response: json: cannot unmarshal string into Go value of type bool

add log code and print

local chunk_fun, err = etcd_cli:watchdir("/watch/test", {timeout = 60, prev_kv = true, fragment = true})

if res.status >= 300 then
    ngx.log(ngx.ERR, "response: " .. res.body_reader());
    return nil, "failed to watch data, response code: " .. res.status
end

 [lua] v3.lua:502: request_chunk(): http request method: POST path: /v3/watch body: {"create_request":{"range_end":"L21pY3Jvcy9zZXJ2aWNm","fragment":"true","key":"L21pY3Jvcy9zZXJ2aWNl","prev_kv":"true"}} query: nil
 [lua] v3.lua:510: request_chunk(): response: {"error":"json: cannot unmarshal string into Go value of type bool","message":"json: cannot unmarshal string into Go value of type bool","code":2}

this should be bool, maybe protobuf can not convert 'true' / 'false' to a bool

-- v3.lua
local prev_kv
if attr.prev_kv then
    prev_kv = attr.prev_kv and 'true' or 'false'
    -- fix: prev_kv = attr.prev_kv and true or false
end

local progress_notify
if attr.progress_notify then
    progress_notify = attr.progress_notify and 'true' or 'false'
    -- fix: progress_notify = attr.progress_notify and true or false
end

local fragment
if attr.fragment then
    fragment = attr.fragment and 'true' or 'false'
    -- fix: fragment = attr.fragment and true or false
end

#23 (comment)

syntax: res, err = cli:push(key:string, val:JSON value [, ttl:int])
to
syntax: res, err = cli:push(dir:string, val:JSON value [, ttl:int])

https://github.com/api7/lua-resty-etcd/blob/master/api_v2.md#push

The TLS authentication for etcd is usually bidirectional, but cosocket is not supported now, but fortunately, there is currently a PR in progress.

we need to fix this first: openresty/lua-nginx-module#997

related issue: apache/apisix#158

    if protocol == "v3" then
        local sub_ver = ver.etcdserver:sub(1, 4)
        local etcd_prefix = prefix_v3[sub_ver] or "/v3beta"
        opts.api_prefix = etcd_prefix .. (opts.api_prefix or "")
        return etcdv3.new(opts)
    end

option.api_prefix not option.preifx

problem

when I use etcd v3 watch method(default params),The following questions occurs:

1. it happens time out per 1minute .
2. There is a lot of connection building with etcd server in per worker progress

log info:

2020/04/15 15:05:34 [info] 9020#9020: *20628390 [lua] v3.lua:502: request_chunk(): http request method: POST path: /v3beta/watch body: {"create_request":{"range_end":"*********","start_revision":6647555,"key":"L2V0Y2QvY2ZnL2dsb2JhbA=="}} query: nil, context: ngx.timer
2020/04/15 15:06:34 [info] 9020#9020: *20628390 [lua] v3.lua:502: request_chunk(): http request method: POST path: /v3beta/watch body: {"create_request":{"range_end":"*********","start_revision":6647555,"key":"L2V0Y2QvY2ZnL2dsb2JhbA=="}} query: nil, context: ngx.timer
2020/04/15 15:07:34 [info] 9020#9020: *20628390 [lua] v3.lua:502: request_chunk(): http request method: POST path: /v3beta/watch body: {"create_request":{"range_end":"*********","start_revision":6647555,"key":"L2V0Y2QvY2ZnL2dsb2JhbA=="}} query: nil, context: ngx.timer
2020/04/15 15:08:34 [info] 9020#9020: *20628390 [lua] v3.lua:502: request_chunk(): http request method: POST path: /v3beta/watch body: {"create_request":{"range_end":"*********","start_revision":6647555,"key":"L2V0Y2QvY2ZnL2dsb2JhbA=="}} query: nil, context: ngx.timer
.......

other info:

etcd endpoints has three ip list such as (1.1.1.1:2379;1.1.1.2:2379;1.1.1.3:2379;)

connection info:
such as worker id: 9020 exits 58 establish connection

root@****:~# netstat -apn|grep 2379 |grep 'ESTABLISHED 9020' |wc -l
58

@YuanSheng Wang Have you met this problem? How to solve

Since in etcd v3 watch would watch all changes before the timeout, compared to only watch the first change in etcd v2. Thus v3 also supports manually cancel the watch before the watch expires.

2020/02/21 04:59:32 [error] 16028#16028: *181 [lua] v2.lua:337: get(): failed to json decode: Expected value but found invalid token at character 1, client: 192.168.0.105, server: localhost, request: "GET /gray HTTP/1.1", host: "192.168.0.107"

1 the set methd is ok

`
local cli, err = require("resty.etcd").new({prototol = "v2", http_host = "http://127.0.0.1:2379"})
local res, err = cli:get('/test/gray/123-456-789')

ngx.print('hello')
`

We should use the same style between different models.

Since there is a bug in 1.3 which was fixed now, i think may be we need to mark that tag 1.3 is not safe to use and public the next release.

etcd client cache is supported ?

Support protocol http forced jump to https

resty-etcd allows setting multiple etcd addresses, when we try to select one of them, we would like to choose a healthy one, skip the unhealthy one automatically and write an error message for this case.

#41 (comment)

Support regular domain name configuration

Could you guys tell where local tab_nkeys = require "table.nkeys" is from ?

thanks

related PR: #32 and #30

https://github.com/api7/lua-resty-etcd/blob/master/lib/resty/etcd/v3.lua#L146-L148

• This regular expression verification does not support complex domain names

example：

• http://gateway-etcd-headless.test.svc.cluster.local:2379

v3 doc need more example, such as api return schema like:

  events = { {
      kv = {
        create_revision = "42",
        key = "/aaa/bbb",
        mod_revision = "43",
        value = "abc",
        version = "2"
      }
    }, {
      kv = {
        key = "/xxx/yyy",
        mod_revision = "44"
      },
      type = "DELETE"
    } },
  header = {
    cluster_id = "14841639068965178418",
    member_id = "10276657743932975437",
    raft_term = "2",
    revision = "44"
  }
}

When I try to install lua-resty-etcd using luarocks I get following error:

luarocks --local install lua-resty-etcd 0.8
Installing http://luarocks.org/repositories/rocks/lua-resty-etcd-0.8-0.rockspec
Cloning into 'lua-resty-etcd'...
warning: Could not find remote branch v0.8 to clone.
fatal: Remote branch v0.8 not found in upstream origin

Error: Failed cloning git repository.

It seems that rockspec refers to non existing tag v0.8:
https://github.com/iresty/lua-resty-etcd/blob/0.8/rockspec/lua-resty-etcd-0.8-0.rockspec#L5

etcd supports automatic TLS as well as authentication through client certificates for both clients to server as well as peer (server to server / cluster) communication.

https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/security.md

Hello everyone,

I was trying to replace Redis with etcd to keep my configurations between multiple regions but I checked api7 etcd library only supports values in json, why?

Hi @membphis I think we could release a new version

@tokers please take a look at this issue when you have time.

Hi.
I cloned the source code.
Installed some dependencies by the following instructions.
cpan Test::Nginx
brew install nginx version 1.19.5

I mkdir /Users/qinghao/Workspace/OpenSource/apisix/lua-resty-etcd/t/v2/t/servroot in my workdir.
And then I runned prove key.t.
After that I met some errors.

My lua version is 5.3.5

I searched it on Google. And it seems lua_socket_log_errors is an Nginx Module provided by openresty? But I'm not very familiar with that. Can anyone give me some clues.

Also, I think there should be some instruction guide for users to learn how to run the test... It takes me some time to figure out https://github.com/openresty/test-nginx has been used since I didn't know it at all before.
I can do that later.

@moonming

we can add a API design first

https://github.com/api7/lua-resty-etcd/blob/master/lib/resty/etcd/v2.lua#L186-L188

ngx.log(ngx.ERR, "failed to check value, got: ", data.body.kvs[1].value,

etcdctl can get the value:

local res, err = cli:get("/offline/999999999999999999999")
for k, v in pairs(res.body.kvs[1]) do
ngx.say(k , "=", v)
end

create_revision=372 mod_revision=372 key=/offline/999999999999999999999 version=1

https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/authentication.md

/lib/resty/etcd/v3.lua line:168: is_auth = not not (user and password)
I want to ask if there is a wrong grammar

etcdv3:get method table has no value prop. on keys created without double quotation marks:

> etcdctl put /foo/baz bar
OK
> etcdctl get /foo/baz    
/foo/baz
bar

content_by_lua_block {
    local etcd = require "resty.etcd"
    local cjson = require "cjson.safe"
    local json_encode = cjson.encode

    local conn, err = etcd.new({
        protocol = "v3",
        http_host = {
            "http://etcd1:2379",
            "http://etcd2:2379",
            "http://etcd3:2379",
        },
        serializer = "raw"
    })

    res, err = conn:get("/foo/baz")

    ngx.say(json_encode(res.body.kvs[1]))
}

{"version":"1","key":"\/foo\/baz","mod_revision":"19","create_revision":"19"}

But if you create key with double quotes:

> etcdctl put /foo/baz \"bar\"
OK
> etcdctl get /foo/baz
/foo/baz
"bar"

{"version":"3","key":"\/foo\/baz","create_revision":"19","mod_revision":"24","value":"bar"}

etcdv3:set method predictably puts values with quotation marks