Code Monkey home page Code Monkey logo

kids's Introduction

kids

Build Status

Kids is a log aggregation system.

It aggregates messages like Scribe and its pub/sub pattern is ported from Redis.

中文文档

Features

  • Real-time subscription
  • Distributed collection
  • Message persistence
  • Multithreading
  • Redis protocol
  • No third-party dependencies

Quickstart

From Source

You need a complier with C++11 support like GCC 4.7 (or later) or Clang.

Download a source release, then:

tar xzf kids-VERSION.tar.gz
cd kids-VERSION
./configure
make
make test  # optional
make install

By default, it will be installed to /usr/local/bin/kids. You can use the --prefix option to specify the installation location. Run ./configure --help for more config options.

Kids comes with some sample config files in samples/, after building, simply run:

kids -c samples/dev.conf

Because kids uses redis protocol, you can use redis-cli to play with it, open another terminal:

$ redis-cli -p 3888
$ 127.0.0.1:3388> PSUBSCRIBE *

In yet another terminal:

$ redis-cli -p 3388
$ 127.0.0.1:3388> PUBLISH kids.testtopic message

redis-cli needs redis to be installed. On Mac, you can run brew install redis to install it. On Linux, run sudo apt-get install redis-tools

Run kids --help for more running options.

Using docker

Do the following:

git clone https://github.com/zhihu/kids.git
cd kids
cp samples/dev.conf debian/kids.conf
docker build -t zhihu/kids .

Now you can run it like this:

docker run -d -p 3388:3388 zhihu/kids

You can also specify the config file like this:

docker run -d -v /path/to/kids/conf:/etc/kids.conf -p 3388:3388 zhihu/kids

Configuration

See configuration.

Run in production

see production.

Developer

You will need

  • build-essential
  • libtool
  • automake
  • c++ compiler with c++ 11 support like gcc4.7+ or Clang

to build kids from source. Run the following to build kids:

./autogen.sh
./configure
make

License

Kids Uses BSD-3, see LICENSE for more details.

FAQ

Q: What is the meaning of "kids"?
A: "kids" is the recursive acronym of "Kids Is Data Stream".

Architecture

image

kids's People

Contributors

clowwindy avatar dccmx avatar fixdocker avatar flaneur2020 avatar hit9 avatar liyichao avatar psjay avatar qingant avatar shanicky avatar xiaogaozi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

kids's Issues

Lack of support for command `PUNSUBSCRIBE `

Hi there,
We don't have support for redis command PUNSUBSCRIBE, so there is not really a way to unsubscribe patterns except reconnecting to KIDS. I would like to submit a PR to fix this issue.

Cheers,
Andy

Parse config file error when set `flush 5s` in store file

kids version is: 1.0.0
*.conf file is like this:

...
store file {
  path /data/kids/logs/[topic];
  name [date].log;
  flush 5s;
}
...

when start kids, I met:

2016/11/07 19:34:43 [FATAL] Parse config file error
Unkonw store field(19): flush 5s

could someone give me help, tks.

master在PutMessage的时候循环对一个worker通知了多次

master在PutMessage的时候,是这样的:
for (auto it : workers_) {
it->AddNotify(worker_id);
}
这样就对msg_wait_to_notify_[worker_id]增加了4次。
请问为啥不是wokers[worker_Id]->AddNofify(),只对msg_wait_to_notify_加1 ?
麻烦大神给讲解一下,有点看不懂了。 :-)

can't build with docker

Environment:

Tencent Cloud Ubuntu Server 18.04.1 LTS 64

Error Message:

ubuntu@VM-0-17-ubuntu:~/kids$ sudo docker build -t zhihu/kids .
Sending build context to Docker daemon 2.438MB
Step 1/9 : FROM debian:wheezy
wheezy: Pulling from library/debian
2b15b7abe8b3: Pull complete
Digest: sha256:2259b099d947443e44bbd1c94967c785361af8fd22df48a08a3942e2d5630849
Status: Downloaded newer image for debian:wheezy
---> 10fcec6d95c4
Step 2/9 : MAINTAINER Li Yichao [email protected]
---> Running in 580c463c606e
Removing intermediate container 580c463c606e
---> 7ebb705db8ce
Step 3/9 : COPY ./docker/sources.list /etc/apt/sources.list
---> 4ef49fd0a7c6
Step 4/9 : RUN apt-get update && apt-get install -y --no-install-recommends build-essential libtool automake
---> Running in fe6def64cffc
Ign http://http.us.debian.org wheezy Release.gpg
Ign http://mirrors.163.com wheezy Release.gpg
Ign http://mirrors.163.com wheezy-proposed-updates Release.gpg
Ign http://mirrors.163.com wheezy/updates Release.gpg
Ign http://mirrors.163.com wheezy Release
Ign http://mirrors.163.com wheezy-proposed-updates Release
Ign http://mirrors.163.com wheezy/updates Release
Ign http://http.us.debian.org wheezy Release
Ign http://security.debian.org wheezy/updates Release.gpg
Ign http://security.debian.org wheezy/updates Release
Err http://security.debian.org wheezy/updates/contrib amd64 Packages

Err http://security.debian.org wheezy/updates/non-free amd64 Packages

Err http://security.debian.org wheezy/updates/contrib amd64 Packages

Err http://http.us.debian.org wheezy/main amd64 Packages
404 Not Found [IP: 208.80.154.15 80]
Err http://security.debian.org wheezy/updates/non-free amd64 Packages

Err http://security.debian.org wheezy/updates/main amd64 Packages

Err http://security.debian.org wheezy/updates/contrib amd64 Packages

Err http://security.debian.org wheezy/updates/non-free amd64 Packages

Err http://security.debian.org wheezy/updates/main amd64 Packages
404 Not Found
Err http://http.us.debian.org wheezy/contrib amd64 Packages
404 Not Found [IP: 208.80.154.15 80]
Err http://security.debian.org wheezy/updates/contrib amd64 Packages
404 Not Found
Err http://http.us.debian.org wheezy/non-free amd64 Packages
404 Not Found [IP: 208.80.154.15 80]
Err http://security.debian.org wheezy/updates/non-free amd64 Packages
404 Not Found
Err http://mirrors.163.com wheezy/main Sources
404 Not Found
Err http://mirrors.163.com wheezy/non-free Sources
404 Not Found
Err http://mirrors.163.com wheezy/contrib Sources
404 Not Found
Err http://mirrors.163.com wheezy/main amd64 Packages
404 Not Found
Err http://mirrors.163.com wheezy/non-free amd64 Packages
404 Not Found
Err http://mirrors.163.com wheezy/contrib amd64 Packages
404 Not Found
Err http://mirrors.163.com wheezy-proposed-updates/main Sources
404 Not Found
Err http://mirrors.163.com wheezy-proposed-updates/contrib Sources
404 Not Found
Err http://mirrors.163.com wheezy-proposed-updates/non-free Sources
404 Not Found
Err http://mirrors.163.com wheezy-proposed-updates/main amd64 Packages
404 Not Found
Err http://mirrors.163.com wheezy-proposed-updates/contrib amd64 Packages
404 Not Found
Err http://mirrors.163.com wheezy-proposed-updates/non-free amd64 Packages
404 Not Found
Err http://mirrors.163.com wheezy/updates/main Sources
404 Not Found
Err http://mirrors.163.com wheezy/updates/contrib Sources
404 Not Found
Err http://mirrors.163.com wheezy/updates/non-free Sources
404 Not Found
Err http://mirrors.163.com wheezy/updates/main amd64 Packages
404 Not Found
Err http://mirrors.163.com wheezy/updates/contrib amd64 Packages
404 Not Found
W: Failed to fetch http://mirrors.163.com/debian/dists/wheezy/main/source/Sources 404 Not Found

W: Failed to fetch http://mirrors.163.com/debian/dists/wheezy/non-free/source/Sources 404 Not Found

W: Failed to fetch http://mirrors.163.com/debian/dists/wheezy/contrib/source/Sources 404 Not Found

W: Failed to fetch http://mirrors.163.com/debian/dists/wheezy/main/binary-amd64/Packages 404 Not Found

W: Failed to fetch http://mirrors.163.com/debian/dists/wheezy/non-free/binary-amd64/Packages 404 Not Found

W: Failed to fetch http://mirrors.163.com/debian/dists/wheezy/contrib/binary-amd64/Packages 404 Not Found

W: Failed to fetch http://mirrors.163.com/debian/dists/wheezy-proposed-updates/main/source/Sources 404 Not Found

W: Failed to fetch http://mirrors.163.com/debian/dists/wheezy-proposed-updates/contrib/source/Sources 404 Not Found

W: Failed to fetch http://mirrors.163.com/debian/dists/wheezy-proposed-updates/non-free/source/Sources 404 Not Found

W: Failed to fetch http://mirrors.163.com/debian/dists/wheezy-proposed-updates/main/binary-amd64/Packages 404 Not Found

W: Failed to fetch http://mirrors.163.com/debian/dists/wheezy-proposed-updates/contrib/binary-amd64/Packages 404 Not Found

W: Failed to fetch http://mirrors.163.com/debian/dists/wheezy-proposed-updates/non-free/binary-amd64/Packages 404 Not Found

W: Failed to fetch http://mirrors.163.com/debian-security/dists/wheezy/updates/main/source/Sources 404 Not Found

W: Failed to fetch http://mirrors.163.com/debian-security/dists/wheezy/updates/contrib/source/Sources 404 Not Found

W: Failed to fetch http://mirrors.163.com/debian-security/dists/wheezy/updates/non-free/source/Sources 404 Not Found

W: Failed to fetch http://mirrors.163.com/debian-security/dists/wheezy/updates/main/binary-amd64/Packages 404 Not Found

W: Failed to fetch http://mirrors.163.com/debian-security/dists/wheezy/updates/contrib/binary-amd64/Packages 404 Not Found

W: Failed to fetch http://mirrors.163.com/debian-security/dists/wheezy/updates/non-free/binary-amd64/Packages 404 Not Found

W: Failed to fetch http://http.us.debian.org/debian/dists/wheezy/main/binary-amd64/Packages 404 Not Found [IP: 208.80.154.15 80]

W: Failed to fetch http://http.us.debian.org/debian/dists/wheezy/contrib/binary-amd64/Packages 404 Not Found [IP: 208.80.154.15 80]

W: Failed to fetch http://http.us.debian.org/debian/dists/wheezy/non-free/binary-amd64/Packages 404 Not Found [IP: 208.80.154.15 80]

W: Failed to fetch http://security.debian.org/dists/wheezy/updates/main/binary-amd64/Packages 404 Not Found

W: Failed to fetch http://security.debian.org/dists/wheezy/updates/contrib/binary-amd64/Packages 404 Not Found

W: Failed to fetch http://security.debian.org/dists/wheezy/updates/non-free/binary-amd64/Packages 404 Not Found

E: Some index files failed to download. They have been ignored, or old ones used instead.
Err http://mirrors.163.com wheezy/updates/non-free amd64 Packages
404 Not Found
The command '/bin/sh -c apt-get update && apt-get install -y --no-install-recommends build-essential libtool automake' returned a non-zero code: 100

Error to build with Dockerfile

Here's the error message.

➜  kids git:(master) sudo docker build -t zhihu/kids .                                                   [sudo] password for tobe:                                                                          [18/131]
Sending build context to Docker daemon 2.293 MB
Sending build context to Docker daemon 
Step 0 : FROM debian:wheezy
Pulling repository debian
479215127fa7: Download complete 
511136ea3c5a: Download complete 
1aeada447715: Download complete 
 ---> 479215127fa7
Step 1 : MAINTAINER Li Yichao <[email protected]>
 ---> Running in 5ce60c1353ee
 ---> a1eeb46c4294
Removing intermediate container 5ce60c1353ee
Step 2 : RUN apt-get update &&  apt-get install -yq --no-install-recommends  build-essential  libtool  automake
 ---> Running in 59f4d348c2c7
Get:1 http://security.debian.org wheezy/updates Release.gpg [836 B]
Get:2 http://security.debian.org wheezy/updates Release [102 kB]
Get:3 http://http.debian.net wheezy Release.gpg [1655 B]
Get:4 http://security.debian.org wheezy/updates/main amd64 Packages [291 kB]
Get:5 http://http.debian.net wheezy-updates Release.gpg [836 B]
Get:6 http://http.debian.net wheezy-updates Release [124 kB]
Ign http://http.debian.net wheezy Release
Get:7 http://http.debian.net wheezy/main amd64 Packages [7625 kB]
Get:8 http://http.debian.net wheezy-updates/main amd64 Packages [3329 B]
Fetched 8148 kB in 47s (172 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
The following extra packages will be installed:
  autoconf autotools-dev binutils bzip2 cpp cpp-4.7 dpkg-dev file g++ g++-4.7
  gcc gcc-4.7 libc-dev-bin libc6-dev libclass-isa-perl libdpkg-perl libgdbm3
  libgmp10 libgomp1 libitm1 libmagic1 libmpc2 libmpfr4 libquadmath0
  libstdc++6-4.7-dev libswitch-perl libtimedate-perl linux-libc-dev m4 make
  patch perl perl-modules
Suggested packages:
  autoconf2.13 autoconf-archive gnu-standards autoconf-doc gettext
  binutils-doc bzip2-doc cpp-doc gcc-4.7-locales debian-keyring g++-multilib
  g++-4.7-multilib gcc-4.7-doc libstdc++6-4.7-dbg gcc-multilib manpages-dev
  automake1.9 flex bison gdb gcc-doc gcc-4.7-multilib libmudflap0-4.7-dev
  libgcc1-dbg libgomp1-dbg libitm1-dbg libquadmath0-dbg libmudflap0-dbg
  libcloog-ppl0 libppl-c2 libppl7 binutils-gold glibc-doc libstdc++6-4.7-doc
  libtool-doc automaken gfortran fortran95-compiler gcj make-doc ed
  diffutils-doc perl-doc libterm-readline-gnu-perl libterm-readline-perl-perl
  libpod-plainer-perl
Recommended packages:
  fakeroot libalgorithm-merge-perl libfile-fcntllock-perl libltdl-dev
The following NEW packages will be installed:
  autoconf automake autotools-dev binutils build-essential bzip2 cpp cpp-4.7
  dpkg-dev file g++ g++-4.7 gcc gcc-4.7 libc-dev-bin libc6-dev
  libclass-isa-perl libdpkg-perl libgdbm3 libgmp10 libgomp1 libitm1 libmagic1
  libmpc2 libmpfr4 libquadmath0 libstdc++6-4.7-dev libswitch-perl
  libtimedate-perl libtool linux-libc-dev m4 make patch perl perl-modules
0 upgraded, 36 newly installed, 0 to remove and 0 not upgraded.
Need to get 46.3 MB of archives.
After this operation, 135 MB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  libgdbm3 libgmp10 libgomp1 libitm1 libmpfr4 libquadmath0 libmpc2 bzip2
  libclass-isa-perl perl-modules perl libswitch-perl m4 patch autoconf
  autotools-dev automake binutils libc-dev-bin libc6-dev cpp-4.7 cpp gcc-4.7
  gcc libstdc++6-4.7-dev g++-4.7 g++ make libtimedate-perl build-essential
  libtool
E: There are problems and -y was used without --force-yes
2015/01/04 10:53:28 The command [/bin/sh -c apt-get update &&  apt-get install -yq --no-install-recommends  build-essential  libtool  automake] returned a non-zero code: 255

是否有性能测试报告?

kids是否有做详细的性能测试分析?我在两台内网机器上分别部署了kids,一台做agent,另一台做server,agent端跑echo2kids程序,server端跑subscribe程序,agent端内存不停地往上飙,CPU占100%~

docker build 错误 [ `doc/Makefile.in' not found]

[root@localhost kids]# docker build -t zhihu/kids .
Sending build context to Docker daemon 1.031 MB
Sending build context to Docker daemon
Step 0 : FROM debian:wheezy
---> b3d362b23ec1
Step 1 : MAINTAINER Li Yichao [email protected]
---> Using cache
---> ea4ce3653add
Step 2 : COPY ./docker/sources.list /etc/apt/sources.list
---> Using cache
---> ffe4209fc1a8
Step 3 : RUN apt-get update && apt-get install -y --no-install-recommends build-essential libtool automake
---> Using cache
---> fa15d470609a
Step 4 : WORKDIR /kids
---> Using cache
---> b79173be6352
Step 5 : COPY . /kids
---> Using cache
---> 2bcbbeb3db40
Step 6 : RUN ./autogen.sh && ./configure && make
---> Running in 6b3b4fff6928
autoreconf: Entering directory .' autoreconf: configure.ac: not using Gettext autoreconf: running: aclocal --force autoreconf: configure.ac: tracing autoreconf: configure.ac: adding subdirectory deps/ae to autoreconf autoreconf: Entering directorydeps/ae'
autoreconf: configure.ac: not using Libtool
autoreconf: running: /usr/bin/autoconf --force
autoreconf: running: /usr/bin/autoheader --force
autoreconf: running: automake --add-missing --copy --force-missing
configure.ac:9: installing ./ar-lib' configure.ac:3: installing./install-sh'
configure.ac:3: installing ./missing' Makefile.am: installing./depcomp'
autoreconf: Leaving directory deps/ae' configure.ac:15: installing./compile'
configure.ac:19: installing ./config.guess' configure.ac:19: installing./config.sub'
configure.ac:4: installing ./install-sh' configure.ac:4: installing./missing'
src/Makefile.am: installing ./depcomp' configure.ac:83: required filedoc/Makefile.in' not found
configure.ac:83: required file `test/Makefile.in' not found
Makefile.am:1: required directory ./doc does not exist
Makefile.am:1: required directory ./test does not exist
autoreconf: automake failed with exit status: 1
INFO[0010] The command [/bin/sh -c ./autogen.sh && ./configure && make] returned a non-zero code: 1

ae库的stop变量不是需要加个volatile吗

void Worker::Stop() {
eventl_->stop = 1;
void *ret;
pthread_join(tid_, &ret);
}

大神,可以请教个问题吗,我看多线程访问了ae_loop的stop变量,不是需要把stop变量加个volatile防止编译器从寄存器里读取值导致死循环吗

赋值操作符自我赋值情况考虑

const Buffer& Buffer::operator=(const Buffer& other) {
Free();
ShallowCopy(other);
return *this;
}

你好,请问这个函数是否没有考虑 a=a这种自我赋值的情况呢

fread错误使用

int ret = fread(data, size, 1, file_);
if (size == 0 || ret == 0) return false;
if (ret == -1) {}

上面是你的filesystem.cc的代码,这段代码有错误,fread不会返回-1,当返回0的时候应该用feof,ferror来检测是什么原因导致返回的数目比1少,发生错误到关闭

引用下面这段话:
RETURN VALUE
fread and fwrite return the number of items successfully read or written (i.e., not the number of
characters). If an error occurs, or the end-of-file is reached, the return value is a short item
count (or zero).

   fread does not distinguish between end-of-file and error, and callers must use feof(3) and ferror(3)
   to determine which occurred.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.