Code Monkey home page Code Monkey logo

pinba_engine's Introduction

1. Short overview
-----------------

Pinba (PHP Is Not A Bottleneck Anymore) is a statistics server using MySQL as an interface. It accumulates and processes data sent over UDP by multiple PHP processes and 
displays statistics in a nice human-readable form of simple "reports", also providing read-only interface to the raw data in order to make possible generation of more sophisticated reports.

Pinba is not a debugging tool in a common sense, since you're not supposed to do debugging on production servers, but its main goal is to help developers to locate bottlenecks in realtime and direct developers' attention to the code that really needs it.

See "How it works" chapter below if you need more details.

2. Installation requirements
--------------------------------

Pinba requires MySQL 5.1.x sources to be built and MySQL 5.1.x installation to run. MySQL 5.1 is the first version which supports pluggable storage engines, 
so older MySQL versions cannot and will never be supported.

Libraries required:
* Google Protocol Buffers - http://code.google.com/p/protobuf
* Judy - http://judy.sf.net
(this one may be already installed in your system, just make sure it's new enough (1.4+ is enough) and check that you have -devel package installed).

Optionally used:
* Hoard memory allocator - http://www.hoard.org

Using --with-hoard option you can add compiled-in Hoard support. It helps to reduce memory consumption (and Pinba is quite memory hungry when it comes to millions of timers). 2x smaller memory footprint with Hoard is what I regularly see.

3. Installation
---------------

Make sure you have all the required libraries installed and proceed with the compilation.

3.1. Compilation
----------------
Unpack Pinba archive and start the good old configure & make procedure:

    # ./configure --with-mysql=/path/to/the/sources/of/mysql-5.1 --with-judy=/judy/prefix --with-protobuf=/protobuf/prefix --with-event=/event/prefix --libdir=/path/to/mysql/plugin/dir
    # make install

--libdir option is required in order to get the library installed into the correct directory (but you can always put the lib there manually). Usually the path looks like <MySQL install prefix>/lib/mysql/plugin, so if you've installed MySQL using /usr/local/mysql prefix, the path should be /usr/local/mysql/lib/mysql/plugin.


3.2. Plugin installation
------------------------

And then in MySQL console execute:

    mysql> INSTALL PLUGIN pinba SONAME 'libpinba_engine.so';

I'd also suggest you to create a separate database, this way:

    mysql> CREATE DATABASE pinba;

And then create the default tables:

    # mysql -D pinba < default_tables.sql

If you did `make install` or used a precompiled package, look for default_tables.sql in /usr/share/pinba_engine directory.

4. How it works
---------------

Each PHP process creates a Protobuf message and sends it to the configured Pinba server on request shutdown. The packet is sent over UDP, so it doesn't affect PHP's performance in any way. This also means some packets may be lost, as UDP is not reliable by its nature, but that should not bother you much.

Pinba server listening to the configured port (see pinba_port setting) reads and decodes arriving Protobuf messages, adding them to the temporary pool. The temporary pool is needed to prevent too frequent locking of the main pool, which might slow down the queries. Once in a while (see pinba_stats_gathering_period directive) Pinba locks down the main pool of records and adds the records from the temporary pool. It also drops outdated records (see pinba_stats_history directive) and updates indexes and base reports in the same moment. Tag reports are also updated, if any. Both main and temporary pools are implemented as cyclic buffers of limited size, created on startup and never re-allocated, so newer records always overwrite older ones.

Well, this is all I had to say so far.
If you have any questions, feel free to ask them in the list - <[email protected]>.
Send an empty email to <[email protected]> in order to subscribe.

Have fun.

5. See also
---------------

https://github.com/badoo/pinba2

An attempt to rethink internal implementation of pinba engine.

pinba_engine's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pinba_engine's Issues

libpinba_engine ignores bind-address directive.

I have bind-address = set in my my.cnf.

port=3306

bind-address = 10.xxx.xxx.xxx

netstat -anlp |grep -i mysqld
tcp 0 0 10.xxx.xxx.xxx:3306 0.0.0.0:* LISTEN 13145/mysqld
udp 0 0 0.0.0.0:30002 0.0.0.0:* 13145/mysqld

Api version error

I tried to install pinba-server plugin into MariaDB 5.5.33 and got error:
MariaDB [(none)]> INSTALL PLUGIN pinba SONAME 'libpinba_engine.so';
ERROR 1126 (HY000): Can't open shared library 'libpinba_engine.so' (errno: 0 API version for STORAGE ENGINE plugin PINBA not supported by this version of the server)

variables 'pinba_%' are not working

If write in my.cnf any variable f.e.:
pinba_stats_history = 100

mysql server doesnt started:
140924 10:08:23 [ERROR] /usr/sbin/mysqld: unknown variable 'pinba_stats_history            =100'
140924 10:08:23 [ERROR] Aborting

OS: ubuntu 12.04
mysql version: 5.5.38
pinba: 1.1.0

mysql> SHOW PLUGINS;
+--------------------------+----------+--------------------+--------------------+---------+
| Name | Status | Type | Library | License |
+--------------------------+----------+--------------------+--------------------+---------+
| binlog | ACTIVE | STORAGE ENGINE | NULL | GPL |
| mysql_native_password | ACTIVE | AUTHENTICATION | NULL | GPL |
| mysql_old_password | ACTIVE | AUTHENTICATION | NULL | GPL |
| MEMORY | ACTIVE | STORAGE ENGINE | NULL | GPL |
| CSV | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MyISAM | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MRG_MYISAM | ACTIVE | STORAGE ENGINE | NULL | GPL |
| ARCHIVE | ACTIVE | STORAGE ENGINE | NULL | GPL |
| PERFORMANCE_SCHEMA | ACTIVE | STORAGE ENGINE | NULL | GPL |
| BLACKHOLE | ACTIVE | STORAGE ENGINE | NULL | GPL |
| FEDERATED | DISABLED | STORAGE ENGINE | NULL | GPL |
| InnoDB | ACTIVE | STORAGE ENGINE | NULL | GPL |
| INNODB_TRX | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_LOCKS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_LOCK_WAITS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CMP | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CMP_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CMPMEM | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CMPMEM_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_BUFFER_PAGE | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_BUFFER_PAGE_LRU | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_BUFFER_POOL_STATS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| partition | ACTIVE | STORAGE ENGINE | NULL | GPL |
| PINBA | ACTIVE | STORAGE ENGINE | libpinba_engine.so | GPL |
+--------------------------+----------+--------------------+--------------------+---------+
24 rows in set (0.00 sec)

mysql> SHOW VARIABLES LIKE 'pinba%';
+------------------------------+---------+
| Variable_name | Value |
+------------------------------+---------+
| pinba_address | |
| pinba_cpu_start | 0 |
| pinba_data_pool_size | 0 |
| pinba_histogram_max_time | 10 |
| pinba_port | 30002 |
| pinba_request_pool_size | 1000000 |
| pinba_stats_gathering_period | 10000 |
| pinba_stats_history | 900 |
| pinba_temp_pool_size | 10000 |
| pinba_temp_pool_size_limit | 1000 |
+------------------------------+---------+

Document or provide a metric for temp pool size

Hi!

The documentation of temp pool size says that the pool size is using a fixed value. However, from the code, I can see that it is now able to grow without an upper bound. Therefore, the provided value seems to be only the initial value.

It would be also convenient to export those values in the info table for monitoring purposes.

MySQL 5.6 Support

Gentoo Linux x86_64, mysql-5.6.12

Previous build with mysql 5.5.31 worked perfectly.
Upgraded up to 5.6.12, rebuild with new mysql sources:

tony2001-pinba_engine-95dacd9 # ./configure --with-mysql=/var/tmp/portage/dev-db/mysql-5.6.12/work/mysql/ --with-judy=/usr --with-event=/usr --libdir=/usr/lib64/mysql/plugin

Getting exception:

mysql> INSTALL PLUGIN pinba SONAME 'libpinba_engine.so';
ERROR 1126 (HY000): Can't open shared library '/usr/lib64/mysql/plugin/libpinba_engine.so' (errno: 13 /usr/lib64/mysql/plugin/libpinba_engine.so: undefined symbol: _ZN7handler22read_multi_range_firstEPP18st_key_multi_rangeS1_jbP17)
mysql>

Memory leak

Memory leak:

[src/threadpool.cc:36]: (error) Memory leak: theQueue
[src/ha_pinba.cc:212]: (error) Memory leak: str_copy

Debian pinba-engine-mysql-5.6 : Depends: mysql-server-core-5.6 (< 5.6.26)

Installing Pinba by DEB packages and getting following error messages

apt-get remove --purge mysql-server

Output:
The following packages have unmet dependencies:
pinba-engine-mysql-5.6 : Depends: mysql-server-core-5.6 (< 5.6.26) but 5.6.27-2 is to be installed

Anyway to skip the version check and continue the installation with mysql-server-5.6.27?

Mysql-server 2.6.26 has vulnerabilities, any suggestions?

Problems with run docker image

docker run -d --net=host --name=pinba tony2001/pinba

check:

docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS                                                                                                NAMES
a99ad76e0309        tony2001/pinba                "/local/mysql/bin/env"   8 seconds ago       Up 8 seconds                                                                                                             pinba

and check again:

docker ps -a
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                         PORTS                                                                                                NAMES
65d5e1c0896b        tony2001/pinba                "/local/mysql/bin/env"   12 minutes ago      Exited (1) 4 minutes ago                                                                                                            jolly_goldberg

Checking whats wrong:

docker logs pinba
2016-06-10 17:27:46 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-06-10 17:27:46 0 [Note] /local/mysql/bin/mysqld (mysqld 5.6.30) starting as process 1 ...

Do you have any ideas?

pinba + mariadb, querie hangs

Hello!
Using pinba-1.0.0 and mariadb-5.5.32.
Extension installed properly and works but when Im trying to do simple query: SELECT * FROM timer LIMIT 15; query just hangs and mysql process eats 100% CPU. Im not able to stop it in proper way, only with SIGKILL.

upd:
interesting details, if Im doing query: SELECT * FROM timer LIMIT 5; it works fine, but if Ill +1 то LIMIT, it will fail:
SELECT * FROM timer LIMIT 6; --> hangs

Request for pinba_clean() feature

Hi,

After 1.1.0, I see we can use PINBA_FLUSH_RESET_DATA with pinba_flush(). But I'd like to have pinba_clean() as well.

I have this scenario:

I start a long running script. In that script, I do many jobs but I only wants to get the data (tags, timer, etc) for each job. I don't want to get the data while initiating the script (before executing jobs) and shutting down (after finishing all the jobs). So as I understand, If I use pinba_flush() after initiating and before executing jobs, it'll send data to pinba server, same for shutting down with pinba.auto_flush. I don't need those data. Therefore, I need pinba_clean() and use it before executing jobs.

Is it ok? What do you think?

Thanks in advance

Can't create any table using ENGINE=PINBA

Hello,

I use Centos 6.5 x86_64.
pinba_engine was compiled and installed successfully.

mysql> show engines \G
Engine: PINBA
Support: YES
Comment: Pinba engine
Transactions: NO
XA: NO
Savepoints: NO

mysql> show plugins \G

Name: PINBA
Status: ACTIVE
Type: STORAGE ENGINE
Library: libpinba_engine.so
License: GPL

However it is not possible to create any table with PINBA engine:

mysql> create table test (id int(8)) ENGINE=PINBA;
Query OK, 0 rows affected (0.00 sec)

mysql> show tables;
Empty set (0.00 sec)

Is there a way I can debug the engine?
BTW, I speak Russian if you'd like to speak it.

unrecognized option: --with-hoard

README asserts that Pinba can be compiled with Hoard. However, latest configure script does not recognize the option "--with-hoard" and the source code of Pinba does not contain references to any of the Hoard functions.

console aggregate don't work

I have no data in pinboard mysql tables.
When I try to run the command "/path/to/pinboard/src/Pinboard/Command/../../../console aggregate"
I get this error
[PDOException]
SQLSTATE[21S01]: Insert value list does not match column list: 1136 Column count doesn't match value count at row 1

Howto debug why protobuf packets droped by pinba server?

i have successfully installed pinba-engine (git clone from master) on ubuntu 15.04 with mysql-5.6

mysql> SHOW STORAGE ENGINES;

+--------------------+---------+----------------------------------------------------------------+--------------+------+------------+
| Engine             | Support | Comment                                                        | Transactions | XA   | Savepoints |
+--------------------+---------+----------------------------------------------------------------+--------------+------+------------+
| PINBA              | YES     | Pinba engine                                                   | NO           | NO   | NO         |

mysql> SHOW VARIABLES LIKE '%pinba%';

+------------------------------+---------+
| Variable_name                | Value   |
+------------------------------+---------+
| pinba_address                |         |
| pinba_cpu_start              | 0       |
| pinba_data_pool_size         | 0       |
| pinba_histogram_max_time     | 10      |
| pinba_port                   | 30002   |
| pinba_request_pool_size      | 1000000 |
| pinba_stats_gathering_period | 10000   |
| pinba_stats_history          | 60      |
| pinba_temp_pool_size         | 10000   |
| pinba_temp_pool_size_limit   | 1000    |
| pinba_timer_pool_size        | 100000  |
+------------------------------+---------+

i try use pynba client for python

import pynba.util
import logging
logger = logging.getLogger('pynba')
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler())

for req in xrange(1,100):
    monitor = pynba.util.ScriptMonitor(
        address=('local.coffee-monitor',30002),
        hostname='localhost',
        scriptname=__name__,
    )
    with monitor.timer(state='test_state'):
        for i in xrange(1,100):
            print 'test pynba BEGIN'
    monitor.send()
print 'test pynba END'

and give good output

test pynba BEGIN
.... 100 times
prepare protobuff
flush
test pynba END

but when i try find my requests in MySQL i see empty requests

all packets send and receive good

local.coffee-monitor> tcpdump -w pynba.pcap -i eth1 -n udp port 30002

tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
99 packets captured
99 packets received by filter
0 packets dropped by kernel

netstat -nap | grep 30002
udp 0 0 0.0.0.0:30002 0.0.0.0:* 11339/mysqld

but not collected into pinba buffer on mysql server
see https://www.cloudshark.org/captures/74b8642c632a

mysql> SELECT * FROM request;
Empty set (0.00 sec)

sorry i can't decode google protobuf from wireshark
how can i get some error messages why pinba drop this request packets?

mysql crash

Hi.

I installed mysql 5.5 and pinba. Also i installed pinboard and when it does aggregation mysql crashes. I enabled full mysql log and found query causes crash:

INSERT INTO ipm_timer
                        (timer_id, request_id, hit_count, value, tag_name, tag_value, created_at)
                    SELECT
                        t.id, t.request_id, t.hit_count, t.value, tag.name as tag_name, tt.value as tag_value, '2015-01-16 21:38:23'
                    FROM
                        timer t
                    JOIN
                        timertag tt ON tt.timer_id = t.id
                    JOIN
                        tag ON tt.tag_id = tag.id
                    WHERE
                        t.request_id IN (0, 1, 3);

I tried to make it simple and that`s result:

SELECT
    *
FROM
    timer t
JOIN
    timertag tt ON tt.timer_id = t.id
JOIN
    tag ta ON tt.tag_id = ta.id;

Also i`ve go same error with percona 5.6

Pinba version: master

crash in ha_pinba:close()

May 29 08:03:51 perfccc mysqld: 120529 8:03:51 - mysqld got signal 11 ;
May 29 08:03:51 perfccc mysqld: This could be because you hit a bug. It is also possible that this binary
May 29 08:03:51 perfccc mysqld: or one of the libraries it was linked against is corrupt, improperly built,
May 29 08:03:51 perfccc mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
May 29 08:03:51 perfccc mysqld: We will try our best to scrape up some info that will hopefully help diagnose
May 29 08:03:51 perfccc mysqld: the problem, but since we have already crashed, something is definitely wrong
May 29 08:03:51 perfccc mysqld: and this may fail.
May 29 08:03:51 perfccc mysqld:
May 29 08:03:51 perfccc mysqld: key_buffer_size=268435456
May 29 08:03:51 perfccc mysqld: read_buffer_size=1048576
May 29 08:03:51 perfccc mysqld: max_used_connections=97
May 29 08:03:51 perfccc mysqld: max_threads=200
May 29 08:03:51 perfccc mysqld: threads_connected=37
May 29 08:03:51 perfccc mysqld: It is possible that mysqld could use up to
May 29 08:03:51 perfccc mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)max_threads = 2107414 K
May 29 08:03:51 perfccc mysqld: bytes of memory
May 29 08:03:51 perfccc mysqld: Hope that's ok; if not, decrease some variables in the equation.
May 29 08:03:51 perfccc mysqld:
May 29 08:03:51 perfccc mysqld: Thread pointer: 0x7fd3f812d0a0
May 29 08:03:51 perfccc mysqld: Attempting backtrace. You can use the following information to find out
May 29 08:03:51 perfccc mysqld: where mysqld died. If you see no messages after this, something went
May 29 08:03:51 perfccc mysqld: terribly wrong...
May 29 08:03:51 perfccc mysqld: stack_bottom = 0x7fd4073afe88 thread_stack 0x40000
May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x29) [0x7fd45ac945d9]
May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(handle_segfault+0x435) [0x7fd45a9ef105]
May 29 08:03:51 perfccc mysqld: /lib/libpthread.so.0(+0xeff0) [0x7fd45a26aff0]
May 29 08:03:51 perfccc mysqld: /usr/lib/mysql/plugin/libpinba_engine.so(ha_pinba::close()+0xab) [0x7fd43fc9ffab]
May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(closefrm(st_table
, bool)+0x10d) [0x7fd45aa4319d]
May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(+0x39dbd9) [0x7fd45aa35bd9]
May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(my_hash_delete+0x334) [0x7fd45ac8a1b4]
May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(close_thread_tables(THD_)+0x244) [0x7fd45aa356f4]
May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(dispatch_command(enum_server_command, THD_, char_, unsigned int)+0x1cc) [0x7fd45aa0125c]
May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(do_command(THD_)+0xea) [0x7fd45aa0293a]
May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(handle_one_connection+0x22d) [0x7fd45a9f59dd]
May 29 08:03:51 perfccc mysqld: /lib/libpthread.so.0(+0x68ca) [0x7fd45a2628ca]
May 29 08:03:51 perfccc mysqld: /lib/libc.so.6(clone+0x6d) [0x7fd458daa86d]
May 29 08:03:51 perfccc mysqld:
May 29 08:03:51 perfccc mysqld: Trying to get some variables.
May 29 08:03:51 perfccc mysqld: Some pointers may be invalid and cause the dump to abort.
May 29 08:03:51 perfccc mysqld: Query (0x7fd3f8022430): is an invalid pointer
May 29 08:03:51 perfccc mysqld: Connection ID (thread ID): 742694
May 29 08:03:51 perfccc mysqld: Status: NOT_KILLED
May 29 08:03:51 perfccc mysqld:
May 29 08:03:51 perfccc mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
May 29 08:03:51 perfccc mysqld: information that should help you find out what is causing the crash.

Versions:
mysql Ver 14.14 Distrib 5.1.58, for debian-linux-gnu (x86_64) using readline 6.1

mysql-server:
Installed: 5.1.58-1~dotdeb.1

pinba-mysql-engine:
Installed: 0.0.6-1.dotdeb.4

php5-pinba:
Installed: 5.3.10-1~dotdeb.1

p95 value is 0

I create report table like this:

CREATE TABLE `tag_info_category_server_name` (
  `tag1_value` varchar(64) DEFAULT NULL,
  `tag2_value` varchar(64) DEFAULT NULL,
  `req_count` int(11) DEFAULT NULL,
  `req_per_sec` float DEFAULT NULL,
  `hit_count` int(11) DEFAULT NULL,
  `hit_per_sec` float DEFAULT NULL,
  `timer_value` float DEFAULT NULL,
  `timer_median` float DEFAULT NULL,
  `index_value` varchar(256) DEFAULT NULL,
  `p90` float DEFAULT NULL,
  `p95` float DEFAULT NULL,
  `p99` float DEFAULT NULL
) ENGINE=PINBA DEFAULT CHARSET=latin1 COMMENT='tagN_info:category,__server_name::90,95,99';

but the value of p95 is always 0.

Dotdeb pinba package doesn't match mysql-server package version for Debian Squeeze

apt-cache policy pinba-mysql-5.5 mysql-server

mysql-server:
Installed: 5.5.31-1dotdeb.0
Candidate: 5.5.31-1
dotdeb.0

pinba-mysql-5.5:
Installed: 1.0.0-0dotdeb.4
Candidate: 1.0.0-0
dotdeb.4

mysql> INSTALL PLUGIN pinba SONAME 'libpinba_engine.so';
ERROR 1126 (HY000): Can't open shared library 'libpinba_engine.so' (errno: 0 API version for STORAGE ENGINE plugin is too different)

2006 MySQL server has gone away

When I execute query:

SELECT t.id, t.hit_count, t.value, tag.name as tag_name, tt.value as tag_value FROM timer t JOIN timertag tt ON tt.timer_id = t.id JOIN tag ON tt.tag_id = tag.id WHERE t.request_id = 4381

with different request_id values after some time I get error 2006 MySQL server has gone away.

In MySQL logs such information:

21:23:28 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=24
max_threads=151
thread_count=23
connection_count=23
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 338508 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x301bcc0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fa85b3ebd98 thread_stack 0x40000
/usr/libexec/mysqld(my_print_stacktrace+0x2e)[0x78b56e]
/usr/libexec/mysqld(handle_fatal_signal+0x493)[0x6742d3]
/lib64/libpthread.so.0(+0xf500)[0x7fa872ee5500]
/usr/lib64/mysql/plugin/libpinba_engine.so(_ZN8ha_pinba14tags_fetch_rowEPhmPm+0x47)[0x7fa86ea31467]
/usr/lib64/mysql/plugin/libpinba_engine.so(_ZN8ha_pinba15read_row_by_keyEPhjPKhji+0x3ff)[0x7fa86ea1d50f]
/usr/lib64/mysql/plugin/libpinba_engine.so(_ZN8ha_pinba10index_readEPhPKhj16ha_rkey_function+0x5a)[0x7fa86ea1d61a]
/usr/libexec/mysqld[0x5a54f4]
/usr/libexec/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x63)[0x5aa283]
/usr/libexec/mysqld[0x59f81f]
/usr/libexec/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x7e)[0x5aa29e]
/usr/libexec/mysqld[0x59f81f]
/usr/libexec/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x7e)[0x5aa29e]
/usr/libexec/mysqld[0x5aaba7]
/usr/libexec/mysqld(_ZN4JOIN4execEv+0xbde)[0x5bc91e]
/usr/libexec/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x162)[0x5bdf82]
/usr/libexec/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x17c)[0x5bea0c]
/usr/libexec/mysqld[0x57e377]
/usr/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x206b)[0x583b7b]
/usr/libexec/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x1bb)[0x5862db]
/usr/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1c4d)[0x5886dd]
/usr/libexec/mysqld(_Z24do_handle_one_connectionP3THD+0xd2)[0x618fb2]
/usr/libexec/mysqld(handle_one_connection+0x50)[0x6190c0]
/lib64/libpthread.so.0(+0x7851)[0x7fa872edd851]
/lib64/libc.so.6(clone+0x6d)[0x7fa8713e511d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fa794004b90): is an invalid pointer
Connection ID (thread ID): 86
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
140519 01:23:28 mysqld_safe Number of processes running now: 0
140519 01:23:28 mysqld_safe mysqld restarted
140519  1:23:29 [Note] Plugin 'FEDERATED' is disabled.
140519  1:23:29 [ERROR] Function 'pinba' already exists
140519  1:23:29 [Warning] Couldn't load plugin named 'pinba' with soname 'libpinba_engine.so'.
140519  1:23:29 InnoDB: The InnoDB memory heap is disabled
140519  1:23:29 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140519  1:23:29 InnoDB: Compressed tables use zlib 1.2.3
140519  1:23:29 InnoDB: Using Linux native AIO
140519  1:23:29 InnoDB: Initializing buffer pool, size = 128.0M
140519  1:23:29 InnoDB: Completed initialization of buffer pool
140519  1:23:29 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 129858137592
140519  1:23:29  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 129858478329
140519  1:23:29  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
InnoDB: Apply batch completed
140519  1:23:29  InnoDB: Waiting for the background threads to start
140519  1:23:30 InnoDB: 5.5.37 started; log sequence number 129858478329
140519  1:23:30 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
140519  1:23:30 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
140519  1:23:30 [Note] Server socket created on IP: '0.0.0.0'.
140519  1:23:30 [Note] Event Scheduler: Loaded 0 events
140519  1:23:30 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.37'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL) by Remi

After mysql rebooting all work as before but after some time this problem repeats.

Can't compile pinba_engine. FreeBSD 9.1, Mysql 5.5.40.

libtool: compile: g++ -DHAVE_CONFIG_H -I. -I/root/mysql-5.5.40/sql -I/root/mysql-5.5.40/include -I/root/mysql-5.5.40/regex -I/root/mysql-5.5.40 -g -O2 -fno-exceptions -fno-rtti -I/usr/local/include -I/usr/local/include -DMYSQL_DYNAMIC_PLUGIN -DNDEBUG -DJUDYERROR_NOTEST -MT main.lo -MD -MP -MF .deps/main.Tpo -c main.cc -fPIC -DPIC -o .libs/main.o
main.cc: In function 'int pinba_collector_init(pinba_daemon_settings)':
main.cc:141: error: 'cpu_set_t' was not declared in this scope
main.cc:141: error: expected ;' before 'mask' main.cc:143: error: 'mask' was not declared in this scope main.cc:143: error: 'CPU_ZERO' was not declared in this scope main.cc:144: error: 'CPU_SET' was not declared in this scope main.cc:145: error: 'pthread_setaffinity_np' was not declared in this scope main.cc:179: error: 'cpu_set_t' was not declared in this scope main.cc:179: error: expected;' before 'mask'
main.cc:181: error: 'mask' was not declared in this scope
main.cc:181: error: 'CPU_ZERO' was not declared in this scope
main.cc:182: error: 'CPU_SET' was not declared in this scope
main.cc:183: error: 'pthread_setaffinity_np' was not declared in this scope
*** [main.lo] Error code 1

Stop in /root/pinba_engine-1.1.0/src.
*** [all] Error code 1

Stop in /root/pinba_engine-1.1.0/src.
*** [all-recursive] Error code 1

Stop in /root/pinba_engine-1.1.0.

Can't restore default_tables.sql dump after installation

Restoring default_tables.sql dump leads to error:

ERROR 1005 (HY000) at line 3: Can't create table 'pinba.request' (errno: 140)

I have Ubuntu 11.10 and stock mysql-server from packages. MySQL sources are fetched from packages too.
I tried pinba_engine both from git master checkout and from http://pinba.org/files/pinba_engine-0.0.6.tar.gz

Engine seems to be correctly installed:

mysql> show plugins;

+------------+----------+----------------+--------------------+---------+
| Name       | Status   | Type           | Library            | License |
+------------+----------+----------------+--------------------+---------+
| binlog     | ACTIVE   | STORAGE ENGINE | NULL               | GPL     |
| partition  | ACTIVE   | STORAGE ENGINE | NULL               | GPL     |
| ARCHIVE    | ACTIVE   | STORAGE ENGINE | NULL               | GPL     |
| BLACKHOLE  | ACTIVE   | STORAGE ENGINE | NULL               | GPL     |
| CSV        | ACTIVE   | STORAGE ENGINE | NULL               | GPL     |
| FEDERATED  | DISABLED | STORAGE ENGINE | NULL               | GPL     |
| MEMORY     | ACTIVE   | STORAGE ENGINE | NULL               | GPL     |
| InnoDB     | ACTIVE   | STORAGE ENGINE | NULL               | GPL     |
| MyISAM     | ACTIVE   | STORAGE ENGINE | NULL               | GPL     |
| MRG_MYISAM | ACTIVE   | STORAGE ENGINE | NULL               | GPL     |
| PINBA      | ACTIVE   | STORAGE ENGINE | libpinba_engine.so | GPL     |
+------------+----------+----------------+--------------------+---------+
11 rows in set (0.00 sec)

Also there is no errors in log:

120317 11:44:57 [Note] Event Scheduler: Purging the queue. 0 events
120317 11:44:57 InnoDB: Starting shutdown...
120317 11:45:00 InnoDB: Shutdown completed; log sequence number 1 3331538341
120317 11:45:00 [Note] /usr/sbin/mysqld: Shutdown complete

120317 11:45:00 [Note] Plugin 'FEDERATED' is disabled.
120317 11:45:00 InnoDB: Initializing buffer pool, size = 8.0M
120317 11:45:00 InnoDB: Completed initialization of buffer pool
120317 11:45:01 InnoDB: Started; log sequence number 1 3331538341
120317 11:45:01 [Note] Event Scheduler: Loaded 0 events
120317 11:45:01 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.61-0ubuntu0.11.10.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)

Pinba crash while running join query

Hi, I compiled and installed pinba from master branch on osx 10.10. Then I ran this query

SELECT t.id,
       t.request_id,
       t.hit_count,
       t.value,
       tag.name AS tag_name,
       tt.value AS tag_value
FROM   timer AS t
       JOIN timertag AS tt
         ON tt.timer_id = t.id
       JOIN tag ON tt.tag_id = tag.id;

And got this error

ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query

I check in the log and see this

01:27:58 UTC - mysqld got signal 4 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=20
max_threads=153
thread_count=19
connection_count=19
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 69180 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fc9380e6a00
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 1ad442e60 thread_stack 0x40000
0   mysqld                              0x000000010b79d6ea my_print_stacktrace + 61
1   mysqld                              0x000000010b5dccbf handle_fatal_signal + 719
2   libsystem_platform.dylib            0x00007fff866e0f1a _sigtramp + 26
3   ???                                 0x00007fc9360a6000 0x0 + 140502171803648
4   libpinba_engine.so                  0x0000000126028006 _ZN8ha_pinba10index_readEPhPKhj16ha_rkey_function + 90
5   mysqld                              0x000000010b51dad2 _ZN7handler17ha_index_read_mapEPhPKhm16ha_rkey_function + 114
6   mysqld                              0x000000010b63cf78 _ZL13join_read_keyP13st_join_table + 355
7   mysqld                              0x000000010b63a36c _Z10sub_selectP4JOINP13st_join_tableb + 262
8   mysqld                              0x000000010b63a8a0 _ZL20evaluate_join_recordP4JOINP13st_join_table + 704
9   mysqld                              0x000000010b63a3d0 _Z10sub_selectP4JOINP13st_join_tableb + 362
10  mysqld                              0x000000010b63a8a0 _ZL20evaluate_join_recordP4JOINP13st_join_table + 704
11  mysqld                              0x000000010b63a3d0 _Z10sub_selectP4JOINP13st_join_tableb + 362
12  mysqld                              0x000000010b638101 _ZN4JOIN4execEv + 733
13  mysqld                              0x000000010b6a1a83 _Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex + 716
14  mysqld                              0x000000010b6a1783 _Z13handle_selectP3THDP13select_resultm + 291
15  mysqld                              0x000000010b677224 _ZL21execute_sqlcom_selectP3THDP10TABLE_LIST + 582
16  mysqld                              0x000000010b670b61 _Z21mysql_execute_commandP3THD + 2937
17  mysqld                              0x000000010b66f24f _Z11mysql_parseP3THDPcjP12Parser_state + 858
18  mysqld                              0x000000010b66caee _Z16dispatch_command19enum_server_commandP3THDPcj + 1017
19  mysqld                              0x000000010b66ec61 _Z10do_commandP3THD + 292
20  mysqld                              0x000000010b62fa74 _Z24do_handle_one_connectionP3THD + 371
21  mysqld                              0x000000010b62f8f4 handle_one_connection + 59
22  mysqld                              0x000000010b976937 pfs_spawn_thread + 311
23  libsystem_pthread.dylib             0x00007fff874772fc _pthread_body + 131
24  libsystem_pthread.dylib             0x00007fff87477279 _pthread_body + 0
25  libsystem_pthread.dylib             0x00007fff874754b1 thread_start + 13

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fc936086810): is an invalid pointer
Connection ID (thread ID): 316
Status: NOT_KILLED

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
141113 08:27:58 mysqld_safe mysqld restarted

Do you know what happened?

Some useful information:
Pinba: master branch
OS: osx 10.10
Percona Server: 5.6.21-70.0

"status" reports

It would be helpful to be able to get stats on status by script_name or by server_name.

Update for hoard 3.9

hoard 3.9 was released in June (and is now also available on github). That version brought a major restructuring of the source code which seems to have somehow removed or split up hoard.h.
Could you take a look to see what's necessary to update the code for hoard 3.9?

Impossible to delete request using binlog

mysql> delete from request;
ERROR 1598 (HY000): Binary logging not possible. Message: Statement-based format required for this statement, but not allowed by this combination of engines
mysql> truncate request;
ERROR 1598 (HY000): Binary logging not possible. Message: Statement-based format required for this statement, but not allowed by this combination of engines

Possibility to log a timer as if it started from the beginning

Hi!

I would like to be able to log a timer without starting it. When I trigger stop(), I would like the time from req_start to be used as start time. This would allow to measure time from the very beginning of a script.

I could just start pinba timer very early, but I would find more convenient to be able to not have too. I suppose that I could implement a method like start(). For example, early()? I can come with a patch if needed.

Mac OS X 10.8.2 compilation problem

Hi,

I'm trying to compile Pinba on my local machine, but with no success. What I've done by this time:

  1. Installed Google Protocol Buffers from sources.
  2. Installed libevent and judy with brew.

The ./configure part went ok.

$ ./configure \
--libdir=/usr/local/mysql/lib/plugin \
--with-protobuf=/usr/local \
--with-mysql=/usr/local/mysql \
--with-event=usr/local/Cellar/libevent/2.0.21 \
--with-judy=/usr/local/Cellar/judy/1.0.5

But when I try to make I got these errors:

$ make
Making all in src
make  all-am
/bin/sh ../libtool --preserve-dup-deps --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I/usr/local/mysql-5.5.28-osx10.6-x86/sql -I/usr/local/mysql-5.5.28-osx10.6-x86/include -I/usr/local/mysql-5.5.28-osx10.6-x86/regex -I/usr/local/mysql-5.5.28-osx10.6-x86     -g -O2 -fno-exceptions -fno-rtti -I/usr/local/include -I/usr/local/Cellar/libevent/2.0.21/include -I/usr/local/Cellar/judy/1.0.5/include -DMYSQL_DYNAMIC_PLUGIN -DGOOGLE_PROTOBUF_NO_RTTI -DNDEBUG -DJUDYERROR_NOTEST -MT pinba-pb.lo -MD -MP -MF .deps/pinba-pb.Tpo -c -o pinba-pb.lo pinba-pb.cc
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I/usr/local/mysql-5.5.28-osx10.6-x86/sql -I/usr/local/mysql-5.5.28-osx10.6-x86/include -I/usr/local/mysql-5.5.28-osx10.6-x86/regex -I/usr/local/mysql-5.5.28-osx10.6-x86 -g -O2 -fno-exceptions -fno-rtti -I/usr/local/include -I/usr/local/Cellar/libevent/2.0.21/include -I/usr/local/Cellar/judy/1.0.5/include -DMYSQL_DYNAMIC_PLUGIN -DGOOGLE_PROTOBUF_NO_RTTI -DNDEBUG -DJUDYERROR_NOTEST -MT pinba-pb.lo -MD -MP -MF .deps/pinba-pb.Tpo -c pinba-pb.cc  -fno-common -DPIC -o .libs/pinba-pb.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I/usr/local/mysql-5.5.28-osx10.6-x86/sql -I/usr/local/mysql-5.5.28-osx10.6-x86/include -I/usr/local/mysql-5.5.28-osx10.6-x86/regex -I/usr/local/mysql-5.5.28-osx10.6-x86 -g -O2 -fno-exceptions -fno-rtti -I/usr/local/include -I/usr/local/Cellar/libevent/2.0.21/include -I/usr/local/Cellar/judy/1.0.5/include -DMYSQL_DYNAMIC_PLUGIN -DGOOGLE_PROTOBUF_NO_RTTI -DNDEBUG -DJUDYERROR_NOTEST -MT pinba-pb.lo -MD -MP -MF .deps/pinba-pb.Tpo -c pinba-pb.cc -o pinba-pb.o >/dev/null 2>&1
mv -f .deps/pinba-pb.Tpo .deps/pinba-pb.Plo
/bin/sh ../libtool --preserve-dup-deps --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I/usr/local/mysql-5.5.28-osx10.6-x86/sql -I/usr/local/mysql-5.5.28-osx10.6-x86/include -I/usr/local/mysql-5.5.28-osx10.6-x86/regex -I/usr/local/mysql-5.5.28-osx10.6-x86     -g -O2 -fno-exceptions -fno-rtti -I/usr/local/include -I/usr/local/Cellar/libevent/2.0.21/include -I/usr/local/Cellar/judy/1.0.5/include -DMYSQL_DYNAMIC_PLUGIN -DGOOGLE_PROTOBUF_NO_RTTI -DNDEBUG -DJUDYERROR_NOTEST -MT ha_pinba.lo -MD -MP -MF .deps/ha_pinba.Tpo -c -o ha_pinba.lo ha_pinba.cc
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I/usr/local/mysql-5.5.28-osx10.6-x86/sql -I/usr/local/mysql-5.5.28-osx10.6-x86/include -I/usr/local/mysql-5.5.28-osx10.6-x86/regex -I/usr/local/mysql-5.5.28-osx10.6-x86 -g -O2 -fno-exceptions -fno-rtti -I/usr/local/include -I/usr/local/Cellar/libevent/2.0.21/include -I/usr/local/Cellar/judy/1.0.5/include -DMYSQL_DYNAMIC_PLUGIN -DGOOGLE_PROTOBUF_NO_RTTI -DNDEBUG -DJUDYERROR_NOTEST -MT ha_pinba.lo -MD -MP -MF .deps/ha_pinba.Tpo -c ha_pinba.cc  -fno-common -DPIC -o .libs/ha_pinba.o
ha_pinba.cc:36:24: error: sql/field.h: No such file or directory
ha_pinba.cc:37:26: error: sql/structs.h: No such file or directory
ha_pinba.cc:38:26: error: sql/handler.h: No such file or directory
In file included from ha_pinba.cc:46:
ha_pinba.h:62: error: 'THR_LOCK' does not name a type
ha_pinba.h:78: error: expected class-name before '{' token
ha_pinba.h:79: error: 'THR_LOCK_DATA' does not name a type
ha_pinba.h:119: error: 'TABLE_SHARE' has not been declared
ha_pinba.h:162: error: use of enum 'ha_rkey_function' without previous declaration
ha_pinba.h:174: error: 'TABLE' has not been declared
ha_pinba.h:174: error: 'HA_CREATE_INFO' has not been declared
ha_pinba.h:177: error: ISO C++ forbids declaration of 'THR_LOCK_DATA' with no type
ha_pinba.h:177: error: expected ';' before '*' token
ha_pinba.h: In member function 'ulonglong ha_pinba::table_flags() const':
ha_pinba.h:131: error: 'HA_NO_AUTO_INCREMENT' was not declared in this scope
ha_pinba.h:131: error: 'HA_BINLOG_ROW_CAPABLE' was not declared in this scope
ha_pinba.h:131: error: 'HA_NO_TRANSACTIONS' was not declared in this scope
ha_pinba.h:131: error: 'HA_STATS_RECORDS_IS_EXACT' was not declared in this scope
ha_pinba.h: In member function 'ulong ha_pinba::index_flags(uint, uint, bool) const':
ha_pinba.h:136: error: 'HA_READ_NEXT' was not declared in this scope
ha_pinba.h:136: error: 'HA_READ_PREV' was not declared in this scope
ha_pinba.h:136: error: 'HA_READ_ORDER' was not declared in this scope
ha_pinba.h:136: error: 'HA_ONLY_WHOLE_INDEX' was not declared in this scope
ha_pinba.h: In member function 'uint ha_pinba::max_supported_record_length() const':
ha_pinba.h:145: error: 'HA_MAX_REC_LENGTH' was not declared in this scope
ha_pinba.h: In member function 'uint8 ha_pinba::table_cache_type()':
ha_pinba.h:151: error: 'HA_CACHE_TBL_NOCACHE' was not declared in this scope
ha_pinba.cc: At global scope:
ha_pinba.cc:86: error: expected initializer before '*' token
ha_pinba.cc:89: error: 'HASH' does not name a type
ha_pinba.cc:94: error: 'pinba_get_table_type' declared as an 'inline' variable
ha_pinba.cc:94: error: 'TABLE' was not declared in this scope
ha_pinba.cc:94: error: 'table' was not declared in this scope
ha_pinba.cc:95: error: expected ',' or ';' before '{' token
make[2]: *** [ha_pinba.lo] Error 1
make[1]: *** [all] Error 2
make: *** [all-recursive] Error 1

Could you please help me? Should I provide any additional info?

License Information

Would it be possible to have Pinba available under BSD-style license? (http://www.linfo.org/bsdlicense.html)

Something like a dual-license with GNU GPL; if you have strong feelings about using GNU GPL.

If possible can we also have license information be more clearly mentioned in various places such as the README. A LICENSE file on the source root (and site) would also be appreciated since it doesn't really matter if you say it's license X if there's not a copy of license X I believe. Currently there's only a passing mention in the FAQ.

Since the license information is so non-existent, I feel I should probably mention this, just to share my experience with the licence. In general I find the GNU GPL license has essentially wide spread ban in most software shops over a wide variety of software disciplines unless it's a tool that comes directly bundled with the operating system or is available. The LGPL has somewhat of a better reputation, but is still quite not that well received.

Basically the issue is that GPL is that a viral license and software that integrates into other software that makes use of it might as well be considered part of the "GPL walled garden." If you're not a piece of software that has to be compiled into other software then this is not really a issue technically, but software houses just can't be bothered one bit about making the difference, since GPL's clauses essentially make any mistake suicidal. As such, more or less, anything whatsoever that uses GPL is much easier to just out right ban. The inaccessible legal babel and overly long length of it doesn't help either.

Important CPU and memory usage

Hi!

We upgraded our installation to the git version from 2013/10/22 without any other changes than updating the default tables. We observe an important grow in memory and CPU usage.

memory
cpu

I think this is expected because of the new features but could we get any performance hint on how Pinba should be expected to perform memory/CPU wise. How is it related to the number of requests, the number of custom tables and their types. What should we watch if we want to reduce memory or CPU usage? Some recipes would be welcome.

Tag versions

Hi!

Could you please tag 1.0.0 and future versions? It would be easier to package Pinba for distributions if we have a version number we can rely on.

Thanks.

Cannot load pinba sql dump after dumping

Get this error:

ERROR 1031 (HY000) at line 44: Storage engine PINBA of the table pinba.info doesn't have this option

This is with MariaDB 10.0.12 built from source. Seems to work ok otherwise.

pinba not compile with libevent-2.0.22

Hello!
I try to compile plugin for centos 6 mysql 5.6.
./configure --with-mysql=/root/mysql-src/mysql-5.6.24 --with-judy=/usr/local/bin/judy-1.0.5 --with-event=/usr/local/bin/libevent-2.0.22 --with-libdir=/usr/lib64/mysql/plugin/
and see:
checking for pthread_setaffinity_np in -lpthread... yes
checking if compiler supports -R... no
checking if compiler supports -Wl,-rpath,... yes
checking for MySQL source code... /root/mysql-src/mysql-5.6.24
checking for MySQL version >= 5.1... ok
checking libevent install prefix... /usr/local/bin/libevent-2.0.22
checking if libevent requires -lrt... configure: error: something went wrong - c annot link libevent, please report (attach you config.log)

What is wrong?

pinba-engine does not compile under osx (homebrew)

Here is a message I get when I'm trying to build pinba.

ha_pinba.cc:2687:15: error: no matching function for call to 'JudyLNext'
                                ppvalue = JudyLNext(D->tag.name_index, &str_hash, NULL);
                                          ^~~~~~~~~
/usr/local/Cellar/judy/1.0.5/include/Judy.h:263:17: note: candidate function not viable: no known conversion from 'uint64_t *' (aka 'unsigned long long *') to 'Word_t *' (aka 'unsigned long *') for 2nd argument
extern PPvoid_t JudyLNext(       Pcvoid_t  PArray, Word_t * PIndex,  P_JE);
                ^
1 error generated.

I have following source packages:

mysql-engine-pinba 1.1.0
mysql 5.6.24
judy 1.0.5
libevent 2.0.22
protobuf 2.6.1

Seems like changing type of str_hash to Word_t solves the issue but I'm not sure it's a correct way of fixing this. Maybe related with 64-bit platform?

Extends tags report in order to mix request and timer tags

I would be useful to mix request and timers tags into tags report.

For this purpose, COMMENT parsing would be upgraded in order to distinguish request tags from the legacy ones.

I suggest to prepend the request tag by a "~" character.
For example, I need to mesure "foo" and "bar", and my request sends these data:

  request:
    tags: foo
    timer:
      - tags: bar

Then the table definition would be:

  CREATE TABLE `tag_info_foo_bar` (
    `foo_value` varchar(64) DEFAULT NULL,
    `bar_value` varchar(64) DEFAULT NULL,
    `req_count` int(11) DEFAULT NULL,
    `req_per_sec` float DEFAULT NULL,
    `hit_count` int(11) DEFAULT NULL,
    `hit_per_sec` float DEFAULT NULL,
    `total_time` float DEFAULT NULL,
    `median_time` float DEFAULT NULL,
    `ru_utime_value` float DEFAULT NULL,
    `ru_stime_value` float DEFAULT NULL,
    `index_value` VARCHAR(256) DEFAULT NULL
  ) ENGINE=PINBA DEFAULT CHARSET=latin1 COMMENT='tag_info:~foo,bar'

where ~foo is the name of the request tag foo, and bar is the name of the timer tag bar.

Pinba engine memory leak?

Hi, almost every week we need to restart the "mysqld" service because it gets out of memory when the Pinba engine is under heavy use.

You can see what happens in our cacti report:
pinba-memory-1
pinba-memory-2
What we are doing now is a manual restart when we observe the system goes out of memory, but of course this is something not sustainable.

Our configuration is:

mysql> show variables like '%pinba%';
+------------------------------+---------+
| Variable_name | Value |
+------------------------------+---------+
| pinba_address | |
| pinba_port | 30002 |
| pinba_request_pool_size | 1000000 |
| pinba_show_protobuf_errors | OFF |
| pinba_stats_gathering_period | 10000 |
| pinba_stats_history | 300 |
| pinba_tag_report_timeout | 600 |
| pinba_temp_pool_size | 10000 |
+------------------------------+---------+
8 rows in set (0.00 sec)

[[email protected] ~]# rpm -qa |grep mysql
mysql55pinba-plugin-pinba-5.5.15-1
mysql55pinba-libs-5.5.15-1
mysql55pinba-server-5.5.15-1
mysql55pinba-5.5.15-1

Server version: 5.5.15-log

Make tag value's size configurable

Hi, I have some features that need tags to be bigger than 65 bytes. Currently I forked your code and make it 255, then compiled and reinstalled

Can we make that value configurable? And what are the cons for that? From my side, I see only UDP buffer sizes will be impacted, it needs to be monitored and increased to prevent losing packets.

Can't initialize plugin

Getting error trying to add plugin:
ERROR 1123 (HY000): Can't initialize function 'pinba'; Plugin initialization function failed.
Is there any way at least to get any error message to see what is actually wrong?
MySQL 5.5.38 on Ubuntu 12.04

UPD: checked out https://groups.google.com/forum/#!topic/pinba-engine-ru/mQDa18eup-c . Is it possible that my issue is the same, different build options? I used mysql-server source from Ubuntu repo to build plugin and then installed mysql-server from the same repo to get the same version. I can compile it myself and use instead of repo version if it can help.
UPD2: tried installing mysql from the same source. Not sure I did it correctly but it works (I'm mean mysql) but plugin still doesn't init
UPD3: [2014-10-08 19:55:37] [PINBA] error: main.cc:112 failed to initialize timer pool (2621440 elements). not enough memory?

Mac OS X 10.8.2 install plugin problem

Hi,

It's me again. I have successfully built libpinga_engine.so and I have these files in my mysql plugin directory /usr/local/mysql/lib/plugin:

-rwxr-xr-x   1 root  wheel   166K 20 мар 21:20 libpinba_engine.0.so
-rw-r--r--   1 root  wheel   201K 20 мар 21:20 libpinba_engine.a
-rwxr-xr-x   1 root  wheel   1,1K 20 мар 21:20 libpinba_engine.la
lrwxr-xr-x   1 root  wheel    20B 20 мар 21:20 libpinba_engine.so -> libpinba_engine.0.so

Now I'm getting an error while trying to install this plugin:

mysql> install plugin pinba soname 'libpinba_engine.so';
ERROR 1126 (HY000): Can't open shared library '/usr/local/mysql/lib/plugin/libpinba_engine.so' (errno: 2 dlopen(/usr/local/mysql/lib/plugin/libpinba_engine.so, 2): no suitable image found.  Did find:
    /usr/local/mysql/lib/plugin/libp)

I did make cmake in mysql source root directory with option -DBUILD_CONFIG=mysql_release to configure it with the same options I have on my machine, and compilation of Pinba engine went without problems.

Support for counters

We currently can simulate counters with timers by stopping a timer immediately after its start and ignore the timer_value in the reports. Although, this is not ideal as some extra data are computed (timing) and we miss an absolute hit_count.

Absolute hit_count for a counter with cyclic buffers may be challenging, but I guess it could be implemented by incrementing counters in the main pool as long as they get updates. When a counter is no longer updated, the counter will be overloaded by new data.

A separate pool may also be setup with a size matching the number of distinct counters the system can track.

Cannot restore tables after installation

Hi!

I'm using percona 5.6 (current version is 5.6.19-67.0) with fedora 20.
So, i'm installing pinba engine with all options from mysqlbug, using same sources version as rpm installation.

Build comes smoothly both for stable and git versions; plugin seems to be successfully installed, but when i'm trying to restore tables, i'm getting errors:

ERROR 1031 (HY000): Table storage engine for 'request' doesn't have this option
Query OK, 0 rows affected, 1 warning (0,00 sec)

ERROR 1031 (HY000): Table storage engine for 'tag' doesn't have this option
Query OK, 0 rows affected, 1 warning (0,01 sec)

And so one...
Have you got any suggestions how can I fix it?

Php segfault

Php get "segmentation fault", when try to authorize on dev version of our site.

Here backtrace:

Program received signal SIGSEGV, Segmentation fault.
zend_hash_index_exists (ht=0x7fc1f883e898, h=419) at /usr/src/debug/php5-source/Zend/zend_hash.c:1209
1209                    if ((p->h == h) && (p->nKeyLength == 0)) {
(gdb) bt
#0  zend_hash_index_exists (ht=0x7fc1f883e898, h=419) at /usr/src/debug/php5-source/Zend/zend_hash.c:1209
#1  0x00007fc1f8636664 in php_timer_resource_dtor (entry=<value optimized out>) at /usr/src/debug/php5-pinba-1.0.0/pinba.cc:232
#2  0x00007fc20151002e in list_entry_destructor (ptr=0x368e5c8) at /usr/src/debug/php5-source/Zend/zend_list.c:184
#3  0x00007fc20150c67e in zend_hash_apply_deleter (ht=0x7fc20190dab0, p=0x3973200) at /usr/src/debug/php5-source/Zend/zend_hash.c:814
#4  0x00007fc20150e010 in zend_hash_graceful_reverse_destroy (ht=0x7fc20190dab0) at /usr/src/debug/php5-source/Zend/zend_hash.c:850
#5  0x00007fc2015003c6 in zend_deactivate () at /usr/src/debug/php5-source/Zend/zend.c:969
#6  0x00007fc2014abb17 in php_request_shutdown (dummy=<value optimized out>) at /usr/src/debug/php5-source/main/main.c:1645
#7  0x000000000040f962 in main (argc=23652424, argv=0x168e920) at /usr/src/debug/php5-fpm-fcgi-5.3.8.20110823/sapi/fpm/fpm/fpm_main.c:1927

Without pinba all works fine.
Php works in fpm mode.

AltLinux 6
nginx-1.2.3
php5-5.3.8
php5-pinba-1.0.0
php5-fpm-fcgi-5.3.8

I have .сore file of php process in to moment of segfault, I can provide it by email

Negative values

I have in log:
[PINBA] warning: pool.cc:401 invalid packet data: req_time=1410.088257, ru_utime=-0.008998, ru_stime=0.001000, doc_size=0.426758
[PINBA] warning: pool.cc:401 invalid packet data: req_time=1410.139648, ru_utime=-0.001000, ru_stime=-0.002000, doc_size=0.443359
[PINBA] warning: pool.cc:401 invalid packet data: req_time=1410.395508, ru_utime=-0.007999, ru_stime=0.001000, doc_size=0.426758
[PINBA] warning: pool.cc:401 invalid packet data: req_time=1410.404907, ru_utime=0.000000, ru_stime=-0.002000, doc_size=0.443359
[PINBA] warning: pool.cc:401 invalid packet data: req_time=1411.169800, ru_utime=0.001000, ru_stime=-0.002000, doc_size=0.443359
[PINBA] warning: pool.cc:401 invalid packet data: req_time=1411.225098, ru_utime=-0.006999, ru_stime=0.001000, doc_size=0.426758

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.