Code Monkey home page Code Monkey logo

query-playback's Introduction

Query Playback

© 2011-2017 Percona LLC. © 2017 Dropbox, Inc. See LICENSE for full text of the GPL.

Query Playback is a tool for replaying the load of one database server to another.

see ./bin/percona-playback --help for details

Build Dependencies

Query Playback is mostly comprised of plugins, some of which have their own build dependencies (and simply won't be built if you don't have the required libraries).

You will need the 'development' packages for all of these (often suffixed by '-devel' for RedHat or '-dev' for Debian derived distributions).

The core of Query Playback requires:

  • libtbb (Intel Threading Building Blocks)
  • Boost >= 1.53
  • pkg-config
  • cmake >= 2.8.7

The plugin that uses the MySQL client library needs:

  • libmysqlclient (MySQL Client Libraries)

Install dependencies (CentOS 7)

yum -y install tbb tbb-devel cmake boost boost-devel

Building

$ mkdir build_dir
$ cd build_dir
$ cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ..
$ make

Test Suite

There are several tests included, these use a dummy database client plugin to mainly test that parsing of the logs is correct.

$ make check

Building release tarball

$ cpack ..

Building release debian package:

$ cpack -G DEB ..

Capturing everything in the slow log

To capture queries for Query Playback to play back, you will need to run MySQL, MariaDB or Percona Server with the slow query log capturing the queries you wish to run. You will need the following options to mysqld:

--slow-query-log --log-slow-admin-statements --log-slow-verbosity=microtime --long-query-time=0

Documentation

You can build the documentation by running make in the docs directory.

query-playback's People

Contributors

dsmythe avatar dveeden avatar fmbiete avatar hrvojem avatar jobcase-plebedev avatar stewartsmith avatar undingen avatar vadimtk avatar vdavidoff avatar vlad-lesin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

query-playback's Issues

Can't make

I'm running OSX. I cloned the repo and tried to follow the instructions. When I run make, I get this:

/Applications/Xcode.app/Contents/Developer/usr/bin/make  all-recursive
Making all in po
make[2]: *** No rule to make target `all'.  Stop.
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

Running configure again, I see this at the bottom:

grep: po/Makefile.in: No such file or directory
config.status: WARNING: po/Makefile.in.in was not created by intltoolize.

Please improve README install instructions

The install instructions are missing many tools like gcc, c++, mysqllib, git, etc. Please make the yum install command a complete command for building playback. Thanks.

CMake Error: your CXX compiler: "CMAKE_CXX_COMPILER-NOTFOUND" was not found.   Please set CMAKE_CXX_COMPILER to a valid compiler path or name.

tcpdump plugin

Hi, how do I build this with the tcpdump plugin? I just built it and ./percona-playback --help shows this:

Loaded Input Plugins: general-log query-log

If there are docs you can point me to, that should be more than sufficient for me to figure it out. Thanks so much~ 😁

PS: The tcpdump plugin wasn't abandoned, was it?

Using playback with more databases

Hi,
I'm tring to use QP to simulate a full load of a DB server with more 300 databases but when I try to launch QP the schema name is mandatory to run, but with this parameter ONLY the queries for specified schema are executed from my slog.log.
How is possible to replicate ALL queries present in my slow.log ?
Thank you very much

Can't complile w/o libdrizzle

This doesn't work if libdrizzle-devel is not installed.

./configure --disable-libdrizzle
make

I think it makes sense to be able to compile it w/o drizzle

make error

0771a4c8a5c1a12dcb74f5b6e26dd69

Installed MySQL Packages:
mysql-community-devel-5.7.28-1.el7.x86_64
mysql-community-libs-5.7.29-1.el7.x86_64
mysql-community-libs-compat-5.7.29-1.el7.x86_64
mysql-community-client-5.7.29-1.el7.x86_64
mysql-community-server-5.7.29-1.el7.x86_64
mysql-community-common-5.7.29-1.el7.x86_64

Can you give me some help

CMake Error: MYSQL_LIB

Fresh CentOS7 install.

    1  yum -y install tbb tbb-devel cmake boost boost-devel gcc gcc-c++ git
    2  yum install http://www.percona.com/downloads/percona-release/redhat/0.1-4/percona-release-0.1-4.noarch.rpm
    4  yum install Percona-Server-client-57 Percona-Server-devel-57.x86_64 -y
[root@vimeo-mysql-test-3 build_dir]# cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ..
-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.8.5
-- Check for working C compiler: /bin/cc
-- Check for working C compiler: /bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /bin/c++
-- Check for working CXX compiler: /bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Could NOT find Doxygen (missing:  DOXYGEN_EXECUTABLE)
-- Boost version: 1.53.0
-- Found the following Boost libraries:
--   system
--   thread
--   regex
--   program_options
--   chrono
Processing plugin error_report
Processing plugin full_report
Processing plugin general_log
Processing plugin mysql_client
Processing plugin null_dbclient
Processing plugin query_log
Processing plugin simple_report
Processing plugin thread_per_connection
Processing plugin thread_pool
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
MYSQL_LIB
    linked by target "mysql_client" in directory /root/query-playback/percona_playback/mysql_client

Compilation failure

[ 36%] Building CXX object percona_playback/query_log/CMakeFiles/query_log.dir/query_log.cc.o
/opt/vadim/query/query-playback/percona_playback/query_log/query_log.cc:55:8: error: ‘atomic’ in namespace ‘boost’ does not name a type
static boost::atomic g_max_behind_ns;
^
/opt/vadim/query/query-playback/percona_playback/query_log/query_log.cc: In function ‘bool parse_time(boost::string_ref, QueryLogEntry::TimePoint&)’:
/opt/vadim/query/query-playback/percona_playback/query_log/query_log.cc:89:27: error: ‘boost::string_ref’ has no member named ‘to_string’
int num_read = sscanf(s.to_string().c_str(), "# Time: %02d%02d%02d %2d:%02d:%02d.%06lld",

Percona-playback looking into test database for each table when executing

Hi Team,

I am doing a load test using percona-playback for MariaDB-10.6 on ubuntu-18, when I am running percona playback it is looking into a test database for all the tables instead of respective databases, can someone help me with this, I am attaching the errors below.

Error during query: Table 'test.' doesn't exist, number of tries 1 of 1
Error during query: Table 'test.
' doesn't exist, number of tries 1 of 1

percona-playback --mysql-max-retries=0 --mysql-user=root --mysql-password=***** --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-socket=/var/run/mysqld/mysqld.sock --db-plugin=libmysqlclient --query-log-file=/usr/local/src/mariadb-slow.log

I used this option as well - --mysql-test-connect=off

it is doing this for all the database tables.
I am attaching the installation procedure below.

https://gist.github.com/zazabe/e5b4f1fd5135b5e7afb7c4a55142a7f7

Thanks,
Jabir

threads leak in thread-per-connection dispatcher plugin

When thread-per-connection dispatcher plugin is used each new connection is served by separate thread. In the case if there is no information about connection termination in input file the thread which serves connection will not be finished and the number of spawned threads and mysql connections will rise. As for tcpdump plugin such situation can be when there is no FIN/RST packet in tcpdump file due to dropped packets during capturing (see https://bugs.launchpad.net/percona-playback/+bug/1034525 ). query-log plugin does not contain the code for detecting connection finishing at all.

The fast workaround: use thread thread-pool dispatcher plugin instead of default thread-per-connection(--dispatcher-plugin=thread-pool).

The possible fixes:

  1. check if it is possible to detect connection finishing in query log and implement corresponsing feature in query-log plugin (see "administrator command: Quit" in query log file)
  2. set time-to-live for inactive threads in thread-per-connection dispatcher plugin

Compile error on Centos 7: undefined reference to symbol 'uncompress'

Hi, when i try to compile i get the following error.

[  4%] Built target db_thread
[  8%] Built target thread_per_connection
[ 12%] Built target error_report
[ 24%] Built target general_log
[ 28%] Built target full_report
[ 32%] Built target query_log
[ 36%] Built target null_dbclient
[ 40%] Built target simple_report
[ 44%] Built target mysql_client
[ 48%] Built target thread_pool
[ 56%] Built target libpercona-playback
Linking CXX executable percona-playback
/bin/ld: /usr/lib64/mysql/libmysqlclient_r.a(my_compress.c.o): undefined reference to symbol 'uncompress'
//usr/lib64/libz.so.1: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [percona-playback] Error 1
make[1]: *** [CMakeFiles/percona-playback.dir/all] Error 2
make: *** [all] Error 2

I read that this is due to zlib and possible -lz option missing in the Makefile but not sure ?

The zlib dev package is installed.

ll /usr/lib64/libz.so.1
lrwxrwxrwx. 1 root root 13 Feb  2 14:50 /usr/lib64/libz.so.1 -> libz.so.1.2.7

Thank you.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.