dalibo / temboard Goto Github PK

PostgreSQL Remote Control

Home Page: https://labs.dalibo.com/temboard

License: Other

Shell 3.36% Python 61.37% JavaScript 4.32% CSS 0.29% HTML 4.10% PLpgSQL 8.09% Makefile 0.78% Dockerfile 0.21% SCSS 0.77% Vue 16.71%

postgresql database rdbms management monitoring administration dashboard

temboard's Introduction

Monitor, optimize and configure multiple PostgreSQL instances.

Manage your PostgreSQL instance fleet from a unique web interface.

Features

Manage hundreds of instances in one interface.
Fleet-wide and per-instance dashboards.
Monitor PostgreSQL with advanced metrics.
Manage running sessions.
Track bloat and schedule vacuum on tables and indexes.
Track slow queries.
Tweak PostgreSQL configuration.

Docker Quickstart

We provide a complete testing environment based on Docker ! Please read our QUICKSTART guide for more details.

Install

temBoard is composed of two services:

A lightweight agent that you need to install on every PostgreSQL server you want to manage.
A central web app to control the agents and collect metrics.

temBoard project provides packages for RHEL and clones as well as Debian. See temboard.readthedocs.io for installation instructions.

Contribute

temBoard is an open project. Any contribution to improve it is welcome.

Want to contribute? Please first read our guide on contributing if you're interested in getting involved.

License

temBoard is available under the PostgreSQL License.

Candy Scordia drew héron garde-bœuf sketches.

temboard's People

Contributors

Stargazers

Watchers

Forkers

vadv daamien bersace nicolasmartin69 keshov segmond pgiraud credativ gridl pierrehilbert garrettsocling madtibo zhuomingliang frbn pavel-ekt vadim0908 databill86 leokhoa orgrim madageomapping dlax suishaojian vtelyver l00ptr yann968 dbc2000 missmoiselle dattatraya-nirmal docdocker rodrigoieh orioquezac sengsok dbwildlife zhaodiankui tilkow suryatmodulus saga1015 pirlgon lpofredc fljdin vidierr bellyfat baekdavid robinportigliatti rave-net abzelev cxz hugues-antoine mariongiusti gulyaissina 94v1c8 krysztophe eh-th

temboard's Issues

Can't modify instance:agent_address

When managing instances it's possible to change values for any field but the agent_address one.

Correlate monitoring and queries

When you click on one of the diagrams on page Supervision / Charts / Last day, could you display the list of the queries that were worked at that moment, looks like on page Activity / Running queries?
On page Activity / Running queries to sort list by clicking on column header.
Notify by email in case of exceeding the set user-defined thresholds.

P/S Thank you for the tool!

Rename the "Supervision" section into "Monitoring"

Even though the word "Supervision" exists in English, it sounds more like a French expression. Seems to me the standard name for this in the industry is "Monitoring" or "Overview" if you want to separate Monitoring 2 parts : "Alerts" + "Overview"

The progress bar is eating my CPU

In the supervision part, when chart data are not available, the "progress bar" is loading my CPU :)
It seems to be related to .progress-bar class.

Thanks

Rename manage -> settings

New UI layout does not scale when using a wide screen

Hi,

PR #81 introduced a very promising new UI layout, nice job.

I have some concerns about the limited size of the div container. Some of the plugins like supervision and activity should really take benefit of using the whole screen size.

What do you think about setting container class size to 100% in temboard.css ?

Thanks,

Update videos in README

They don't match the new layout.

Publish documenation on RTFD

Redirect HTTP to HTTPS

Timezone in supervision

The supervision shows date in UTC.

temboard=# select * from metric_cpu_current order by datetime desc limit 4;
        datetime        | host_id | cpu  |                  record
------------------------+---------+------+------------------------------------------
 2017-03-08 14:25:02+00 |       1 | cpu3 | (,00:01:00.318868,2560,1970,55490,110,0)
 2017-03-08 14:25:02+00 |       1 | cpu2 | (,00:01:00.318846,2600,1700,55500,120,0)

In the following picture, the last item is one hour behind the system date on top of the screen.

Remove `table_func` related documentation

See doc/temboard-repository-setup.md and doc/temboard-repository-setup.md.
#83 (comment)
#83 (comment)

Convert the agent API documentation from apidocjs to a format supported by readthedoc.

Showing swap activity on dashboard ?

(improvement)

Can you add a graph for swap activity on dashboard?
We use it to know if the server suffers.

Regards

Add 'include_dir' parameter in the configuration file.

The goal is to move specific configuration section from each plugin into a new configuration file (separated from the main configuration file).

Add a new API for getting agent's state.

It could be nice to have an API (/state) providing informations about the agent:

start/reload time
configuration path
log file path
home path
version
the user running the agent

role is not available in template variables when asking for login in instance

The role (user) is not provided when the user is asked to log in for an instance.
It prevents us to display the manage and logout buttons.

Remove "PG" in instance information block

"PG data" -> "Data"
"PG Port" -> "Port"
"PG Version" -> "Version"

500 error when trying on pgconf/hba

Currently when trying to load the tab for pg_hba.conf I get a 500 error.
I get the following error in my console:

repository_1  | ERROR:  unterminated quoted identifier at or near ""
repository_1  |                     SELECT rolname
repository_1  |                     FROM pg_user
repository_1  |                     JOIN pg_auth_members
repository_1  |                     ON (pg_user.usesysid=pg_auth_members.member)
repository_1  |                     JOIN pg_roles
repository_1  |                     ON (pg_roles.oid=pg_auth_members.roleid) ORDER BY rolname ASC
repository_1  |                 " at character 1
repository_1  | STATEMENT:  "
repository_1  |                     SELECT rolname
repository_1  |                     FROM pg_user
repository_1  |                     JOIN pg_auth_members
repository_1  |                     ON (pg_user.usesysid=pg_auth_members.member)
repository_1  |                     JOIN pg_roles
repository_1  |                     ON (pg_roles.oid=pg_auth_members.roleid) ORDER BY rolname ASC
repository_1  |

IPv6 support.

[supervision] Error with /server/0.0.0.0/2345/supervision/data/blocks

I get this error when going to /server/0.0.0.0/2345/supervision/week

The ''Filesystems size'' and ''Filesystems usage'' graphs are not displaid.

2016-10-31 17:42:15,405 temboard[8671]: [temboard] INFO: 200 GET /server/0.0.0.0/2345/supervision/data/blocks?start=2016-10-30T17:42:13&end=2016-10-31T17:42:13 (90.78.76.123) 446.22ms
2016-10-31 17:42:15,411 temboard[8671]: [temboard] DEBUG: Traceback (most recent call last):
2016-10-31 17:42:15,416 temboard[8671]: [temboard] DEBUG:   File "/usr/local/lib/python2.7/dist-packages/temboard-0.0.1-py2.7.egg/temboardui/plugins/supervision/__init__.py", line 254, in get_data_probe
2016-10-31 17:42:15,417 temboard[8671]: [temboard] DEBUG:     data = get_tblspc_size(self.db_session, instance.hostname, instance.pg_port, start_time, end_time)
2016-10-31 17:42:15,417 temboard[8671]: [temboard] DEBUG:   File "/usr/local/lib/python2.7/dist-packages/temboard-0.0.1-py2.7.egg/temboardui/plugins/supervision/chartdata.py", line 303, in get_tblspc_size
2016-10-31 17:42:15,417 temboard[8671]: [temboard] DEBUG:     cur.copy_expert(q_copy.format(col=col, tablename=tablename, hostname=hostname, port=port, start_datetime=start_datetime, end_datetime=end_datetime, col_type=col_type), data_buffer)
2016-10-31 17:42:15,417 temboard[8671]: [temboard] DEBUG: ProgrammingError: query-specified return tuple and crosstab function are not compatible
2016-10-31 17:42:15,418 temboard[8671]: [temboard] DEBUG: 
2016-10-31 17:42:15,418 temboard[8671]: [temboard] ERROR: query-specified return tuple and crosstab function are not compatible

Plan ?

Hi,

I think this project is really cool.
But I am wondering what is next ? Do you have a plan in mind ?

Benoit

UI refactoring

In more modern way now we know where we are going to.

Display listening socket with protocol

During startup, temboard UI logs lines like:

[temboard] INFO: Starting temboardui on 0.0.0.0:8888

It would be better to log

[temboard] INFO: Starting temboardui on https://0.0.0.0:8888

So it's clear what protocol we should use on browser, and this URL is clickable :)

Rename plugin settings

To be renamed to configuration.

Use "temBoard" everywhere

Usage of the brand name "temBoard" (snake case) should be consistent all over the code (especially in documentation).

Internal error on running queries (pg 9.6)

Hello

I use the master branch of temboard-agent

On dashboard on postgresql 9.6 instance, the action Activity>Running queries doesn't work (internal error 500)

On the temboard-agent log, I see the postgresql error sql 42703 (undefined_column)

I suppose the problem is the "waiting" column of pg_stat_activity. This column doesn't exists on 9.6

PS: the action works on my 9.4 instance

Best regards

Release 1.0a1

Preparing 1.0, we should release a pre-version to check temboard delivrability : tarball generation, upload on pypi, metadatas, install from pip, debian/rpm package from tarball, etc.

Validate python syntax - Flake8

We should run a linter (ie. Flake8) on our code.

the file ``share/supervision.sql`` is missing

I can't find the init SQL script for the supervision plugin :

As explain here:
http://temboard.readthedocs.io/en/latest/temboard-repository-setup/#installation

The file should be in share/supervision.sql

Clean whitespace

clean trailing spaces.
drop tabs in HTML, CSS and JS files.

Review packaging to allow the deployment of several agent on the same host.

Maintenance plugin.

Complete functional tests for the agent

The main goal is to have functional tests covering the whole REST API of the agent

Bad handling of "Restart PostgreSQL" in the Settings / Configuration tool

Hi there,

I use the 3 docker's quickstart thing to test temBoard deeper... when I go to Settings / Configuration tool of the 9.6 instance (port 5435 of the docker's images), and set shared_buffers from 128MB to 96MB, it asks me if I want to restart the PG server to apply changes... When I press OK, it nevers goes back to a normal state, like if the PG never restarts.

But it did the restart, applying the changes, so I have to dismiss, and I see it's applyied.

To me its a bad handling of a monitored PG instance ?

Here's a video of what I have, to show you this with an example.

Add 'exclude_probes' parameter to supervision plugin configuration. This parameter will contain a list of probes we don't want to run.

ERROR: Invalid FQDN: localhost

I'm having trouble with the temboard agent. When I start the agent I get the following error every second.

2016-10-22 09:59:38,364 temboard-agent[25756]: [dashboard_collector] ERROR: Invalid FQDN: localhost

I didn't modify the dashboard section in the temboard-agent.conf file... I'm using the default self signed SSL certificates

Can't add an instance into a group

I successfully created an instance.

On the /manage/instances page, I can click on the edit button which display the Update instance properties form.

When I try to add the instance into a group I get an "internal error" message and the following log:

2016-10-28 21:30:09,088 temboard[20449]: [temboard] INFO: 200 GET /json/manage/instance/0.0.0.0/2345 (90.78.76.123) 63.16ms
2016-10-28 21:30:17,140 temboard[20449]: [temboard] INFO: Posting instance.
2016-10-28 21:30:17,184 temboard[20449]: [temboard] DEBUG: {u'pg_port': u'5434', u'new_agent_address': u'0.0.0.0', u'hostname': u'redtail.taadeem.net', u'pg_data': u'/var/lib/postgresql/9.5/main', u'agent_key': u'NOTMYACTUALKEY', u'memory_size': u'2063396864', u'groups': [u'plop'], u'plugins': [u'activity', u'dashboard', u'supervision', u'settings'], u'new_agent_port': u'2345', u'pg_version': u'PostgreSQL 9.5.4 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.2) 5.4.0 20160609, 64-bit', u'cpu': u'1'}
2016-10-28 21:30:17,187 temboard[20449]: [temboard] DEBUG: Traceback (most recent call last):
2016-10-28 21:30:17,188 temboard[20449]: [temboard] DEBUG:   File "/usr/local/lib/python2.7/dist-packages/temboard-0.0.1-py2.7.egg/temboardui/handlers/manage/instance.py", line 128, in post_instance
2016-10-28 21:30:17,189 temboard[20449]: [temboard] DEBUG:     instance_groups = get_groups_by_instance(self.db_session, instance.agent_address, instance.agent_port)
2016-10-28 21:30:17,190 temboard[20449]: [temboard] DEBUG:   File "/usr/local/lib/python2.7/dist-packages/temboard-0.0.1-py2.7.egg/temboardui/application.py", line 383, in get_groups_by_instance
2016-10-28 21:30:17,190 temboard[20449]: [temboard] DEBUG:     return session.query(InstanceGroups).filter(InstanceGroups.agent_address == unicode(agent_address), InstanceGroups.agent_port == agent_port).order_by(InstanceGroup.group_name).all()
2016-10-28 21:30:17,191 temboard[20449]: [temboard] DEBUG: NameError: global name 'InstanceGroup' is not defined
2016-10-28 21:30:17,191 temboard[20449]: [temboard] ERROR: global name 'InstanceGroup' is not defined
2016-10-28 21:30:17,192 temboard[20449]: [temboard] INFO: Failed.

PITR Backup plugin.

Based on pitrery ?

Split temboard-agent in its own project

Todo

Create project https://github.com/dalibo/temboard-agent
Push temboard-agent code
Clean temboard #104
update docker dalibo/docker#11
Update both README.md

Error when posting new group

I can't create a new group. The creation form hangs forerver:

Here's the log trace

2016-10-26 14:27:22,978 temboard[18043]: [temboard] INFO: Posting new group.
2016-10-26 14:27:22,991 temboard[18043]: [temboard] DEBUG: {u'new_group_name': u'test', u'description': u'test'}
2016-10-26 14:27:23,087 temboard[18043]: [temboard] DEBUG: Traceback (most recent call last):
2016-10-26 14:27:23,089 temboard[18043]: [temboard] DEBUG:   File "/usr/local/lib/python2.7/dist-packages/temboard-0.0.1-py2.7.egg/temboardui/handlers/manage/group.py", line 164, in post_group
2016-10-26 14:27:23,090 temboard[18043]: [temboard] DEBUG:     group_kind)
2016-10-26 14:27:23,093 temboard[18043]: [temboard] DEBUG:   File "/usr/local/lib/python2.7/dist-packages/temboard-0.0.1-py2.7.egg/temboardui/application.py", line 105, in add_group
2016-10-26 14:27:23,094 temboard[18043]: [temboard] DEBUG:     raise TemboardUIError(400, "Group '%s' ('%s') already exists." % (group_name, group_kind))
2016-10-26 14:27:23,095 temboard[18043]: [temboard] DEBUG: TemboardUIError: Group 'test' ('role') already exists.
2016-10-26 14:27:23,095 temboard[18043]: [temboard] ERROR: Group 'test' ('role') already exists.
2016-10-26 14:27:23,096 temboard[18043]: [temboard] INFO: Failed.

Automated alerting system.

As a part of supervision plugin.

Change logo color on the login page

CLI tool to request the agent API.

Decrease on disk space usage to store collected data by supervision plugin.

The current disk space used to store these data is about 10MB / day / postgres cluster. This rate is clearly to high.

agent-install-sources.md : /usr/share/temboard-agent doesn't exist

I've found a small problem while installing from source.

I successfull run $ sudo python setup.py install but then the command below fails:

$ sudo cp /usr/share/temboard-agent/temboard-agent.conf.sample /etc/temboard-agent/temboard-agent.conf

Because /usr/share/temboard-agent/ doesn't exist

I remplaced the command with this one:

$ sudo cp share/temboard-agent.conf.sample /etc/temboard-agent/temboard-agent.conf

And the install process continues just fine

Get rid of glyphicons

[supervision] Error with some graph

When I go to ```/server/0.0.0.0/2345/supervision/week``, the graphs won't load and I get the error belown in the log

2016-10-31 11:14:24,376 temboard[20449]: [temboard] INFO: 500 GET /server/0.0.0.0/2345/supervision/data/waiting_locks?start=2016-10-24T11:14:22&end=2016-10-31T11:14:22 (90.78.76.123) 140.75ms
2016-10-31 11:14:24,381 temboard[20449]: [temboard] DEBUG: Traceback (most recent call last):
2016-10-31 11:14:24,382 temboard[20449]: [temboard] DEBUG:   File "/usr/local/lib/python2.7/dist-packages/temboard-0.0.1-py2.7.egg/temboardui/plugins/supervision/__init__.py", line 246, in get_data_probe
2016-10-31 11:14:24,383 temboard[20449]: [temboard] DEBUG:     data = get_locks(self.db_session, instance.hostname, instance.pg_port, start_time, end_time)
2016-10-31 11:14:24,383 temboard[20449]: [temboard] DEBUG:   File "/usr/local/lib/python2.7/dist-packages/temboard-0.0.1-py2.7.egg/temboardui/plugins/supervision/chartdata.py", line 208, in get_locks
2016-10-31 11:14:24,384 temboard[20449]: [temboard] DEBUG:     cur.copy_expert(query % (tablename, hostname, port, start.strftime('%Y-%m-%dT%H:%M:%S'), end.strftime('%Y-%m-%dT%H:%M:%S')), data_buffer)
2016-10-31 11:14:24,384 temboard[20449]: [temboard] DEBUG: ProgrammingError: relation "supervision.metric_locks_10m" does not exist
2016-10-31 11:14:24,385 temboard[20449]: [temboard] DEBUG: LINE 1: ...) AS access_exclusive, SUM(siread) AS siread FROM supervisio...
2016-10-31 11:14:24,386 temboard[20449]: [temboard] DEBUG:                                                              ^
2016-10-31 11:14:24,387 temboard[20449]: [temboard] DEBUG: 
2016-10-31 11:14:24,387 temboard[20449]: [temboard] ERROR: relation "supervision.metric_locks_10m" does not exist
LINE 1: ...) AS access_exclusive, SUM(siread) AS siread FROM supervisio...
                                                             ^

2016-10-31 11:14:24,390 temboard[20449]: [temboard] INFO: 500 GET /server/0.0.0.0/2345/supervision/data/locks?start=2016-10-24T11:14:22&end=2016-10-31T11:14:22 (90.78.76.123) 174.98ms

The supervision.sql is correctly loaded

temboard=# \d
                       List of relations
   Schema    |             Name              | Type  |  Owner   
-------------+-------------------------------+-------+----------
 application | access_role_instance          | table | temboard
 application | groups                        | table | temboard
 application | instance_groups               | table | temboard
 application | plugins                       | table | temboard
 application | role_groups                   | table | temboard
 application | roles                         | table | temboard
 supervision | hosts                         | table | temboard
 supervision | instances                     | table | temboard
 supervision | metric_bgwriter               | table | temboard
 supervision | metric_blocks                 | table | temboard
 supervision | metric_cpu                    | table | temboard
 supervision | metric_db_size                | table | temboard
 supervision | metric_filesystems_size       | table | temboard
 supervision | metric_loadavg                | table | temboard
 supervision | metric_locks                  | table | temboard
 supervision | metric_memory                 | table | temboard
 supervision | metric_process                | table | temboard
 supervision | metric_replication            | table | temboard
 supervision | metric_sessions               | table | temboard
 supervision | metric_tblspc_size            | table | temboard
 supervision | metric_temp_files_size_db     | table | temboard
 supervision | metric_temp_files_size_tblspc | table | temboard
 supervision | metric_vacuum_analyze         | table | temboard
 supervision | metric_wal_files              | table | temboard
 supervision | metric_xacts                  | table | temboard
(25 rows)

temboard-agent-0.0.1-9.el7.centos
PostgreSQL 9.6

When will support PostgreSQL 9.6?

Agent log:

2017-01-19 12:27:10,995 temboard-agent[29002]: [supervision_collector_worker] INFO: Loaded probe: memory
2017-01-19 12:27:11,018 temboard-agent[29002]: [supervision_collector_worker] ERROR: Unable to run probe "sessions" on "main" on database "postgres"

PG log:

< 2017-01-19 12:27:11.018 MSK > ERROR:  column "waiting" does not exist at character 105
< 2017-01-19 12:27:11.018 MSK > STATEMENT:  select
          current_timestamp as datetime,
          d.datname as dbname,
          coalesce(sum((state = 'active' and not waiting)::integer), 0) as active,
          coalesce(sum((state = 'active' and waiting)::integer), 0) as waiting,
          coalesce(sum((state = 'idle')::integer), 0) as idle,
          coalesce(sum((state = 'idle in transaction')::integer), 0) as idle_in_xact,
          coalesce(sum((state = 'idle in transaction (aborted)')::integer), 0) as idle_in_xact_aborted,
          coalesce(sum((state = 'fastpath function call')::integer), 0) as fastpath,
          coalesce(sum((state = 'disabled')::integer), 0) as disabled,
          coalesce(sum((query = '<insufficient privilege>')::integer), 0) as no_priv
        from pg_database d
          left join pg_stat_activity a on (d.oid = a.datid)
        where d.datallowconn
        group by d.datname

Regards,