Hi,
I'm using the below command to start MySQL Router exporter, unable to find the metrics mentioned as flags except for metadata_status.

cmd: /usr/bin/mysqlrouter_exporter --url=http://localhost:8443/ --user=user --pass=pass -k --collect-interval=5 --collect.metadata.status --collect.route.connections.byte_from_server --collect.route.connections.byte_to_server --collect.route.connections.time_started --collect.route.connections.time_connected_to_server --collect.route.connections.time_last_sent_to_server --collect.route.connections.time_received_from_server

mysqlrouter_exporter version: 4.0.5
Please help if any wrong usage of command.

Hi, Thanks for making this exporter.

I have difficulties in connecting mysqlrouter.

I am trying to using ssl certification like below.
But I don't know how to configure mysqlrouter_exporter service file when using ssl certification.
when using ssl certification in mysqlrouter, how should i setup starting flag of exporter or environment variables?
It looks like there is no pem file environment variables (only URL, PASS, USER variables exists)

• mysqlrouter.conf
  [http_server]
  port=11002
  ssl=1
  ssl_cert=/var/lib/mysqlrouter/router-cert.pem
  ssl_key=/var/lib/mysqlrouter/router-key.pem

• [Service] in mysqlrouter_exporter.service file
  [Service]
  Type=simple
  Restart=always
  RestartSec=15
  Environment="MYSQLROUTER_EXPORTER_URL=http://127.0.0.1:10001\"
  Environment="MYSQLROUTER_EXPORTER_USER=histudy"
  Environment="MYSQLROUTER_EXPORTER_PASS=dailystudy2*"
  ExecStart=/opt/mysqlrouter_exporter/mysqlrouter_exporter \
  --listen-port=10102 \
  --collect.metadata.status \
  --collect.route.connections.byte_from_server \
  --collect.route.connections.byte_to_server \
  --collect.route.connections.time_started \
  --collect.route.connections.time_connected_to_server \
  --collect.route.connections.time_received_from_server

This issue is related to the time_wait issue that resulted in the implementation of graceful shutdowns.

I am still noticing connections in TIME_WAIT when I shutdown mysqlrouter-exporter:

root@mysql-router-k8s-0:/# netstat -nat  | grep 49152                                                                                                                                                              
tcp        0      0 10.1.241.240:49152      10.1.241.239:3306       TIME_WAIT                                                                                                                                      
root@mysql-router-k8s-0:/# netstat -ap | grep 49152                                                                                                                                                                
tcp        0      0 mysql-router-k8s-:49152 mysql-k8s-0.mysql-:3306 TIME_WAIT   

The address 10.1.241.239:3306 is for the mysql pod that I have in my deployment:

Unit                         Workload  Agent  Address       Ports  Message    
mysql-k8s/0*                 active    idle   10.1.241.239         Primary    
mysql-router-k8s/0*          active    idle   10.1.241.240      
mysql-test-app/0*            active    idle   10.1.241.193      
self-signed-certificates/0*  active    idle   10.1.241.247      

My question would be where in the router-exporter code do we connect with MySQL? Or is there a way MySQL can connect to the router exporter? If so, how? I have traced all the code in this repo as well as mysqlrouter-go repo, but fail to see connections with MySQL

Some additional context: mysql-router-k8s has an integration with mysql-k8s. mysql-router-k8s is using a self-signed TLS certificate

Hi
I'm having trouble running mysql router exporter
mysqlrouter uses certificates, so I tried using --skip-tls-verify like this :

mysqlrouter_exporter.service :

[Service]
Type=simple
Environment="MYSQLROUTER_EXPORTER_URL=http://10.10.10.80:8443"
ExecStart=/usr/local/bin/mysqlrouter_exporter
--skip-tls-verify

mysqlrouter.conf :

[http_server]
port=8443
ssl=1
ssl_cert=/var/lib/mysqlrouter/router-cert.pem
ssl_key=/var/lib/mysqlrouter/router-key.pem

When I restart there is a problem like this, systemctl status mysqlrouter_exporter -l :

● mysqlrouter_exporter.service - mysqlrouter-exporter
   Loaded: loaded (/etc/systemd/system/mysqlrouter_exporter.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2022-12-05 17:48:57 WIB; 9s ago
     Docs: https://github.com/rluisr/mysqlrouter-exporter
  Process: 80964 ExecStart=/usr/local/bin/mysqlrouter_exporter --skip-tls-verify (code=exited, status=1/FAILURE)
 Main PID: 80964 (code=exited, status=1/FAILURE)

Dec 05 17:48:56 mysql-router-test2 systemd[1]: Started mysqlrouter-exporter.
Dec 05 17:48:57 mysql-router-test2 mysqlrouter_exporter[80964]: 2022/12/05 17:48:57 failed to create mysql router client. err: Get "http://10.10.10.80:8443/api/20190715/swagger.json": EOF
Dec 05 17:48:57 mysql-router-test2 systemd[1]: mysqlrouter_exporter.service: main process exited, code=exited, status=1/FAILURE
Dec 05 17:48:57 mysql-router-test2 systemd[1]: Unit mysqlrouter_exporter.service entered failed state.
Dec 05 17:48:57 mysql-router-test2 systemd[1]: mysqlrouter_exporter.service failed.

This is w.r.t to fix provided in previous PR #9 :

i m not able to see version i m using, i know i have downloaded latest release 2.23

tmp$ ./mysqlrouter_exporter -v
the required flag --url' was not specified tmp$ ./mysqlrouter_exporter --version the required flag --url' was not specified

tmp$./mysqlrouter_exporter
the required flag `--url' was not specified

without any flag it should show help section and for -v | --version flag it must show version

Issue

The exporter starts a server on listen-port using http.ListenAndServe. However, there is no attempt at a graceful shutdown when a SIGTERM is received (see this post for some ideas of how a SIGTERM can be reacted with a server.Shutdown())

We have an issue in our project where we use this project (mysqlrouter_exporter). As the trace in the issue indicates, we run into bind address in use error intermittently when we start exporter, stop it and then restart it. Further digging revealed that when we stop exporter, sometimes, due to an ungraceful shutdown, we have connections in TIME_WAIT:

root@mysql-router-k8s-0:/# netstat -nat  | grep 49152
tcp        0      0 10.1.241.229:49152      10.1.241.230:3306       TIME_WAIT

It is exactly under these conditions that we run into the bind address in use error. When the

Suggested Solution

To gracefully shutdown the server as indicated by the above linked post when a SIGTERM is received

Potential followups

I will attempt to create a PR with a proposed solution for this issue

Hi,
We need new flag --skip="metrics list separated by comma that are to be skippped"
e.g --skip=metrics1,metrics2,metrics3

There are some metrics which are creating lot of noise and consuming lot of system resources such as CPU,
so we wish to scrap out all those metrics hence a skip flag.

Hello,
I've installed the mysqlrouter_exporter on a router. It starts but I get the following error 202 got 404:
sudo systemctl status mysqlrouter_exporter.service

● mysqlrouter_exporter.service - mysqlrouter-exporter
     Loaded: loaded (/etc/systemd/system/mysqlrouter_exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-12-15 16:59:52 PST; 3s ago
       Docs: https://github.com/rluisr/mysqlrouter-exporter
   Main PID: 13100 (mysqlrouter_exp)
      Tasks: 4 (limit: 6970)
     Memory: 1.9M
        CPU: 58ms
     CGroup: /system.slice/mysqlrouter_exporter.service
             └─13100 /usr/local/bin/mysqlrouter_exporter --skip-tls-verify

Dec 15 16:59:52 ia-se-vumr2-112 systemd[1]: Started mysqlrouter-exporter.
Dec 15 16:59:52 ia-se-vumr2-112 mysqlrouter_exporter[13100]: 2023/12/15 16:59:52 Start exporter on 0.0.0.0:49152/metrics
Dec 15 16:59:52 ia-se-vumr2-112 mysqlrouter_exporter[13100]: [mysqlrouter_exporter ERROR] response status code is not 200 got 404
Dec 15 16:59:55 ia-se-vumr2-112 mysqlrouter_exporter[13100]: [mysqlrouter_exporter ERROR] response status code is not 200 got 404

Here's the config file:

[Unit]
Description=mysqlrouter-exporter
Documentation=https://github.com/rluisr/mysqlrouter-exporter
After=network-online.target

[Service]
Type=simple
Environment="MYSQLROUTER_EXPORTER_URL=https://10.10.10.10:8443"
Environment="MYSQLROUTER_EXPORTER_USER=user"
Environment="MYSQLROUTER_EXPORTER_PASS=password"
Environment="MYSQLROUTER_EXPORTER_SERVICE_NAME=mysqlrouter-test-mr2"

ExecStart=/usr/local/bin/mysqlrouter_exporter --skip-tls-verify

[Install]
WantedBy=multi-user.target

Am I missing something or something slightly off on the configuration?
Thanks!

Not seeing some of the metrics that the graph is looking for:

mysqlrouter_route_connections_byte_from_server

mysqlrouter_route_connections_byte_to_server

Hi,
We are again hitting the same issue of HIGH CPU & Memory utilisation
0efd986164:~$ grep mysqlrouter_metadata_status /tmp/myrtr | wc -l
575598

Over the time, this metric is accumulating and there is gradual increase in cpu, so we want to cut down metric router_metadata_status.

so can we just have 1 skip flag:
--skip=
so in this case we don't need 3 to 4 skip flag and in turn reduces the dev effort of creating more flags
NOTE:
This is nothing to do with HIGH traffic coz rest of prometheus exporter are not showing so much high CPU utilisation etc.
we are having docker image of mysqlrouter_exporter running on kubernetes POD
we have given max resources to this mysqlrouter_exporter POD so that it doesn't run out of capacity while collecting metrics nor it restarts at any moment.
below is the configuration of running mysqlrouter_exporter:

/opt/mysqlrouter_exporter --skip-tls-verify \
                          --skip.collect.route.connections.byte_from_server \
			  --skip.collect.route.connections.byte_to_server \
			  --skip.collect.route.connections.time_started \
			  --skip.collect.route.connections.time_connected_to_server \
                          --skip.collect.route.connections.time_last_sent_to_server \
			  --skip.collect.route.connections.time_received_from_server \
			  --listen-port=8444

so instead can we just have 1 flag
--skip="metrics1,metrics2,metrics3 etc"
mysql-router_exporter will stop collecting these all metrics

Since we are monitoring each and every k8s pod we are always alerted w.r.t high cpu utilisation by this mysqlrouter_exporter POD.
Let me know if anything else is more needed from my end or any workaround you can provide.

Thank you.

The exporter I used is the latest version 4.0.5
mysqlrouter version: MySQL Enterprise - Commercial 8.0.34

When I stopped mysqlrouter, still get all the route are health.
Please help to look into this issue.

Thank you.

Hi, I'm facing a problem. After I started using mysql router exporter few weeks ago (used as sidecar container of mysqlrouter pod on k8s), mysql router was crashed some times with this stacktrace (it happend after 7-10 days running):
I was wondering it happend because of mysql router exporter or not. I tried with both 8.0.32 and 8.0.33 mysql router versions.

Application got fatal signal: 11
stack_bottom = 0 thread_stack 0x0
/usr/lib64/mysqlrouter/private/libmysqlharness.so.1(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x7f147816bd41]
/usr/lib64/mysqlrouter/private/libmysqlharness.so.1(+0xd87f1) [0x7f14781567f1]
/lib64/libpthread.so.0(+0x12d00) [0x7f1477d8cd00]
/usr/lib64/mysqlrouter/private/libmysqlrouter_routing.so.1(MysqlRoutingClassicConnection::get_server_address[abi:cxx11]() const+0x42) [0x7f1474a85882]
/usr/lib64/mysqlrouter/private/libmysqlrouter_routing.so.1(void concurrent_map<MySQLRoutingConnectionBase*, std::shared_ptr<MySQLRoutingConnectionBase>, std::hash<MySQLRoutingConnectionBase*> >::for_each<ConnectionContainer::get_all_connections_info()::{lambda(std::pair<MySQLRoutingConnectionBase* const, std::shared_ptr<MySQLRoutingConnectionBase> > const&)#1}>(ConnectionContainer::get_all_connections_info()::{lambda(std::pair<MySQLRoutingConnectionBase* const, std::shared_ptr<MySQLRoutingConnectionBase> >/usr/lib64/mysqlrouter/private/libmysqlrouter_routing.so.1(MySQLRouting::get_connections()+0x38) [0x7f1474a5e6e8]
/usr/lib64/mysqlrouter/private/libmysqlrouter_routing.so.1(MySQLRoutingAPI::get_connections() const+0x29) [0x7f1474a807e9]
/usr/lib64/mysqlrouter/rest_routing.so(RestRoutingConnections::on_handle_request(HttpRequest&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)+0x1de) [0x7f1474b974ae]
/usr/lib64/mysqlrouter/rest_api.so(RestApi::handle_paths(HttpRequest&)+0x729) [0x7f1474c14379]
/usr/lib64/mysqlrouter/http_server.so(HttpRequestRouter::route(HttpRequest)+0x16e) [0x7f1474c8cb2e]
/usr/lib64/mysqlrouter/http_server.so(+0x37e0c) [0x7f1474c8ce0c]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0xd6b5) [0x7f14780056b5]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x3076c) [0x7f147802876c]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x18b75) [0x7f1478010b75]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x34e5d) [0x7f147802ce5d]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x34e86) [0x7f147802ce86]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x1ff42) [0x7f1478017f42]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x20767) [0x7f1478018767]
/usr/lib64/mysqlrouter/http_server.so(HttpRequestThread::wait_and_dispatch()+0x35) [0x7f1474c7b5d5]
/lib64/libstdc++.so.6(+0xc2ba3) [0x7f1476514ba3]
/lib64/libpthread.so.0(+0x81da) [0x7f1477d821da]
/lib64/libc.so.6(clone+0x43) [0x7f1475b2ae73]

We are using mysqlrouter_exporter
version: 2.2.1 commit: eb1138d date: 2020-08-20T05:11:26Z

we are seeing HIGH amount of CPU usage when mysql-router_exporer is scraping the metrics

I can see 37,418 lines of this particular metric
cat route_metric.txt | grep route_connections_time_started -c
37418
E.g:
route_connections_time_started{destination_address="mysql-host1:3306",name="some_cluster",router_hostname="mysql-router-7465d4cb76-xmz4s",source_address="10.10.10.20:60678"} 1.614528416e+12

i don't want this particular metric at all
how do i scrap out this metric

sh-4.2# ./mysqlrouter_exporter --help
--port Listen port. Default 49152
--version Show version
--skip-tls-verify Skip TLS Verification

Need some more flags to scrap in the metrics i need
just like how mysqld_exporter has it -> https://github.com/prometheus/mysqld_exporter

how should look docker compose ?

  mysql-router:
    image: container-registry.oracle.com/mysql/community-router:8.0.36
    hostname: "mysql-router{{.Task.Slot}}"
    environment:
      MYSQL_HOST: mysql1
      MYSQL_PORT: 3306
      MYSQL_USER: ${DB_ROOT_USER}
      MYSQL_PASSWORD: ${DB_ROOT_PASSWORD}
      MYSQL_INNODB_CLUSTER_MEMBERS: 2
      MYSQL_ROUTER_BOOTSTRAP_EXTRA_OPTIONS: --conf-set-option=DEFAULT.max_connections=2048 --conf-set-option=DEFAULT.max_total_connections=2048

  mysqlrouter-exporter:
    image: ghcr.io/rluisr/mysqlrouter_exporter:v5.0.0
    hostname: "mysqlrouter-exporter{{.Task.Slot}}"
    command: --service-name=mysql-router --skip-tls-verify
    environment:
      MYSQLROUTER_EXPORTER_URL: https://mysql-router{{.Task.Slot}}:8443

1 error
the required flag `--service-name' was not specified
added --service-name=mysql-router
(wasn't mentioned that MYSQLROUTER_EXPORTER_SERVICE_NAME is mandatory in readme)

2 error
failed to create mysql router client. err: Get "https://mysql-router1:8443/api/20190715/swagger.json": tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match mysql-router1
added --skip-tls-verify

3 error
Start exporter on 0.0.0.0:49152/metrics
[mysqlrouter_exporter ERROR] response status code is not 200 got 401

what does it want ?
curl -s -k https://mysql-router1:8443/api/20190715/swagger.json works
could be issue with too new version 8.0.36 ?

If there is demand, I will work on this.
If you can commit, you can create a PR.

Hi, I'm using mysqlrouter 8.0.32 and newest router exporter image and I have a problem
When I turn off mysql backend servers, then the API of the router reports correctly
curl -k -u username:password https://192.168.1.10:8443/api/20190715/routes/bootstrap_rw/health {"isAlive":false}
But the exporter still reports 1 (active)
mysqlrouter_route_health{name="bootstrap_rw",router_hostname="mysqlrouter-ABCXYZ"} 1

My exporter args:
/app -k -p 9105 --collect.metadata.status "true" --collect.route.connections.byte_from_server "true" --collect.route.connections.byte_to_server "true" --collect.route.connections.time_started "true" --collect.route.connections.time_connected_to_server "true" --collect.route.connections.time_last_sent_to_server "true" --collect.route.connections.time_received_from_server "true"
Please take a look

Continue from this topic: #28 (I still got the problem after applying new image)
I dig a little bit deeper and found another problem:
After changing mysql innodb cluster from online => offline

• exporter has alot of logs:

[mysqlrouter_exporter ERROR] response status code is not 200 got 500
[mysqlrouter_exporter ERROR] response status code is not 200 got 500
[mysqlrouter_exporter ERROR] response status code is not 200 got 500

• I added a nginx pod between router-exporter and router-api and I can see the url

[10/May/2023:09:14:37 +0000] localhost 127.0.0.1 "GET /api/20190715/routes/bootstrap_ro/health HTTP/1.1" 500 - "-" "Go-http-client/1.1" "-" "-" "-" "127.0.0.1:8443" 500 0.052 197 195 0.052
[10/May/2023:09:14:39 +0000] localhost 127.0.0.1 "GET /api/20190715/routes/bootstrap_ro/health HTTP/1.1" 500 - "-" "Go-http-client/1.1" "-" "-" "-" "127.0.0.1:8443" 500 0.044 197 195 0.047
[10/May/2023:09:14:41 +0000] localhost 127.0.0.1 "GET /api/20190715/routes/bootstrap_ro/health HTTP/1.1" 500 - "-" "Go-http-client/1.1" "-" "-" "-" "127.0.0.1:8443" 500 0.048 197 195 0.048

• I curl directly to API URL and found that API Still reponse {"isAlive":false} but with status code 500

curl -k -u user:password https://192.168.1.10:8443/api/20190715/routes/bootstrap_ro/health -vvv
*   Trying 192.168.1.10:8443...
* Connected to 192.168.1.10 (192.168.1.10) port 8443 (#0)
....
....
< HTTP/1.1 500 Internal Error
< Content-Type: application/json
< Date: Wed, 10 May 2023 09:20:45 GMT
< Content-Length: 17
< 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host 192.198.1.10 left intact
{"isAlive":false}

• I think the problem from this code: https://github.com/rluisr/mysqlrouter-go/blob/df2dd1c6513c172d0dddfb6835ace54ba2637795/request.go#L34
• Now we need to accept 500 return code to get result, not just 200
  Please take a look.

Hello,
how can I skip TLS verification for mysqlrouter_exporter chart

I m using SELF Signed CERTs on my mysqlrouter

Aug 19 08:29:40 p35-bastion.someorg.com mysqlrouter_exporter[10180]: 2020/08/19 08:29:40 failed to create mysql router client. err: Get "https://p35-bastion.someorg.com:8443/api/20190715/swagger.json": x509: certificate signed by unknown authority

i don't see any place to add 'insecure_skip_verify' flag for this exporter service

mysqlrouter_exporter --help
--port        Listen port. Default 49152
--version     Show version