Bug Report: 'show vitess_replication_status' always says ReplicationLag is 0 about vitess HOT 3 CLOSED

Hi @wiebeytec,

Can you please provide a test case demonstrating this with the local examples? It should be relatively easy to do given the same basic process you showed here. It's hard to follow what you're seeing as you are mixing various things together here as your first output is for vt_0000000300 and the next for vic-eu-central-1-fakelive-0000000401.

I have been able to repeat one thing that you showed:

{"EventStreamRunning":"No","EventApplierRunning":"No","LastError":""}   | 0

That's because when replication is not running the lag is unknown. MySQL represents that with NULL for Seconds_Behind_Source. As an integer, that becomes the zeroval. I will change that handling to use a string instead so that we also display NULL in this case. The test case for that is:

git checkout main && make build

pushd examples/local 

./101_initial_cluster.sh

../common/scripts/vtorc-down.sh

command mysql -u root --socket=${VTDATAROOT}/vt_0000000102/mysql.sock --binary-as-hex=false -e "stop slave"

mysql < ../common/insert_commerce_data.sql

sleep 5

mysql -e "show vitess_replication_status"

The results::

+----------+-------+------------+------------------+-----------+-------------------+-------------------------------------------------------------------------+----------------+-----------------------------------------+
| Keyspace | Shard | TabletType | Alias            | Hostname  | ReplicationSource | ReplicationHealth                                                       | ReplicationLag | ThrottlerStatus                         |
+----------+-------+------------+------------------+-----------+-------------------+-------------------------------------------------------------------------+----------------+-----------------------------------------+
| commerce | 0     | REPLICA    | zone1-0000000100 | localhost | localhost:17101   | {"EventStreamRunning":"Yes","EventApplierRunning":"Yes","LastError":""} | 0              | {"state":"OK","load":0.00,"message":""} |
| commerce | 0     | RDONLY     | zone1-0000000102 | localhost | localhost:17101   | {"EventStreamRunning":"No","EventApplierRunning":"No","LastError":""}   | 0              | {"state":"OK","load":0.00,"message":""} |
+----------+-------+------------+------------------+-----------+-------------------+-------------------------------------------------------------------------+----------------+-----------------------------------------+

I'll try some more to repeat the behavior you're seeing but a test case would be very helpful. If I cannot repeat it then I will only fix what I was able to repeat.

Thank you!

from vitess.

Hi @wiebeytec ,

When implementing this show vitess_replication_status command I used the tablet's replication lag stats variable (managed within the discover package's tablet healthcheck) instead of the seconds_behind_[master|source] value from mysqld precisely because the stats variable will estimate the lag when replication is not running (this value is used to determine the lag and whether or not the tablet should be serving or not). I've confirmed that this stats variable is no longer working as expected, which we can also see in the tablet's HTTP status page output where it's also used:

❯ curl -s http://localhost:15102/debug/status | grep -A1 "Replication Lag"
    <td>Replication Lag</td>
    <td>0s</td>

I will investigate that underlying issue further and open a PR to fix it.

Thanks again!

from vitess.

My reference to vt_0000000300 was merely to explicitly use a config file name where the skip_slave_start was located , but not specific to one managed mysql. It was to prevent saying something vague like 'the mysql config'. That could have been more clear; true.

Anyway, going by your last comment, all confusion seems cleared up 👍

from vitess.