Internal server error 500 after upgrading from 5.3.0 to 5.3.2 about emqx HOT 7 CLOSED

Here comes logs but data/configs/cluster.hocon file is not present, added data/configs/cluster-override.conf instead.
emqx.zip

NOTE: After upgrade and collect logs, when running with 5.3.2, I've disabled the file log and all mqtt authentication, and when back to 5.3.1, emqx crashes. I'm now on 5.3.2 because I can't go back. I suppose that the changes on configuration I've made, makes new configuration incompatible with 5.3.1 and causes the crash. Is possible go back to 5.3.1?

from emqx.

I can confirm that the problem is with 5.3.2, version 5.3.1 works fine.

from emqx.

Hi.

1. Are there any logs you could share? If they could be set to debug before trying to upgrade, that would be best.
2. Could you share data/configs/cluster.hocon? Please double check and censor any sensitive information before sharing it.

from emqx.

Thanks for the extra info.

I managed to reproduce your issue by starting on 5.3.0, creating an HTTP bridge, and then using the same config on 5.3.2. That produces the following log when starting EMQX:

2023-12-19T16:48:11.708766+00:00 [error] msg: action_references_nonexistent_bridges, mfa: emqx_rule_engine:with_parsed_rule/3(490), hint: this rule will be disabled, nonexistent_bridge_ids: #{nonexistent_bridge_ids => [{http,p1}]}, rule_id: <<"web">>

However, I could not observe the same issue when trying to access the bridges API: the bridge is return correctly when listing all bridges or getting its config.

Did you observe the error while trying to list the bridges in a "mixed cluster"? That is: a cluster with some nodes in 5.3.0, some in 5.3.2? That could explain how you experience it.

This particular error should be resolved in 5.4.0 (still unreleased), and considering the whole cluster is running the same (new) version.

I suppose that the changes on configuration I've made, makes new configuration incompatible with 5.3.1 and causes the crash. Is possible go back to 5.3.1?

Automatically downgrading the configuration file is not supported, I'm afraid. If you changed something in 5.3.2, then it probably saved the new state in the new format. You would need to inspect the logs when EMQX crashes and see which configurations it complains about, and then try to manually fix the config before starting it in an older EMQX version.

from emqx.

Did you observe the error while trying to list the bridges in a "mixed cluster"? That is: a cluster with some nodes in 5.3.0, some in 5.3.2? That could explain how you experience it.

No, I only have one node, but in /mnesia folder, I got many [email protected] folders created during my tests and with older versions of emqx, and I don't know if this can cause any issue, or I can simply delete the unused.

I suppose that the changes on configuration I've made, makes new configuration incompatible with 5.3.1 and causes the crash. Is possible go back to 5.3.1?

Automatically downgrading the configuration file is not supported, I'm afraid. If you changed something in 5.3.2, then it probably saved the new state in the new format. You would need to inspect the logs when EMQX crashes and see which configurations it complains about, and then try to manually fix the config before starting it in an older EMQX version.

Ok, no problem I only want to know if there is an easy way, I have backups (emqx ctl data export).
An easy, and fast, way to restore previous configurations for downgrades after "failed" updates, will be appreciated. In the config folder there are many cluster-override.conf.xxx.bak (dated) files, but I don't know If I can use this files as substitution of my config or need someting more...

from emqx.

No, I only have one node, but in /mnesia folder, I got many [email protected] folders created during my tests and with older versions of emqx, and I don't know if this can cause any issue, or I can simply delete the unused.

Those shouldn't affect your single node. If you are sure the folders are not used by your node (i.e., they don't match the name of such single node or any other in the cluster), then you can remove them.

Ok, no problem I only want to know if there is an easy way, I have backups (emqx ctl data export).
An easy, and fast, way to restore previous configurations for downgrades after "failed" updates, will be appreciated. In the config folder there are many cluster-override.conf.xxx.bak (dated) files, but I don't know If I can use this files as substitution of my config or need someting more...

Both methods should work (using an older .bak file that was created with the older version, or importing a file generated by emqx ctl data export).

from emqx.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

from emqx.