Comments (7)
Update - Didn't even think about the fact that when running replicants, MQTT requests hit the replicants, not the core :)
Adding the same env var to the replicantsTemplate seems to have resolved my problem. Appreciate your support @zmstone!
---
apiVersion: apps.emqx.io/v2beta1
kind: EMQX
metadata:
name: emqx
spec:
image: emqx:5.5.0
coreTemplate:
spec:
replicas: 3
env:
- name: EMQX_MQTT__MAX_PACKET_SIZE
value: 256MB
bootstrapAPIKeys:
- key: some_api_key
secret: some_api_secret
replicantTemplate:
spec:
replicas: 3
env:
- name: EMQX_MQTT__MAX_PACKET_SIZE
value: 256MB
listenersServiceTemplate:
spec:
type: LoadBalancer
loadBalancerIP: 10.1.50.13
dashboardServiceTemplate:
spec:
type: LoadBalancer
loadBalancerIP: 10.1.50.14
from emqx.
To inspect the config value in effect, you can execute this command:
$ emqx eval 'emqx_config:get_zone_conf(default, [mqtt, max_packet_size])'
268435456
If it's indeed 256MB in your case, it's likely that the client has sent some bytes which were parsed into a frame length greater than this.
from emqx.
If it's indeed 256MB in your case, it's likely that the client has sent some bytes which were parsed into a frame length greater than this.
emqx@emqx-core-bfd48bd6c-0:/opt/emqx$ emqx eval 'emqx_config:get_zone_conf(default, [mqtt, max_packet_size])'
268435456
Thanks for your reply, @zmstone. Indeed it is set properly, it seems. Is there a trace log that will show the payload size? From what I can tell, the client is simply sending some JSON (which is not nearly 256MB in size). It's also strange that I can fix the issue with 100% success rate simply by pressing "save" in the EMQX UI.
from emqx.
Did some more digging to find out if the message was indeed going rogue and sending 256MB+
Packet capture taken on the client side is below.
Client: 10.42.2.183
EMQX: 10.43.247.45
The summary is that no single packet is larger than a couple dozen Kilobytes (Largest single packert is roughly 20KB). Entire TCP conversation is only 100-200KB total.
Thoughts on how I can debug further?
from emqx.
The trace logs do not include the actual length received (this is something we can improve, we will work on a fix).
Could you share the pcap file so we can have a closer look?
Another question, are you running EMQX nodes in a cluster?
The changes made from UI applies to all nodes in the cluster, but environment variable is not shared.
You mentioned that you are setting EMQX_MQTT__MAX_PACKET_SIZE
environment variable, but also changing settings from the UI.
from emqx.
I deploy EMQX in a clustered environment via the emqx-operator
:
---
apiVersion: apps.emqx.io/v2beta1
kind: EMQX
metadata:
name: emqx
spec:
image: emqx:5.5.0
coreTemplate:
spec:
replicas: 3
env:
- name: EMQX_MQTT__MAX_PACKET_SIZE
value: 256MB
Does this deploy the env var to all nodes (I assume so, but sanity check)? Is there a better way to set this via the operator?
Pcap attached.
from emqx.
Thanks @SlothCroissant
I have sent a logging enhancement here: #12530 to help troubleshooting.
from emqx.
Related Issues (20)
- can't get real ipaddress of clients HOT 4
- receive a huge of connect and disconnect events from one client with no reasons HOT 10
- runq_overload alert not cleared sometimes HOT 3
- connection_shutdown with reason: #{hint => invalid_password HOT 3
- MQTT Egress and Ingress to same topic structure causes looping HOT 7
- JWT Payload Fails to pass the verification after the `Claim` and `Expected Value` are configured HOT 4
- emqx4.4.7 Node is already running HOT 9
- esockd_connection_sup Error Interpretation HOT 3
- Shared subscription sticky strategy stops working after a rollout restart of cluster HOT 10
- 设备频繁断开连接,设备断开原因是 bad_frame_header HOT 3
- Broker doesn't send PUBLISH to QUIC client until QUIC client sends keepalive HOT 8
- client is in endless online and offline switching
- 共享订阅客户端主动断开连接后,`emqx`仍然会向离线的客户端推送消息,导致消息顺序错乱 HOT 2
- Unable to login the dashboard HOT 9
- 为什么不提供windows版下载 HOT 2
- JWKS SSL opts in per-listener authentication not applied correctly in EMQX 5.7.1+ HOT 5
- Auto subscribe based on client id pattern HOT 4
- 管理端访问 主题 带 / 的功能时报错 HOT 2
- Emqx manual clustering fail when windows emqx broker attempting to join the linux broker HOT 4
- AWK not found in docker HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from emqx.