Very slow publishing data more 100 MB. about aiomqtt HOT 14 CLOSED

OK. I will do an MR a little later)

from aiomqtt.

Hi)
I opened the MR) Look please)

from aiomqtt.

Hi again. :) Thanks for raising this issue. Let me have a look.

I've never tested with packets of that size to be honest. The specification let's the messages be as large as 256 MiB but people usually send much smaller messages (well under 1 MiB).

In any case, let's rule out some potential bottlenecks first.

1. What kind of transfer speeds do you get with alternative MQTT clients on the same network?
2. Is the server and client on the same machine? (that would be good for network traffic)
3. Is the machine a recent desktop/laptop computer? (and not, say, a Raspberry Pi or something)

I'm particularly interested in (1). It'll give us a rough estimate of what to expect in therms of transfer speeds of large messages.

Any idea why this might be the case? Could it be the small size of the TCP allocated buffer?

That's a good guess. 👍 As a quick test, you can try to increase this to 20480. Let me know what you figure out. :)

from aiomqtt.

By the way, the only reason that SNDBUF is 2048 is because I got that setting from the loop_asyncio.py example in paho.mqtt. I don't know why they change it from the default.

from aiomqtt.

The SNDBUF default value on my system is 212992! That's 104 times bigger than 2048. Maybe we shouldn't change it from the default.

from aiomqtt.

I tried increasing the buffer (20048) and it really increased the transfer rate. Now my file with TLS enabled was transferred in less than 3 minutes. Previously, I generally went into a timeout.

And, my answers:
1.For example, through a mosquito_pub - instantly transmitted (though without TLS)

it is a pity that there are no timestamps in the logs ((
2. Yes, this is a localhost,
3. This is a desktop (x86)

from aiomqtt.

Another local test. I increased the buffer to 1 MB and the file was transferred (with TLS) to the broker - instantly.

from aiomqtt.

Can we move the buffer setting to the init or, mb, some setter?

from aiomqtt.

Thanks for doing the tests. 👍 Seems to me like the best approach is to simply remove the setting altogether and just use the default value. It was a mistake on my part to change this setting to begin with (I just copied the line from the paho.mqtt example code). 😓

Do you want to do a PR? :)

from aiomqtt.

I can remove the _set_client_socket_defaults method from the module client.py and remove string _set_client_socket_defaults(client_socket) from the connect method.

I'm also very interested in why the Paho creators at changing the buffer parameter in the example? Very strange. O_o

from aiomqtt.

You're right, we can remove the entire _set_client_socket_defaults method altogether. Nice. 👍 I'm ready to merge in a PR. :)

I'm also very interested in why the Paho creators at changing the buffer parameter in the example. Very strange. O_o

To be fair, the usual MQTT message size is very small. It's a reasonable optimization if you can make such assumptions. Still, it shouldn't go into the example code like that. Very strange indeed.

from aiomqtt.

Yes, guys could make a comment on this.
Ok, but if this is an optimization, is it definitely worth removing this method? Maybe it's better to set the size to init ? And set 2048 by default.

It is true that usually big data is not transmitted via mqtt, but there is an exception)

from aiomqtt.

Good point. :) If you want to, you can add a keyword argument to Client.__init__ like socket_options, that is a list/dict/something with various socket options. This way, our users can supply any socket option that they want to use. 👍

In any case, I think it's a good idea to remove the hard-coded 2048 value that we currently have.

from aiomqtt.

I merged your pull request. Thanks again! :)

from aiomqtt.