client or server seems to break after a certain amount of requests about leshan HOT 9 CLOSED

I forgot to add the tool and command I'm using for testing. It's httperf, the command used is:

httperf --server=localhost --port=8080 --uri=/api/clients/liberator/6/0/1?format=TLV --num-conns=1 --rate=1 --num-calls=100000 --timeout=1

The numer 65536 is important, because 65536 = 2^16, so somewhere there a variable limiting the amount of requests possible. Maybe the is a modulo operation missing somewhere, so far I haven't found the source of the problem.

EDIT: I found out that the problem might be caused by californium, and referenced this issue from the issue I created on their repository.

from leshan.

I tested it on my side and get the same results than you.
I tried to test with ab:
ab -c 5 -n 100000 "http://localhost:8081/api/clients/liberator/6/0/1?format=TLV"

I get some request timeout arround 65536 requests, then after some
minutes the requests is accepted again. (I bet this time is CoAP
EXCHANGE_LIFETIME)
So I suspect the "CoAP deduplicator" which is responsible to detect
duplicate message based on MID (16 bytes).
see CoAP spec: https://tools.ietf.org/html/rfc7252#section-4.5

To validate this theory, I tried to run the test with deduplicator by
changing the californium config (Californium.properties).
Setting DEDUPLICATOR=NO_DEDUPLICATOR.

With this config no more timeout. This confirms my supposition.
(This is clearly not recommended to remove deduplicator in production)

from leshan.

@sbernard31 Thank you very much for your feedback. Since you seem to also be a part of the californium team, do you have a suggestion what I can do in this situation to keep the system stable? From what I can tell, there seems to be no way around implementing a custom deduplicator that "never" breaks.

from leshan.

The spec (§2.1) say that :
"The Message ID is compact; its 16-bit size enables up to about 250 messages per second from one endpoint to another with default protocol parameters."

I suppose this covers a lot of real use case especially at client side, If you want more than that maybe coapTCP will be more adapted(No message ID). But the spec is still in draft and I'm not sure there is (mature) implementation for now.

The spec said the protocol limitation is about 250 messages per second from one endpoint to another. This is a protocol limitation. This also means thats you could send 500 messages per seconds to 2 different peer.

If you look at the spec (§4.4), there is several way to implement this. Californium choose the more simple one.
"In the simplest case, a CoAP endpoint generates Message IDs by keeping a single Message ID variable, which is changed each time a new Confirmable or Non-confirmable message is sent, regardless the destination address or port."
So I suppose that Californium is limited to 250 messages per seconds no matter the number of peers. This is an implementation limitation and this will be probably an issue at server side for Leshan.

I made a quick test using httperf (--wlog), 1 leshan server and 2 leshan clients:

httperf --server=localhost --port=8081 --wlog=y,urls.httperf --num-conns=1 --rate=1 --num-calls=100000 --timeout=1 

(see how to create urls.httperf file)
This fails at 65536 requests too. So it seems to confirm the limitation.

from leshan.

See eclipse-californium/californium#57 for additional discussion ...

from leshan.

@sbernard31, @sophokles73

Thank you very much for your help and responses. We are going to discuss this matter internally to figure out how to proceed.

I believe this issue here can be closed, since the observed behaviour is caused by Californium and a result of the requirements of our application, which seem to go beyond the intended purpose of coap.

from leshan.

@Endebert @sbernard31 @sophokles73
Storing standard properties in file Californium.properties
Is this step necessary?
When I was running at Android studio, I found that this file could not be created.
Here is the log from logcat

05-05 15:11:31.280 D/mwqi: Loading OMA standard object models
05-05 15:11:31.397 D/mwqi: min.zhu objectId=0
05-05 15:11:31.398 D/mwqi: min.zhu objectId=1
05-05 15:11:31.398 D/mwqi: min.zhu objectId=3
05-05 15:11:31.398 D/mwqi: min.zhu objectId=5
05-05 15:11:31.398 D/mwqi: min.zhu objectId=6
05-05 15:11:31.400 D/mwqi: min.zhu leshanClientBuilder localAddress=/0.0.0.0:0
05-05 15:11:31.401 D/mwqi: min.zhu leshanClientBuilder localSecureAddress=/0.0.0.0:0
05-05 15:11:31.401 D/mwqi: min.zhu leshanClient endpoint=min.zhuand localAddress=/0.0.0.0:0and localSecureAddress=/0.0.0.0:0
05-05 15:11:31.401 D/mwqi: min.zhu leshanClient Create CoAP non secure endpoint localAddress=/0.0.0.0:0
05-05 15:11:31.404 I/NetworkConfig: Storing standard properties in file Californium.properties
05-05 15:11:31.407 W/NetworkConfig: Error while storing properties to /Californium.properties
java.io.FileNotFoundException: Californium.properties: open failed: EROFS (Read-only file system)
at libcore.io.IoBridge.open(IoBridge.java:456)
at java.io.FileOutputStream.(FileOutputStream.java:87)
at java.io.FileOutputStream.(FileOutputStream.java:72)
at java.io.FileWriter.(FileWriter.java:42)
at org.eclipse.californium.core.network.config.NetworkConfig.store(NetworkConfig.java:218)
at org.eclipse.californium.core.network.config.NetworkConfig.store(NetworkConfig.java:205)
at org.eclipse.californium.core.network.config.NetworkConfig.createStandardWithFile(NetworkConfig.java:170)
at org.eclipse.californium.core.network.config.NetworkConfig.getStandard(NetworkConfig.java:125)
at org.eclipse.californium.core.network.CoapEndpoint.(CoapEndpoint.java:166)
at org.eclipse.leshan.client.californium.LeshanClient.(LeshanClient.java:87)
at org.eclipse.leshan.client.californium.LeshanClientBuilder.build(LeshanClientBuilder.java:120)

        at android.view.View.performClick(View.java:4780)
        at android.view.View$PerformClick.run(View.java:19866)
        at android.os.Handler.handleCallback(Handler.java:739)
        at android.os.Handler.dispatchMessage(Handler.java:95)
        at android.os.Looper.loop(Looper.java:135)
        at android.app.ActivityThread.main(ActivityThread.java:5254)
        at java.lang.reflect.Method.invoke(Native Method)
        at java.lang.reflect.Method.invoke(Method.java:372)
        at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:903)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:698)

Caused by: android.system.ErrnoException: open failed: EROFS (Read-only file system)
at libcore.io.Posix.open(Native Method)
at libcore.io.BlockGuardOs.open(BlockGuardOs.java:186)
at libcore.io.IoBridge.open(IoBridge.java:442)
at java.io.FileOutputStream.(FileOutputStream.java:87) 
at java.io.FileOutputStream.(FileOutputStream.java:72) 
at java.io.FileWriter.(FileWriter.java:42) 
at org.eclipse.californium.core.network.config.NetworkConfig.store(NetworkConfig.java:218) 
at org.eclipse.californium.core.network.config.NetworkConfig.store(NetworkConfig.java:205) 
at org.eclipse.californium.core.network.config.NetworkConfig.createStandardWithFile(NetworkConfig.java:170) 
at org.eclipse.californium.core.network.config.NetworkConfig.getStandard(NetworkConfig.java:125) 
at org.eclipse.californium.core.network.CoapEndpoint.(CoapEndpoint.java:166) 
at org.eclipse.leshan.client.californium.LeshanClient.(LeshanClient.java:87) 
at org.eclipse.leshan.client.californium.LeshanClientBuilder.build(LeshanClientBuilder.java:120) 

        at android.view.View.performClick(View.java:4780) 
        at android.view.View$PerformClick.run(View.java:19866) 
        at android.os.Handler.handleCallback(Handler.java:739) 
        at android.os.Handler.dispatchMessage(Handler.java:95) 
        at android.os.Looper.loop(Looper.java:135) 
        at android.app.ActivityThread.main(ActivityThread.java:5254) 
        at java.lang.reflect.Method.invoke(Native Method) 
        at java.lang.reflect.Method.invoke(Method.java:372) 
        at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:903) 
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:698) 

05-05 15:11:31.447 D/mwqi: Starting Leshan client ...
05-05 15:11:31.448 I/CoapServer: Starting server
05-05 15:11:31.448 I/CoapEndpoint: Starting endpoint at /0.0.0.0:0
05-05 15:11:31.459 I/DTLSConnector: DTLS connector listening on [{0}] with MTU [{1}] using (inbound) datagram buffer size [{2} bytes]
05-05 15:11:31.464 I/CoapEndpoint: Starting endpoint at /0.0.0.0:0
05-05 15:11:31.475 D/mwqi: Leshan client started.

Although the client has started.But did not successfully connect to the server.
Connected clients: 0
Is it because this property file does not create the cause of success?

from leshan.

@zmsoft please use the mailing list for asking questions or create a new issue. Your inquiry doesn't seem related to this issue (which is BTW already closed), or is it?

from leshan.

@sophokles73 Thank you very much for your help and responses.

from leshan.