SHA1 checksum fails on 12.X.X even though messages are equivalent about autobahn-testsuite HOT 9 OPEN

For binary messages, the equivalence of messages is the identity, and hence hashes MUST match. For text messages, the echo'ing peer might do some wrangling / normalization allowed by Unicode, but this is still dubious and the actual messages would need to be analyzed. IOW: it MUST work with at least binary messages (hashes must matcH) otherwise your testee is bogus.

from autobahn-testsuite.

@oberstet For the binary messages I can 100% confirm that the resulting data is correct if I send the same data to a different server and make exact binary comparisons. Maybe there is an issue with the compression config on the client? Would the comparison fail if the compressed data is incorrect or is only the uncompressed data compared?

from autobahn-testsuite.

the hashes are computed over the uncompressed message payload. I don't think this is the reason. My bet is your testee language (which one btw?) does some Unicode normalization/wrangling ... which might be allowed (Unicode has some "equivalence" definitions), but it might also be invalid/disallowed, and then your testee is buggy. Anyways, more investigation would be needed to find out ..

from autobahn-testsuite.

@oberstet It's a thin Swift wrapper on top of libwebsockets. I did not try to run libwebsockets directly through Autobahn as it's pretty battle tested already. The Swift implementation is just a wrapper and doesn't do any data manipulation.

I don't know where to start to be honest with debugging as simply replicating the test suite with a Javascript websocket server leads to correct results. Would the reports help? I can upload them.

from autobahn-testsuite.

Just to repeat the interesting bit: It's dependent on the OS but deterministic. On macOS 13.0+ all of the tests always pass. On Linux (Ubuntu 20.04 and 22.04) with Swift 5.7 some fail, with Swift 5.8 others fail.

from autobahn-testsuite.

@oberstet

This is Swift 5.8 on Ubuntu 22.04. Case 12.3.18 is the only 12.X case that's failing here. Might help to debug.

https://transfer.sh/c8BbWePeow/autobahn-libwebsockets-swift.tar

from autobahn-testsuite.

Would the reports help? I can upload them.

thanks, but I don't have time to dig in ..

On macOS 13.0+ all of the tests always pass. On Linux (Ubuntu 20.04 and 22.04) with Swift 5.7 some fail, with Swift 5.8 others fail.

this sounds like a bug in swift

I don't know where to start to be honest with debugging

write a simple swift program to do the following:

1. serialize unicode strings (you need fancy strings) on system A into bytes
2. copy the bytes to system B
3. deserialize bytes into string
4. and then reverse back to system A

data must roundtrip to the single bit.

stuff like this. what I want to say: under the hypothesis that this is some swift implementation bug related to fancy unicode strings and normalization/whatever-wrangling, you should be able to prove that using a simple swift program without any websocket or what.

hope this helps

from autobahn-testsuite.

should note: I'd do the simple roundtrip (unicode=>serialize=>deserialize) both with two different systems A and B (those you mentioned above), and do roundtrip on A-A and B-B (self roundtrip) ... actually, the roundtrip could work on A-A but not on B-B ... for certain unicode strings only that is (it might work with simple us-ascii all the time etc)

so follow a systematic bisection approach to identify the issue in a simplified setting (no websocket or what, just plain swift programs and text/binary)

from autobahn-testsuite.

This is Swift 5.8 on Ubuntu 22.04. Case 12.3.18 is the only 12.X case that's failing here.

mmh. the test data used in those tests is here

https://github.com/crossbario/autobahn-testsuite/tree/master/autobahntestsuite/autobahntestsuite/testdata

this is used to generate the 12.* test cases

and test subscases 12.N.*

https://github.com/crossbario/autobahn-testsuite/blob/09cfbf74b0c8e335c6fc7df88e5c88349ca66879/autobahntestsuite/autobahntestsuite/case/case12_x_x.py#L231C13-L231C22

from autobahn-testsuite.