Comments (7)
the testsuite has a builtin webserver: just add the webport option
Awesome, I didn't know that, thanks!
the report files are generated, stored and served from within the docker container, so that should be good (independent of your test env)?
Yes definitely, if the container can serve those files via HTTP, this is just perfect. I was talking about the absence of such HTTP API, in which case I would have to read the files manually.
this is one of the HTTP endpoint that are indeed used to control the fuzzing server - but that is HTTP not WebSocket
Oh. It did look like a websocket API though here:
https://github.com/crossbario/autobahn-testsuite/blob/master/autobahntestsuite/autobahntestsuite/fuzzing.py#L309
the problem with putting the control onto the same WebSocket connection that is being fuzzed is obvious. So the only sane way would be to run a 2nd WebSocket connection only for the control. Possible, sure. But that is not implemented right now ..
Totally agreed, and I actually used additional websocket connections for those endpoints (which seemed to work fine on most client implementations I'm testing right now). So now I'm confused as to this HTTP vs websocket controls, it really did look like it worked as a websocket connection 😆 In any case, I will try with the HTTP server, thanks!
from autobahn-testsuite.
the JSON reports are just files generated, and so can be fetched via HTTP I guess ... long ago, I forgot;) what I'm sure: there is no builtin websocket protocol for retrieving the test results. the websocket connection is the one under test, and only used for that - not for controlling/results-access
from autobahn-testsuite.
@oberstet thanks for your response!
the JSON reports are just files generated, and so can be fetched via HTTP I guess
I'm not aware of an HTTP server running in the autobahn testsuite container. Is there one? Or are you talking about running a separate HTTP server next to the autobahn container?
My main problem is that the file system is not always accessible depending on the platform. I'm working on a Kotlin multiplatform client, and the tests could go and read the JSON reports as files on JVM, nodeJS and some native platforms, but in JS browser or iOS I don't have access to the file system. That's why it would have been way easier to get these JSONs as responses in some endpoint since we're already calling "control endpoints" like /updateReports
.
what I'm sure: there is no builtin websocket protocol for retrieving the test results.
Then what is /getCaseStatus?casetuple=$case
meant for?
the websocket connection is the one under test, and only used for that - not for controlling/results-access
AFAICT there are control/results endpoints: /getCaseInfo
, /getCaseStatus
, /updateReports
.
Is that an internal API or something meant for some other usage?
from autobahn-testsuite.
the testsuite has a builtin webserver: just add the webport
option
and start the docker container exposing 8080 with -p 8080:8080
the result looks ugly (the index pages .. the report pages are rendered nicely), but works;)
My main problem is that the file system is not always accessible depending on the platform.
the report files are generated, stored and served from within the docker container, so that should be good (independent of your test env)?
Then what is /getCaseStatus?casetuple=$case meant for?
this is one of the HTTP endpoint that are indeed used to control the fuzzing server - but that is HTTP not WebSocket. the problem with putting the control onto the same WebSocket connection that is being fuzzed is obvious. So the only sane way would be to run a 2nd WebSocket connection only for the control. Possible, sure. But that is not implemented right now ..
from autobahn-testsuite.
In any case, I will try with the HTTP server, thanks!
yes, from what I understand, that should do what you want (get at the test result json) and is simple to do.
So now I'm confused as to this HTTP vs websocket controls
I'm sorry, you're right;) long ago I looked into the code. anyways, it indeed works by using the HTTP path given by the fuzzed client when opening a WebSocket connection: if no "fuzzing case" is to be run, use that path to multiplex to the respective control endpoint.
and after that, immediately close the connection. so either run a test case or run a cmd and end immediately is the logic once a new websocket connection is opened. so no fuzzing and control at the same time - even though only over 1 server, and websocket.
from autobahn-testsuite.
So I tried and I can confirm that the HTTP server can indeed serve the JSON reports. I just need to GET /cwd/reports/clients/index.json
or other JSON files at the same path.
I also tested the endpoints mentioned above (/getCaseStatus
, /updateReports
etc.), and as expected they are websocket connection endpoints and not supported by the HTTP server. When trying to call them via HTTP, I get a 404 error page:
<html>
<head><title>404 - No Such Resource</title></head>
<body>
<h1>No Such Resource</h1>
<p>File not found.</p>
</body>
</html>
Using the one-shot websocket connection (immediately closed) works most of the time, it's what I did so far to get the test status and to generate reports. However sometimes the connection is abruptly closed without returning a text frame with the test result (couldn't pinpoint the exact problem here, though, but it seems to mostly happen when testing several agents in parallel - hence my question in #119).
Apart from that, the only drawback of /getCaseStatus
is that it only returns the status of the test (OK, FAILED, etc.) but no extra information.
By contrast, with the HTTP server giving me the JSON files, I need to run updateReports
after each test to then be able to fetch the JSON file for the test and read detailed results. Depending on the number of tests, it might be a problem to rewrite these files constantly. I also wonder about synchronization issues for parallel tests. Still better than nothing, though :)
from autobahn-testsuite.
I also tested the endpoints mentioned above (/getCaseStatus, /updateReports etc.),
yes, those the URLs used when opening the websocket connection, and since every websocket connection starts as HTTP, this is where the confusion comes from. those URLs cannot be used to continue with HTTP/GET after the initial HTTP open.
By contrast, with the HTTP server giving me the JSON files, I need to run updateReports after each test to then be able to fetch the JSON file for the test and read detailed results.
yes, unfort. that's how it works (results are just collected in a server class and only updateReports will process that, and process all collected results).
from autobahn-testsuite.
Related Issues (20)
- Port to Python 3 HOT 1
- Add tests for unusual payload lengths
- Python3 support and docker image HOT 1
- Case 7.13.2: We send 65535, not 65536
- Test 13 WebSocket Compression (different parameters)
- Case 7.9.6: Close code 1014 is now valid HOT 1
- update autobahn-testsuite docker image
- autobahn-testsuite support for IPv6 HOT 1
- Support for Server Name Indication
- Parallel test support? HOT 1
- ModuleNotFoundError: No module named '_version' HOT 2
- Problem running fuzzingserver in virtualenv HOT 8
- Remove case 7.7.8 (close code 1010 may not be used by server) HOT 4
- Minecraft websockets' differential vs standard implementation that passes Autobahn? HOT 2
- Document CI integration and testsuite Docker usage HOT 3
- Python 3 support? HOT 2
- SHA1 checksum fails on 12.X.X even though messages are equivalent HOT 9
- externalPort check cannot be disabled (as the testsuite uses old autobahn ..) HOT 9
- the page linked from project's main page is not working
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from autobahn-testsuite.