technikempire / citadelcore Goto Github PK

This website returns a 502 on current version of CitadelCore.
It tries to connect twice via websockets and then gracefully degrades to XHR.
wss://ws-eu.pusher.com/app/a677eaa2ad4202ba89fc?protocol=7&client=js&version=4.2.1&flash=false

To replicate after first load, run this in the javascript console

delete localStorage.pusherTransportEncrypted;

I fixed this in our own repo by adding Upgrade, Connection, and Sec-WebSocket-Version to ForbiddenWsHeaders as the implementation of websockets we're using adds those headers itself. This seemed to break whatsapp web, however.

Title says everything.

Coverity complains about accessing a property (getter function) of a variable that is initialized as a null reference.

V6HttpEndpoint = httpListenOptions.IPEndPoint;

httpListenOptions is declared as null and gets defined later on based on IPV4/IPV6 settings. We should wrap this assignment in a null check.

The engine, via header forwarding, wrongly passes negotiated websocket session keys upstream, leading to 400-series responses (access denial and such).

The HTTP and HTTPS endpoint public properties return the same inner private object. Not super critical, but still wrong.

This seems to be a different issue than #26 as a 502 is not returned. The server closes the connection without warning when connecting via onshape.com.

If I take the websocket URL and paste it into the websocket echo test, it connects fine.

@TechnikEmpire you can contact @montesound if you want to test the behavior.

Get rid of m_x type variables and replace with standard _x for members.

Possible causes:

• The Microsoft util that performs stream copying.
• Kestrel's automatic chunked encoding system.
• Stream copying buffer is insufficient (too small). [Best guess ATM]

Performance drops can be observed on things such as google maps.

We do not return from the control block, and therefore block actions without requesting content inspections on responses will fail!

Related to #31

We need to handle errors and correctly shut down even when one side or both sides of the volley error out.

This is needed to pass #29

Need to review our forbidden headers list and ensure that we're stripping hop-by-hop headers we need to strip, not necessarily all of them, despite the proxy rule that we must.

Right now there's a rather low limit set on request body sizes. Remove it.

We don't have any content inspection callbacks available for websockets. We need to implement them, including the upcoming 3.0.x "stream" inspection.

Problems accessing web.whatsapp.com

We should have the originating/receiving machine-local software readily available in the message callbacks.

The callbacks for directing the next action are mangled in such a way that:

• Requesting response inspection will never result in a response inspection callback.
• Request headers are sent to the callbacks for the response messages.
• We completely ignore the proxy next action directive on responses.

We need to specify the domain and path of cookies. I'm not sure why we're not getting this info out of our http context object but anyway. We can default to "/" for path and use the DNS host from the request string to populate those fields and this solves that problem at least for now.

We install the root CA into the root X509Store for the local machine in order to establish OS trust. The question is, will this function the same on different platforms? If not, we just have to modify our abstract proxy server to request a platform specific implementation.

Presently we just have the built in error handling middleware print the raw error to the response stream. This can be useful, but it'd be nice to be able to expose this to customization, dynamic especially.

We should allow the ability to inspect streams as they pass through the proxy. This would involve adding an extra ProxyNextAction that will allow you to stream-inspect.

A valid requirement/use case that this would address is where people want to inspect large streams such as streamed videos. You can't handle a 300+mb stream in a binary fashion, so you permit it to pass but get to look at the stream as it moves forward. In this case, the user would be empowered to be able pull arbitrary frames out of the video stream and inspect/classify the video content.

This shouldn't be hard. Add an enum, and wherever we use the stream copy utilities internally, we would instead manually loop the stream and then invoke an extra callback when our copy buffer fills up before sending it out. We would also have an out param in this callback that would enable the user to stop or modify the stream. (maybe we don't need modifiability in the callback initially).

In the case of decoding video frames. I've already had feedback from licensing people at MPEG-LA that as long as I leverage platform provided decoders, my use of patented codecs is covered because the provider (say, Microsoft and their media components) pays for patent use which includes the end user, and the software would fall under that "end user" clause. Don't use ffmpeg though, they specifically named it as an example where you are not covered and would thus be infringing and wow nothx.

Right now we can only inject data and then drop the connection. We ideally would like to be able to inject and keep the connection alive.

We should have a callback where users can modify headers and rewrite URL's before actually sending the request outbound. Shouldn't be too hard, and would be really powerful for things like forcing safe search on and what have you.

These issues always arise from the way that .NET splits headers up into different, typed containers in HttpWebRequest/Response but not in ASP.NET request and response objects.

When we set the expires header, it already exists. Need to remove it first, then add it (when issuing a block action response).

HTTP2 is not universally complete in .NET. We should strip all headers and values that advertise it's available, as well as other protocols like QUIC.

We should also strip accept-encoding headers and let the http client set those by itself.

We should also strip any google specific encoding headers, like their custom compression algos and such.

Kestrel 2.2.x will include HTTP/2 support, so watch for that and be ready to upgrade so we automatically inherit it.

I'm assuming that WsClient inherits from IDisposable, since Coverity has flagged it as a dangling resource/leak. It needs to be explicitly disposed if this is the case. Perhaps we can just wrap the whole thing in a using statement, or we can do a try/finally.

Right now, whether a user has requested it or not, we fully load request bodies into memory before processing them. Instead, when a user wants to ignore it, we should simply wrap the input stream as StreamContent so we write-through directly to the server, rather than buffering everything into memory first.

aspnet/WebSockets#19
https://github.com/dotnet/corefx/issues/15430

The autobahn test on CitadelCore only fails here but that's just because the whole thing is missing.

We ignore zero-length request and response bodies right now. Unfortunately some web API's can complain about this and break some sites. We have to create zero length payloads in order to attach content related headers because Microsoft is super strict and won't let us attach content related headers when there is no content body itself.

Document offline coverity build process:

• Need build tools command prompt + cov-build in PATH
• Travis-CI doesn't support this with .NET core presently
• SonarQube analysis process (offline server, but need a way to export issues into Github tickets if possible)
• Codacity (pretty sure this is automatic)
• May be able to move coverity auto-build over to AppVeyor by issuing custom commands.
• Command example:
  cov-build --dir cov-int msbuild /t:Rebuild /p:Configuration=Release /p:Platform="Any CPU" /p:AllowUnsafeBlocks="True"
• For CitadelCore.Windows, need a SLN doesn't include the "Tests" program.
• Need real unit tests.

On service point manager set max concurrent connections high and we get 10x throughput.

This is flagged as a bug because user action on a provided functionality should not error. Must change the object to swap the real stream for a ClosedStream object just like the TLS SNI adapter does.

Obviously there's an issue with the timezone settings when generating the timestamp and validity date(s) of the certs. Easiest solution is to make the dates retroactive. We could hunt down the exact time conversion issue but the former is a guaranteed solution.

Some odd requests are giving 500 internal server error, which means the proxy has had an error while handling that request. However, when directly visiting the links of failed requests, there's no error. Perhaps something to do with referer maybe?

Title explains it all.

We pass false to the leaveInnerStreamOpen prorperty when constructing SSL streams around our peek streams (SNI capable streams). This causes some requests to randomly fail. It would appear that these are typically requests that have very small payload responses, so I'm not sure if the peek stream is getting closed because we're peeking to the end of the stream.

Anyway, changing leaveInnerStreamOpen to true seems to resolve it. If this problem persists, we need to replace our stream peeking mechanism. Looks like first class SNI support is in the pipeline for corefx, but I'm not gonna hold my breath for that. Whenever it does come though, we should use the built in SNI.

We can cut down on copies to all of our callbacks by using Memory. Don't use Span because Span can't survive continuations. On that note, we may want at least some of our current and even planned callbacks to be task based.

We need to add the basic constraints extension to flag the cert as a CA otherwise Firefox, the only browser in the universe that cares about this, will reject the certs we spoof.

Enable super bad protocols on the client side (SSL3 etc). As long as we keep them disabled on the upstream connections we're fine. We'd actually be doing clients a favour by transparently upgrading the connection (as the deprecated protocols get intercepted and handled locally, so never get to leave the client machine).

The volley attempts to shut down the already-shut-down endpoint in both the client and server IO tasks.

We need a generic error handler callback. Ultimately we want users of the library to have complete control over the internet experience of their users. So, if SSL fails, for example, there should be a callback/handler just like the message begin/end functions where library users can handle it however they would like.

This should go for websocket handshake failures, parsing errors etc. Anywhere in the code where we abandon a connection due to an error, the user should get to handle this (optionally) in a callback.

Given that this handler could be called even for non-http messages (eg. Websockets), we should explore exposing a generic stream or something. Something to abstract away the protocol but give control over content still.

Some standard-hating webapps send headers with HTTP verbs that are not legal, and C#/MS fights us on those, so we came up with a dirty work around. However, this is not accurate. We need to modify the header, if present, on the request.Content.Headers property of the request if there is actual content, but we need to modify the header directly on the request.Headers properties when no content has actually been sent.

Right now, we end up breaking these requests that are technically, by the book illegal.

Our response stream automatically decompresses on the upstream. So when we are passing data back we need to both remove the content-encoding header and possibly adjust Content-Length. If we can assume that all connections are http 1.1 or later on the client side then we don't need to set a content length and can just purge that header, and allow kestrel to automatically chunk the response.

If not, that is if it's a http 1.0 connection, then we need to collect the whole response in a buffer, set the content length properly and then pipe out the response in a non-chunked fashion.

We need a configuration class to avoid an ever-bloating constructor.

Microsoft's built in "cross-platform" websocket implementation doesn't work on Windows 7. Need to remove that plugin/addon and replace it with code written by someone competent.

This is with CitadelCore 2.2.3

2018-06-01 09:58:12.8820 INFO Request https://billing.vispnet.ca/projects/piwi/index.php?NEWSESSION/ whitelisted by rule @@||billing.vispnet.ca in category /default/whitelist/rules.
2018-06-01 09:58:12.8820 INFO Connection from local:59476 -> remote:443 outbound is our process. WindowsDiverter.cs::HandleNewTcpConnection() #616
2018-06-01 09:58:15.8880 INFO Connection from local:59476 -> remote:443 outbound is our process. WindowsDiverter.cs::HandleNewTcpConnection() #616
2018-06-01 09:58:21.8890 INFO Connection from local:59476 -> remote:443 outbound is our process. WindowsDiverter.cs::HandleNewTcpConnection() #616
2018-06-01 09:58:23.5133 INFO Connection from local:59477 -> remote:80 outbound is our process. WindowsDiverter.cs::HandleNewTcpConnection() #616
2018-06-01 09:58:33.8971 ERROR An error occurred while sending the request. at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.Http.HttpClient.d__59.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at CitadelCore.Net.Handlers.FilterHttpResponseHandler.d__6.MoveNext()Unable to connect to the remote server at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 66.244.214.7:443 at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
From FilterHttpResponseHandler.cs::Handle() #263
2018-06-01 09:58:34.1942 INFO Request https://billing.vispnet.ca/favicon.ico whitelisted by rule @@||billing.vispnet.ca in category /default/whitelist/rules.
2018-06-01 09:58:34.1942 INFO Connection from local:59478 -> remote:443 outbound is our process. WindowsDiverter.cs::HandleNewTcpConnection() #616
2018-06-01 09:58:37.1967 INFO Connection from local:59478 -> remote:443 outbound is our process. WindowsDiverter.cs::HandleNewTcpConnection() #616
2018-06-01 09:58:43.1977 INFO Connection from local:59478 -> remote:443 outbound is our process. WindowsDiverter.cs::HandleNewTcpConnection() #616
2018-06-01 09:58:46.2524 INFO RESPONSE WindowsDiverter.cs::HandleNewTcpConnection() #678
2018-06-01 09:58:46.2524 INFO Connection from local:59357 -> remote:59479 outbound is our process. WindowsDiverter.cs::HandleNewTcpConnection() #616