kitura / kitura-net Goto Github PK

Kitura networking

License: Apache License 2.0

Swift 82.17% Objective-C 0.80% C 16.81% Shell 0.21%

kitura http fastcgi request response socket headers swift networking

kitura-net's Introduction

Kitura-Net

The Kitura-net module contains logic for sending and receiving HTTP requests. It also contains the structure for listening on a port and sending requests to a delegate for processing. It can be used to create HTTP/CGI servers on specific ports, and provides HTTP functionality.

We expect most of our users to require higher level concepts such as routing, templates and middleware, these are not provided in Kitura-net, if you want to use those facilities you should be coding at the Kitura level, for this please see the Kitura project. Kitura-net underpins Kitura which offers a higher abstraction level to users.

Kitura-net utilises the BlueSocket framework, the BlueSSLService framework and CCurl.

Features

Port Listening
FastCGI Server support
HTTP Server support (request and response)

Usage

Add dependencies

Add the Kitura-net package to the dependencies within your application’s Package.swift file. Substitute "x.x.x" with the latest Kitura-net release.

.package(url: "https://github.com/Kitura/Kitura-net.git", from: "x.x.x")

Add KituraNet to your target's dependencies:

.target(name: "example", dependencies: ["KituraNet"]),

Import package

import KituraNet

Prerequisites

As of Kitura-net 3.0.0, Swift 5.1+ is required.

Contributing to Kitura-net

All improvements to Kitura-net are very welcome! Here's how to get started with developing Kitura-net itself.

Clone this repository.

$ git clone https://github.com/Kitura/Kitura-net && cd Kitura-net

Build and run tests.

$ swift test

You can find more info on contributing to Kitura in our contributing guidelines.

API Documentation

For more information visit our API reference.

Community

We love to talk server-side Swift, and Kitura. Join our Slack to meet the team!

License

This library is licensed under Apache 2.0. Full license text is available in LICENSE.

kitura-net's People

Contributors

Stargazers

Watchers

Forkers

codestergit nacho4d es-kumagai collinhundley korzonek neotron complexityclass yoseob c-l-o-u-d jberstler timbodeit kazu0620 lfaoro vherrin ianpartridge gormster naithar david-coyle-sjc tardieu gtaban damian-kolakowski leonhartley modelkit raszi ymanton steadyequipment terhechte michielcuijpers bdhernand na-gupta swizzlr euwars bouke ivanandriollo bytethenoodle emiliopavia adellibovi dmirain ralischer klaas cpageler93 isalva prakaashkpk shmuelk c1ira mgiger kphrx shihabmehboob zig1375 nhardman kyemaloy97 nocturnalsolutions saihemak voynovia bikram990 seabaylea nethraravindran pushkarnk makleso6 geertberkers djones6 rudraniw sugarandcandy shoaibpathan spriter liamnichols artmartiros borojevicd azurecloudmonk enzosterro kiturakommunity dannys42-contrib kitura-legacy yjosephides isabella232 millertechnologyperu bobbyski jefferyq7

kitura-net's Issues

HTTPStatusCode enum is missing registered values

The public enum HTTPStatusCode is missing a number of values that are registered with IANA:
https://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml

The ones that I spotted are:

423: Locked
425: Too Early
426: Upgrade Required

There may be others missing - it would be worth comparing against the IANA list.

Whilst adding cases to this public enum is technically a breaking change, we do not realistically expect someone to exhaustively switch over it (ie. without a default case), and so we feel this is a reasonable change to make as a bug fix.

This change needs to be made to both Kitura-net and Kitura-NIO.

Kitura-net ignores an Expect: 100-continue header

I noticed that when posting data with curl over a certain size (in my experiments, > 1024 bytes) there was a 1 second (or occasionally 2 second) delay. I tracked this down to the behaviour of the Expect: 100-continue header which Kitura-net does not handle.

Here's 'normal' behaviour, where the entire payload is received as a single HTTP message:

[2019-03-01T09:35:34.568Z] [DEBUG] [HTTPServer.swift:288 listen(listenSocket:socketManager:)] Accepted HTTP connection from: 192.168.1.35:53472
[2019-03-01T09:35:34.569Z] [DEBUG] [IncomingSocketHandler.swift:114 init(socket:using:)] Socket connection 7 established
[2019-03-01T09:35:34.638Z] [DEBUG] [IncomingSocketHandler.swift:158 handleRead()] Read 1162 bytes from socket 7
[2019-03-01T09:35:34.639Z] [DEBUG] [IncomingSocketHandler.swift:158 handleRead()] Read 0 bytes from socket 7
[2019-03-01T09:35:34.639Z] [DEBUG] [IncomingHTTPSocketProcessor.swift:137 process] Processing buffer of size 1162 for socket 7
[2019-03-01T09:35:34.639Z] [DEBUG] [IncomingHTTPSocketProcessor.swift:241 parse(_:from:completeBuffer:)] HTTP parser parsed 1162 out of 1162 bytes for socket 7
[2019-03-01T09:35:34.639Z] [DEBUG] [IncomingHTTPSocketProcessor.swift:242 parse(_:from:completeBuffer:)] Buffered bytes:
POST /post HTTP/1.1
User-Agent: curl/7.35.0
Host: gruffalo10G:8080
Accept: */*
Content-Type: application/raw
Content-Length: 1024

1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef
[2019-03-01T09:35:34.639Z] [VERBOSE] [HTTPServerRequest.swift:333 parsingCompleted()] HTTP request from=192.168.1.35; proto=http;
[2019-03-01T09:35:34.640Z] [DEBUG] [IncomingSocketHandler.swift:337 write(from:)] writing data
[2019-03-01T09:35:34.640Z] [DEBUG] [IncomingSocketHandler.swift:353 write(from:length:)] writing bytes with length 123

Sending 1 byte more (1025 bytes payload), curl sends the headers and body separately, the former with a Expect: 100-continue header:

[2019-03-01T09:35:42.416Z] [DEBUG] [HTTPServer.swift:288 listen(listenSocket:socketManager:)] Accepted HTTP connection from: 192.168.1.35:53474
[2019-03-01T09:35:42.417Z] [DEBUG] [IncomingSocketHandler.swift:114 init(socket:using:)] Socket connection 9 established
[2019-03-01T09:35:42.417Z] [DEBUG] [IncomingSocketHandler.swift:158 handleRead()] Read 160 bytes from socket 9
[2019-03-01T09:35:42.417Z] [DEBUG] [IncomingSocketHandler.swift:158 handleRead()] Read 0 bytes from socket 9
[2019-03-01T09:35:42.417Z] [DEBUG] [IncomingHTTPSocketProcessor.swift:137 process] Processing buffer of size 160 for socket 9
[2019-03-01T09:35:42.417Z] [DEBUG] [IncomingHTTPSocketProcessor.swift:241 parse(_:from:completeBuffer:)] HTTP parser parsed 160 out of 160 bytes for socket 9
[2019-03-01T09:35:42.417Z] [DEBUG] [IncomingHTTPSocketProcessor.swift:242 parse(_:from:completeBuffer:)] Buffered bytes:
POST /post HTTP/1.1
User-Agent: curl/7.35.0
Host: gruffalo10G:8080
Accept: */*
Content-Type: application/raw
Content-Length: 1025
Expect: 100-continue


[2019-03-01T09:35:43.418Z] [DEBUG] [IncomingSocketHandler.swift:158 handleRead()] Read 1025 bytes from socket 9
[2019-03-01T09:35:43.418Z] [DEBUG] [IncomingSocketHandler.swift:158 handleRead()] Read 0 bytes from socket 9
[2019-03-01T09:35:43.418Z] [DEBUG] [IncomingHTTPSocketProcessor.swift:137 process] Processing buffer of size 1025 for socket 9
[2019-03-01T09:35:43.418Z] [DEBUG] [IncomingHTTPSocketProcessor.swift:241 parse(_:from:completeBuffer:)] HTTP parser parsed 1025 out of 1025 bytes for socket 9
[2019-03-01T09:35:43.418Z] [DEBUG] [IncomingHTTPSocketProcessor.swift:242 parse(_:from:completeBuffer:)] Buffered bytes:
1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef1
[2019-03-01T09:35:43.418Z] [VERBOSE] [HTTPServerRequest.swift:333 parsingCompleted()] HTTP request from=192.168.1.35; proto=http;
[2019-03-01T09:35:43.419Z] [DEBUG] [IncomingSocketHandler.swift:337 write(from:)] writing data
[2019-03-01T09:35:43.419Z] [DEBUG] [IncomingSocketHandler.swift:353 write(from:length:)] writing bytes with length 123

Note there is a 1 second delay between the parsing of the first 160 byte transmission (headers) and the second 1025 bytes (body). Kitura-net did not respond to the Expect header, as it was waiting for the message to be completely sent, and curl eventually decided to send it anyway.

Errors in removeIdleSockets

In my server log I see these errors:

[IncomingSocketManager.swift:252 removeIdleSockets(removeAll:)] epoll_ctl failure. Error code=1. Reason=Operation not permitted

It seems that the removeIdleSockets is getting the EPERM exception when cleaning up idle connections.

I'm running my server using this Docker container. It is deployed on Amazon ECS.

My server is mostly a websocket server. There is a memory leak I'm trying to investigate and I wonder if this might be related.

PATCH Request not passing Body

I am using the ClientRequest to create a request to a REST API
The body goes through with:
httpRequest.write(from: httpBody)

When using httpMethod PATCH, the body string is not passed across
When using httpMethod PUT or POST, the body string goes through...

Gracefully close connections on server shutdown

See Kitura/SwiftyRequest#69

The analysis of an issue involving failures from a shared URLSession when the server is repeatedly stopped/started concluded that the issue relates to when the HTTPServer is stopped, and any open connections are abruptly closed. Curl doesn't seem to handle this well.

This was resolved for Kitura-NIO under Kitura/Kitura-NIO#216 where we concluded that connections should be closed gracefully on server stop. The same should be applied to Kitura-net.

Should use symbolic name for error instead of hardcoding value.

In HttpServerSpi.swift, line 60, the BlueSocketError errorCode property is being checked for the hard coded -9994 value for BlueSocket.SOCKET_ERR_ACCEPT_FAILED. It should be using the symbolic name instead. While this value hasn't changed recently, others have and it's generally a better idea to use the symbolic name instead of the actual value.

Study: Does the HTTP smuggling attack affect Kitura-net?

Investigate if Kitura-net iis vulnerable to the HTTP smuggling attack described here.

SSLRead() return error -9806

When sending requests to certain sites using HTTPS, I get a nil response. One such site is maps.googleapis.com.

Enabling CURLOPT_VERBOSE prints this after some header information:

SSLRead() return error -9806

The internet tells me that this might be an issue with CURL and App Transport Security on macOS. The headers returned are from Google, which means that the server is being reached, but it appears that CURL and/or macOS doesn't trust Google's SSL certificate.

Does anybody know how to fix this issue?

Undeprecate `urlComponents`?

fe65945 deprecated urlComponents because of a memory leak in Swift Foundation. That bug was resolved a few months ago. It seems like urlComponents should no longer leak, thus the deprecation warning could be dropped? I use urlComponents to spit out the query parameters for a URL.

ClientRequest does not handle credentials where one of the pair is blank

Basic HTTP Authentication can be:

Username: myusername
Password: <blank>

Username: <blank>
Password: mypassword

The creation of the URI should allow for this. Examples with above information:

https://myusername:@mydomain.com
https://:[email protected]

This fragment:

        // Adding support for Basic HTTP authentication
        let user = self.userName ?? ""
        let pwd = self.password ?? ""
        var authenticationClause = ""
        if (!user.isEmpty && !pwd.isEmpty) {
          authenticationClause = "\(user):\(pwd)@"
        }

        url = "\(theSchema)\(authenticationClause)\(hostName)\(port)\(path)"

    }

Should be:

        // Adding support for Basic HTTP authentication
        let user = self.userName ?? ""
        let pwd = self.password ?? ""
        var authenticationClause = ""
        // If either the user or pwd are non-empty, add the authenticationClause
        if (!user.isEmpty || !pwd.isEmpty) {
          authenticationClause = "\(user):\(pwd)@"
        }

        url = "\(theSchema)\(authenticationClause)\(hostName)\(port)\(path)"

    }

urlString is never set in IncomingMessage

Kitura-net tests are intermittently crashing with swift 4 snapshot

See https://travis-ci.org/IBM-Swift/Kitura-net/builds/243906452

Validate curl 7.59 fix for chunked encoding

Kitura/Kitura#1465 was raised to track a bug in curl relating to receiving a response with a chunked encoding, which broke in curl 7.57 and was fixed in curl 7.59, but which is shipped in Ubuntu 18.04 (ships curl 7.58).

We need to validate that the 7.59 fix does actually resolve the problem we have with ClientRequest and the duplicate callbacks.

Kitura-net fails to build

Using the correct version of Swift (2016-03-01), Kitura-net fails with many errors, which breaks all Kitura projects, including Kitura-Sample.

The first error is:

Compiling Swift Module 'KituraNet' (12 sources)
<module-includes>:1:9: note: in file included from <module-includes>:1:
#import "shim.h"
        ^
/Users/ben/Sync/Sync2/development/Kitura-Sample/Packages/CHttpParser-0.0.1/shim.h:22:10: error: 'http_parser.h' file not found

Here's my Swift version:

$ which swift
/Library/Developer/Toolchains/swift-latest.xctoolchain/usr/bin/swift
$ ls -la /Library/Developer/Toolchains/
total 8
drwxr-xr-x  4 root  wheel  136 Mar 27 16:24 .
drwxr-xr-x  4 root  wheel  136 Feb 23 08:50 ..
drwxr-xr-x  6 root  wheel  204 Mar 27 16:24 swift-DEVELOPMENT-SNAPSHOT-2016-03-01-a.xctoolchain
lrwxr-xr-x  1 root  wheel   81 Mar 27 16:24 swift-latest.xctoolchain -> /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-03-01-a.xctoolchain
$ swift --version
Apple Swift version 3.0-dev (LLVM b361b0fc05, Clang 11493b0f62, Swift 24a0c3de75)
Target: x86_64-apple-macosx10.9

Error when using SSL

I switched over to using Kitura with SSL (and self-signed certificates). I'm getting no errors specific to my server, but I do get reports of:

[2017-05-19T20:50:50.263-06:00] [ERROR] [IncomingSocketHandler.swift:148 handleRead()] Read from socket (file descriptor 8) failed. Error = Error code: -9806(0x-264E), ERROR: SSLRead, code: -9806, reason: errSSLClosedAbort.

This is not causing failures, but doesn't look good.

Suggestions?
Thanks,
Chris.

Asynchronous ClientRequest

Is there any plan to implement an asynchronous version of ClientRequest? Or, is the plan to defer this functionality to NSURLRequest once it becomes available?

It just occurred to me that ClientRequest is actually synchronous, which causes some concern for my application. Since completion handlers are called before the end() method returns, this can result in infinite recursion for applications designed for async networking.

HTTP/2 support

Hi, I am looking at some greenfield development and would like to make use of HTTP/2 PUSH_PROMISE. Any idea if/when http/2 will be supported in Kitura or do I need to make a mess and try to write it? I am seeing support in WKWebView on the IOS client side lots of server-side but would like to see it somewhere via Swift on the server side as in Kitura. https://github.com/http2/http2-spec/wiki/Implementations

Compiler Error

When trying to compile a Kitura project, Kitura-net fails to compile because of a dependency on Kitura-HttpParserHelper which as been recently deprecated. The compiler fails repeatedly with:

/Packages/Kitura-HttpParserHelper-0.4.0/utils.h:22:10: error: 'http_parser.h' file not found

missing required modules: 'CCurl', 'CHTTPParser'

I'm trying to import a project and compile in XCODE (macOS/Cocoa) and I'm getting this error.

Here is my write up on Stack Overflow -- https://stackoverflow.com/questions/47167257/embedding-frameworks-from-swiftpm-into-cocoa-application-xcode

Using Kitura-net 1.7.0

[ClientRequest.swift:346 end(close:)] ClientRequest error. Failed to parse response. Error=Parsed fewer bytes than were passed to the HTTP parser

I am getting the above message with version 1.7.8. With the exact same code I do not get any error on 1.7.1

The url that I am trying to send a request to is a slackbot. The url and parameters are identical and one version success and the other fails.

Can no longer build since net/helper modules moved to inline functions

Now that the curl and http helper .c/.h files have been flattened into just .h files with inline functions, I wonder how we should be building this? Since there are no longer any .c files as currently referenced in the Makefile we can't make a .a file to link in, but the swift package/build mechanism doesn't currently allow for any way to specify custom compiler flags which are needed to find eg. pcre and http-parser headers at compile time and libs at link time.

Listening on the wrong port

Hi!
I wanted to replicate the Kitura-TodoList project with my own classes.
The only major difference is, that in the func setupRoutes(router: Router, messages: MessagesArray) {}
I have provided implementation for router.use, router.all, router.get, router.options, but I have not provided implementation of router.post, router.patch, router.delete... Would that make a difference?

I've modified the provided ones (from Kitura-TodoList) and now after my new project succesfully compiles, whenever I type in browser: http://localhost:8090/messages in the browser I see: Cannot GET /messages. and in the terminal I see INFO: spiListen(_:port:) (...)/Packages/Kitura-net-0.4.0/Sources/KituraNet/HttpServerSpi.swift line 51 - Accepted connection from: 127.0.0.1on port 36054 UnexpectedEOF

Why would the server accept connection on port 36054 if I explicitly try to connect on 8090? Or maybe I'm doing sth wrong?

EDIT: If I try localhost:8090 I get unknown displayed in the browser window.

Any help will be appreciated!
Thanks :)

FastCGI server doesn't send files more than 64 kB.

When tried to implement test SPA using Bootstrap framework source stored along with FastCGI server resources, the resulting page layout was corrupted.
After some investigation it turned out that file "/bootstrap-3.3.7-dist/css/bootstrap.min.css" was failed to be sent.
The error message was: "Failed to send response to the client".
In closer look, the problem was in failed condition (FastCGIRecordCreate.swift:297):

guard data.count <= 65535 else {
    throw FastCGI.RecordErrors.oversizeData
}

According to FastCGI protocol, the response payload really can't be more than 64 KB, but if the file is larger it could be chunked into smaller packets that would fit 64 KB.

Looks like the call that caused this failing state is (FastCGIServerResponse.swift:244):

try socket.write(from: getMessage(buffer: data))

The proposed solution is to chunk data into parts less than 64 kB and write those parts as message.

Compiling fails in Xcode

Description:

I am trying to build a Kitura-based application in Xcode. The problem is that the Kitura-net dependency fails building. The compiler constantly throws the no such module CCurl error. The linker also outputs : "ditto: can't get real path for source '/Users/User/Library/Developer/Xcode/DerivedData/Projectheftrmxirszwjsegjokkskppwzwl/Build/Intermediates/Project.build/Debug/KituraNet.build/Objects-normal/x86_64/KituraNet-Swift.h'"

Building and running works perfectly with Terminal swift build command.

Steps to reproduce:

Open Xcode and make sure it is set to use the DEVELOPMENT-SNAPSHOT-2016-05-03-aSwift snapshot
Create a dummy swift package in terminal using the same Swift version.
Add the latest Kitura package as a dependency.
Run swift build : building should complete with no issue.
Run the swift build --generate-xcodeproj and open the generated project
Switch to the executable scheme and hit ⌘B.
Building fails.

Configuration:

Xcode 7.3 public release configured with the May 3rd snapshot.

Note:

The failure propagates to libraries that have Kitura-net as a dependency, making it impossible to compile the whole project (you have to use terminal) and very hard to get code completion to work.

See Full Xcode Build Log

ClientRequest init(options, callback) generates invalid URI

If a ClientRequest object is created with an array of ClientRequestOptions then the resulting URI that is created is missing '://' after the schema element.

Easiest way to recreate this is to parse a URL with URLParser and then use the fields from it as ClientRequestOptions and the resulting URL won't match the original.

Swap HTTPParser Swift package for http-parser

We have been working a pure Swift port of the 'C' http-parser project.

https://github.com/smithmicro/HTTPParser

Swapping HTTPParser in for http-parser would get Kitura closer to pure Swift. Please review the performance of HTTPParser in the readme to see if it would meet the requirements of Kitura.

I will work on a PR, but I wanted to alert you to this new project and gauge interest.

Failing to send HEAD requests with Kitura-net

Whenever HEAD method is used request remains hanging indefinitely.

Streams for Kitura?

My use case: I want to have my HTTP server be able to receive requests containing files, but not store those files on my server in any fashion. Instead, I want to stream those files off to other services. E.g., to Google Drive. My rationale for not storing files on my server has several motivations:

It ought to be more efficient to stream the incoming (uploaded) file off to something like Google Drive, without first storing that upload to a file in the server-local file system;
For purposes of server scaling, storing files in a local file system is not necessarily a good idea-- those files may be removed upon a server reboot and make recovery from errors more difficult;
I don't want to have to pay for temporary file storage on a server for these files.

What I'm looking for is a Swift implementation of Streams in the sense of Node.js. E.g., the standard HTTP request objects in Node.js (see https://nodejs.org/api/http.html#http_class_http_incomingmessage) implement a stream.Readable interface (see https://nodejs.org/api/stream.html#stream_class_stream_readable) which for example make a pipe method available where you can pipe the HTTP request to a writable stream. Busboy provides an example of this usage-- see http://schempy.com/2015/03/11/streaming_file_uploads_with_nodejs/

It looks like that at present Kitura doesn't support this kind of functionality. Is there interest in adding this? I'd be looking for architecture ideas, to do the development myself, and looking for code reviews on PRs. Any and all input would be accepted.

I've done some reading through the Kitura-net code and it looks like a place to start might be with the ServerRequest protocol, analogous to the way in which Node.js does this. Perhaps ServerRequest could optionally support streaming methods and properties. Or perhaps this could just be a different protocol, say, ReadableStream, which is implemented by some classes implementing ServerRequest. I'm not sure if a different server type would be necessary. e.g., comparable to the HTTP or FastCGI servers.

Please let me know your thoughts on this. This is not for the company I work for, rather it's in support of a personal project. (I'm converting this project-- https://github.com/crspybits/SMSyncServer over to using Swift).

Port Kitura Request/Response tests to Kitura-net

Kitura tests provide a better coverage, especially in catching regressions, than Kitura-net tests themselves. Some tests from Kitura could be replicated on Kitura-net.

Implement test for ServerLifecycleListener.performClientConnectionFailCallbacks

#221 appeared to remove coverage of the ServerLifecycleListener.performClientConnectionFailCallbacks() function, but this was really because of an error handling fix I just submitted to BlueSSLService (Kitura/BlueSSLService#42).

We should have a test that deliberately fails to initialize a connection, and checks that the fail callback is called.

HTTP extension recommedations

It would be ideal to allow simple extensions of the HTTP class for other methods such as DELETE.
This would be a clean extension for 'delete' but ClientRequestOptions does allow a full URL:

extension HTTP {
    class func delete(_ url: String, callback: ClientRequest.Callback) {
        var options: [ClientRequest.Options] = []
        options.append(.method("delete"))
        options.append(.url(url))   // Error: Type 'ClientRequestOptions' has no member 'url'
        let req = HTTP.request(options, callback: callback)
        req.end()
    }
}

Another suggestion would to create an request function that allows for both url and options. This would require extending the underlying ClientRequest class.

extension HTTP {
    class func delete(_ url: String, callback: ClientRequest.Callback) {
        var options: [ClientRequest.Options] = []
        options.append(.method("delete"))
        // Error: Cannot invoke 'request' with an argument list of type ...
        let req = HTTP.request(url, options: options, callback: callback)
        req.end()
    }
}```

Allow a way to deal with unsigned certificates

I am trying to use Kitura-net to do a POST but the request fails because the host is using an unsigned certificate. Unfortunately, while the underlying libcurl should support this scenario, Kitura-net does not expose a way to control it.

Thread sanitizer finds issues

I tried running my server with thread sanitizer enabled and it flagged several data races in InconingSocketManager.swift. My server uses websockets, which might be related to these data races.

They are easy to repro for me, but here are some of them it found for reference:

WARNING: ThreadSanitizer: data race (pid=67210)
  Write of size 1 at 0x7b1c00068df0 by thread T14:
  * #0 WSSocketProcessor.inProgress.setter WSSocketProcessor.swift (KituraWebSocket:x86_64+0x10b89)
    #1 protocol witness for IncomingSocketProcessor.inProgress.setter in conformance WSSocketProcessor WSSocketProcessor.swift (KituraWebSocket:x86_64+0x1380c)
    #2 IncomingSocketHandler.handleCancel() IncomingSocketHandler.swift:398 (KituraNet:x86_64+0xbb14b)
    #3 partial apply IncomingSocketHandler.swift (KituraNet:x86_64+0xbcad4)
    #4 thunk for @callee_owned () -> () FastCGIServer.swift (KituraNet:x86_64+0x3d34c)
    #5 __tsan::dispatch_callback_wrap(void*) <null>:5353648 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x662e1)
    #6 _dispatch_call_block_and_release <null>:5353648 (libdispatch.dylib:x86_64+0x9c3c)

  Previous read of size 1 at 0x7b1c00068df0 by thread T1:
  * #0 WSSocketProcessor.inProgress.getter WSSocketProcessor.swift (KituraWebSocket:x86_64+0x10b0a)
    #1 protocol witness for IncomingSocketProcessor.inProgress.getter in conformance WSSocketProcessor WSSocketProcessor.swift (KituraWebSocket:x86_64+0x13763)
    #2 IncomingSocketManager.removeIdleSockets(removeAll:) IncomingSocketManager.swift:242 (KituraNet:x86_64+0xbf1c0)
    #3 IncomingSocketManager.handle(socket:processor:) IncomingSocketManager.swift:137 (KituraNet:x86_64+0xbde71)
    #4 HTTPServer.handleClientConnection(clientSocket:socketManager:) HTTPServer.swift:278 (KituraNet:x86_64+0x71183)
    #5 HTTPServer.listen(listenSocket:socketManager:) HTTPServer.swift:223 (KituraNet:x86_64+0x6c09c)
    #6 closure #1 in HTTPServer.listen(on:) HTTPServer.swift:133 (KituraNet:x86_64+0x69aee)
    #7 partial apply for closure #1 in HTTPServer.listen(on:) HTTPServer.swift (KituraNet:x86_64+0x69c8c)
    #8 thunk for @callee_owned () -> () FastCGIServer.swift (KituraNet:x86_64+0x3d34c)
    #9 _dispatch_client_callout <null>:5353648 (libdispatch.dylib:x86_64+0x1d1e)
    #10 _dispatch_client_callout <null>:5353648 (libdispatch.dylib:x86_64+0x1d1e)

  Issue is caused by frames marked with "*".

  Location is heap block of size 112 at 0x7b1c00068dd0 allocated by thread T5:
    #0 malloc <null>:5353680 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x48b3a)
    #1 swift_slowAlloc <null>:5353680 (libswiftCore.dylib:x86_64+0x344478)
    #2 WSSocketProcessor.__allocating_init(connection:) WSSocketProcessor.swift (KituraWebSocket:x86_64+0x11584)
    #3 WSConnectionUpgradeFactory.upgrade(handler:request:response:) WSConnectionUpgradeFactory.swift:73 (KituraWebSocket:x86_64+0x36f8)
    #4 protocol witness for ConnectionUpgradeFactory.upgrade(handler:request:response:) in conformance WSConnectionUpgradeFactory WSConnectionUpgradeFactory.swift (KituraWebSocket:x86_64+0x4e8b)
    #5 ConnectionUpgradeFactory.upgrade(handler:request:response:) ConnectionUpgradeFactory.swift:54 (KituraNet:x86_64+0x20185)
    #6 ConnectionUpgrader.upgradeConnection(handler:request:response:) ConnectionUpgrader.swift:77 (KituraNet:x86_64+0x22788)
    #7 IncomingHTTPSocketProcessor.parsingComplete() IncomingHTTPSocketProcessor.swift:245 (KituraNet:x86_64+0x8c9d0)
    #8 IncomingHTTPSocketProcessor.parse(_:) IncomingHTTPSocketProcessor.swift:222 (KituraNet:x86_64+0x8bcd0)
    #9 IncomingHTTPSocketProcessor.process(_:) IncomingHTTPSocketProcessor.swift:98 (KituraNet:x86_64+0x8a5b3)
    #10 protocol witness for IncomingSocketProcessor.process(_:) in conformance IncomingHTTPSocketProcessor IncomingHTTPSocketProcessor.swift (KituraNet:x86_64+0x8deec)
    #11 IncomingSocketHandler.handleReadHelper() IncomingSocketHandler.swift:167 (KituraNet:x86_64+0xb2d5d)
    #12 IncomingSocketHandler.handleRead() IncomingSocketHandler.swift:139 (KituraNet:x86_64+0xb0984)
    #13 closure #1 in IncomingSocketHandler.init(socket:using:) IncomingSocketHandler.swift:105 (KituraNet:x86_64+0xaff37)
    #14 partial apply for closure #1 in IncomingSocketHandler.init(socket:using:) IncomingSocketHandler.swift (KituraNet:x86_64+0xaff8d)
    #15 thunk for @callee_owned () -> () FastCGIServer.swift (KituraNet:x86_64+0x3d34c)
    #16 __tsan::dispatch_callback_wrap(void*) <null>:5353680 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x662e1)
    #17 _dispatch_client_callout <null>:5353680 (libdispatch.dylib:x86_64+0x1d1e)

  Thread T14 (tid=5477528, running) is a GCD worker thread

  Thread T1 (tid=5475065, running) is a GCD worker thread

  Thread T5 (tid=5475102, finished) is a GCD worker thread

SUMMARY: ThreadSanitizer: data race WSSocketProcessor.swift in WSSocketProcessor.inProgress.setter

WARNING: ThreadSanitizer: data race (pid=67210)
  Write of size 8 at 0x7b2400009178 by thread T14:
  * #0 IncomingSocketHandler.processor.setter IncomingSocketHandler.swift (KituraNet:x86_64+0xadd4f)
    #1 IncomingSocketHandler.handleCancel() IncomingSocketHandler.swift:402 (KituraNet:x86_64+0xbb577)
    #2 partial apply IncomingSocketHandler.swift (KituraNet:x86_64+0xbcad4)
    #3 thunk for @callee_owned () -> () FastCGIServer.swift (KituraNet:x86_64+0x3d34c)
    #4 __tsan::dispatch_callback_wrap(void*) <null>:5353696 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x662e1)
    #5 _dispatch_call_block_and_release <null>:5353696 (libdispatch.dylib:x86_64+0x9c3c)

  Previous read of size 8 at 0x7b2400009178 by thread T1:
  * #0 IncomingSocketHandler.processor.getter IncomingSocketHandler.swift (KituraNet:x86_64+0xadc43)
    #1 IncomingSocketManager.removeIdleSockets(removeAll:) IncomingSocketManager.swift:242 (KituraNet:x86_64+0xbee0a)
    #2 IncomingSocketManager.handle(socket:processor:) IncomingSocketManager.swift:137 (KituraNet:x86_64+0xbde71)
    #3 HTTPServer.handleClientConnection(clientSocket:socketManager:) HTTPServer.swift:278 (KituraNet:x86_64+0x71183)
    #4 HTTPServer.listen(listenSocket:socketManager:) HTTPServer.swift:223 (KituraNet:x86_64+0x6c09c)
    #5 closure #1 in HTTPServer.listen(on:) HTTPServer.swift:133 (KituraNet:x86_64+0x69aee)
    #6 partial apply for closure #1 in HTTPServer.listen(on:) HTTPServer.swift (KituraNet:x86_64+0x69c8c)
    #7 thunk for @callee_owned () -> () FastCGIServer.swift (KituraNet:x86_64+0x3d34c)
    #8 _dispatch_client_callout <null>:5353696 (libdispatch.dylib:x86_64+0x1d1e)
    #9 _dispatch_client_callout <null>:5353696 (libdispatch.dylib:x86_64+0x1d1e)

  Issue is caused by frames marked with "*".

  Location is heap block of size 133 at 0x7b2400009120 allocated by thread T1:
    #0 malloc <null>:5353728 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x48b3a)
    #1 swift_slowAlloc <null>:5353728 (libswiftCore.dylib:x86_64+0x344478)
    #2 IncomingSocketHandler.__allocating_init(socket:using:) IncomingSocketHandler.swift (KituraNet:x86_64+0xaea3c)
    #3 IncomingSocketManager.handle(socket:processor:) IncomingSocketManager.swift:120 (KituraNet:x86_64+0xbdb8e)
    #4 HTTPServer.handleClientConnection(clientSocket:socketManager:) HTTPServer.swift:278 (KituraNet:x86_64+0x71183)
    #5 HTTPServer.listen(listenSocket:socketManager:) HTTPServer.swift:223 (KituraNet:x86_64+0x6c09c)
    #6 closure #1 in HTTPServer.listen(on:) HTTPServer.swift:133 (KituraNet:x86_64+0x69aee)
    #7 partial apply for closure #1 in HTTPServer.listen(on:) HTTPServer.swift (KituraNet:x86_64+0x69c8c)
    #8 thunk for @callee_owned () -> () FastCGIServer.swift (KituraNet:x86_64+0x3d34c)
    #9 _dispatch_client_callout <null>:5353728 (libdispatch.dylib:x86_64+0x1d1e)
    #10 _dispatch_client_callout <null>:5353728 (libdispatch.dylib:x86_64+0x1d1e)

  Thread T14 (tid=5477528, running) is a GCD worker thread

  Thread T1 (tid=5475065, running) is a GCD worker thread

SUMMARY: ThreadSanitizer: data race IncomingSocketHandler.swift in IncomingSocketHandler.processor.setter

WARNING: ThreadSanitizer: data race (pid=67210)
  Write of size 4 at 0x7b440001a064 by thread T13:
  * #0 Socket.socketfd.setter Socket.swift (Socket:x86_64+0x1bcc6)
    #1 Socket.close(withSSLCleanup:) Socket.swift:3376 (Socket:x86_64+0x70f27)
    #2 Socket.close() Socket.swift:1557 (Socket:x86_64+0x38edc)
    #3 IncomingSocketHandler.handleCancel() IncomingSocketHandler.swift:395 (KituraNet:x86_64+0xbb034)
    #4 partial apply IncomingSocketHandler.swift (KituraNet:x86_64+0xbcad4)
    #5 thunk for @callee_owned () -> () FastCGIServer.swift (KituraNet:x86_64+0x3d34c)
    #6 __tsan::dispatch_callback_wrap(void*) <null>:1594000 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x662e1)
    #7 _dispatch_call_block_and_release <null>:1594000 (libdispatch.dylib:x86_64+0x9c3c)

  Previous read of size 4 at 0x7b440001a064 by thread T1:
  * #0 Socket.socketfd.getter Socket.swift (Socket:x86_64+0x1bc4a)
    #1 IncomingSocketManager.handle(socket:processor:) IncomingSocketManager.swift:121 (KituraNet:x86_64+0xbdbeb)
    #2 HTTPServer.handleClientConnection(clientSocket:socketManager:) HTTPServer.swift:278 (KituraNet:x86_64+0x71183)
    #3 HTTPServer.listen(listenSocket:socketManager:) HTTPServer.swift:223 (KituraNet:x86_64+0x6c09c)
    #4 closure #1 in HTTPServer.listen(on:) HTTPServer.swift:133 (KituraNet:x86_64+0x69aee)
    #5 partial apply for closure #1 in HTTPServer.listen(on:) HTTPServer.swift (KituraNet:x86_64+0x69c8c)
    #6 thunk for @callee_owned () -> () FastCGIServer.swift (KituraNet:x86_64+0x3d34c)
    #7 _dispatch_client_callout <null>:1594000 (libdispatch.dylib:x86_64+0x1d1e)
    #8 _dispatch_client_callout <null>:1594000 (libdispatch.dylib:x86_64+0x1d1e)

  Issue is caused by frames marked with "*".

  Location is heap block of size 292 at 0x7b440001a040 allocated by thread T1:
    #0 malloc <null>:1594032 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x48b3a)
    #1 swift_slowAlloc <null>:1594032 (libswiftCore.dylib:x86_64+0x344478)
    #2 Socket.__allocating_init(fd:remoteAddress:path:) Socket.swift (Socket:x86_64+0x24845)
    #3 Socket.acceptClientConnection(invokeDelegate:) Socket.swift:1402 (Socket:x86_64+0x33861)
    #4 HTTPServer.listen(listenSocket:socketManager:) HTTPServer.swift:196 (KituraNet:x86_64+0x6b2f4)
    #5 closure #1 in HTTPServer.listen(on:) HTTPServer.swift:133 (KituraNet:x86_64+0x69aee)
    #6 partial apply for closure #1 in HTTPServer.listen(on:) HTTPServer.swift (KituraNet:x86_64+0x69c8c)
    #7 thunk for @callee_owned () -> () FastCGIServer.swift (KituraNet:x86_64+0x3d34c)
    #8 _dispatch_client_callout <null>:1594032 (libdispatch.dylib:x86_64+0x1d1e)
    #9 _dispatch_client_callout <null>:1594032 (libdispatch.dylib:x86_64+0x1d1e)

  Thread T13 (tid=5475740, running) is a GCD worker thread

  Thread T1 (tid=5475065, running) is a GCD worker thread

SUMMARY: ThreadSanitizer: data race Socket.swift in Socket.socketfd.setter

WARNING: ThreadSanitizer: data race (pid=67210)
  Read of size 1 at 0x7b1c00010780 by thread T1:
  * #0 WSSocketProcessor.inProgress.getter WSSocketProcessor.swift (KituraWebSocket:x86_64+0x10b0a)
    #1 protocol witness for IncomingSocketProcessor.inProgress.getter in conformance WSSocketProcessor WSSocketProcessor.swift (KituraWebSocket:x86_64+0x13763)
    #2 IncomingSocketManager.removeIdleSockets(removeAll:) IncomingSocketManager.swift:242 (KituraNet:x86_64+0xbf1c0)
    #3 IncomingSocketManager.handle(socket:processor:) IncomingSocketManager.swift:137 (KituraNet:x86_64+0xbde71)
    #4 HTTPServer.handleClientConnection(clientSocket:socketManager:) HTTPServer.swift:278 (KituraNet:x86_64+0x71183)
    #5 HTTPServer.listen(listenSocket:socketManager:) HTTPServer.swift:223 (KituraNet:x86_64+0x6c09c)
    #6 closure #1 in HTTPServer.listen(on:) HTTPServer.swift:133 (KituraNet:x86_64+0x69aee)
    #7 partial apply for closure #1 in HTTPServer.listen(on:) HTTPServer.swift (KituraNet:x86_64+0x69c8c)
    #8 thunk for @callee_owned () -> () FastCGIServer.swift (KituraNet:x86_64+0x3d34c)
    #9 _dispatch_client_callout <null>:1058032 (libdispatch.dylib:x86_64+0x1d1e)
    #10 _dispatch_client_callout <null>:1058032 (libdispatch.dylib:x86_64+0x1d1e)

  Previous write of size 1 at 0x7b1c00010780 by thread T20:
    [failed to restore the stack]

  Issue is caused by frames marked with "*".

  Location is heap block of size 112 at 0x7b1c00010760 allocated by thread T20:
    #0 malloc <null>:1058064 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x48b3a)
    #1 swift_slowAlloc <null>:1058064 (libswiftCore.dylib:x86_64+0x344478)
    #2 WSSocketProcessor.__allocating_init(connection:) WSSocketProcessor.swift (KituraWebSocket:x86_64+0x11584)
    #3 WSConnectionUpgradeFactory.upgrade(handler:request:response:) WSConnectionUpgradeFactory.swift:73 (KituraWebSocket:x86_64+0x36f8)
    #4 protocol witness for ConnectionUpgradeFactory.upgrade(handler:request:response:) in conformance WSConnectionUpgradeFactory WSConnectionUpgradeFactory.swift (KituraWebSocket:x86_64+0x4e8b)
    #5 ConnectionUpgradeFactory.upgrade(handler:request:response:) ConnectionUpgradeFactory.swift:54 (KituraNet:x86_64+0x20185)
    #6 ConnectionUpgrader.upgradeConnection(handler:request:response:) ConnectionUpgrader.swift:77 (KituraNet:x86_64+0x22788)
    #7 IncomingHTTPSocketProcessor.parsingComplete() IncomingHTTPSocketProcessor.swift:245 (KituraNet:x86_64+0x8c9d0)
    #8 IncomingHTTPSocketProcessor.parse(_:) IncomingHTTPSocketProcessor.swift:222 (KituraNet:x86_64+0x8bcd0)
    #9 IncomingHTTPSocketProcessor.process(_:) IncomingHTTPSocketProcessor.swift:98 (KituraNet:x86_64+0x8a5b3)
    #10 protocol witness for IncomingSocketProcessor.process(_:) in conformance IncomingHTTPSocketProcessor IncomingHTTPSocketProcessor.swift (KituraNet:x86_64+0x8deec)
    #11 IncomingSocketHandler.handleReadHelper() IncomingSocketHandler.swift:167 (KituraNet:x86_64+0xb2d5d)
    #12 IncomingSocketHandler.handleRead() IncomingSocketHandler.swift:139 (KituraNet:x86_64+0xb0984)
    #13 closure #1 in IncomingSocketHandler.init(socket:using:) IncomingSocketHandler.swift:105 (KituraNet:x86_64+0xaff37)
    #14 partial apply for closure #1 in IncomingSocketHandler.init(socket:using:) IncomingSocketHandler.swift (KituraNet:x86_64+0xaff8d)
    #15 thunk for @callee_owned () -> () FastCGIServer.swift (KituraNet:x86_64+0x3d34c)
    #16 __tsan::dispatch_callback_wrap(void*) <null>:1058064 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x662e1)
    #17 _dispatch_client_callout <null>:1058064 (libdispatch.dylib:x86_64+0x1d1e)

  Thread T1 (tid=5475065, running) is a GCD worker thread

  Thread T20 (tid=5486776, finished) is a GCD worker thread

SUMMARY: ThreadSanitizer: data race WSSocketProcessor.swift in WSSocketProcessor.inProgress.getter

Listening on port

Hi!
I wanted to replicate the Kitura-TodoList project with my own classes. I've modified the provided ones and now after my new project succesfully compiles, whenever I type in browser: http://localhost:8090/messages in the browser I see: Cannot GET /messages. and in the terminal I see INFO: spiListen(_:port:) (...)/Packages/Kitura-net-0.4.0/Sources/KituraNet/HttpServerSpi.swift line 51 - Accepted connection from: 127.0.0.1on port 36054 UnexpectedEOF

Why would the server accept connection on port 36054 if I explicitly try to connect on 8090? Or maybe I'm doing sth wrong?
Any help will be appreciated!
Thanks :)

EXC_BAD_INSTRUCTION on IncomingSocketHandler.swift line 197 (only Mac OS X)

On our development server, which is running Mac OS X 10.12.6, we are getting EXC_BAD_INSTRUCTION. So far, it was not a huge problem, as we were only using OS X for development. We are currently looking at moving our production server from bluemix to OS X, so it is essential that we sort it out.

The crash happens after the server has run for a couple of hours. As the server is used for development the traffic is minimal. I am not 100% sure, but when the server crashes I believe that there is no traffic at all. So far, I have never managed to get it running for more than 1 day without getting the crash. The only way to replicate the crash is if I wait for a couple of hours. So far I have not managed to replicate it in any other way.

The crash happens on IncomingSocketHandler.swift line 197. The method that is causing the crash is handleBufferedReadData(). The method is only run when in OSX, iOS, tvOS and under GCD_ASYNC (which I am not sure what it means). That is probably the reason that we do not get it under Linux.

A screenshot from xcode has been attached for more information.

POST command fails with updates from PR 54

PR #54 added a "Connection: close" header, which cases certain POST commands using HTTP.request() to fail.

Here is the verbose output from CURL when used with HTTP.request().

*   Trying 10.0.0.1...
* Connected to a.example.com (10.0.0.1) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.example.com
> POST /exampleapi HTTP/1.1
Host: a.example.com
Accept: */*
Content-Type: application/json
Connection: close
Content-Length: 39624
Expect: 100-continue

* Done waiting for 100-continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=UTF-8
< Vary: X-Origin
< Vary: Referer
< Date: Thu, 11 Aug 2016 21:14:20 GMT
< Server: ESF
...
< Accept-Ranges: none
< Vary: Origin,Accept-Encoding
< Connection: close
< 
* SSLRead() return error -9806
* Closing connection 0

Commenting out line 290 of ClientRequest.swift allows HTTP POST requests to work.
//headers["Connection"] = "close"

To make the KituraNet HTTP class more general purpose, adding the "Connection: close" header should be optional.

Valid port numbers not representable

The ClientRequest.Options enum assumes that port numbers will be represented by a signed 16-bit integer. However, valid port numbers are actually unsigned 16-bit integers.

This means that port numbers 32768-65535 (which are supposed to be valid) cannot be represented. In particular, it means that all dynamic ports won't work.

ClientResponse body

How can I get access to ClientResponse body? Sorry, I couldn't find anything in the documentation about response's body (only status code).

Kitura-net 0.13 fails to build CHttpParser

We're trying to use the 0.13 tag of Kitura-Net in our SDKs, however it fails with below error. Note that 0.12 works without similar issues.

'Error' is ambiguous for type lookup in this context.

Hello,

When I import Kitura-net (KituraNet), I get errors like below.

'Error' is ambiguous for type lookup in this context.

KituraNet causes this error because it has a "public" enum named Error on Error.swift.

As you know, Swift 3 uses Error instead of NSError now.

So, I think it should be fixed.

EXC_BAD_ACCESS when reading HTTPServerRequest.method with SwiftMetricsKitura

I've encountered an intermittent crash when running load against a Kitura server with SwiftMetricsKitura HTTP monitoring enabled. There seem to be two distinct types of crash, though I believe their root cause may be related.

A crash happens frequently when SwiftMetrics is attempting to access the HTTPServerRequest.method value in SwiftMetricsKitura.finished().

To make it clearer which field is causing the issue, I added a little extra debug (below) to access the field outside of the HTTPData constructor:

A crash in HeadersContainer.append() where we try to free a pointer that has already been freed.

Both crashes can be eliminated by avoiding reuse of the HTTPParser, instead allocating a new one in IncomingHTTPSocketProcessor.process().

So I believe the corruption / bad behaviour is somehow related to the lifecycle of the HTTP parser. I'm currently of the opinion that SwiftMetrics itself is behaving correctly, but is exposing an issue in Kitura-net by retaining a reference to the HTTPServerRequest beyond the point at which the response has been sent.

Nil dereference timing window in removeIdleSockets

There appears to be a timing window in IncomingSocketManager.removeIdleSockets where a field can become nil between checking for nil and then force-unwrapping it:

    if !removeAll && handler.processor != nil  &&  (handler.processor!.inProgress  ||  maxInterval < handler.processor!.keepAliveUntil) {
        continue
    }

I ran into this while trying to debug a different problem (Kitura/Kitura#1034). The field handler.processor has become nil, and Xcode reports that we've attempted to dereference a nil on this line, despite explicitly checking that this is not the case.

[2017-07-19T14:28:25.257+01:00] [WARNING] [Session.swift:43 init(secret:cookie:store:)] Using default in-memory session store
[2017-07-19T14:28:25.264+01:00] [INFO] [HTTPServer.swift:117 listen(on:)] Listening on port 8080
objc[1705]: Class _NSZombie_Foundation._NSSwiftData is implemented in both ?? (0x101c0d020) and ?? (0x101809f50). One of the two will be used. Which one is undefined.
objc[1705]: Class _NSZombie_CFSet is implemented in both ?? (0x101c0eb70) and ?? (0x10180beb0). One of the two will be used. Which one is undefined.
objc[1705]: Class _NSZombie__SwiftNativeNSError is implemented in both ?? (0x1566c6f20) and ?? (0x15781ccf0). One of the two will be used. Which one is undefined.
fatal error: unexpectedly found nil while unwrapping an Optional value
2017-07-19 14:34:11.516973+0100 HelloMiddleware[1705:10981] fatal error: unexpectedly found nil while unwrapping an Optional value
Current stack trace:
0    libswiftCore.dylib                 0x000000010077f130 swift_reportError + 129
1    libswiftCore.dylib                 0x000000010079bb50 _swift_stdlib_reportFatalError + 60
2    libswiftCore.dylib                 0x000000010058c250 specialized specialized StaticString.withUTF8Buffer<A> ((UnsafeBufferPointer<UInt8>) -> A) -> A + 342
3    libswiftCore.dylib                 0x0000000100706e90 partial apply for (_fatalErrorMessage(StaticString, StaticString, file : StaticString, line : UInt, flags : UInt32) -> Never).(closure #2) + 109
4    libswiftCore.dylib                 0x000000010058c250 specialized specialized StaticString.withUTF8Buffer<A> ((UnsafeBufferPointer<UInt8>) -> A) -> A + 342
5    libswiftCore.dylib                 0x00000001006b99a0 specialized _fatalErrorMessage(StaticString, StaticString, file : StaticString, line : UInt, flags : UInt32) -> Never + 96
6    KituraNet                          0x00000001001c6410 IncomingSocketManager.removeIdleSockets(removeAll : Bool) -> () + 2424
7    KituraNet                          0x00000001001c5dd0 IncomingSocketManager.handle(socket : Socket, processor : IncomingSocketProcessor) -> () + 533
8    KituraNet                          0x00000001001e7ad0 HTTPServer.listen(listenSocket : Socket, socketManager : IncomingSocketManager) -> () + 1281
9    KituraNet                          0x00000001001e6fb0 HTTPServer.(listen(on : Int) throws -> ()).(closure #1) + 72
10   KituraNet                          0x00000001001bfb10 thunk + 39
11   libdispatch.dylib                  0x0000000100d4e0bf _dispatch_block_async_invoke_and_release + 75
12   libdispatch.dylib                  0x0000000100d38784 _dispatch_client_callout + 8
13   libdispatch.dylib                  0x0000000100d3a76d _dispatch_root_queue_drain + 1710
14   libdispatch.dylib                  0x0000000100d3a6ab _dispatch_worker_thread3 + 114
15   libsystem_pthread.dylib            0x0000000100daf2c7 _pthread_wqthread + 1299
16   libsystem_pthread.dylib            0x0000000100daf2a8 start_wqthread + 13
(lldb)

Running into SIGPIPE

Hey,

I'm using Kitura-net because I have a Linux app that's doing a lot of networking. Originally, I was using Swift Foundation's networking, however I ran into too many problems with it (the macOS version worked fine, the Swift one not). I've switched to kitura-net for my networking and it works a lot better. However, after a certain amount of requests, I'm running into a SIGPIPE:

(my LLDB session output)

* thread #1: tid = 20019, 0x00007ffff671035d libpthread.so.0`__libc_write + 45, name = 'CSPTest', stop reason = signal SIGPIPE
    frame #0: 0x00007ffff671035d libpthread.so.0`__libc_write + 45
libpthread.so.0`__libc_write:
->  0x7ffff671035d <+45>: movq   (%rsp), %rdi
    0x7ffff6710361 <+49>: movq   %rax, %rdx
    0x7ffff6710364 <+52>: callq  0x7ffff670fe80            ; __pthread_disable_asynccancel
    0x7ffff6710369 <+57>: movq   %rdx, %rax
(lldb) up
frame #1: 0x00007ffff78807a6 libcrypto.so.1.0.0`___lldb_unnamed_symbol566$$libcrypto.so.1.0.0 + 38
libcrypto.so.1.0.0`___lldb_unnamed_symbol566$$libcrypto.so.1.0.0:
    0x7ffff78807a6 <+38>: movl   $0xf, %esi
    0x7ffff78807ab <+43>: movq   %rax, %rbx
    0x7ffff78807ae <+46>: movq   %rbp, %rdi
    0x7ffff78807b1 <+49>: callq  0x7ffff787e650            ; BIO_clear_flags
(lldb) up
frame #2: 0x00007ffff787e84b libcrypto.so.1.0.0`BIO_write + 107
libcrypto.so.1.0.0`BIO_write:
    0x7ffff787e84b <+107>: testl  %eax, %eax
    0x7ffff787e84d <+109>: jle    0x7ffff787e856            ; <+118>
    0x7ffff787e84f <+111>: movslq %eax, %rdx
    0x7ffff787e852 <+114>: addq   %rdx, 0x58(%rbx)
(lldb) up
frame #3: 0x00007ffff7b9cc12 libssl.so.1.0.0`___lldb_unnamed_symbol79$$libssl.so.1.0.0 + 114
libssl.so.1.0.0`___lldb_unnamed_symbol79$$libssl.so.1.0.0:
    0x7ffff7b9cc12 <+114>: movl   0x11c(%rbx), %ecx
    0x7ffff7b9cc18 <+120>: cmpl   %ecx, %eax
    0x7ffff7b9cc1a <+122>: je     0x7ffff7b9cc90            ; <+240>
    0x7ffff7b9cc1c <+124>: testl  %eax, %eax
(lldb) up
frame #4: 0x00007ffff7b9e7f3 libssl.so.1.0.0`___lldb_unnamed_symbol85$$libssl.so.1.0.0 + 51
libssl.so.1.0.0`___lldb_unnamed_symbol85$$libssl.so.1.0.0:
    0x7ffff7b9e7f3 <+51>: movl   %eax, %ebp
    0x7ffff7b9e7f5 <+53>: movq   0x80(%rbx), %rax
    0x7ffff7b9e7fc <+60>: testl  %ebp, %ebp
    0x7ffff7b9e7fe <+62>: jle    0x7ffff7b9e890            ; <+208>
(lldb) up
frame #5: 0x00007ffff7b9acf2 libssl.so.1.0.0`___lldb_unnamed_symbol53$$libssl.so.1.0.0 + 178
libssl.so.1.0.0`___lldb_unnamed_symbol53$$libssl.so.1.0.0:
    0x7ffff7b9acf2 <+178>: movq   0x80(%rbx), %rax
    0x7ffff7b9acf9 <+185>: movl   0x1d4(%rax), %ecx
    0x7ffff7b9acff <+191>: testl  %ecx, %ecx
    0x7ffff7b9ad01 <+193>: jne    0x7ffff7b9acd0            ; <+144>
(lldb) up
frame #6: 0x00007ffff60e139e libcurl.so.4`___lldb_unnamed_symbol610$$libcurl.so.4 + 30
libcurl.so.4`___lldb_unnamed_symbol610$$libcurl.so.4:
    0x7ffff60e139e <+30>: movq   0x10(%rbx), %rdi
    0x7ffff60e13a2 <+34>: callq  0x7ffff609be60            ; symbol stub for: SSL_set_connect_state
    0x7ffff60e13a7 <+39>: movq   0x10(%rbx), %rdi
    0x7ffff60e13ab <+43>: callq  0x7ffff609c740            ; symbol stub for: SSL_free
(lldb) up
frame #7: 0x00007ffff60b238b libcurl.so.4`___lldb_unnamed_symbol168$$libcurl.so.4 + 267
libcurl.so.4`___lldb_unnamed_symbol168$$libcurl.so.4:
    0x7ffff60b238b <+267>: movq   0x60(%rbp), %rdi
    0x7ffff60b238f <+271>: callq  0x7ffff60c3da0            ; ___lldb_unnamed_symbol303$$libcurl.so.4
    0x7ffff60b2394 <+276>: testb  %al, %al
    0x7ffff60b2396 <+278>: jne    0x7ffff60b23d8            ; <+344>
(lldb) up
frame #8: 0x00007ffff60b5c02 libcurl.so.4`___lldb_unnamed_symbol184$$libcurl.so.4 + 322
libcurl.so.4`___lldb_unnamed_symbol184$$libcurl.so.4:
    0x7ffff60b5c02 <+322>: testl  %eax, %eax
    0x7ffff60b5c04 <+324>: jne    0x7ffff60b5c90            ; <+464>
    0x7ffff60b5c0a <+330>: movq   $0x0, (%r12)
    0x7ffff60b5c12 <+338>: movq   %rbp, %rdi
(lldb) up
frame #9: 0x00007ffff60c57db libcurl.so.4`___lldb_unnamed_symbol316$$libcurl.so.4 + 2411
libcurl.so.4`___lldb_unnamed_symbol316$$libcurl.so.4:
    0x7ffff60c57db <+2411>: movq   0x18(%rsp), %rdi
    0x7ffff60c57e0 <+2416>: testq  %rdi, %rdi
    0x7ffff60c57e3 <+2419>: je     0x7ffff60c5144            ; <+724>
    0x7ffff60c57e9 <+2425>: leaq   0x234258(%rip), %rax
(lldb) up
frame #10: 0x00007ffff60c5f01 libcurl.so.4`curl_multi_perform + 145
libcurl.so.4`curl_multi_perform:
    0x7ffff60c5f01 <+145>: cmpl   $-0x1, %eax
    0x7ffff60c5f04 <+148>: je     0x7ffff60c5ef0            ; <+128>
    0x7ffff60c5f06 <+150>: cmpb   $0x0, 0x6f0(%rbx)
    0x7ffff60c5f0d <+157>: je     0x7ffff60c5f2c            ; <+188>
(lldb) up
frame #11: 0x00007ffff60bd533 libcurl.so.4`curl_easy_perform + 259
libcurl.so.4`curl_easy_perform:
    0x7ffff60bd533 <+259>: testl  %eax, %eax
    0x7ffff60bd535 <+261>: jne    0x7ffff60bd590            ; <+352>
    0x7ffff60bd537 <+263>: movl   0x14(%rsp), %eax
    0x7ffff60bd53b <+267>: testl  %eax, %eax
(lldb) up
frame #12: 0x000000000076adb6 CSPTest`CurlInvoker.(ptr=<unavailable>, self=<unavailable>, rc=<unavailable>) -> CURLcode).(closure #1) + 70 at ClientRequest.swift:484

I did add a signal handler for SIGPIPE in my main.swift:

signal(SIGPIPE, SIG_IGN)

but that didn't seem to help. Is there something else I can do?

Cheers & Thanks,
Benedikt

Build failure with latest trunk Swift toolchain "swift-DEVELOPMENT-SNAPSHOT-2016-06-06-a.xctoolchain"

Kitura-Sample/Packages/Kitura-net-0.16.0/Sources/KituraNet/ClientRequest.swift:466:47: error: cannot convert value of type '(UnsafeMutablePointer!, Int, Int, UnsafeMutablePointer!) -> Int' to expected argument type '(@convention(c) (UnsafeMutablePointer?, Int, Int, UnsafeMutablePointer?) -> Int)!'
curlHelperSetOptReadFunc(handle, ptr) { (buf: UnsafeMutablePointer!, size: Int, nMemb: Int, privateData: UnsafeMutablePointer!) -> Int in
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Use ephemeral ports in KituraNet tests

The tests that create an HTTP server always bind to and listen on port number 8080. Though this is fine when we run the tests sequentially, we run into port collisions when we try to run them in parallel (using swift test --parallel). We must ideally be using ephemeral ports to avoid this issue.

The other option is to configure SO_REUSEPORT by default, but with this tests that verify the presence and absence of SO_REUSEPORT become irrelevant.

Ephemeral ports may lie in the range starting 16000 and up to 65535. Using the sysctl command on Linux and macOS, it is possible to limit them to a given range. In the context of Kitura, limiting them to a range so that the port number can be accommodated in an UInt16 essential because ClientRequest.Options.port is an UInt16.

Swift debugger does not show variable values when importing KituraNet in CLion

I have a code in CLion:

import Foundation

func d() -> String {
    let a = "Hello";
    var b = a + " ";
    b = b + "World";
    var c = b + "!"

    return c
}

print(d());

If I set breakpoint at line let a = "Hello"; and press F8, I see all variable values.
But if I add import KituraNet, then all values won't be shown.

PS: I tried on macOS Sierra

HTTP request to a Unix Socket

Hi,

I was thinking that it would be nice to be able to make HTTP request on a Unix Socket.
There are libraries which use HTTP and sockets to expose their API.

Is this something in scope for this module? If so I would like to prepare a PR with it.

Thanks

Crashes when calling CloudKit log-in.

In order to obtain a "web auth token", Apple's CloudKit (https://developer.apple.com/library/content/documentation/DataManagement/Conceptual/CloudKitWebServicesReference/SettingUpWebServices/SettingUpWebServices.html#//apple_ref/doc/uid/TP40015240-CH24-SW1) sends a redirect, which uses HTTP status code 421. However, KituraNet 's HTTPStatusCode has no case for 421, so it's init.(rawValue:) method returns nil. While KituraNet should support all known status codes, this would have been fine, had a force-unwrapped optional not been used in ClientResponse's var status didSet. Please fix the ASAP. In conjunction with the lack of support for ECDSA, it makes logging in to iCloud impossible with the current Kitura tools.

The appropriate code for the didSet would be:

didSet {
statusCode = HTTPStatusCode(rawValue: status) ?? .unknown
}

Force-unwrapped optionals should never be used in code which needs to be reliable, and a web service needs to be reliable. The only exception is functions with no inputs. Not just no arguments, no inputs of any kind. Completely isolated functions (computed properties) which always produce exactly the same value and have a unit test hooked up to verify that they produce that value. This is the only safe way to force-unwrap an optional.

swift build failing on Ubuntu 16.04

swift build is failing on Ubuntu 16.04 with the following output:

clay@clay-ellis-ubuntu-linux-vm:~/Documents/Udacity/Projects/Kitura-Net$ swift build
Fetching https://github.com/IBM-Swift/LoggerAPI.git
Fetching https://github.com/IBM-Swift/BlueSocket.git
Fetching https://github.com/IBM-Swift/CCurl.git
Fetching https://github.com/IBM-Swift/BlueSSLService.git
Fetching https://github.com/IBM-Swift/CEpoll.git
Fetching https://github.com/IBM-Swift/BlueSignals.git
Fetching https://github.com/IBM-Swift/OpenSSL.git
Cloning https://github.com/IBM-Swift/OpenSSL.git
Resolving https://github.com/IBM-Swift/OpenSSL.git at 0.3.5
Cloning https://github.com/IBM-Swift/CCurl.git
Resolving https://github.com/IBM-Swift/CCurl.git at 0.4.0
Cloning https://github.com/IBM-Swift/BlueSignals.git
Resolving https://github.com/IBM-Swift/BlueSignals.git at 0.9.48
Cloning https://github.com/IBM-Swift/BlueSSLService.git
Resolving https://github.com/IBM-Swift/BlueSSLService.git at 0.12.45
Cloning https://github.com/IBM-Swift/BlueSocket.git
Resolving https://github.com/IBM-Swift/BlueSocket.git at 0.12.56
Cloning https://github.com/IBM-Swift/CEpoll.git
Resolving https://github.com/IBM-Swift/CEpoll.git at 0.1.0
Cloning https://github.com/IBM-Swift/LoggerAPI.git
Resolving https://github.com/IBM-Swift/LoggerAPI.git at 1.7.0
Compile Swift Module 'Signals' (1 sources)
Compile Swift Module 'Socket' (3 sources)
Compile Swift Module 'LoggerAPI' (1 sources)
Compile Swift Module 'SSLService' (1 sources)
Compile CHTTPParser utils.c
Compile CHTTPParser http_parser.c
Linking CHTTPParser
Compile Swift Module 'KituraNet' (35 sources)
<module-includes>:1:10: note: in file included from <module-includes>:1:
#include "shim.h"
         ^
/home/clay/Documents/Udacity/Projects/Kitura-Net/.build/checkouts/CCurl.git--3963518514570097934/shim.h:20:9: error: 'curl/curl.h' file not found
#import <curl/curl.h>
        ^
/home/clay/Documents/Udacity/Projects/Kitura-Net/Sources/KituraNet/ClientRequest.swift:18:8: error: could not build Objective-C module 'CCurl'
import CCurl
       ^
<module-includes>:1:10: note: in file included from <module-includes>:1:
#include "shim.h"
         ^
/home/clay/Documents/Udacity/Projects/Kitura-Net/.build/checkouts/CCurl.git--3963518514570097934/shim.h:20:9: error: 'curl/curl.h' file not found
#import <curl/curl.h>
        ^
/home/clay/Documents/Udacity/Projects/Kitura-Net/Sources/KituraNet/ClientRequest.swift:18:8: error: could not build Objective-C module 'CCurl'
import CCurl
       ^
<module-includes>:1:10: note: in file included from <module-includes>:1:
#include "shim.h"
         ^
/home/clay/Documents/Udacity/Projects/Kitura-Net/.build/checkouts/CCurl.git--3963518514570097934/shim.h:20:9: error: 'curl/curl.h' file not found
#import <curl/curl.h>
        ^
/home/clay/Documents/Udacity/Projects/Kitura-Net/Sources/KituraNet/ClientRequest.swift:18:8: error: could not build Objective-C module 'CCurl'
import CCurl
       ^
<module-includes>:1:10: note: in file included from <module-includes>:1:
#include "shim.h"
         ^
/home/clay/Documents/Udacity/Projects/Kitura-Net/.build/checkouts/CCurl.git--3963518514570097934/shim.h:20:9: error: 'curl/curl.h' file not found
#import <curl/curl.h>
        ^
/home/clay/Documents/Udacity/Projects/Kitura-Net/Sources/KituraNet/ClientRequest.swift:18:8: error: could not build Objective-C module 'CCurl'
import CCurl
       ^
<module-includes>:1:10: note: in file included from <module-includes>:1:
#include "shim.h"
         ^
/home/clay/Documents/Udacity/Projects/Kitura-Net/.build/checkouts/CCurl.git--3963518514570097934/shim.h:20:9: error: 'curl/curl.h' file not found
#import <curl/curl.h>
        ^
/home/clay/Documents/Udacity/Projects/Kitura-Net/Sources/KituraNet/ClientRequest.swift:18:8: error: could not build Objective-C module 'CCurl'
import CCurl
       ^
<module-includes>:1:10: note: in file included from <module-includes>:1:
#include "shim.h"
         ^
/home/clay/Documents/Udacity/Projects/Kitura-Net/.build/checkouts/CCurl.git--3963518514570097934/shim.h:20:9: error: 'curl/curl.h' file not found
#import <curl/curl.h>
        ^
/home/clay/Documents/Udacity/Projects/Kitura-Net/Sources/KituraNet/ClientRequest.swift:18:8: error: could not build Objective-C module 'CCurl'
import CCurl
       ^
<module-includes>:1:10: note: in file included from <module-includes>:1:
#include "shim.h"
         ^
/home/clay/Documents/Udacity/Projects/Kitura-Net/.build/checkouts/CCurl.git--3963518514570097934/shim.h:20:9: error: 'curl/curl.h' file not found
#import <curl/curl.h>
        ^
/home/clay/Documents/Udacity/Projects/Kitura-Net/Sources/KituraNet/ClientRequest.swift:18:8: error: could not build Objective-C module 'CCurl'
import CCurl
       ^
<module-includes>:1:10: note: in file included from <module-includes>:1:
#include "shim.h"
         ^
/home/clay/Documents/Udacity/Projects/Kitura-Net/.build/checkouts/CCurl.git--3963518514570097934/shim.h:20:9: error: 'curl/curl.h' file not found
#import <curl/curl.h>
        ^
/home/clay/Documents/Udacity/Projects/Kitura-Net/Sources/KituraNet/ClientRequest.swift:18:8: error: could not build Objective-C module 'CCurl'
import CCurl
       ^
<unknown>:0: error: build had 1 command failures
error: exit(1): /usr/local/swift-3.1.1-RELEASE-ubuntu16.04/usr/bin/swift-build-tool -f /home/clay/Documents/Udacity/Projects/Kitura-Net/.build/debug.yaml

Can not access body of ClientResponse

I'm trying to build a request handler that performs a fairly common scenario:

router.get("/test") {
  request, response, next in
  # send .get request to another service to get a json
  # handle json received in a response
  # send back some parts of that json as a response to /test request
}

My first approach was to use Http.request method that takes a closure taking ClientResponse:

_ = Http.request("url") {
    res in
    # do something with response
  }.end()

However current implementation of ClientResponse is quite limited. There is no parsing as in case of ServerResponse, and I can not even access the body of the response (the raw buffer I guess).

Am I approaching the entire thing in a wrong way or it's just that ClientResponse needs some development? If it's the latter I'm happy to help if you provide some guidance.