finagle / featherbed Goto Github PK

Sometimes one wishes to do single call to a large number of servers. e.g. Say poll 100 server IPs at random in a DataCenter every 10 mins or whatever. http://[somehost]/ping for 100 different hosts.

The concept is an HTTP Client to some host is created, used to invoke a single HTTP GET and then that Client is no longer needed or retained.

According to the Finagle folk there is a close () method on the HTTP Client. Did not get firm confirmation that this close () must be invoked to ensure proper resource clean up. But it was implied.

In addition the close () should not called until any inflight requests are complete. Again not solidly confirmed but believed to be so.

PlayWS doc does discuss that a close () must be used in their implementation otherwise one gets a "cant' create native thread" due to resource leaking as each instantiated client creates its own dedicated thread pool.

See https://www.playframework.com/documentation/2.5.x/ScalaWS Using WSClient.

I suspect that Finagle HTTP Client also requires explicit management for those clients not created at application startup for the life of the application.

Assuming the above is all true and required then to use ephemeral one-shot HTTP Clients something like the following needs to occur.

1. Create a client to the some host:port.
2. Invoke a GET request return a Future[Response]
3. Twitter Future ensure () method to close () the client when the request eventually completes.

Pseudo code

def onshotCallAHost (url: URL)  = {
    val client = featherbed.Client (url) // create a one time use client
    val req    = client.get ("").send[Response] ()
    req.ensure { client.close () }  // Close the client when request completes, successful or not.
    req   // return the Future[Response] out into the world which down the road will complete
} 

The current featherbed.Client does not allow access to the underlying Finagle HTTP Client nor expose a delegating close () method to the underlying Finagle httpClient to implement the above sequence.

Featherbed currently ships with an existing typelevel specification of a JSON encoder for an "application/json" content type.

It would be helpful to include some documentation for handling alternative specifications such as "application/vnd.somevendor+json; version=1" (as opposed to non-JSON content type encodings such as "text/xml").

error is explained in #23.

There was a change in Twitter Common 6.34.0 that is causing a dynamic class load failure in Featherbed 0.2.1-SNAPSHOT at Finagle at 6.35.0. I'm not sure how common this is or is simply highly specific to my dependency stack (extensive) and the ensuing evictions (hellish).

However, I have found Finagle 6.38.0 is a been stable least upper bound which compiles clean and has been solid for me so far with light duty Featherbed usage.

Like to bump 0.2.1-SNAPSHOT up to Finagle 6.38.0

When implementing a new API client, it is very useful to have verbose logging of what you're sending and receiving (so you can debug & easily send logs to API vendors to report problems without resorting to some external tool). I'm happy to take a stab at adding it, but don't want to introduce a logging framework unless you agree with it. Do you prefer to add a logging framework (which one) and configure logging levels in the traditional way, or would it be better to have something like client.get(path).verboselog(true)?

I suggest the following steps:

• Introduce code coverage tool (https://codecov.io/?)
• More tests (scalacheck/specs2)
• Introduce code style standard (scalastyle)
• Benchmarks (??? if really needed)

scala> import java.net.URL
import java.net.URL
scala> val client = new featherbed.Client(new URL("http://www.wontfindmeindns.com"))
Jul 15, 2017 9:36:07 PM com.twitter.finagle.Init$$anonfun$5 apply$mcV$sp
INFO: Finagle version 6.44.0 (rev=ef94604c6db76959610eeb8fb2bb06810022061f) built at 20170421-125957
client: featherbed.Client = featherbed.Client@40b23107
scala> Jul 15, 2017 9:36:09 PM com.twitter.finagle.DnsResolver$$anonfun$apply$3$$anonfun$apply$5 apply
INFO: Failed to resolve www.wontfindmeindns.com. Error java.net.UnknownHostException: www.wontfindmeindns.com: nodename nor servname provided, or not known
Jul 15, 2017 9:36:09 PM com.twitter.finagle.InetResolver$$anonfun$toAddr$2 apply
INFO: Resolution failed for all hosts in ArraySeq((www.wontfindmeindns.com,80,Map()))
Jul 15, 2017 9:36:09 PM com.twitter.finagle.loadbalancer.LoadBalancerFactory$StackModule$$anonfun$8 apply
INFO: www.wontfindmeindns.com:80: name resolution is negative (local dtab: Dtab())
Jul 15, 2017 9:36:13 PM com.twitter.finagle.DnsResolver$$anonfun$apply$3$$anonfun$apply$5 apply
INFO: Failed to resolve www.wontfindmeindns.com. Error java.net.UnknownHostException: www.wontfindmeindns.com
Jul 15, 2017 9:36:13 PM com.twitter.finagle.InetResolver$$anonfun$toAddr$2 apply
INFO: Resolution failed for all hosts in ArraySeq((www.wontfindmeindns.com,80,Map()))
Jul 15, 2017 9:36:13 PM com.twitter.finagle.loadbalancer.LoadBalancerFactory$StackModule$$anonfun$8 apply
INFO: www.wontfindmeindns.com:80: name resolution is negative (local dtab: Dtab())
Jul 15, 2017 9:36:18 PM com.twitter.finagle.DnsResolver$$anonfun$apply$3$$anonfun$apply$5 apply
INFO: Failed to resolve www.wontfindmeindns.com. Error java.net.UnknownHostException: www.wontfindmeindns.com
Jul 15, 2017 9:36:18 PM com.twitter.finagle.InetResolver$$anonfun$toAddr$2 apply
INFO: Resolution failed for all hosts in ArraySeq((www.wontfindmeindns.com,80,Map()))
Jul 15, 2017 9:36:18 PM com.twitter.finagle.loadbalancer.LoadBalancerFactory$StackModule$$anonfun$8 apply
INFO: www.wontfindmeindns.com:80: name resolution is negative (local dtab: Dtab())
Jul 15, 2017 9:36:23 PM com.twitter.finagle.DnsResolver$$anonfun$apply$3$$anonfun$apply$5 apply
INFO: Failed to resolve www.wontfindmeindns.com. Error java.net.UnknownHostException: www.wontfindmeindns.com: nodename nor servname provided, or not known
// this will go on forever

Even worse: if you then make a request to this client, the Future never completes (with either a success or failure)

See my comment on fefa97b

[error] /Users/brandon/Documents/workspace/schoox-api-scalawrapper/src/main/scala/com/schoox/api/Api.scala:36: class Client in package featherbed cannot be accessed in package featherbed
[error]   private val client = new featherbed.Client(baseUrl)

Not an issue but help question - I'm following https://finagle.github.io/featherbed/doc/06-building-rest-clients.html to deserialize the response to case class. But I getting error In order to decode a request to com.twitter.finagle.http.Response, it must be known that a decoder exists to com.twitter.finagle.http.Response from.

1. The doc https://finagle.github.io/featherbed/doc/04-response-decoding-and-validation.html clearly says To specify that a response should be decoded, use the send[T] method to initiate the request:

import java.net.URL

import com.twitter.finagle.http.Response
import featherbed.circe._
import io.circe.generic.auto._

import scala.concurrent.{Future, Promise}
import com.twitter.util.{
  Return,
  Throw,
  Future => TwitterFuture,
  Try => TwitterTry
}
import io.finch.syntax.scalaFutures._

import scala.util.{Failure, Success, Try}

object HttpClient {

  implicit def twitterToScalaTry[T](t: TwitterTry[T]): Try[T] = t match {
    case Return(r) => Success(r)
    case Throw(ex) => Failure(ex)
  }

  implicit def twitterToScalaFuture[T](tf: TwitterFuture[T]): Future[T] = {
    val scalaPromise = Promise[T]()
    tf.respond(t => scalaPromise.complete(t))
    scalaPromise.future
  }

}

abstract class HttpClient(baseResourceLocator: String) {
  val httpClient = new featherbed.Client(new URL(baseResourceLocator))
}

http client example;

class CustomerHttpClient(baseResourceLocator: String)
  extends HttpClient(baseResourceLocator) {

  import HttpClient._

  val heartbeatEndpoint = httpClient.get("heartbeat").accept("application/json")

  private val validationEndpoint =
    (userToken: String, accessToken: String) =>
      httpClient
        .get(s"/bestbuy/online/users/$userToken")
        .withHeader("access_token", accessToken)


  def heartbeat: Future[Response] = heartbeatEndpoint.send[Response]()

  def customerIsValid: (String, String) => Future[Response] =
    (userToken: String, accessToken: String) =>
      validationEndpoint(userToken, accessToken)
        .accept("application/json")
        .send[Response]()
}

Error

[error] /Users/prayagupd/duwamish-os/chat-server/chatServerApi/src/main/scala/com/duwamish/client/CustomerHttpClient.scala:59: In order to decode a request to com.twitter.finagle.http.Response, it must be known that a decoder exists to com.twitter.finagle.http.Response from
[error] all the content types that you Accept, which is currently AcceptTypes.
[error] You may have forgotten to specify Accept types with the `accept(..)` method,
[error] or you may be missing Decoder instances for some content types.
[error]       .send[Response]()
[error]                      ^

2. Also used val heartbeatEndpoint = httpClient.get("heartbeat").accept(Coproduct."application/json") does not work.

3. I randomly added import featherbed.support.DecodeAll._ as well as I see send function expects it; but does not solve the problem.

    def send[K]()(implicit
      canBuild: CanBuildRequest[GetRequest[Accept]],
      decodeAll: DecodeAll[K, Accept]
    ): Future[K] = sendRequest[K](canBuild, decodeAll)

Similar to #70 but does not help.

Thanks.

Using the "git method" seems to fail, and featherbed is not published on maven.

In the meantime I've cloned this repo and run sbt publishLocal to get it into my local .ivy2

It'd be good to get the basic import featherbed.Client statement for featherbed into the README as well, it was a little confusing to see it fully-qualified everywhere in the docs.

Do you think featherbed should support making a websockets client?
Or do you think this belongs to another project?

@jeremyrsmith is already aware of this, but just wanted to leave this here in case someone else had trouble using the new release.

If you're using "io.github.finagle" % "featherbed_2.11" % "0.3.0", somehow tut has become a dependency.

Workaround: add "tpolecat Repo" at "http://dl.bintray.com/tpolecat/maven/" to your resolvers in build.sbt. You'll just be downloading some jars you don't actually need.

See #16 for reference

I would rather have the query string itself be agnostic to its content (rather than locking it into "foo=bar&baz=buzz" for example). Also, I don't think you can necessarily treat params in that fashion as a Map, because a particular key may occur more than once, which Map doesn't allow.

This is taken straight from the documentation and it appears that map is not a valid method on a request, which seems silly, but there it is! Maybe it has to do with how I compiled the project (See #33)?

val client = featherbed.Client(new URL("http://localhost:8888/"))
// client: featherbed.Client = featherbed.Client@4a02fb3c

val req = client.get("")
// req: client.GetRequest[shapeless.:+:[String("*/*"),shapeless.CNil]] = GetRequest(http://localhost:8888/,List(),UTF-8)

val result = Await.result { req map { r => r.contentString } }
// <console>:15: error: value map is not a member of client.GetRequest[shapeless.:+:[String("*/*"),shapeless.CNil]]
//        val result = Await.result { req map { r => r.contentString } }
//                                        ^

In a complicated API, even when handling errors explicitly, there are times when you still get a stacktrace like the one below, which doesn't help you find the error (especially in complicated workflows where the exact call that failed is hard to determine):

featherbed.request.ErrorResponse: Error response received
	at featherbed.request.RequestTypes$RequestSyntax$$anonfun$featherbed$request$RequestTypes$RequestSyntax$$handleRequest$1.apply(RequestSyntax.scala:123)
	at featherbed.request.RequestTypes$RequestSyntax$$anonfun$featherbed$request$RequestTypes$RequestSyntax$$handleRequest$1.apply(RequestSyntax.scala:84)
	at com.twitter.util.Future$$anonfun$flatMap$1.apply(Future.scala:1089)
	at com.twitter.util.Future$$anonfun$flatMap$1.apply(Future.scala:1088)
	at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:107)
       ...

At least InvalidResponse includes the reason in the message, but ErrorResponse is a complete black box if it gets thrown in a place you didn't explicitly handle it.

I'm wondering if some information from the Request and/or Response might be helpful in constructing the message (here), or if you have other suggestions on how this might be made easier (without wrapping all code in try-catches).

I'm happy to help with implementation, but wanted to see if there was a better way to handle this before I dove in.

Not found any way to configure HTTP client, in particular I need to setup request timeout.

It would be really nice to build for Scala 2.10 as well.

Currently the main thing preventing this is the use of a small macro which converts literal strings for accept into a Coproduct of singleton types. This could conceptually be replaced with usage of SingletonProductArgs, but that breaks type inference in IDEs (at least in IntelliJ) which I see as a pretty big problem. The specialized macro avoids this problem.

It's possible the macro could be made to build for 2.10 using macro-compat, but I initially haven't had success with that.

Hi, is there a plan to update featherbed to Scala 2.13?

It seems like I can only do get, post, put, delete or head requests.
In the current project I'm working on, I will need the support for the PATCH request.

Do you think it can be added easily?

Currently accept requires the use of a Shapeless Coproduct type syntax:

client.get("foo").accept[Coproduct.`"application/json","text/xml"`.T]

This is less than ideal. Probably, a better syntax could be achieved by accepting a product of singleton types and converting that to an inferred Coproduct at compile time, which would look more like:

client.get("foo").accept("application/json", "text/xml")

If that's not possible, some common types could be assigned type aliases like this (or something):

val jsonWitness = Witness("application/json")
type JSON = jsonWitness.T

val textXmlWitness = Witness("text/xml")
type `text/xml` = textXmlWitness.T

And then they could be combined using Coproduct combinator:

client.get("foo").accept[JSON :+: `text/xml` :+: CNil]

which would be slightly less objectionable. Would that be a good idea?

GET, POST, and PUT requests should support content streaming.

GET requests should allow content to be streamed from the resource.

POST and PUT requests should allow content to be streamed to the resource.

Finagle has various mechanisms to support this; featherbed should identify a good standard and add the support for that standard.

GET requests should be able to take key/value parameters, which would become the query string of the URL.

I'm trying to get rid of boilerplate methods like this one:

def fetchStuff: Future[List[Stuff]] = {
  val request = someClient.get("stuff").accept("application/json")
  request.send[List[Stuff]]
}

Tried this as well as replacing T in return type with List[T] but to no avail:

def fetch[T](endpoint: String): Future[List[T]] = {
  val request = someClient.get(endpoint).accept("application/json")
  request.send[List[T]]
}

Error:(30, 17) In order to decode a request to T, it must be known that a decoder exists to T from
all the content types that you Accept, which is currently String("application/json") :+: shapeless.CNil.
You may have forgotten to specify Accept types with the `accept(..)` method,
or you may be missing Decoder instances for some content types.
    request.send[T]

Is there any solution or should I just keep my boilerplate code?

This compiles

    val req = 
      .put("foo/bar")
      .withContent("Hello world", "text/plain")
      .accept("text/plain")

    req.send[String]()

But this doesn't

    val req = client
      .put("foo/bar")
      .accept("text/plain")
      .withContent("Hello world", "text/plain")

    req.send[String]()

with because can't find implicit CanBuildRequest instance