facebookarchive / nifty Goto Github PK
View Code? Open in Web Editor NEWThrift on Netty
License: Apache License 2.0
Thrift on Netty
License: Apache License 2.0
This will make sure that people trying out facebook/swift 0.14.0-SNAPSHOT can get the dependent JARs easily. For now had to manually clone the repo and build locally.
excuse my poor english , i google this question for a long time ,but nothing can clear about this. expect your answers~ thanks
There is a small mistake on the README.md
// Create the server transport
final NettyServerTransport server = new NettyServerTransport(serverDef,
new NettyConfigBuilder(),
new DefaultChannelGroup(),
new HashedWheelTimer());
replace by
// Create the server transport
final NettyServerTransport server = new NettyServerTransport(serverDef,
new NettyServerConfigBuilder(),
new DefaultChannelGroup(),
new HashedWheelTimer());
I'm sending a thrift struct of < 300MB.
libraryDependencies ++= Seq(
"com.facebook.nifty" % "nifty-core" % "0.18.0",
"com.facebook.nifty" % "nifty-client" % "0.18.0",
"com.facebook.swift" % "swift-service" % "0.18.0",
"com.facebook.swift" % "swift-codec" % "0.18.0",
"com.facebook.swift" % "swift-annotations" % "0.18.0"
)
Using java8 VM, oracle JDK.
import io.airlift.units.DataSize.Unit.GIGABYTE;
import io.airlift.units.DataSize
val serverConfig = new ThriftServerConfig()
.setBindAddress(config.schedulerEndpoint.ip)
.setPort(config.schedulerEndpoint.port)
.setMaxFrameSize(new DataSize(2, GIGABYTE)) // the default is 64MB
val server = new ThriftServer(processor, serverConfig)
Runtime.getRuntime.addShutdownHook(new Thread() {
override def run() = {
Try(server.close) match {
case Success(p) => logger.info(s"Successfully stopped server")
case Failure(exn) => logger.error(s"Error stoping service $exn")
}
}
});
server.start
org.jboss.netty.handler.codec.frame.TooLongFrameException: Maximum frame size of -2147483648 exceeded
at com.facebook.nifty.codec.DefaultThriftFrameDecoder.tryDecodeFramedMessage(DefaultThriftFrameDecoder.java:102)
at com.facebook.nifty.codec.DefaultThriftFrameDecoder.decode(DefaultThriftFrameDecoder.java:68)
at com.facebook.nifty.codec.DefaultThriftFrameDecoder.decode(DefaultThriftFrameDecoder.java:33)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at com.facebook.nifty.codec.DefaultThriftFrameCodec.handleUpstream(DefaultThriftFrameCodec.java:42)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at com.facebook.nifty.core.ChannelStatistics.handleUpstream(ChannelStatistics.java:79)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:124)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:124)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
import com.facebook.nifty.codec.{
ThriftFrameCodecFactory,
DefaultThriftFrameCodec}
import org.apache.thrift.protocol.TProtocolFactory
import org.jboss.netty.channel.ChannelHandler
val frameFactory = new ThriftFrameCodecFactory(){
override def create(maxFrameSize: Int,
defaultProtocolFactory: TProtocolFactory): ChannelHandler = {
import com.google.common.base.Verify.verify
verify(maxFrameSize > 0, s"Frame size ($maxFrameSize) is negative!")
new DefaultThriftFrameCodec(size, defaultProtocolFactory)
}
}
import com.facebook.nifty.core.NiftyTimer
import com.google.common.collect.ImmutableMap
import org.apache.thrift.protocol.TBinaryProtocol
import com.facebook.nifty.codec.ThriftFrameCodecFactory
import com.facebook.nifty.duplex.TDuplexProtocolFactory
val server = new ThriftServer(processor,
serverConfig,
new NiftyTimer("thrift"),
ImmutableMap.of("framed",
frameFactory.asInstanceOf[ThriftFrameCodecFactory]),
ImmutableMap.of("binary",
TDuplexProtocolFactory.fromSingleFactory(new TBinaryProtocol.Factory())),
ThriftServer.DEFAULT_WORKER_EXECUTORS,
ThriftServer.DEFAULT_SECURITY_FACTORY)
However, this fails trying to create teh ChannelHandler
verify(maxFrameSize > 0, s"Frame size ($maxFrameSize) is negative!")
I'm using the thrift 0.9.2 compiler for python, the swift 0.18.0 compiler for java.
Any tips, suggestions would be greatly appreciated. Note that this doesn't happen between a C++ service and the python client.
Thanks in advance!
does it still active for this project, I saw there's no update for a while.
The ThriftServerModule binds a nifty timer out of a Provides method:
@Provides
@ThriftServerTimer
@Singleton
public Timer getThriftServerTimer()
{
return new NiftyTimer("thrift");
}
and the Timer is annotated with @PreDestroy. However, it gets never hooked in the lifecycle and therefore when the server is shut down, it leaks its timer. This is not a big deal for an actual server but for Unit tests which start and stop a lot of services, they accumulate timers (and associated resources).
I am using Swift and setting up a thrift client using Nifty HttpClientConnector in the following way.
HttpClientConnector connector = new HttpClientConnector(URI.create("http://xxxxx:8181/trackerservice"));
ThriftClientManager clientManager = new ThriftClientManager(new ThriftCodecManager());
thriftClient = clientManager.createClient(connector, TrackerService.class).get();
When calling remote procedures which returns a large result set I get an exception:
org.apache.thrift.transport.TTransportException: org.jboss.netty.handler.codec.frame.TooLongFrameException: HTTP content length exceeded 16777216 bytes
How can I change max frame size on my connection?
For example, how long a request spends in the queue until it starts being processed
The RSA cert checked into netty-examples/src/main/resources/rsa.crt will expire in early 2017, which will probably cause the test to start failing. We should regenerate the cert with a longer expiration (say, 10000 days) before then.
The nifty-core-fbcode
module has a dependency on the internal com.facebook:thrift
artifact. This has to be resolved before the open source release.
Will this project continue to be updated?
I am basing this question on the fact that netty is thrift version independent. There are some breaking changes when you shift thrift versions.
In order to monitor a service, a number of counters need to be readable. Currently, almost nothing is accessible. Active_worker_threads and request_queue_size can be read using the ThreadPoolExecutor.
Num_connection is accessible nettyServerTransport.getMetrics().getChannelCount(), but only after hacking the visibility of some field with reflection. Average_queue_time and num_killed_requests (because of expireTimeout) are not accessible at all.
Please provide an API to easily access to the following counters:
We haven't looked into whether this functionality from apache works on nifty yet, we should check whether it works and make any necessary adjustments.
Name of all boss threads is "nifty-server-boss-%d". Name of all worker threads is "nifty-server-worker-%d". I can't add identifier for the threads. How can I change the name of the threads? I need a way to set niftyName of NettyConfigBuilderBase.
It seems NiftyDispatcher does not catch RejectedExecutionException if submit() fails. This could happen, say, the exe configured by the application code is using a bounded queue. Without this we cannot prevent a thrift server already having performance issues from going into an OOM state, further exacerbating the problem.
What is the recommended way of adding SSL to Nifty?
Are there any plans to upgrade netty version?
Currently any exception thrown in the channel pipeline will close the client connection. There are some types of exceptions that we could respond to with a TApplicationException instead of closing the channel (e.g. TooLongFrameException... we can't send the correct response if it's too long, but we can send a TApplicationException indicating the method failed instead of closing the channel.
libthrift support non-blocking and asynchronous processor in - 0.9.1. Right now nifty support TNonblockingServerSocket?
please excuse my poor English! i expect your answer~tks.
Client object can only connect to one host and generally we initiate a single client and use that across client application. But usual practice has multiple set of servers for every service. Is there a support for dynamic host set in the client object with zookeeper monitor? Or is it going to supported?
Does nifty plan to support Thrift0.9.1?
Sometimes server fails with the following exception:
EXCEPTION, please implement org.jboss.netty.handler.codec.http.HttpChunkAggregator.exceptionCaught() for proper handling.
org.jboss.netty.channel.ConnectTimeoutException: connection timed out: /10.11.10.35:8082
at org.jboss.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:137)
I would be nice to be able to handle it as netty suggests
Netty 4 is nearing completion so, it would be lovely to see that up for use.
Ubuntu 12 clean install nifty using maven 3.1,errors occur:
Caused by: java.lang.ClassNotFoundException: org.sonatype.aether.version.VersionConstraint
at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:244)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:230)
... 64 more
What is the best way to expose Channel Statistics for gauging metrics (to ultimately log to Graphite)? Ideally, this would be implemented without modifying or duplicating existing Nifty classes.
how can i get the client ip address from my service implement?
TNiftyAsyncClientTransport has a close() method but doesn't implement Closeable.
if my project use jdk7 , how I use nifty
private void onSendTimeoutFired(Request request)
{
cancelAllTimeouts();
WriteTimeoutException timeoutException = new WriteTimeoutException("Timed out waiting " + getSendTimeout() + " to send data to server");
fireChannelErrorCallback(request.getListener(), new TTransportException(TTransportException.TIMED_OUT, timeoutException));
}
Why cancel all timeouts when one request timeout?
nifty-core/src/main/java/com/facebook/nifty/core/NiftyExceptionLogger.java contains the following line:
log.error(exceptionEvent.getCause(), "Exception triggered on channel connected to %s", remoteAddress);
It is probably not at the error severity to log a broken connection. It would show up as ERROR on log4j based logs. This class of severity is usually reserved for things that shouldn't happen in production, but broken connection happens all the time in normal operations.
Is there a way to add compression to transport layer?
Upgrading netty
NiftyClient currently has a shutdown method. If you were to add Closable to Nifty and rename this method to close, the NiftyClient could be used in Java 7 try with resources statements.
It would be helpful if the connection refused exception message contained the target host and port. For example:
org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
at com.facebook.nifty.client.NiftyClient.connectSync(NiftyClient.java:117)
....
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
... 1 more
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.