brikis98 / ping-play Goto Github PK

One of the features in Facebooks BigPipe is to label pagelets with different priorities. For example, the
most important pagelets above the fold, such as the top three news feed items, get the highest priority, and are rendered before anything else, no matter what order they get sent down the wire. Other pagelets above the fold are in a second priority group, data below the fold is in a third priority group, and non-essential features, such as chat, are in the lowest priority group.

I don't seem to be able to get .scala.stream templates to work using Play Java version. Have you tried this?

Could it be that build.sbt trick:
play.Keys.templatesTypes ++= Map("stream" -> "ui.HtmlStreamFormat")

play.Keys.templatesImport ++= Vector("root.ui.HtmlStream", "root.ui.HtmlStream._", "root.ui.StaticContent")

works only in Play Scala?

To track the performance of a BigPipe page, you will want to track (at least):

1. The time when HTTP request was received
2. The time when Play app sent back the first bytes of the response
3. The time when each pagelet arrived in the browser.
4. The time it took to load each pagelet's dependencies
5. The time it took to render each pagelet

There are existing tools to measure (1) and (2), but we should add server-side hooks for (3), and client-side hooks for (3), (4), and (5). The client-side hooks could even have an optional Google Analytics integration.

I'm not sure if it is a Java 8 or Scala 2.11 thing but the escaping doesn't work for me. The dashes just become dashes.

scala> "asdf-asdf".replaceAll("-", "\u002d")
res2: String = asdf-asdf

Here is my fixed Pagelet implementation that seems to work:

case class HtmlStreamPagelet(id: String, streamFuture: Future[HtmlStream]) extends Pagelet {
  override def renderServerSide(implicit ec: ExecutionContext): HtmlStream = {
    throw new UnsupportedOperationException(s"Server-side rendering is not supported for ${getClass.getName}")
  }

  override def renderClientSide(implicit ec: ExecutionContext): HtmlStream = {
    val wrapperStreamFuture = streamFuture.map { stream =>
      val escapedEnumerator = stream.enumerator.map { html =>
        Html(html.body.replaceAllLiterally("--", "\\u002d\\u002d"))
      }
      views.stream.pageletClientSide(HtmlStream.fromHtmlEnumerator(escapedEnumerator), id, PageletContentType.html)
    }
    HtmlStream.flatten(wrapperStreamFuture)
  }
}

hey,
i want to install ping-play on a new play project, i added this to my SBT

//ping-play BIG PIPE
libraryDependencies += "com.ybrikman.ping" %% "big-pipe" % "0.0.13"

TwirlKeys.templateFormats ++= Map("stream" -> "com.ybrikman.ping.scalaapi.bigpipe.HtmlStreamFormat")
TwirlKeys.templateImports ++= Vector("com.ybrikman.ping.scalaapi.bigpipe.HtmlStream", "com.ybrikman.ping.scalaapi.bigpipe._")
//END BIG-PIPE

activator compile
then
activator eclipse

but eclipse can't find any of the big pipe files.
please help .

also where can i find the bagpipe.js file ?

The only catch is that Activator expects the activator.properties file in the root of your SBT project, so when SBT runs, it'll fire up the default project, which for ping-play is the aggregate rootProject and not the sample apps. Is it possible to have two activator.properties files, one in each sample app, even though the build details are in the directory above?

(Play Java). We are observing a very interesting problem, when a runtime exception occurs and we are chunking then runtime exception is not propagated to logs, this works if I use serial mode, i.e. without using fold.

I think the problem may be related to this method:
def toChunks(): Chunks[Html] = {
val utf8 = Codec.javaSupported("utf-8")

new Chunks[Html](Writeable.writeableOf_Content(utf8, ContentTypeOf.contentTypeOf_Html(utf8))) {
  def onReady(out: Out[Html]) {
    enumerator.run(Iteratee.foreach { html =>
      if (!html.toString.isEmpty) {
        out.write(html)
      }
    }).onComplete(_ => out.close())
  }
}

}

What actually happens when RuntimeException is thrown while Iteratee is processing? Is this exception propagated? I could create a sample Play Java app demonstrating this problem but maybe you have Play Java with your pagelets implementation around. It is worth to mention I am using Play 2.3.1 and converted all your code to use it (e.g migration to twirl). Have you also observed this problem that RuntimeExceptions are not logged when in chunking mode?

Hello, first of all thanks for sharing your work, it was an eye opener :)

I re-implemented your template streaming solution (without JS / SEO needs) as seen in your slides (124 to 142) based on the latest version of your code and it works differently than in your presentation.
The HtmlStream in the body parameter gets rendered at the end of the template and I really can't find why.

I just created a views/wvyp/wvypStreaming.scala.stream :

@(body: HtmlStream)
<html>
  <head>
    <link rel="stylesheet" href="/assets/stylesheets/wvyp.css"/>
  </head>
  <body>
    <div class="wvyp">
      <h2>Who's Viewed Your Profile</h2>
      @body
    </div>
  </body>
</html>

and modified the WvypStream controller :

def index = Action {
    val wvypCountFuture = ServiceClient.makeServiceCall("wvyp")
    val searchCountFuture = ServiceClient.makeServiceCall("search")

    val wvypCountHtmlFuture = wvypCountFuture.map(str => views.html.wvyp.wvypCount(str.toInt))
    val searchCountHtmlFuture = searchCountFuture.map(str => views.html.wvyp.searchCount(str.toInt))

    val wvypStream = HtmlStream(wvypCountHtmlFuture)
    val searchStream = HtmlStream(searchCountHtmlFuture)

    val body = wvypStream.andThen(searchStream)

    Ok.chunked(views.stream.wvyp.wvypStreaming(body))
}

The 2 streams are correctly rendered as async chunks but after the end of the content of the wvypStreaming template !

<html>
  <head>
    <link rel="stylesheet" href="/assets/stylesheets/wvyp.css"/>
  </head>
  <body>
    <div class="wvyp">
      <h2>Who's Viewed Your Profile</h2>

    </div>
  </body>
</html>

<p class="wvyp-count">
  <span class="large-number">56</span>
  <span>Your profile has been viewed by <b>56</b> people in the past 3 days</span>
</p>

<p class="search-count">
  <span class="large-number">10</span>
  <span>Your have shown up in search results <b>10</b> times in the past 3 days</span>
</p>

Do you think I did something wrong or is it a "side-effect" of using play 2.3 ?

To be truly standalone, each pagelet should be able to specify its CSS and JS dependencies. The dependencies that are known statically can be concatenated & deduped by the controller and included in the head or foot of the document, as appropriate. The dependencies that are only known dynamically should be loaded by the pagelet itself. For example, the BigPipe.onPagelet method could be:

var BigPipe.onPagelet = function(placeholderId, contentId, cssDependencies, jsDependencies) { ... }

Using a tool like require.js, the onPagelet code could lod the cssDependencies and jsDepenencies before rendering this pagelet.

Java controllers no longer work in this app after the upgrade to Play 2.4. This has nothing to do with the streaming code (even a simple return ok("hi") controller won't work), but rather, something to do with using the Scala compile time dependency injection with a Java controller. I've asked a question on the Play mailing list to try to track this one down. Until it's resolved, visiting the URL for the Java example will show the following error:

[error] p.c.s.n.PlayDefaultUpstreamHandler - Exception caught in Netty
scala.MatchError: Right((play.core.routing.HandlerInvokerFactory$JavaActionInvokerFactory$$anon$14$$anon$3@43355ebf,play.api.DefaultApplication@2ac109ad)) (of class scala.util.Right)
    at play.core.server.netty.PlayDefaultUpstreamHandler.messageReceived(PlayDefaultUpstreamHandler.scala:150) ~[play-netty-server_2.11-2.4.0.jar:2.4.0]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.3.Final.jar:na]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.3.Final.jar:na]
     at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.3.Final.jar:na]
        at com.typesafe.netty.http.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:62) [netty-http-pipelining-1.1.4.jar:na]
 at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.10.3.Final.jar:na]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.3.Final.jar:na]
     at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.3.Final.jar:na]
        at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) [netty-3.10.3.Final.jar:na]
       at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.3.Final.jar:na]

When a pagelet arrives in the browser, check the location of its placeholder div. If that div is not currently visible (e.g. it's below the fold), don't waste CPU cycles rendering the content. Instead, attach a scroll event handler and only render the content if/when the user scrolls to it. This can dramatically reduce load time for tall pages with lots of content/images below the fold.

I am trying to extend the loader/Filters.java with the HTMLCompressorFilter from

https://github.com/mohiva/play-html-compressor

and standard Play GzipFilter, but do not see any change in the returned HTML code.

It might be possible, that chunked encoding does not support a combination with gzip-encoding, but a text-compressed HTML code should be possible.

Hi Jim,

After implementing in one of my projects some of the concepts that you presented at ping conf, I wanted to run your example.

When launched the first time after a play clean, the JVM shuts down due to an OutOfMemoryError. When relaunching after that, it works correctly.

This might help you:

[info] play - Application started (Dev)
[info] play - Starting application default Akka system.
Uncaught error from thread [play-akka.actor.default-dispatcher-5] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enable
d for ActorSystem[play]
java.lang.OutOfMemoryError: PermGen space
[ERROR] [01/31/2014 13:00:36.096] [play-akka.actor.default-dispatcher-5] [ActorSystem(play)] Uncaught error from thread [play-akka
.actor.default-dispatcher-5] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
java.lang.OutOfMemoryError: PermGen space
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at ui.Pagelet$.render(Pagelet.scala:46)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at ui.Pagelet$.renderStream(Pagelet.scala:58)
        at ui.Pagelet$.render(Pagelet.scala:46)
        at ui.Pagelet$.renderStream(Pagelet.scala:58)
        at ui.Pagelet$$anonfun$renderStream$1.apply(Pagelet.scala:70)
        at ui.Pagelet$$anonfun$renderStream$1.apply(Pagelet.scala:70)
        at ui.Pagelet$$anonfun$renderStream$1.apply(Pagelet.scala:70)
        at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:253)
        at ui.Pagelet$$anonfun$renderStream$1.apply(Pagelet.scala:70)
        at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:253)
        at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:249)
        at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:249)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:29)
        at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:29)
        at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
        at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
        at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
        at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
        at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
        at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
        at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
        at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
        at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Uncaught error from thread [play-akka.actor.default-dispatcher-6] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[play]
[ERROR] [01/31/2014 13:00:42.981] [play-akka.actor.default-dispatcher-6] [ActorSystem(play)] Uncaught error from thread [play-akjava.lang.OutOfMemoryError: PermGen space
ka.actor.default-dispatcher-6] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
java.lang.OutOfMemoryError: PermGen space

This is because the base image, jeanblanchard/docker-busybox-java, is old, and apparently the image it depends on, progrium/busybox, has changed the download URL for packages. As a result, bash fails to install, and the test code cannot run. I've filed jeanblanchard/docker-busybox-java#8 to request a new build of that image.

Fantastic webinar and exciting technology!

I tried to update the project to Play 2.4.x at

https://github.com/mantsak/ping-play/tree/2.4.x

My knowledge is not that deep to get the application running. I changed the Controller classes from objectto class, but there is still some coding left. It would be great if this repositary can be updated. If you implement the changes in your code, I will delete my fork. Or update my fork with a pull request to keep your version according to your presentation.

hi, i want to learn more about ping-play, so i want to place sample-app-common from the example inside my java application, since i don't have experience in scala.
i am a bit struggling with the conversion, is it possible to have also a java version ?

thanks

I noticed a funny behaviour when hitting reload on the browser very fast.

The Promises take up to a second and when I hit reload faster, they get spooled in the background.
This is a normal behaviour, since the code is still working asynchronously to get those respoonses.

When I wait another 5 seconds for a new reload I receive all those spooled promises back in the new requested page. So, this last page contains not only the regular requested Pagelets, but also many Pagelets from previous (at that time incomplete) requests.

This might be a security issue, if another user receives responses who were originally intended for another user.

A solution might be to "kill" previous Promises at the beginning/end of the Controller, but this probably has to be request or user based.

I'm still watching the behaviour and implemented some logging with timestamps in the Controller to see what really happens.

Could you please provide license for this project because we would like to use parts of your work in our project and we are unsure if we are allowed to do so?

I have performed some work locally to make this work with latest Play if one needs help ask me, in a nutshell what is required in build.sbt:

TwirlKeys.templateImports in Compile ++= Vector("root.ui.HtmlStream", "root.ui.HtmlStream._")

TwirlKeys.templateFormats in Compile += ("stream" -> "ui.HtmlStreamFormat")

If anybody has problems mention this here I can walk you though changes needed to be Play 2.3.x compatible.

they've migrated to akka streams.. lib doesn't work anymore :/

In my app I have pages that use BigPipe and pages that don't.

I don't want to duplicate basic template tag in both format. e.g. analytics.scala.stream / analytics.scala.html or footer.scala.stream / footer.scala.html

In a classic html template I use the standard syntaxe :

in index.scala.html : @footer()
in build.sbt : TwirlKeys.templateImports ++= Seq("tags._")

If I do the same in detail.scala.stream I get the following error :

not found: value footer

I can use something like that

@{
  HtmlStream.fromHtml(views.html.tags.footer())
}

but it's a bit ugly and I have the feeling we can do better.

Next level to the question : I have a template with header/menu/footer. I want to use it for stream and html pages.

@(bigPipe   : BigPipe
, enterprise: Pagelet
, investors : Pagelet
, comments  : Pagelet
, documents : Pagelet
)(implicit request: RequestHeader, lang:Lang)

@scripts = {
    <script src="/assets/com/ybrikman/ping/big-pipe.js"></script>
}


@main("meta.application.index.title"
    , "meta.application.index.description"
    , "meta.application.index.keyword"
    , scripts) {


    <h1>@{m("meta.project.detail.title")}</h1>

        @bigPipe.render { pagelets =>
          <table class="wrapper">
            <tr>
              <td>@pagelets(enterprise.id)</td>
              <td>@pagelets(investors.id)</td>
            </tr>
            <tr>
              <td>@pagelets(comments.id)</td>
              <td>@pagelets(documents.id)</td>
            </tr>
          </table>
        }
}

The above doesn't work for the same reason : not found: value main
If I try to use the following I get : expected start of definition

@HtmlStream.fromHtml(views.html.main("meta.application.index.title"
    , "meta.application.index.description"
    , "meta.application.index.keyword"
    , scripts) {


    <h1>@{m("meta.project.detail.title")}</h1>

        @bigPipe.render { pagelets =>
          <table class="wrapper"> <!-- expected start of definition here -->
            <tr>
              <td>@pagelets(enterprise.id)</td>
              <td>@pagelets(investors.id)</td>
            </tr>
            <tr>
              <td>@pagelets(comments.id)</td>
              <td>@pagelets(documents.id)</td>
            </tr>
          </table>
        }
})