jroper / play-iteratees-extras Goto Github PK

Extra iteratees for Play

License: Other

Scala 100.00%

play-iteratees-extras's Introduction

Play Iteratees Extras

This is an unofficial library for extra Play iteratees that you may find useful.

Since it is unofficial, don't expect any support, bug fixes or updates in any sort of timely manner. Also don't expect any sort of backwards compatibility between releases. But do expect to find some iteratees that you may find very useful in your projects.

Currently the library contains the following tools:

Character encoding enumeratee for decoding streams of byte arrays
Some useful combinator style iteratees for parsing streams of character arrays
An error reporting enumeratee for including context to error messages when using combinators
A full JSON parser, allowing streaming parsing of JSON without writing all the JSON into memory (using a JSON enumeratee parser)

To use with Play 2.5, simply add the following library dependency to your build:

libraryDependencies += "com.typesafe.play.extras" %% "iteratees-extras" % "1.6.0"

play-iteratees-extras's People

Contributors

Stargazers

Watchers

Forkers

adelaalfar oluies jboga onsails sidneyshek richdougherty almoehi andrewlague inexplicable wertlex leerisk manuelbernhardt mikesname miraisolutions digideskio jtjeferreira npersad

play-iteratees-extras's Issues

Support Scala 2.12

Any plans on releasing a version for Scala 2.12?

JsonIteratees not working with Enumeratee.grouped

I'm trying to use your JsonIteratees to process a chunked http response of JSON Objects on the fly.
sample data:

{"key": "value1"}{"key": "value2"}{"key": "value3"}{"key": "value4"}{"key": "value5"} [etc ...]

So I want to process the stream in such way, that it passes along JsObject values each time it parses a complete key/value object.
So I'm using this Enumeratee in my pipeline which parses the JS objects and passes them on to the remaining enumeratees.

Enumeratee.grouped(play.extras.iteratees.JsonIteratees.jsSimpleObject) &>

Everything works well, but when receiving Input.EOF I get this exception:

[error]    RuntimeException: : Premature end of input, expected '{'  (Iteratee.scala:348)
[error] play.api.libs.iteratee.Iteratee$$anonfun$run$1$$anonfun$apply$10.apply(Iteratee.scala:348)
[error] play.api.libs.iteratee.Iteratee$$anonfun$run$1$$anonfun$apply$10.apply(Iteratee.scala:345)
[error] play.api.libs.iteratee.ImmediateIteratee$$anonfun$fold$2.apply(Iteratee.scala:595)
[error] play.api.libs.iteratee.ImmediateIteratee$$anonfun$fold$2.apply(Iteratee.scala:595)
[error] play.api.libs.iteratee.Execution$$anon$1.execute(Execution.scala:43)
[error] play.api.libs.iteratee.Execution$$anon$2.execute(Execution.scala:70)
[error] play.api.libs.iteratee.Execution$$anon$2.execute(Execution.scala:70)

I checked the source and the jsSimpleObject parser has line expect("{") where it expects the next char of a new object and obviously fails when receiving Input.EOF

Any idea how to fix this ?
thx !

StackOverflowError in Iteratee.scala

I am getting the following error using the reactive json parser when importing close to 100,000 items.

Could it be related to the memory leak fix in Future.flatMap ? SI-7336

play.api.Application$$anon$1: Execution exception[[ExecutionException: Boxed Error]]
    at play.api.Application$class.handleError(Application.scala:293) ~[play_2.10.jar:2.2.2-RC4]
    at play.api.DefaultApplication.handleError(Application.scala:399) [play_2.10.jar:2.2.2-RC4]
    at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$3.apply(PlayDefaultUpstreamHandler.scala:261) [play_2.10.jar:2.2.2-RC4]
    at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$3.apply(PlayDefaultUpstreamHandler.scala:261) [play_2.10.jar:2.2.2-RC4]
    at scala.Option.map(Option.scala:145) [scala-library.jar:na]
    at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2.applyOrElse(PlayDefaultUpstreamHandler.scala:261) [play_2.10.jar:2.2.2-RC4]
java.util.concurrent.ExecutionException: Boxed Error
    at scala.concurrent.impl.Promise$.resolver(Promise.scala:55) ~[scala-library.jar:na]
    at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:47) ~[scala-library.jar:na]
    at scala.concurrent.impl.Promise$KeptPromise.<init>(Promise.scala:324) ~[scala-library.jar:na]
    at scala.concurrent.Promise$.failed(Promise.scala:134) ~[scala-library.jar:na]
    at scala.concurrent.Future$.failed(Future.scala:464) ~[scala-library.jar:na]
    at play.api.libs.iteratee.internal$.eagerFuture(package.scala:27) ~[play-iteratees_2.10.jar:2.2.2-RC4]
java.lang.StackOverflowError: null
    at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$16.apply(Iteratee.scala:501) ~[play-iteratees_2.10.jar:2.2.2-RC4]
    at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$16.apply(Iteratee.scala:501) ~[play-iteratees_2.10.jar:2.2.2-RC4]
    at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$16.apply(Iteratee.scala:501) ~[play-iteratees_2.10.jar:2.2.2-RC4]
    at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$16.apply(Iteratee.scala:501) ~[play-iteratees_2.10.jar:2.2.2-RC4]
    at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$16.apply(Iteratee.scala:501) ~[play-iteratees_2.10.jar:2.2.2-RC4]
    at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$16.apply(Iteratee.scala:501) ~[play-iteratees_2.10.jar:2.2.2-RC4]

Parsing Performance Non-Linear

I'm noticing fairly substantial performance degradation when streaming large data sets of JsArray. Is this expected? My assumption was that streaming should have a fairly constant (i.e. linear, or O(n)) performance characteristics. Is this a known limitation of the library? Note that I do not notice the degradation when using either a vanilla Play Enumerator or simply the Encoding.decode() transformation. It is only noticeable when applying the JsonEnumeratees.jsArray transformation.

I cloned the repo and updated the performance test to demonstrate the issue. I've also included a helper script for generating the larger json files (assumes you are running the script from the ./src/test/resources dir.

The updated test and script are found here and here.

A printout of the result is below:

[info] Running play.extras.iteratees.JsonPerformanceTest 
Warmup: 161ms
<----- Vanilla Enum Performance Test ---->
10x: 40ms
100x: 196ms
200x: 340ms
400x: 479ms
800x: 885ms
<----- Decoding Performance Test ---->
10x: 21ms
100x: 186ms
200x: 346ms
400x: 640ms
800x: 1288ms
<----- Json Parsing Performance Test ---->
10x: 1584ms
100x: 18204ms
200x: 44301ms
400x: 129456ms
800x: 417772ms

You can also find a graphical representation of the results here (the Enumerator.fromFile and Encoding.decode() runs are on the secondary axis).

Note that I didn't include the resource files since they are fairly large, but it should be trivial to generate using the test script.

Huge thank you for putting the library together - it was exactly what I was looking for when building out a series of Json transformation services in Play. I just need to get this performance issue figured out as I'm working with fairly sizable streams (100,000s - MMs of records).

Gzip.gunzip throws java.util.zip.DataFormatException: invalid stored block lengths

Gzip.gunzip enumeratee throws "java.util.zip.DataFormatException: invalid stored block lengths" when I'm trying to decompress any content created by gzip command line tool. It works perfect with content produced by Gzip.gzip enumeratee but fails for any file created by gunzip.

Exception in thread "main" java.util.zip.DataFormatException: invalid stored block lengths
at java.util.zip.Inflater.inflateBytes(Native Method)
at java.util.zip.Inflater.inflate(Inflater.java:259)
at play.filters.gzip.Gzip$$anon$5.play$filters$gzip$Gzip$$anon$$inflateUntilNeedsInput(Gzip.scala:235)
at play.filters.gzip.Gzip$$anon$5$$anonfun$play$filters$gzip$Gzip$$anon$$step$2.apply(Gzip.scala:218)
at play.filters.gzip.Gzip$$anon$5$$anonfun$play$filters$gzip$Gzip$$anon$$step$2.apply(Gzip.scala:211)

GzipFilter falls over with NoSuchElementException: key not found: Transfer-Encoding response is not chunked

GzipFilter.isNotChunked is missing a get call on the header map.

jroper / play-iteratees-extras Goto Github PK

play-iteratees-extras's Introduction

Play Iteratees Extras

play-iteratees-extras's People

Contributors

Stargazers

Watchers

Forkers

play-iteratees-extras's Issues

Support Scala 2.12

JsonIteratees not working with Enumeratee.grouped

StackOverflowError in Iteratee.scala

Parsing Performance Non-Linear

Gzip.gunzip throws java.util.zip.DataFormatException: invalid stored block lengths

GzipFilter falls over with NoSuchElementException: key not found: Transfer-Encoding response is not chunked

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent