Asynchronous browser#get? about scala-scraper HOT 4 OPEN

In some common situations which there are a large number of concurrent jobs, simply wrapping ".get" inside a Future seems problematic as the Future would take a spot in execution context's thread-pool. Is there any plan for building an async version? If not, I can get started and make a pull request.

from scala-scraper.

I'm planning on working this weekend on extending the Browser interface to support async browser implementations and providing a naive factory method that wraps a non-async Browser by wrapping calls in futures. Once I do that, a pull request with a proper async implementation would be very helpful :)

from scala-scraper.

Hi @frankandrobot! It's a good point and the Browser implementations are probably not thread-safe. I'll get to improve both the browser implementations and the API later.

Nevertheless, you can easily load several pages in parallel by just wrapping the operations in a Future, like:

import scala.concurrent.ExecutionContext.global
import scala.concurrent.Future

val doc: Future[Document] = Future { JsoupBrowser().get("http://example.com") }

You can implement your custom loader that returns a Document, you really just need to implement its interface (location, root and toHtml methods). You will probably have less work if you use jsoup or HtmlUnit for parsing the HTML though, as scala-scraper already provides Document instances wrapping the models for these libraries.

from scala-scraper.

from scala-scraper.