Introduce ignoreContentType for JsoupBrowser about scala-scraper HOT 4 CLOSED

oh, so easy. Thanks, it is working!

from scala-scraper.

Hi @parius! What exactly are you trying to do? Jsoup cannot read PDF files, so this restriction seems reasonable. Or is it the case that the website is sending a wrong Content-Type header?

from scala-scraper.

While parsing a page, I would like to follow a link on it (e.g. https://de.indeed.com/rc/clk?jk=b30cb1d8cced1cdf) but I don't know yet what is there. If it would be not html, I would like to get the location (final permanent link), content type and be able to dump it on disk.

With pure JSoup and ignoreContentType I could achieve this. I was thinking of extending JsoupBrowser, but it not that open for this case

from scala-scraper.

Got it 👍 You can extend JsoupBrowser for that, though:

import net.ruippeixotog.scalascraper.browser.JsoupBrowser
import org.jsoup.Connection

val browser = new JsoupBrowser {
  override def requestSettings(conn: Connection) = conn.ignoreContentType(true)
}

See #60 for another example. Let me know if this helps.

from scala-scraper.