Hello I am trying to remove some elements How can I achive that like JSoup rem

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

Hi again <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard

No remove method? about scala-scraper HOT 2 CLOSED

ruippeixotog commented on May 27, 2024

No remove method?

from scala-scraper.

Comments (2)

ruippeixotog commented on May 27, 2024

Hi @emrekanca! Sorry for the long delay. Currently scala-scraper only scrapes web pages to an immutable model - there are no methods to modify the elements, only to retrieve them. For now you should use jsoup directly or use scala-scraper, convert it to HTML with .outerHtml and parse it using jsoup in order to modify it.

from scala-scraper.

ruippeixotog commented on May 27, 2024

Hi again @emrekanca. Through the new typed element API introduced in version 2.0.0-RC2, it is now possible to remove elements by accessing the underlying HTML library:

scala> val browser = JsoupBrowser.typed()
browser: net.ruippeixotog.scalascraper.browser.JsoupBrowser = net.ruippeixotog.scalascraper.browser.JsoupBrowser@3868bd2b

scala> val doc = browser.parseString("""<div><p>abc</p><span>def</span>""")
doc: net.ruippeixotog.scalascraper.browser.JsoupBrowser.JsoupDocument =
JsoupDocument(<html>
 <head></head>
 <body>
  <div>
   <p>abc</p>
   <span>def</span>
  </div>
 </body>
</html>)

scala> val elem = doc >> pElement("div")
elem: net.ruippeixotog.scalascraper.browser.JsoupBrowser.JsoupElement =
JsoupElement(<div>
 <p>abc</p>
 <span>def</span>
</div>)

// This is browser specific. There will be an equivalent operation in HtmlUnit
scala> elem.underlying.children.select("span").remove()
res1: org.jsoup.select.Elements = <span>def</span>

scala> doc
res2: net.ruippeixotog.scalascraper.browser.JsoupBrowser.JsoupDocument =
JsoupDocument(<html>
 <head></head>
 <body>
  <div>
   <p>abc</p>
  </div>
 </body>
</html>)