When scraping this link I get data like the string below: <a href="http://www.inc.com/

The result seems correct for me. The code: <div class="highlight highlight-source-

Strange encoding when scraping Inc.com about readability HOT 6 OPEN

OKNoah commented on June 8, 2024

Strange encoding when scraping Inc.com

from readability.

Comments (6)

cuecusp commented on June 8, 2024 1

When I use this code, I don't get absolute image url. I think when response.body is passed to readability it doesn't understand the main url. How to solve it?

from readability.

OKNoah commented on June 8, 2024

I think this is caused by JSDOM not decrypting a compressed page.

Related to this: jsdom/jsdom#648

from readability.

luin commented on June 8, 2024

The result seems correct for me. The code:

read('http://www.inc.com/gene-marks/the-one-way-to-tell-if-you-re-a-successful-entrepreneur.html?cid=sf01001',
function(err, article, meta) {
  console.log(article.content);
});

And output:

<div class="article-body inc_editable" data-editable="true" data-editor-class="InlineTextEditor" data-label="Body" data-content-type="article" data-content-id="59175" data-fieldname="inc_clean_text"><p>I'm not a very successful entrepreneur.&nbsp; Are you?</p><span>...

from readability.

OKNoah commented on June 8, 2024

Ah, OK I think the problem was I scraped the data with JSDOM then used readability on the HTML.

from readability.

OKNoah commented on June 8, 2024

Nope, I still see the results above.

I now use Needle to decompress the page first, and that works.

from readability.

chrisribe commented on June 8, 2024

Yep using Needle to get the content works better.
Here is the link:
https://github.com/tomas/needle

And a code example:

var needle = require('needle');
var read = require('node-readability');

var url = "http://www.inc.com/gene-marks/the-one-way-to-tell-if-you-re-a-successful-entrepreneur.html?cid=sf01001";
needle.get(url, function(error, response) {
  if (!error && response.statusCode == 200){

        read(response.body,
        function(err, article, meta) {
          var dom = article.document;
          var html = '<html><head><meta charset="utf-8"><title>'+dom.title+'</title></head><body><h1>'+article.title+'</h1>'+article.content+'</body></html>';
          console.log(html);
        });
  }
});