spekulatius / phpscraper Goto Github PK

This library is locked to Symfony 5, I'll submit a PR to allow Symfony 6 as well.

It would be beneficial to return the URL of the sitemap.xml file directly.

See https://symfony.com/doc/current/http_client.html#concurrent-requests

Is there a way to click on links using CSS selectors instead of name or URL?

Hello,

How to make a condition if the timeout exceeds a certain number of seconds?

Thank

Symfony\Component\HttpClient\Exception\TransportException: SSL certificate problem: unable to get local issuer certificate for ***

Getting an error when trying to get data from a link that doesn't have ssl

Details TBC

It would be great if the most common link types could be interpreted directly. To mind jump:

• http / https
• mailto
• ftp
• tel
• tg
• # - page internal anchors
• javascript

Should we split the library and docs into 2 different repos?
Because when installing the library, we only use the phpscraper.php file, and the website directory is not necessary.

The image URL should be returned, if this the only part of the anchor texts.

spekulatius/phpscraper 0.5.2 requires symfony/dom-crawler ^5.0 -> found symfony/dom-crawler[v5.0.0, ..., v5.4.3] but the package is fixed to v6.0.3 (lock file version) by a partial update and that version does not match. Make sure you list it as an argument for the update command.

Hi

Im getting an

SSL connect error for "https://bbc.in/3cNMnkw".

when connecting to some websites. I'm writing a tweetdeck style system and so will scrape the site that the tweet links to to pick up the details so that a nicer link can be generated. Any ideas on how to solve this ? Most work fine, its just a few that fail.

Also Im sometimes getting a "This browser is no longer supported" coming through as a description from the page. I notice you are passing through 'Mozilla/5.0 (compatible; PHP Scraper/0.x; +https://phpscraper.de)' as the user agent. Is it worth altering the user agent ? I dont think I can pass it in as a parameter anywhere from what I can see?

Thanks.

When i run the sample code:

$web = new \Spekulatius\PHPScraper\PHPScraper();
$web->go('https://www.google.com/');
echo $web->title;

It return:

Spekulatius\PHPScraper\Core::setHttpClient(): Argument #1 ($httpClient) must be of type Symfony\Component\HttpClient\CurlHttpClient, Symfony\Component\HttpClient\NativeHttpClient given, called in C:\www\web-crawer\vendor\spekulatius\phpscraper\src\PHPScraper.php on line 108

Environments

PHP: 8.1.13
PHPScraper: 1.0.1

In my branch I've removed the magic __get and __call methods, and moved what was core into the phpscraper, so now there is only one class.

After a while I got tired of find/replace, so I created a rector rule to change the properties to method.

Is there a demo repository that uses I can use to test my branch? Tests are passing, except for the ones related to internal/external links, which I'll address in another issue.

How do I check if the scraped page is an error page?
It would be helpful to have the http status for that - something like

$web->go('https://httpstat.us/404');
echo $web->status;     // prints 404

Just before posting, I found the solution myself:

$web->client->getResponse()->getStatusCode()

It would be great to have this added to the documentation.
Or maybe add $web->status as shortcut?

Allow to receive various feeds sites might provide:

• rss
• sitemap.xml
• index.json
• ?

Inclusive the respective content.

Could be implemented similar to CSV (simple & header).

I wanted to know the response code of the url but could not get access.

Looking at Goutte under the hood, I think I could get it if I could do something like:

$web->getClient()->getInternalResponse()->getStatusCode()

So, I would like a function to get access to the $client, like:

public function getClient()
{
   return $this->client;
}

Is there data scraping using classes as a reference, or DOM structure? Thank you.

Navigating through web pages works wonderful.

But i dont see a way for those ones

<a href="#" onclick="Postback('xxxx','2');return false;">2</a>

Previously, there was an issue about exposing the Goutte/Client via core that was fixed but never released.

My code was depending on it via a dev branch.

In the latest version, this functionality was removed. I recommend that it be added back in.

In UsesGoutte.php,

/**
 * Retrieve the client
 *
 * @param \Goutte\Client $client
 */
public function client(): GoutteClient
{
    return $this->client;
}

I am using this example https://phpscraper.de/examples/extract-keywords.html with PHP 8.1.8

Deprecated: strpos(): Passing null to parameter #1 ($haystack) of type string is deprecated in /Users/khanakia/D1/www/php/scrap_php/vendor/spekulatius/phpscraper/src/phpscraper.php on line 858
PHP Deprecated:  strpos(): Passing null to parameter #1 ($haystack) of type string is deprecated in /Users/khanakia/D1/www/php/scrap_php/vendor/spekulatius/phpscraper/src/phpscraper.php on line 859

#16 proposes adding support for JSONLD.

There isn't only JSONLD - structured data can be provided also in the microdata notation, and: good news - there is a project which parses microdata and converts it to the same data structure as JSONLD: https://github.com/yusufkandemir/microdata-parser

So it should be possible to use both and treat it just like an additional JSONLD block!

A first test:

$jsonlddata = \YusufKandemir\MicrodataParser\Microdata::fromHTML($web->client->getResponse()->getContent(), $web->currentUrl())->toArray();

Internally this project uses an own DOM document class derived from DOMDocument. It has a function to import a DOMDocument - but Symphonys response class doesn't allow to access the DOMDocument.

I did a small test, but didn't fiddle it out how to pass the DOMDocument without reparsing - my try which didn't work:

$dom = new \DOMDocument('1.0', 'UTF-8');
$dom->importNode($web->filterFirst('//*')->getNode(0), true);
$jsonlddata = \YusufKandemir\MicrodataParser\Microdata::fromDOMDocument($dom);

What makes sense?
Adding separate PHPScraper functions for JSONLD and microdata? Or mixing both automatically? (my opinion: mixing)

How should support for microdata look like? Adding the other project to PHPScraper? Extending the existing classes or porting the whole functionality to PHPScraper?

Problem

Running composer require spekulatius/phpscraper fails on macOS 10.15.5 with the following:

[RuntimeException]                                                                                                                                                                                                          
 Failed to extract spekulatius/phpscraper: (50) '/usr/bin/unzip' -qq '/xxx/xxx/vendor/composer/tmp-123dcc14bbc3272649fdd489b0ecc9a3' -d '/xxx/xxx/vendor/composer/74030a62'  
                                                                                                                                                                                                                            
error:  cannot create /xxx/xxx/vendor/composer/74030a62/spekulatius-PHPScraper-fb4f590/tests/resources/assets/katze-+?-++-+?.jpg                                                                    
        Illegal byte sequence                                                                                                                                                                                               
error:  cannot create /xxx/xxx/vendor/composer/74030a62/spekulatius-PHPScraper-fb4f590/tests/resources/assets/???.jpg                                                                               
        Illegal byte sequence                                                                                                                                                                                               
error:  cannot create /xxx/xxx/vendor/composer/74030a62/spekulatius-PHPScraper-fb4f590/websites/test-pages/assets/katze-+?-++-+?.jpg                                                                
        Illegal byte sequence                                                                                                                                                                                               
error:  cannot create /xxx/xxx/vendor/composer/74030a62/spekulatius-PHPScraper-fb4f590/websites/test-pages/assets/???.jpg                                                                           
        Illegal byte sequence                                                                                                                                                                                               

It seems like the unzip command within composer fails due to the strange characters.

I'm working on a project that needs to be constantly changing proxies and taking a look in the Goute client, the underlying implementation already supports it.

Research

1. HttpBrowser, which Goute is based on, supports a custom client, implementing the HttpClientInterface (source code)

...
class HttpBrowser extends AbstractBrowser
{
    private $client;

    public function __construct(HttpClientInterface $client = null, History $history = null, CookieJar $cookieJar = null)
...
        $this->client = $client ?? HttpClient::create();
...

2. The HttpClient::create() supports proxy using the $defaultOptions parameter, that gets passed to the selected HttpClient (source code)

        public static function create(array $defaultOptions = [], int $maxHostConnections = 6, int $maxPendingPushes = 50): HttpClientInterface

Implementation details

The idea is to expose this functionality though a setProxy function in the core class. The library will continue to dynamically select the httpClient accordingly.

I just made a POC in my fork with all necessary code to make this work (a6589da).

public function setProxy(string $proxy)
{
    $httpClient = HttpClient::create([
        'proxy' => $proxy
    ]);

    $this->client = new Client($httpClient);

    return $this;
}

How to use

$web = new phpscraper;
$web->__call('setProxy', [
    'http://user:[email protected]:3128',
]);

If this feature gets approved, I will open a PR with it. If anything needs to be changed, let me know.

How can I get all element with class for example "card_header_f3"?

Allow to select presets of common browsers (chrome-like, Firefox, Safari) presets with agent and maybe other headers using simple function calls:

$web->firefox();
$web->chrome();
$web->safari();

These should be chainable as well.

Allow for returning of a lists set (unorder, order) with each list items below.

Currently unsure how to handle more HTML within the

See https://symfony.com/doc/current/http_client.html#streaming-responses

I'm trying to integrate this into an app that is already locked to psr/log:^3.

jeremykendall/php-domain-parser version 5 is locked to psr/log:^1

There are 2 tests that fail when bumping to the new version. I can fix and submit a PR.

Implement basic CSV parsing functionality.

API

As usual, keep it simple:

Direct Call

$data = $web->parseCsv('https://test-pages.phpscraper.de/1.csv');

Chained

$data = $web->go('https://test-pages.phpscraper.de/1.csv')->parseCsv()

Separate Calls

$web->go('https://test-pages.phpscraper.de/1.csv');
$data = $web->parseCsv();

Associative Array vs. Simple Array

While the CSVs are flat by nature, they can be resolved into an associative array. This should be considered as an option. Suitable naming TBC.

Consideration

• https://www.php.net/manual/en/function.str-getcsv.php

Motivation

I saw some references to an API service in the documentation. I don't know if you plan to make the implementation open-source as well, but I really liked the idea of abstracting the extraction process.

Combined with the proxy feature, it provides an incredibly powerful tool for gathering data.

Proposal

The idea is to implement a setApi method in the phpscraper class that supports various APIs. I went with the "namespace builder" approach, to load the API code as needed.

public $api = null;
...
public function setApi($api)
{
    $apiClass = __NAMESPACE__ . '\apis\\' . $api;

    $this->api = new $apiClass($this->core);

    return $this;
}

Then we will just need to create a file inside the new apis folder with the corresponding implementation, inside the namespace.

src/apis/example_api.php

namespace spekulatius\apis;

class example_api
{
    protected $core = null;

    public function __construct(core &$core)
    {
        $this->core = $core;
    }
...
}

Implementation example

I've implemented an API to scrap products from Mercado Libre (the biggest online marketplace in Latin America).

$web = new phpscraper;
$web->setApi('mercado_libre');

$web->go('https://mercadolivre.com.br/product_url');
$productData = $web->api->getProduct();

More details about the api mercado libre API here.

What do you think?

Hi

Its a great project, we use that library, its great 👍 thanks for all the work!

We are not sure what the versioning concept of this library looks like, also we could not find information about what has changed (a changelog), there are only tags: https://github.com/spekulatius/PHPScraper/tags. So it would be nice to read something about what is the versioning strategy and when are breaks expected.

Of course semver would be nice with an 1.0.0 release and a CHANGELOG.md and UPGRADE.md. My personal example would look like this https://github.com/nadar/quill-delta-parser/blob/master/CHANGELOG.md and this https://github.com/nadar/quill-delta-parser/blob/master/UPGRADE.md

Thanks and keep up to great work!

Sets to discover certain data (e.g. URLs, etc.) from markup and plain text content

How can I scrape a website that requires authentication?

That is, I want to start with at https://jardinado.herokuapp.com/login, fill in my credentials, and THEN start scraping the site.

That is, I want the $goutteClient to execute something like this first, then scrape:

            if ($username) {
                $crawler = $gouteClient->request('GET', $url = $baseUrl . "/login", [
                ]);

// select the form and fill in some values
                $form = $crawler->selectButton('login-btn')->form();
                $form['_username'] = 'user';
                $form['_password'] = 'pass';

// submit that form
                $crawler = $gouteClient->submit($form);
                $response = $gouteClient->getResponse();

Now that cookies are set, when I fetch a url that requires login I should get the page instead of the 302 (redirect to login).

I'm not sure how to implement this within the context of phpscraper. One idea would be to expose the goutte client.

Is this method used? Where is charset() defined?

    /**
     * Get the header collected as an array
     *
     * @return array
     */
    public function headers()
    {
        return [
            'charset' => $this->charset(),
            'contentType' => $this->contentType(),
            'viewport' => $this->viewport(),
            'canonical' => $this->canonical(),
            'csrfToken' => $this->csrfToken(),
        ];
    }

See https://symfony.com/doc/current/http_client.html#caching-requests-and-responses

Hello, I have a problem, I want to have all the links from // * [@ id = "content"] 'how do I do?

It would make sense to parse the structured data JSON provided by some sites within the head-tag. This way the already accessed information from the meta tags could be made more robust and possibility extended later on.

Ref: https://developers.google.com/search/docs/data-types/article

I am not sure if I'm simply missing something, but after installing phpscraper and simply writing this:
<?php require 'vendor/autoload.php'; $web = new \spekulatius\phpscraper; ?>
Brings this error.
Fatal error: Default value for parameters with a class type can only be NULL in .../vendor/symfony/browser-kit/AbstractBrowser.php on line 136
I'm using vanilla php with XAMPP.

Greetings,

I would like to know how I can skip errors when the site does not load correctly or is not working, for example:

$scrap = $web->go('https://asdsadsadassd.com');

if ( $scrap )
{
echo 'ok';
}

To benefit from simpler analysis it makes sense to provide a keyword extraction system out-of-the-box. Rake looks promising.

E.g.

// GET request
$web->get('https://...');

// POST request
$response = $web->post('https://...', [
  'param' => 'first param',
]);

// ...

This could be done either directly in PHPScraper or built upon another specialized lib such as Symfony HTTP. Exposing the functionality of the existing dependency sounds like a reasonable way to go, if the idea is of interest.