This is a small crawler package for the site leboncoin.fr.
leboncoin.fr is one of the most famous classified ads website in france. Whatever what you're looking for, it is probably there. It has a lots of ads and is very fast and simple to use.
However, the first problem comes when you need to exploit the search results in a way that the site can't help you. In fact, the search results page is pretty poor in terms of data. For exemple, it'd be so cool to get the mileage when I'm looking for a car or the surface when I'm looking for a flat.
The second problem is that saving a search is really a pain on the actual site. All the searches you want to save give one single result page. That's pretty dumb, but anyway.
And the third and last problem is that you are forced to use selected values with some criterias. For example, when I was looking for a motorcycle, I was looking for those with a bigger engine than 1200cc. The fact that the biggest value available in the input is 1000cc and because there are tons of ads with 1000cc motorcycle, it made search much more complicated. I did send an email to ask for an additional value, but I didn't get any answer (which I didn't expect anyway). So I had to change the value in the query string every single request... What a waste of time...
So for all those little reasons, I decided to write my good old web scraper to be able to extract the data from the site to anywhere (a DB, an array, a json, an api, who knows...).
- PHP 5.6 | 7, HHVM (PHP 5.4 and 5.5 may work)
- [optional] PHPUnit to execute the test suite
$ composer require rfussien/leboncoin-crawlerSuper easy !!!
(new Lbc\GetFrom)->search('<search_result_url>');
// or with detailed ads
(new Lbc\GetFrom)->search('<search_result_url>', true);example of output:
{
"page": 2,
"links": {
"current": "http://www.leboncoin.fr/ventes_immobilieres/offres/basse_normandie/calvados/?pe=11&sqs=10&ros=5&ret=1&f=p&o=2",
"previous": "http://www.leboncoin.fr/ventes_immobilieres/offres/basse_normandie/calvados/?pe=11&sqs=10&ros=5&ret=1&f=p&o=1",
"next": "http://www.leboncoin.fr/ventes_immobilieres/offres/basse_normandie/calvados/?pe=11&sqs=10&ros=5&ret=1&f=p&o=3"
},
"total_ads": 604,
"total_page": 18,
"category": "ventes_immobilieres",
"location": null,
"search_area": "basse_normandie",
"sort_by": "date",
"type": "part",
"ads": {
"602701721": {
"id": "602701721",
"title": "Maison F7 à EVRECY",
"price": 200000,
"url": "http://www.leboncoin.fr/ventes_immobilieres/602701721.htm?ca=4_s",
"created_at": "2015-01-10 19:01",
"thumb": "http://193.164.197.40/thumbs/808/808ae4f91c5bf1871b96f16bccb3751eeb0baec4.jpg",
"nb_image": 3,
"placement": "Evrecy / Calvados",
"type": "part"
},
[...]
"755560430": {
"id": "755560430",
"title": "Maison Atypique ( Esprit Loft ) 145 m2",
"price": 243000,
"url": "http://www.leboncoin.fr/ventes_immobilieres/755560430.htm?ca=4_s",
"created_at": "2015-01-11 19:01",
"thumb": "http://193.164.196.60/thumbs/aa3/aa336ba634f7e5f43b6c016358afa2510e42aa0b.jpg",
"nb_image": 3,
"placement": "Caen / Calvados",
"type": "part"
}
}
}(new Lbc\GetFrom)->ad('<ad_url>');
// or
(new Lbc\GetFrom)->ad('<ad_id>', '<ad_category>');example of output:
{
"id": "602701721",
"category": "ventes_immobilieres",
"thumbs": [
"http://193.164.197.40/thumbs/808/808ae4f91c5bf1871b96f16bccb3751eeb0baec4.jpg",
"http://193.164.196.60/thumbs/1b4/1b40871304534d25c99c7b3baeda07c16c8b48cd.jpg",
"http://193.164.196.30/thumbs/152/15251eb4128758c6d0c44523b6733ee9d5ea3749.jpg"
],
"pictures": [
"http://193.164.197.40/images/808/808ae4f91c5bf1871b96f16bccb3751eeb0baec4.jpg",
"http://193.164.196.60/images/1b4/1b40871304534d25c99c7b3baeda07c16c8b48cd.jpg",
"http://193.164.196.30/images/152/15251eb4128758c6d0c44523b6733ee9d5ea3749.jpg"
],
"title": "Maison F7 à EVRECY",
"cp": "14210",
"city": "Evrecy",
"price": 200000,
"criterias": {
"type_de_bien": "Maison",
"pieces": "7",
"surface": "140 m2",
"ges": "F (de 56 à 80)",
"classe_energie": "D (de 151 à 230)"
},
"description": "Baisse de prix pour une maison à [...] sur un terrain de 576 m². AGENCE S'ABSTENIR."
}There are a bunch of features if you digg a bit in the sources.
$ composer testPlease see CONTRIBUTING and CONDUCT for details.
If you discover any security related issues, please email :author_email instead of using the issue tracker.
The MIT License (MIT). Please see License File for more information.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent