benjaminpochat / demos Goto Github PK

Extend / enhance scraping process to analyze french intercommunalités web sites as well.
Less numerous than commune, easier to roll over in a short time, in the perspective of and to detect and alerting for new council reports.

The scraping/classification process should be optimized to avoid classifying the same document several times. For doing that, the documents already classified (their SHA-1 identifier) need to be persisted in the database.

Permettre un dépoiement avec Docker, notamment pour raspberry (voir les ébauches de docker file présents dans git)

In the PDF converting process logs, we found many errors like this one, whereas the url is correct and can be downloaded successfully in a classic web browser.

020-01-01 21:11:52 ERROR PdfConverter:41 - An error occurs while converting http://mairie-larbresle.fr/public/files/0/mairie/conseil_municipal/comptes_rendus/2013/crcm09-07-2013_57fb82f263f07.pdf into text.
java.io.IOException: Error: End-of-File, expected line
at org.apache.pdfbox.pdfparser.BaseParser.readLine(BaseParser.java:1123)
at org.apache.pdfbox.pdfparser.COSParser.parseHeader(COSParser.java:2595)
at org.apache.pdfbox.pdfparser.COSParser.parsePDFHeader(COSParser.java:2566)
at org.apache.pdfbox.pdfparser.PDFParser.parse(PDFParser.java:219)
at org.demos.pdfconverter.process.PdfConverter.getPDDocument(PdfConverter.java:50)
at org.demos.pdfconverter.process.PdfConverter.convert(PdfConverter.java:36)
at org.apache.kafka.streams.kstream.internals.AbstractStream.lambda$withKey$1(AbstractStream.java:103)
at org.apache.kafka.streams.kstream.internals.KStreamMapValues$KStreamMapProcessor.process(KStreamMapValues.java:40)
at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:117)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:201)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:180)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:133)
at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:87)
at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:366)
at org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:199)
at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:420)
at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:890)
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:805)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:774)
2020-01-01 21:11:52 INFO WebDocumentFilterByContent:18 - Document found at url http://mairie-larbresle.fr/public/files/0/mairie/conseil_municipal/comptes_rendus/2013/crcm09-07-2013_57fb82f263f07.pdf is dropped from the processing stream and won't be sent to the classification process, because its text content is null.

The solution might be given here :
https://stackoverflow.com/questions/50953924/pdfbox-ioexception-end-of-file-expected-line

When scraping web sites with calendar, scraping process loops infinitely.
For instance : https://www.mairie-hiersac.fr/agenda-hiersac-196004.html

For example, when scraping www.fargues-saint-hilaire.fr, the following link points to a pdf file that is not analyzed during the scraping process : https://www.fargues-saint-hilaire.fr/la-commune/vie-municipale/compte-rendu-de-conseil-municipal/category/3-compte-rendus-de-l-annee-2018?download=2:pv-de-la-seance-du-lundi-22-janvier-2018

The classification process works only for PDF containing text.
Some of official reports contain text scanned as images embedded in the PDF. These reports cannot be referenced by Demos. An OCR module to convert text as images in real images would improve the coverage of the documents analysed.

https://tika.apache.org/ seems to offer this feature... There might be some other open source candidates for that...

The current web scraper, based on scrapy only works for static html web sites.
As most of the web sites are base on dynamic javascript code, to be able to scrape all web site, scrapy should be replaced by a js scraped. Apify (https://sdk.apify.com/) is the reference for js web scraper.

Déplacer la fonction de génération de la roc curve pour qu'on puisse l'exécuter avec la commande suivante :
demos test roc
plutot que
demos train roc

The scraping process needs to be optimized to avoid selecting randomly the local government to be analyzed. One simple rule is : analyzing in priority the local government's web site that has not been analyzed since a long time. This needs to persist in the database the date of the last scan for each web site.

With a limit of 300Mo, the pdf-converter service fails in OutOfMemoryError when converting Firminy's PLU PDF documents. After it fails, swarm restart the service, but the same message is read from kafka queue, and it fails again and again.
2019-12-25T21:59:40.224334935Z demos_pdf-converter.log

Ne pas écraser les fichier suivants lors de l'exécution des tests unitaires (src/test/script/unittest.sh) :

• src/main/resources/mlp_model.h5
• src/main/resources/feature_selector.pkl
• src/main/resources/vectoriser.pkl

In the ui, one should be able to select a local government :

• whether he write the separator or not : writing "Saint Tropez" or "Saint-Tropez" should both return "Saint-Tropez"
• whether he write the french accents or not : writing "Périgueux" or "Perigueux" should both return "Périgueux"

Initialiser la liste des sites web des communes françaises sur https://www.data.gouv.fr/fr/datasets/service-public-fr-annuaire-de-l-administration-base-de-donnees-locales/
Ces données permettront de savoir quels sites doivent etre scrapés

Les infos à restituer sont :

• le nombre de documents classifiés
• le nombre de communes qui possèdent des documents classifiés

The error is given below
demos-scraper | 2019-06-12 00:23:39,771 - demos - INFO - gc.garbage =
demos-scraper | Unhandled Error
demos-scraper | Traceback (most recent call last):
demos-scraper | File "/app/src/main/python/process/archiving/archiver.py", line 24, in _crawl_local_governments_web_sites
demos-scraper | crawling_process.crawl()
demos-scraper | File "/app/src/main/python/process/crawling/crawling_process.py", line 22, in crawl
demos-scraper | self.crawler_process.start()
demos-scraper | File "/usr/local/lib/python3.5/site-packages/scrapy/crawler.py", line 291, in start
demos-scraper | reactor.run(installSignalHandlers=False) # blocking call
demos-scraper | File "/usr/local/lib/python3.5/site-packages/twisted/internet/base.py", line 1261, in run
demos-scraper | self.mainLoop()
demos-scraper | --- ---
demos-scraper | File "/usr/local/lib/python3.5/site-packages/twisted/internet/base.py", line 1273, in mainLoop
demos-scraper | self.doIteration(t)
demos-scraper | File "/usr/local/lib/python3.5/site-packages/twisted/internet/epollreactor.py", line 218, in doPoll
demos-scraper | l = self._poller.poll(timeout, len(self._selectables))
demos-scraper | File "/app/src/main/python/process/pdf_converter/pdf_converter.py", line 29, in timeout_handler
demos-scraper | raise Exception("timeout reached (" + str(self._timeout) + 's)')
demos-scraper | builtins.Exception: timeout reached (300s)

The technical documentation (readme.md) is messy. It should be structured in the same way the code structure is structured :

• training
• scraping
• gui
• core
• package

A graphical element is missing in the presentation part of the web app to make the page more attractive.

Il existe http://www.hirsingue.fr/ qui n'est pas détecté comme site officiel par LocalGovernmentInitializer (car non redirrigé depuis http://hirsingue.fr). Du coup, c'est hirsingue.com qui est détecté comme site officiel, à tort.

permettre par exemple de sucharger le serveur BDD défini dans config.yml de la manière suivante :
demos --database_host DBSERVER train collect
ou
demos train collect --database_host DBSERVER

The problem seems to appear when 2 scrapers run concurrently.