benjaminpochat / demos Goto Github PK
View Code? Open in Web Editor NEWDemos is a CivicTech project for scraping official city web sites, and archiving deliberations in a centralized database
License: GNU Affero General Public License v3.0
Demos is a CivicTech project for scraping official city web sites, and archiving deliberations in a centralized database
License: GNU Affero General Public License v3.0
Extend / enhance scraping process to analyze french intercommunalités web sites as well.
Less numerous than commune, easier to roll over in a short time, in the perspective of and to detect and alerting for new council reports.
The scraping/classification process should be optimized to avoid classifying the same document several times. For doing that, the documents already classified (their SHA-1 identifier) need to be persisted in the database.
Permettre un dépoiement avec Docker, notamment pour raspberry (voir les ébauches de docker file présents dans git)
In the PDF converting process logs, we found many errors like this one, whereas the url is correct and can be downloaded successfully in a classic web browser.
020-01-01 21:11:52 ERROR PdfConverter:41 - An error occurs while converting http://mairie-larbresle.fr/public/files/0/mairie/conseil_municipal/comptes_rendus/2013/crcm09-07-2013_57fb82f263f07.pdf into text.
java.io.IOException: Error: End-of-File, expected line
at org.apache.pdfbox.pdfparser.BaseParser.readLine(BaseParser.java:1123)
at org.apache.pdfbox.pdfparser.COSParser.parseHeader(COSParser.java:2595)
at org.apache.pdfbox.pdfparser.COSParser.parsePDFHeader(COSParser.java:2566)
at org.apache.pdfbox.pdfparser.PDFParser.parse(PDFParser.java:219)
at org.demos.pdfconverter.process.PdfConverter.getPDDocument(PdfConverter.java:50)
at org.demos.pdfconverter.process.PdfConverter.convert(PdfConverter.java:36)
at org.apache.kafka.streams.kstream.internals.AbstractStream.lambda$withKey$1(AbstractStream.java:103)
at org.apache.kafka.streams.kstream.internals.KStreamMapValues$KStreamMapProcessor.process(KStreamMapValues.java:40)
at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:117)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:201)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:180)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:133)
at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:87)
at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:366)
at org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:199)
at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:420)
at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:890)
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:805)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:774)
2020-01-01 21:11:52 INFO WebDocumentFilterByContent:18 - Document found at url http://mairie-larbresle.fr/public/files/0/mairie/conseil_municipal/comptes_rendus/2013/crcm09-07-2013_57fb82f263f07.pdf is dropped from the processing stream and won't be sent to the classification process, because its text content is null.
The solution might be given here :
https://stackoverflow.com/questions/50953924/pdfbox-ioexception-end-of-file-expected-line
When scraping web sites with calendar, scraping process loops infinitely.
For instance : https://www.mairie-hiersac.fr/agenda-hiersac-196004.html
For example, when scraping www.fargues-saint-hilaire.fr, the following link points to a pdf file that is not analyzed during the scraping process : https://www.fargues-saint-hilaire.fr/la-commune/vie-municipale/compte-rendu-de-conseil-municipal/category/3-compte-rendus-de-l-annee-2018?download=2:pv-de-la-seance-du-lundi-22-janvier-2018
The classification process works only for PDF containing text.
Some of official reports contain text scanned as images embedded in the PDF. These reports cannot be referenced by Demos. An OCR module to convert text as images in real images would improve the coverage of the documents analysed.
https://tika.apache.org/ seems to offer this feature... There might be some other open source candidates for that...
The current web scraper, based on scrapy only works for static html web sites.
As most of the web sites are base on dynamic javascript code, to be able to scrape all web site, scrapy should be replaced by a js scraped. Apify (https://sdk.apify.com/) is the reference for js web scraper.
Déplacer la fonction de génération de la roc curve pour qu'on puisse l'exécuter avec la commande suivante :
demos test roc
plutot que
demos train roc
The scraping process needs to be optimized to avoid selecting randomly the local government to be analyzed. One simple rule is : analyzing in priority the local government's web site that has not been analyzed since a long time. This needs to persist in the database the date of the last scan for each web site.
With a limit of 300Mo, the pdf-converter service fails in OutOfMemoryError when converting Firminy's PLU PDF documents. After it fails, swarm restart the service, but the same message is read from kafka queue, and it fails again and again.
2019-12-25T21:59:40.224334935Z demos_pdf-converter.log
Ne pas écraser les fichier suivants lors de l'exécution des tests unitaires (src/test/script/unittest.sh) :
In the ui, one should be able to select a local government :
Initialiser la liste des sites web des communes françaises sur https://www.data.gouv.fr/fr/datasets/service-public-fr-annuaire-de-l-administration-base-de-donnees-locales/
Ces données permettront de savoir quels sites doivent etre scrapés
Les infos à restituer sont :
The error is given below
demos-scraper | 2019-06-12 00:23:39,771 - demos - INFO - gc.garbage =
demos-scraper | Unhandled Error
demos-scraper | Traceback (most recent call last):
demos-scraper | File "/app/src/main/python/process/archiving/archiver.py", line 24, in _crawl_local_governments_web_sites
demos-scraper | crawling_process.crawl()
demos-scraper | File "/app/src/main/python/process/crawling/crawling_process.py", line 22, in crawl
demos-scraper | self.crawler_process.start()
demos-scraper | File "/usr/local/lib/python3.5/site-packages/scrapy/crawler.py", line 291, in start
demos-scraper | reactor.run(installSignalHandlers=False) # blocking call
demos-scraper | File "/usr/local/lib/python3.5/site-packages/twisted/internet/base.py", line 1261, in run
demos-scraper | self.mainLoop()
demos-scraper | --- ---
demos-scraper | File "/usr/local/lib/python3.5/site-packages/twisted/internet/base.py", line 1273, in mainLoop
demos-scraper | self.doIteration(t)
demos-scraper | File "/usr/local/lib/python3.5/site-packages/twisted/internet/epollreactor.py", line 218, in doPoll
demos-scraper | l = self._poller.poll(timeout, len(self._selectables))
demos-scraper | File "/app/src/main/python/process/pdf_converter/pdf_converter.py", line 29, in timeout_handler
demos-scraper | raise Exception("timeout reached (" + str(self._timeout) + 's)')
demos-scraper | builtins.Exception: timeout reached (300s)
The technical documentation (readme.md) is messy. It should be structured in the same way the code structure is structured :
A graphical element is missing in the presentation part of the web app to make the page more attractive.
Il existe http://www.hirsingue.fr/ qui n'est pas détecté comme site officiel par LocalGovernmentInitializer (car non redirrigé depuis http://hirsingue.fr). Du coup, c'est hirsingue.com qui est détecté comme site officiel, à tort.
permettre par exemple de sucharger le serveur BDD défini dans config.yml de la manière suivante :
demos --database_host DBSERVER train collect
ou
demos train collect --database_host DBSERVER
The problem seems to appear when 2 scrapers run concurrently.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.