adipasquale / insee-brut Goto Github PK

http://api.insee.pw/api/v1/root_documents
=>
http://api.insee.pw/v1/root_documents

• in prod plug the prepare documents script after scrapings
• move what was done in python with the augmenter to an API job ? or keep it in python, but that's yet another project to maintain ...

maybe on netlify ?

look if there are other non-root types than Series

follow https://blog.dipasquale.fr/en/2018/12/17/incremental-scraping-with-scrapy-and-mongo/

pages with hierarchical navigation in "famille" and "categories"

example :
the root document with id_insee 3530678.
https://insee.fr/fr/statistiques/3530678
looks like

• In the list, the document points to this : https://insee.fr/fr/statistiques/ID_INSEE
• while other series documents point to https://insee.fr/fr/statistiques/series/ID_INSEE
• this last url doesn't work at all for this serie

it looks like this page might be the only one of this kind :
looking for {"famille.libelleFr": "Séries chronologiques"} only yields this document.

maybe they just push it to the top of the results when they update any serie inside it

on the outermost left panel : root documents. infinite scroll and search bar
on the outermost right panel : the currently selected document infos, tables, report

in middle panels : sub-documents for "Series" and other nested document types

either with algolia or a hosted ElasticSearch

We can add a download link on all Serie items.

https://insee.fr/fr/statistiques/serie/telecharger/ID_BANK

It's the same link that is generated from the 'telecharger' onglet on these pages eg. https://insee.fr/fr/statistiques/serie/010540695#Telechargement :

when you don't put any parameters, it seems to give you a 'full' version, which is what we want.
ex https://insee.fr/fr/statistiques/serie/telecharger/010540695

ideas :

• insee RAW
• insee brut

list models and explain fields