Code Monkey home page Code Monkey logo

scrapping_vdm's Introduction

scrapping_vdm

Petit programme de scrapping du site vdm

########## Notes générales ###############

Installation des bibliothèques : pip install -r requirements.txt

Notice d'utilisation : En lançant "python app.py" via la console.

Les URLs sont : 127.0.0.1/api/posts -> affiche les 200 derniers posts 127.0.0.1/api/posts/ -> récupère le post d'ID donné, si il existe 127.0.0.1/api/posts/author= -> récupère tous les postes d'un auteur donné 127.0.0.1/api/posts/from=<2017-09-28T09:00:00Z>&to=<2017-12-31T00:00:00Z> -> récupère tous les postes entre ces deux dates, dates à mettre a ce format la spécifiquement.

############################################# Courte description des fichiers : #############################################

app.py -> contient l'api flask qui gère l'affichage des informations extractor.py -> contient le scrappeur qui va chercher l'information sur le site VDM form.py -> contient le code qui demande la ligne de commande ("fetch") testing.py -> contient les test unitaires des fonctions principales. Pour les lancer, il faut utiliser la commande "python testing.py"

############################################# Dockerisation : ############################################# Ne fonctionne pas encore, problème avec la gestion du serveur local, et des inputs utilisateurs.

scrapping_vdm's People

Contributors

chamawix avatar

Watchers

 avatar

scrapping_vdm's Issues

Fetch not working on osx ?

Hi,

I have this error trying to enter the fetch command.
Could you provide an endpoint or fix this issue ?

Please enter the "fecth" command to fetch all the data :
fetch
not from linux bash

Content and Date are missing

Voici le return de /api/posts :

{'count': 200, 'post': [{'content': '', 'date': '', 'id': 1, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 2, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 3, 'author': 'Sebastien Folin'}, {'content': '', 'date': '', 'id': 4, 'author': 'Mark E. Smith'}, {'content': '', 'date': '', 'id': 5, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 6, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 7, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 8, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 9, 'author': 'Lassie'}, {'content': '', 'date': '', 'id': 10, 'author': 'The Chronic'}, {'content': '', 'date': '', 'id': 11, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 12, 'author': 'Edouard Leclerc'}, {'content': '', 'date': '', 'id': 13, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 14, 'author': 'Polly'}, {'content': '', 'date': '', 'id': 15, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 16, 'author': 'Crocodile Dundee'}, {'content': '', 'date': '', 'id': 17, 'author': 'Kevin Carter'}, {'content': '', 'date': '', 'id': 18, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 19, 'author': 'Wikiki Girl'}, {'content': '', 'date': '', 'id': 20, 'author': 'Erin Brokovitch'}, {'content': '', 'date': '', 'id': 21, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 22, 'author': u'S\xe9bastien Folin'}, {'content': '', 'date': '', 'id': 23, 'author': 'William Portail'}, {'content': '', 'date': '', 'id': 24, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 25, 'author': 'DJ Calebut'}, {'content': '', 'date': '', 'id': 26, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 27, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 28, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 29, 'author': 'Mimi'}, {'content': '', 'date': '', 'id': 30, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 31, 'author': 'youpla'}, {'content': '', 'date': '', 'id': 32, 'author': 'hommebrioche'}, {'content': '', 'date': '', 'id': 33, 'author': 'Anonyme'}, {'content': '', 'date': '', 'id': 34, 'author': 'momoleconnard'}, {'content': '', 'date': '', 'id': 35, 'author': 'zed'}, {'content': '', 'date': '', 'id': 36, 'author': 'zed'}, {'content': '', 'date': '', 'id': 37, 'author': 'Shemselnihar'}, {'content': '', 'date': '', 'id': 38, 'author': 'Shemselnihar'}, {'content': '', 'date': '', 'id': 39, 'author': 'Sylvie20'}, {'content': '', 'date': '', 'id': 40, 'author': 'quent1'}, {'content': '', 'date': '', 'id': 41, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 42, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 43, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 44, 'author': 'Anonyme'}, {'content': '', 'date': '', 'id': 45, 'author': u'Rat\xe9'}, {'content': '', 'date': '', 'id': 46, 'author': u'Rat\xe9'}, {'content': '', 'date': '', 'id': 47, 'author': "Pline l'Ancien"}, {'content': '', 'date': '', 'id': 48, 'author': 'Simone'}, {'content': '', 'date': '', 'id': 49, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 50, 'author': 'Pennywise'}, {'content': '', 'date': '', 'id': 51, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 52, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 53, 'author': 'MacGyver'}, {'content': '', 'date': '', 'id': 54, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 55, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 56, 'author': 'Yvan Lendl'}, {'content': '', 'date': '', 'id': 57, 'author': 'Hawkeye Pierce'}, {'content': '', 'date': '', 'id': 58, 'author': 'Simone'}, {'content': '', 'date': '', 'id': 59, 'author': 'Franky le Borgne'}, {'content': '', 'date': '', 'id': 60, 'author': 'Anna Wintour'}, {'content': '', 'date': '', 'id': 61, 'author': 'Barbara Windsor'}, {'content': '', 'date': '', 'id': 62, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 63, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 64, 'author': 'Stan Lee'}, {'content': '', 'date': '', 'id': 65, 'author': 'Monsieur Connard'}, {'content': '', 'date': '', 'id': 66, 'author': 'Johnny Hardon'}, {'content': '', 'date': '', 'id': 67, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 68, 'author': 'Professeur Tournesol'}, {'content': '', 'date': '', 'id': 69, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 70, 'author': 'Barack Obama'}, {'content': '', 'date': '', 'id': 71, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 72, 'author': 'Jean-Luc'}, {'content': '', 'date': '', 'id': 73, 'author': u'\xc9milie'}, {'content': '', 'date': '', 'id': 74, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 75, 'author': 'Peter Priapic'}, {'content': '', 'date': '', 'id': 76, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 77, 'author': 'Hawkeye Pierce'}, {'content': '', 'date': '', 'id': 78, 'author': 'Dogbowl'}, {'content': '', 'date': '', 'id': 79, 'author': ''}, {'content': '', 'date': '', 'id': 80, 'author': 'Lucky Luke'}, {'content': '', 'date': '', 'id': 81, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 82, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 83, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 84, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 85, 'author': 'Vidal Sassoon'}, {'content': '', 'date': '', 'id': 86, 'author': u'St\xe9phane Plaza'}, {'content': '', 'date': '', 'id': 87, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 88, 'author': 'Cheeky Nandos'}, {'content': '', 'date': '', 'id': 89, 'author': 'Britney Bi**h'}, {'content': '', 'date': '', 'id': 90, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 91, 'author': 'Donald Trompette'}, {'content': '', 'date': '', 'id': 92, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 93, 'author': 'Snoop Dogg'}, {'content': '', 'date': '', 'id': 94, 'author': 'Roger Canigou'}, {'content': '', 'date': '', 'id': 95, 'author': 'VDM Educ'}, {'content': '', 'date': '', 'id': 96, 'author': 'Hawkeye'}, {'content': '', 'date': '', 'id': 97, 'author': 'Frank Beard'}, {'content': '', 'date': '', 'id': 98, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 99, 'author': u'G\xe9nie'}, {'content': '', 'date': '', 'id': 100, 'author': u'J\xe9j\xe9'}, {'content': '', 'date': '', 'id': 101, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 102, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 103, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 104, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 105, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 106, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 107, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 108, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 109, 'author': u'No\xebl et Liam'}, {'content': '', 'date': '', 'id': 110, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 111, 'author': ''}, {'content': '', 'date': '', 'id': 112, 'author': 'Jean-Michel Pilote'}, {'content': '', 'date': '', 'id': 113, 'author': 'Ghost'}, {'content': '', 'date': '', 'id': 114, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 115, 'author': 'Hawkeye'}, {'content': '', 'date': '', 'id': 116, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 117, 'author': ''}, {'content': '', 'date': '', 'id': 118, 'author': 'Booby McTits'}, {'content': '', 'date': '', 'id': 119, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 120, 'author': 'Le Twitto'}, {'content': '', 'date': '', 'id': 121, 'author': 'Jean-Michel Festival'}, {'content': '', 'date': '', 'id': 122, 'author': 'Shakira'}, {'content': '', 'date': '', 'id': 123, 'author': 'Peppa Pig'}, {'content': '', 'date': '', 'id': 124, 'author': 'Pablo Escobar'}, {'content': '', 'date': '', 'id': 125, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 126, 'author': u'Une personne outr\xe9e'}, {'content': '', 'date': '', 'id': 127, 'author': 'Mitch Buchannan'}, {'content': '', 'date': '', 'id': 128, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 129, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 130, 'author': ''}, {'content': '', 'date': '', 'id': 131, 'author': 'Pepito'}, {'content': '', 'date': '', 'id': 132, 'author': 'Canard WC'}, {'content': '', 'date': '', 'id': 133, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 134, 'author': 'Hawkeye'}, {'content': '', 'date': '', 'id': 135, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 136, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 137, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 138, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 139, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 140, 'author': 'Le Twitto'}, {'content': '', 'date': '', 'id': 141, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 142, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 143, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 144, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 145, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 146, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 147, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 148, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 149, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 150, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 151, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 152, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 153, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 154, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 155, 'author': 'VDM'}, {'content': '', 'date': '', 'id': 156, 'author': 'Ned Flanders'}, {'content': '', 'date': '', 'id': 157, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 158, 'author': 'Jojo'}, {'content': '', 'date': '', 'id': 159, 'author': u'VDMM\xe9dics'}, {'content': '', 'date': '', 'id': 160, 'author': 'Le Twitto'}, {'content': '', 'date': '', 'id': 161, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 162, 'author': 'Lili'}, {'content': '', 'date': '', 'id': 163, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 164, 'author': 'Lulu'}, {'content': '', 'date': '', 'id': 165, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 166, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 167, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 168, 'author': 'La Bikeuse'}, {'content': '', 'date': '', 'id': 169, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 170, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 171, 'author': ''}, {'content': '', 'date': '', 'id': 172, 'author': 'MC Hammer'}, {'content': '', 'date': '', 'id': 173, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 174, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 175, 'author': 'Samantha'}, {'content': '', 'date': '', 'id': 176, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 177, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 178, 'author': 'Jojo'}, {'content': '', 'date': '', 'id': 179, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 180, 'author': 'Le Twitto'}, {'content': '', 'date': '', 'id': 181, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 182, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 183, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 184, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 185, 'author': u'VDM culott\xe9e'}, {'content': '', 'date': '', 'id': 186, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 187, 'author': 'Le Fou du Volant'}, {'content': '', 'date': '', 'id': 188, 'author': 'Vic666'}, {'content': '', 'date': '', 'id': 189, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 190, 'author': 'Le Breton'}, {'content': '', 'date': '', 'id': 191, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 192, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 193, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 194, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 195, 'author': 'Samantha'}, {'content': '', 'date': '', 'id': 196, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 197, 'author': 'Julie'}, {'content': '', 'date': '', 'id': 198, 'author': 'Jean Valjean'}, {'content': '', 'date': '', 'id': 199, 'author': 'Alan'}, {'content': '', 'date': '', 'id': 200, 'author': 'Miss Camping'}]}

VDM Auto processing

Bonjour,
Merci pour ta réactivité, le code de collecte et de requétage semble fonctionner.
Maintenant, pour aller plus loin dans le test, serait-il possible d'ajouter une analyse automatique des VDM portant sur le contenu de la VDM ?
Libre à toi de choisir ce qui te semble pertinent.
N’hésites pas à poser des questions dans cette issue afin que l’exercice soit le plus interactif possible.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.