nltk.download('averaged_perceptron_tagger')

nieuwe versie van marthe:
Appendix about Selenium_updatedversion.docx

in boek en .ipynb

best practice:

import statsmodels as sm
import statsmodels.formula.api as smf

mod = smf.ols(formula='Lottery ~ Literacy + Wealth * Region', data=df)
res = mod.fit()
print(res.summary())

https://github.com/RaRe-Technologies/bounter

toevoegen als extra tip

naar dit tutorial verwijzen:
https://github.com/vtraag/4TU-CSS/blob/master/presentations/traag/notebook/Network.ipynb

https://opendata.tweedekamer.nl/nieuws/kamer-maakt-parlementaire-data-toegankelijker

benoemen dat linux steeds populairder wordt en inmiddels ook binnen windows kan:
https://blogs.msdn.microsoft.com/commandline/2017/10/11/whats-new-in-wsl-in-windows-10-fall-creators-update/

editors

• emacs
• vi
• geany (iets als textwrangler/notepad++)

staat op harde schijf

New book on Data Science with Python:
[https://github.com/jakevdp/PythonDataScienceHandbook/blob/master/README.md]

https://www.kaggle.com/abhishek/approaching-almost-any-nlp-problem-on-kaggle

in at least one example, .readlines() is used, which is an inefficient way to read from a file.

code example still contains python2-style u"..."-string

should point to page number (some pages earlier)

moet zijn:
nltk.download("vader_lexicon")

make sure that people check whether they have IPython>5

zo niet:
sudo apt-get remove ipython3
sudo pip3 install ipython

overal in boek en slides aanpassen om verwarring met het optellen van integers ( i += 1) te vermijden

• andere modellen toevoegen (Logistic Regression, SVM)
• precision/recall toevoegen

comparison_classifiers.txt

https://www.goodreads.com/book/show/16179186-scraping-for-journalists
bradshaw book "scraping for journalists"

• add jupyter notebook
• add pandas

When there is a line/paragraph page within the results of an XPATH, the /text() function might not function properly, as it sees each part as a seperate element.
Fix:
leave away the /text() in the xpath itself and use the .text_content() method later on:

reviews = tree.xpath('//div/div/div[2]/div[*]/div[2]/p[1]')
print (len(reviews),"reviews scraped. Showing the first 60 characters of each:")
i=0
for review in reviews:
    print("Review",i,":",review.text_content())
    i+=1

Add this as alternative solution to XPATH-chapter

on p. 58, also install the following tagger

nltk.download('hmm_treebank_pos_tagger')

opnemen bij installatie-instructies aan het begin:
sudo apt-get install python3-dev

misschien leuke corpus voor oefening:

https://freedom-to-tinker.com/2016/09/14/all-the-news-thats-fit-to-change-insights-into-a-corpus-of-2-5-million-news-headlines/

• verschillende kleuren voor bash en python code
• overzichtje maken van commando's die in de bash moeten worden uitgevoerd

adres klopt niet (andere server)

explain what to do if accidently chosen wrong setting

sudo dpkg-reconfigure keyboard-configuration

Model: 'Algemeen 105 toetsen internationaal
Oorsprong van het toetsenbord (Engels VS)
Indeling: Engels (VS)
standaard
geen samenstelling
X-server: nee

taal (Nederlands) --> maakt niet uit

github-link

eerst:
sudo pip3 install tweepy

en dan de minimale code:
(maar je wilt wsl niet printen maar naar een bestand schrijven)

from tweepy.streaming import StreamListener
from tweepy import OAuthHandler
from tweepy import Stream

consumer_key = "..."
consumer_secret = "..."
access_token = "..."
access_token_secret = "..."


class StdOutListener(StreamListener):
    """ A listener handles tweets that are received from the stream.
    This is a basic listener that just prints received tweets to stdout.
    """
    def on_data(self, data):
        print(data)
        return True

    def on_error(self, status):
        print(status)

l = StdOutListener()
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

stream = Stream(auth, l)
stream.filter(track=['basketball'])

voor handleiding Big Data

• andere benadering: R. https://rstudio-pubs-static.s3.amazonaws.com/31867_8236987cf0a8444e962ccd2aec46d9c3.html

when reading the data, labels are integers: 1 and -1
when trying to calculate precision, the code uses strings '1', '-1'
--> this cannot work, change either first to strings or second to ints.

http://nbviewer.jupyter.org/github/JosPolfliet/pandas-profiling/blob/master/examples/meteorites.ipynb

regel 5 moet zijn:

... delimiter = "\t")

er is nu een tweede editie:

http://www.webpages.uidaho.edu/~stevel/504/Mining-the-Social-Web-2nd-Edition.pdf

Csvkit: a handy set of #Python tools for quick CSV hacking from command line (Excel/SQL/JSON...) http://csvkit.readthedocs.io

maar ook: import.io

xpath checker eruit
voorbeelden kloppen niet meer

eventueel ook regexp eruit?

misschien css selector erin?

don't call the listener l, it can be mistaken for 1

see inca-code

https://nlpforhackers.io/topic-modeling/

we geven twee taggers, maar eentje blijkt niet meer te bestaan. checken

head -5 sanders.tsv has the color of python, should be the terminal color

best (from your computer),
theo :)

To mount a directory from your "real" computer, the so-called host, in the file system of the VM, you can do the following:
Select the folder you want to share in Virtual Box (see screenshot). Select permanent, but NOT auto-mounting (screenshot).
Remember the name of the share (in my example, Desktop)

Within the VM, now create a folder where you want the content of your host folder to appear. For example, this could be:

mkdir /home/damian/myrealcomputer

Now, use the following line to mount the share (in my example, it is called Desktop) to that folder:

sudo mount -t vboxsf -o uid=$UID,gid=$(id -g) Desktop /home/damian/myrealcomputer/

zie ook mail.
oplossing:

import csv

data = open('_chat.txt').readlines()
timestamps=[e[:17] for e in data]
text_unparsed =[e[19:] for e in data]
namen = [e.split(':')[0] for e in text_unparsed]
text = [":".join(e.split(':')[1:]) for e in text_unparsed]

output = zip(timestamps,namen,text)
with open('output.csv',mode='w') as fo:
  writer=csv.writer(fo)
  writer.writerows(output)
'''

should be:

import nltk
nltk.download(’vader_sentiment’)