medialab / ural Goto Github PK

tmtc @boogheta

It would be nice to have a function able to extract and iterate over urls in the given piece of text.

https://www.bfmtvregain-de-popularite-pour-emmanuel-macron-et-edouard-phi...

https://l.facebook.com/l.php?u=http%3A%2F%2Fwww.chaos-controle.com%2Farchives%2F2013%2F10%2F14%2F28176300.html&h=AT0iUqJpUTMzHAH8HAXwZ11p8P3Z-SrY90wIXZhcjMnxBTHMiau8Fv1hvz00ZezRegqmF86SczyUXx3Gzdt_MdFH-I4CwHIXKKU9L6w522xwOqkOvLAylxojGEwrp341uC-GlVyGE2N7XwTPK9cpP0mQ8PIrWh8Qj2gHIIR08Js0mUr7G8Qe9fx66uYcfnNfTTF1xi0Us8gTo4fOZxAgidGWXsdgtU_OdvQqyEm97oHzKbWfXjkhsrzbtb8ZNMDwCP5099IMcKRD8Hi5H7W3vwh9hd_JlRgm5Z074epD_mGAeoEATE_QUVNTxO0SHO4XNn2Z7LgBamvevu1ENBcuyuSOYA0BsY2cx8mPWJ9t44tQcnmyQhBlYm_YmszDaQx9IfVP26PRqhsTLz-kZzv0DGMiJFU78LVWVPc9QSw2f9mA5JYWr29w12xJJ5XGQ6DhJxDMWRnLdG8Tnd7gZKCaRdqDER1jkO72u75-o4YuV3CLh4j-_4u0fnHSzHdVD8mxr9pNEgu8rvJF1E2H3-XbzA6F2wMQtFCejH8MBakzYtTGNvHSexSiKphE04Ci1Z23nBjCZFsgNXwL3wbIXWfHjh2LCKyihQauYsnvxp6fyioStJSGgyA9GGEswizHa20lucQF0S0F8H9-

Currently normalize_url trashes trailing slashes, although it is not a so uncommon practice that a website would serve two different things at http://www.domain.tld/path/ and http://www.domain.tld/path

So I'm not sure this normalising should remain, or at least maybe it could be optional only, depending on the usage to be done further on (for instance requesting share values on facebook ;) )

http://bc-marfeel-com.cdn.ampproject.org/c/s/bc.marfeel.com/www.capital.fr/a-la-une/actualites/francois-fillon-face-aux-geants-du-numerique-pas-de-faiblesse-1199748?marfeeltn=amp

http://www-irishtimes-com.cdn.ampproject.org/c/s/www.irishtimes.com/business/transport-and-tourism/talks-stall-between-ryanair-and-irish-pilots-union-1.3466248?mode=amp

http://cdn.ampproject.org/c/www.journaldunet.com/solutions/dsi/1165409-qu-est-ce-que-le-datalake-le-nouveau-concept-big-data-en-vogue/?output=amp

?wpamp

• puny code [done]
• sorting query params [done]
• protocol option
• authentication [done]
• option for trailing / [done]
• case? [done]
• https://en.wikipedia.org/wiki/URL_normalization

@boogheta @farjasju:

should we change extract_urls and extract_urls_from_html to urls_from_text and urls_from_html? And later add links_from_html with the link's title & url in a tuple?

• https://www.jungewelt.de/loginFailed.php?ref=/artikel/312583.kampf-gegen-macron-au%C3%9Ferparlamentarisch-f%C3%BChren.html
• http://www.milletpress.com/Detail_EN.aspx?Jiamre=2038&T=%E2%80%8BTurkey%20Plan%20Assassination%20of%20Kurdish%20Activists%20Abroad

Having a function able to tell whether the given mixed value is a url would be nice. There should be an option supporting urls without protocols also?

What URL-related heuristics do we need to centralize here?

Ping @boogheta.

https://www.facebook.com/astucerie/posts/428202057564823
https://www.facebook.com/permalink.php?story_fbid=1354978971282622&id=598338556946671
https://www.facebook.com/groups/175634843342347/permalink/235340200705144

Need to check tld also to be sure and don't get false positives as much as possible.

It would be nice to have a helper able to display a url in a squeezed space (such as a terminal window for instance).

It would be nice to have a function able to extract and iterate over urls pointed by html a tags only.

Example:
http://127.0.0.1/economie/2019/01/08/un-journaliste-poursuit-richard-ferrand-pour-lavoir-bloque-sur-twitter/

Traceback (most recent call last):
  File "bin/export_users_urls_lrumedia.py", line 56, in <module>
    media = trie.match(l) or ""
  File "/home/boo/.virtualenvs/gazouilloire-polarisation/lib/python2.7/site-packages/ural/lru/trie.py", line 33, in match
    stems = self.__lru_stems(url)
  File "/home/boo/.virtualenvs/gazouilloire-polarisation/lib/python2.7/site-packages/ural/lru/stems.py", line 93, in lru_stems
    return lru_stems_from_parsed_url(urlsplit(full_url), tld_aware=tld_aware)
  File "/home/boo/.virtualenvs/gazouilloire-polarisation/lib/python2.7/site-packages/ural/lru/stems.py", line 49, in lru_stems_from_parsed_url
    tld = '.'.join(domain_parts[non_zero_i:])
TypeError: 'NoneType' object has no attribute '__getitem__'

• canonicalize = browser will see those variants as the same
• normalize = those variants are the same permalink
• key = those keys are not urls anymore but are useful to compute statistical aggregates

It would be nice for this library to expose a CLI tool able to read lists of urls from data (such as CSV files, typically) and able to perform web/url related tasks such as:

• normalize the urls
• ensure or strip the protocols
• extract domain/subdomain/tld/...
• fetch html
• fetch json (maybe with some way to query inside)
• scrape?
• get FB like button data
• use dragnet to extract content
• etc.

cc @farjasju @boogheta

https://www.youtube.com/attribution_link?a=bWdZjuszr4I&u=%2Fwatch%3Fv%3Df5ZJLklAIQc%26feature%3Dshare

https://www.google.com/url?sa=t&source=web&rct=j&url=https%3A%2F%2Fm.youtube.com%2Fwatch%3Fv%3D4iJBsjHMviQ&ved=2ahUKEwiBm-TO3OvkAhUnA2MBHQRPAR4QwqsBMAB6BAgDEAQ&usg=AOvVaw0i7y2_fEy3nwwdIZyo_qug

https://addons.mozilla.org/en-US/firefox/addon/link-cleaner/

What to do?

http://enqueteur.cgdd.developpement-durable.gouv.fr/index.php/561714?lang=fr%0D

Should we also strip? What about spaces?

Relying on phylacterys TrieDict implementation we should, with the addition of a simplified lru method, add the simple LRUTrie.

Message error

...\ural\urls_from_html.py in urls_from_html(string)
     29     for a_tag in re.finditer(HTML_URL_RE, string):
     30         url = a_tag.group(3)
---> 31         yield clean_link(url)

...\ural\urls_from_html.py in clean_link(link)
     13 def clean_link(link):
     14     """Removes leading and trailing whitespace and punctuation"""
---> 15     return link.strip("\t\r\n '\"\x0c")
     16 
     17 

AttributeError: 'NoneType' object has no attribute 'strip'

List of website where error happened

• https://www.privacyshield.gov/participant?id=a2zt000000001K2AAI
• https://developer.github.com/v3/users/
• https://www.privacyshield.gov/
• https://www.jamsadr.com/file-an-eu-us-privacy-shield-claim
• https://www.privacyshield.gov/article?id=ANNEX-I-introduction
• https://creativecommons.org/publicdomain/zero/1.0/
• https://github.community
• https://twitter.com/github
• https://www.facebook.com/GitHub

Such as ], <br> etc.

add a strip_protocol option to the normalize_url function :

normalize_url(http://www.google.com/page.html, strip_protocol = False)
>>> "http://google.com"