gwu-libraries / uriscrape Goto Github PK

2.0 5.0 0.0 71 KB

Scrape URIs from Telegram channel transcripts in PDF files

License: MIT License

Python 100.00%

telegram pdf-scraping telegram-channel-transcripts

uriscrape's Introduction

uriscrape

Scrapes URIs from Telegram channel transcripts in PDF files. Typically URIs will take the form of something like these examples:

https://t.me/joinchat/AAAAAEOs3wFD4Mv6SN4hlQ

(tg://join?invite=AAAAAEOs3wFD4Mv6SN4hlQ)

https://drive.google.com/open?id=0B_3xyna6XV4GMHNPU0VVWHZKRXc

https://archive.org/details/Rumiyah13UR_201709
(https://archive.org/details/Rumiyah13UR_201709)

(tg://search_hashtag?hashtag=%D8%A6%DB%95%D9%84%DA%BE%D8%A7%D9%8A%D8%A7%D8%AA)

Running the program

usage: `python uriscrape.py transcript`

positional arguments:
  transcript         filepath to transcript pdf or directory

optional arguments:
  None yet...

Output file

urls.xlsx - All found URIs, including columns/variables as follows:

File: PDF file processed
Access_Date: Date/time the program was run. May be important for documenting when the program attempted to resolve URIs
Post_Date: Date of the post, as derived from the date labels in the Telegram transcript
URL: URL as found
Site_Reached: True/False - whether the URI was able to be resolved
Unshortened URL: Unshortened URL (e.g. https://youtu.be/lqXwyl89xU4 -> unshortens to https://www.youtube.com/watch?v=lqXwyl89xU4&feature=youtu.be )
Status: Error code, if an error was encountered in trying to access the URI
Type: Classification of the link
Hashtag: Hashtag, if the link is a Telegram hashtag link
Channel: Channel, if the link is a Telegram join link
Account: Account, if the link is a Telegram account link
Domain: Full server daomain (e.g. www.youtube.com)
Primary_Secondary: Just the primary and secondary portions of the domain (e.g. youtube.com)

uriscrape's People

Stargazers

Watchers

uriscrape's Issues

Handle invalid URLs

Recover gracefully from errors such as:

(http://[mailto:[email protected])
Traceback (most recent call last):
  File "uriscrape.py", line 195, in <module>
    url_domain = domain(cleaned_url)
  File "uriscrape.py", line 56, in domain
    return urlparse(url).netloc
  File "/Users/kerchner/anaconda3/lib/python3.6/urllib/parse.py", line 368, in urlparse
    splitresult = urlsplit(url, scheme, allow_fragments)
  File "/Users/kerchner/anaconda3/lib/python3.6/urllib/parse.py", line 418, in urlsplit
    raise ValueError("Invalid IPv6 URL")
ValueError: Invalid IPv6 URL```

De-duplicate where second link has other text suffixed

Example from Sheikh Ayman Al Zawahiri supporter group 6-15-18 1057.pdf

https://www.facebook.com/groups/235369163906382/
https://www.facebook.com/groups/235369163906382TelegramShahada

Include PDF file name in output spreadsheet

i.e. name of file where link was found

Aggregate analysis

Generate a list of unique external links, and their counts (how many appear in the data set)

Include date/time stamp

i.e. date/time when running the script. This is important because, among other reasons, we try to access http(s) links to see if we can resolve them.

Try to resolve links, include response status

Indicate whether external site was reachable or not when expanding URL

Add this as a variable

De-duplicate where tg_channel link is in between

There are links where we have:
URL
Channel URL (at the bottom of the page)
URL

and the URLs are not being deduplicated because a channel link is in the middle. Skip the channel links earlier in the logic.

Create separate variables for channel and account

As parsed out of associated link types. Similar to how Hashtag is parsed out.

include date from within transcript

Give extracts unique filenames

Add in datetime

Deduplicate pairs of links

Links appear in pairs - deduplicate

There also can be triplicates: Some twitter links, some youtube links. Generally when embedded.

Remove channel identifier from observations

This refers to the channel identifier that appears in the lower left of the PDF

De-duplicate where the only difference is http vs. https

Example is in Sheikh Ayman Al Zawahiri supporter group 6-15-18 1057.pdf file - see

http://www.thedailystar.net/frontpage/ex-cpb-leader-shot-dead-munshiganj-1589782
https://www.thedailystar.net/frontpage/ex-cpb-leader-shot-dead-munshiganj-1589782

[Analysis] Join with table of channel ids and join hashes

Generate output as XLSX so that non-Roman characters render in Excel

[Analysis] Generate A->B network table from join links

Classify links

Potential classifications (TBD):

Telegram
- hashtags
- join links
- account pages (?)
- "other"
External

Is t.me/XYZ a hashtag or an account?

Currently, this pattern
https://t.me/KhilafahNews437
is being categorized as tg_account but it seems to often be followed immediately by
tg://search_hashtag?hashtag=KhilafahNews
Not sure how to intepret the number that seems to be present on the t.me link but not on the latter?