pushshift / telegram Goto Github PK

Thank you very much for this great project.

It seems to be a really well made way of collecting data that also supports efficient search.

I really would like to use it and I was thinking that a few minimal examples showing how to collect data, add a list of channels, query the database, etc. would be very helpful.

Also, is it possible to load the zenodo dataset (https://zenodo.org/record/3607497) into the PostgreSQL database?

Thanks,

Alex

The structure of a Telegram message can get complex when there is media attached to the message. In the meantime, to start enabling searches, a basic Elasticsearch mapping should be created that stores the channel, author, creation time, message text and other basic fields.

The ES mapping will be improved over time, but for the time being, a basic ES mapping will suffice to enable basic searches.

There is currently no LICENSE file

Telegram allows users to edit their messages. Determining if a message was edited from an original message is relatively easy since there is a field in the message object that signifies if a message was edited.

However, there needs to be logic in place to compare a message that has already been edited to detect additional future edits to that same message. Within the database, the original message will be stored along with all edited versions.

For example, if a message is an original message (unedited), the edited field in the message object can be examined to see if a message has been edited. If the message is then later edited again, further logic needs to be introduced to detect if the message was edited more than once.

All edits to a message will be stored in the database as they are detected during rescans, etc.

For the channels.py script, add method(s) that will enable discovery of new channels. Ideally, we would have the capability of discovering all publicly available channels. Also, being able to discover new channels as they are added to Telegram would be extremely beneficial.

A public API endpoint that allows researchers to search channels / messages is crucial for this project.

Adding an endpoint that allows users of the API to suggest new channels for ingest would be very beneficial and help crowdsource discovery of existing and new channels.

We will need some logic in place to avoid abuse of the endpoint, etc. Perhaps by limiting the number of suggested channels by IP address, etc.