For different data sources, we collect data differently.

grobid_client --input "PATH/TO/PDF/FILES/" --output "PATH/TO/SAVE/XML/FILES/" processFulltextDocument

save_folder: ./data/collection/tweets # folder to save tweets
users: # list of users for tweet crawl
  - EITHealth
  - EU_Commission

from_users: True # if enabled then tweets for given list of users will be crawled
have_user_mentioned: False # if enabled then tweets which mentions the user will be crawled

from_mentioned_users: False # If true then looks for tweet jsons in save_folder for all the users and crawl tweets
  # for mentioned users in the tweets.
mentioned_by_params:
  top_n_users: 400 # Get tweets for the top_n_users mentioned by a particular user. It is used when mentioned_by_user is enabled
  interaction_level: 3 # Save those mentioned users who have mentioned original user interaction_level times

get_images: False # extract images from tweets of users
# Optional, comment them out if not required. All the tweets will be mined in that case.
since: 2022-10-01
until: 2022-10-30

python src/data/collection/tweets.py -config configs/data/collection/tweet.yaml

Data collected for tweets using snscrape are in format ready for preprocessing, so we do not need extraction process for twitter data.
For scientific articles, we get files in tei.xml format after converting it from pdf in previous step. We use sci_articles.py script to extract required information from xml. We list information that we need from scientific article in config file. Example config:

type: sci
corpus_name: EIST # this will be used to save processed file as {corpus_name}.json
info_to_extract: # list of information to extract from xml file
  - doi
  - title
  - abstract
  - text
  - location
  - year
  - authors

Following command is used :

python src/data/extraction/sci_articles.py -config configs/data/extraction/sci_articles.yaml

A json file will be created which will contain extracted information for all the xml files.

{
    "1": {
        "file_name": "-It-s-not-talked-about---The-risk-of-failure-_2020_Environmental-Innovation-",
        "doi": "10.1016/j.eist.2020.02.008",
        "title": "Environmental Innovation and Societal Transitions",
        "abstract": "Scholars of sustainability transition have given much attention to local experiments in 'protected spaces' where system innovations can be initiated and where learning about those innovations can occur. However, local project participants' conceptions of success are often different to those of transition scholars; where scholars see a successful learning experience, participants may see a project which has failed to \"deliver\". This research looks at two UK case studies of energy retrofit projects -Birmingham Energy Savers and Warm Up North, both in the UK, and the opportunities they had for learning. The findings suggest that perceptions of failure and external real world factors reducing the capacity to experiment, meant that opportunities for learning were not well capitalised upon. This research makes a contribution to the sustainability transitions literature which has been criticised for focusing predominantly on successful innovation, and not on the impact of failure.",
        "text": {
            "Introduction": " A transition away from the use of fossil fuels to heat and power inefficient homes is urgent if the worst climate change predictions are to be avoided. Such a transition is complex, long term and uncertain, requiring change in multiple subsystems which are locked-in to high carbon usage (Unruh, 2000). Scholars of tran ..."

We go through an additional step to verify the data extracted for scientific articles from xml by validating it using a metatdata(csv) for articles generated using Scopus. We use following command to perform this task

python src/utils/verify_sci_articles.py -json_file EIST.json -csv_file EIST_metadata.csv

This validation step can also be done along with extraction if we add metadata_csv key to the data extraction config.

We use Bertopic for topic modelling. Our pipeline requires data in following format:

{
  "text": [
    "list of text to be used for topic modelling"
  ]
}

Data should be in json format and it must contain a key text which will be used for topic modelling. Value of this key is list of string which represent document. Data json can have other keys as well for example we have title, id, class etc. as metadata for each document in text.

To run topic modelling we use train config, snippet of config is shown below:

type: sci
model_name: EIST
train:
  supervised: False
  n_gram_range: # Give a range here. In example here, we use ngram from 1-3
    - 1
    - 3
  embedding_model:
    use: sentence_transformer
    combine: False
    name: paraphrase-multilingual-mpnet-base-v2 # allenai/scibert_scivocab_uncased
    max_seq_len: 512

This config is used along with topic_model.py script to perform topic modelling.

python src/train/topic_model.py -config configs/train/base.yaml

Currently, evaluation is only performed to get optimal number of topics for given dataset. It can be done independent of training pipeline,

python src/evaluate/using_lda.py -config configs/evaluate/lda.yaml

or can be integrated to train pipeline by adding find_optimal_topic_first key to train config as :

...
find_optimal_topic_first:
  config_path: ./configs/evaluate/lda.yaml
...

Note: This section is only validated against scientific articles pipeline yet.

There are several insights that can be made from the topics. Topic loading can be used to understand temporal trends like hot topics, cold topics, evergreen topics etc in the data. We can obtain this using the following command:

python src/analyze/temporal_trends.py

Successful execution of this command will create multiple files,

csvs
  sci
    topic_landscape.csv
    descriptive_stats.csv
    temporal_landscape.csv
plots
  sci
    loading_heatmap.jpg
    hot_topics.jpg
    cold_topics.jpg
    ...