- Clone repo
- BeautifulSoup? Required? $ pip install beautifulsoup4
- Yaml $ pip install pyyaml
Change the paths in the create-articles.yaml file in configurations folder
$> cd [repo]
$> cd scripts/create-articles/
$> python3 create-articles.py ../../configurations/create-articles.yaml > log.txt
For windows run the following command
python [path to create-articles.yaml file] > [log filename.txt]
This will generate a log.txt file in scripts/create-articles folder
This will generate dataset.yaml in artifacts/articles folder
This section provides details on how to consume the articles data and explore it for analysis. Following are the are ways to do it.
- Make sure that you have the articles folder made after running the above steps. it should be the
outputFolderPath
. - Find the folder
artifacts/articles/
. where you will find the articles data files in{files}.yaml
format. - Open the
Yaml-to-Dataframe.ipynb
notebook and give the path to the folder mentioned in Step-2. Follow rest of the instructions within the notebook, and you will have apandas
dataframe for exploration.
- Download the data file from the LINK - This data has articles uptill
31-12-2017
- Extract the compressed file using 7zip.
- Paste the data file in the working directory.
- Open the
Yaml-to-Dataframe.ipynb
notebook, Scroll down to theQuick Way : Load data file
- Execute the cell and you will have the dataframe to explore.