- Python (3.9 or later)
- Pipenv
- Java (1.8 or later)
pipenv install
pipenv run unit-test
pipenv run integration-test
pipenv run packager
pipenv run linter
At HelloFresh we have a big recipes archive that was created over the last 8 years. It is constantly being updated either by adding new recipes or by making changes to existing ones. We have a service that can dump archive in JSON format to selected s3 location. We are interested in tracking changes to see available recipes, their cooking time and difficulty level.
There are recipe files for this under resources/recipes/*.json
with historical data.
Reads *.json
files and transforms it to parquet
format, casting fields with the proper type.
Historical recipes *.json
files, this is a recipe example:
{
"name": "Creamy Cheese Grits with Chilies",
"ingredients": "4-1/2 cups Water\n1/2 teaspoon Salt\n1 cup Grits (quick Or Regular)\n1/2 can (10 Ounce Can) Rotel (tomatoes And Chilies)\n1 can (4 Ounce Can) Chopped Green Chilies\n8 ounces, weight Monterey Jack Cheese, Grated\n4 ounces, weight Cream Cheese, Cut Into Cubes\n1/4 teaspoon Cayenne Pepper\n1/4 teaspoon Paprika\n Black Pepper To Taste\n1 whole Egg Beaten",
"url": "http://thepioneerwoman.com/cooking/2010/10/creamy-cheese-grits-with-chilies/",
"image": "http://static.thepioneerwoman.com/cooking/files/2010/10/5079611293_ff628b6e0c_z.jpg",
"cookTime": "PT45M",
"recipeYield": "8",
"datePublished": "2010-10-14",
"prepTime": "PT5M",
"description": "I have a good, basic recipe for cheese grits in my cookbook, but last night I was feeling feisty. I was cooking steaks. B..."
}
*.parquet
files applying the following:
-
cookTime
andprepTime
are converted from text to number (minutes) -
datePublished
andrecipeYield
are casted to date and number respectively.
Please make sure to package the code before submitting the spark job (pipenv run packager
)
pipenv run spark-submit
--master local
--py-files dist/hellofresh-0.1.0-py3.9.egg
jobs/hellofresh_job.py
INGEST
<INPUT_FILE_PATH>
<OUTPUT_PATH>
pipenv run spark-submit --master local --py-files dist/hellofresh-0.1.0-py3.9.egg jobs/hellofresh_job.py INGEST resources/recipes resources/transformed/recipes
This task is responsible for:
- Extract only recipes that have beef as one of the ingredients.
- Calculate average cooking time duration per difficulty level.
- Persist dataset as CSV to the output folder.
The *.parquet
files ingested previously.
The dataset have 2 columns: difficulty,avg_total_cooking_time. Total cooking time duration = cookTime + prepTime
Criteria for levels based on total cook time duration:
easy - less than 30 mins
medium - between 30 and 60 mins
hard - more than 60 mins.
pipenv run spark-submit
--master local
--py-files dist/hellofresh-0.1.0-py3.9.egg
jobs/hellofresh_job.py
CALC
<INPUT_PATH>
<OUTPUT_PATH>
pipenv run spark-submit --master local --py-files dist/hellofresh-0.1.0-py3.9.egg jobs/hellofresh_job.py CALC resources/transformed/recipes resources/output/recipes
Recipes CSV
File.
difficulty,avg_total_cooking_time
medium,45.0
hard,194.3913043478261
easy,19.625