How easy is it to read large text files with pyspark? In this repository I show you some source codes that you can use to practice python with spark and text files.
- For this example, I'll use these text files found in the "1_input" folder:
- Both files are large, not because of the number of rows but because of the number of characters per row:
- I want two data from these files, the supply number and the flag if this supply has digital delivery:
- The supply number is in position 2 and has 7 characters.
- The "digital delivery" flag is at position 13 337 and has 1 characters.
- To upload these files and merge both, I'll be using Python and Spark.