The scrapysharp-dt2020 from leagify

Upgrade to LTS release.

Move system out println statements to use log file instead

This program is currently very "chatty" on the command line.
Changing these system out print statements to append to a log file, maybe in a logs directory, would be nice.
Different log levels would be nice, as there is certainly some verbose output that isn't always needed.

Move README file from src to root of directory

When adding tests, we moved files from the root of the folder to src/scrapysharp/.

One of them was the README:
https://github.com/Leagify/scrapysharp-dt2020/blob/master/src/scrapysharp/README.md

This should be moved back into the root of the repository so that GitHub can show off the information appropriately.

Consider converting height from X'X" to inches

Scraping and joining this field is leading to adding extra quotes, such as "6'1""" or """6'1""""""". It seems like 73 would be easier to handle.

Create csv output from scraped information

Consider using CsvHelper to implement.

Output should be similar to draft prospects csvs from last year, found here.

Write to log if height cannot be converted to inches correctly.

Right now, the height conversion just takes in the height and converts to inches, or outputs 0 if it can't be converted.

scrapysharp-dt2020/Program.cs

Line 423 in e263d63

return 0;

It might be nice to write to the log when a height can't be converted. In that case, we may want to refactor convertHeightToInches to take two arguments, the height and the playerName, then write something like:

Player playerName height of originalHeight not converted properly, entering 0 instead.

to the log file, similar to this:

scrapysharp-dt2020/Program.cs

Line 259 in e263d63

    
           File.AppendAllText($"logs{Path.DirectorySeparatorChar}Mismatches.log", $"{s.rank}, {s.name}, {s.college}" + Environment.NewLine);

Upgrade .NET Console experience.

Integrate some of the cool stuff from this article

Slight page format change.

There are two main changes to the page, as of 12-18-19.

Change number one- The column that previously had the player's alternate position ranking now indicates their grade, including whether or not they red-shirted.

Change number two is that declarations for the draft are now being color coded by line number:

Underclassmen Tracker (Most Recent Update Dec 18)
White Shading - Seniors
Yellow Shading - Underclassmen who have declared for the 2020 NFL Draft
Blue Shading - Underclassmen who have not made a decision yet

I'm going to see if I can access the shading of the line easily to add that contextual information.

Update .gitignore file so that compiled output isn't tracked.

I tried using a .gitignore file from www.gitignore.io to prevent this repository from tracking bin and obj files, but it doesn't seem to be working. This is particularly troublesome when trying to work on this repo in Linux or on a Mac, because the output in those directories appears to be OS-specific.

It would be nice if this worked correctly, and the appropriate files were removed from being tracked.

Add xUnit test project and first simple test

Using xUnit, create the appropriate test directory and one simple test that returns true.

I'm thinking that using the CLI might be a good option. Here's a Microsoft article about setting everything up:
https://docs.microsoft.com/en-us/dotnet/core/tutorials/testing-with-cli

Set up GitPod for this repository for easier development access.

I've heard really good things about GitPod.

I think it would be nice to set GitPod up on this repository, since open-source developers get 100 free hours on GitPod.

The instructions on how to get started are here: https://www.gitpod.io/docs/10_getting_started/

You can open this repository in GitPod by going here: https://gitpod.io/#https://github.com/Leagify/scrapysharp-dt2020

It is running the repo in a Linux container (Ubuntu 18.04, I think). However, the docker container that opens doesn't have .NET Core 3.0 installed, so running the code doesn't appear to be possible.

We can change this by setting up a .gitpod.yml file, and directions for that are here: https://www.gitpod.io/docs/41_config_gitpod_file/

Courtesy of GitPod workspaces on GitHub, there appears to be a prebuilt .NET Core environment here:
https://github.com/gitpod-io/workspace-images/blob/1440ffd296d9b6a2fe82a46d5c2b9e1532b9ccf2/dotnet/Dockerfile#L1-L9

It looks like this was added in this PR: gitpod-io/workspace-images#49

It looks like that installs .NET Core 2.2, but this project uses .NET Core 3.0

There's also a .NET VNC image, but I'm not sure why that should be used instead.

We need to figure out a way to configure a .NET Core 3.0 GitPod environment to get this going.

I think that this application is cross-platform friendly, so I hope that there are no code changes necessary to build, but that is a guess.

Hopefully that will help get everyone up and running.

Create release with executable

I think that dot net core 3 lets you make an executable that's less annoying. Consider also creating a linux executable.

Get Region from StatesToRegions.csv and join with LINQ

There is a StatesToRegions.csv file in the info directory. There are two columns - state and region, both strings.
We should read in that file into a new mapped class, maybe called Regions.
It will look similar to this:

scrapysharp-dt2020/Program.cs

Lines 119 to 126 in 8d98052

    
           // Get Schools and the States where they are located. 
        
           List<School> schoolsAndConferences; 
        
           using (var reader = new StreamReader($"info{Path.DirectorySeparatorChar}SchoolStatesAndConferences.csv")) 
        
           using (var csv = new CsvReader(reader)) 
        
           { 
        
               csv.Configuration.RegisterClassMap<SchoolCsvMap>(); 
        
               schoolsAndConferences = csv.GetRecords<School>().ToList(); 
        
           }

After we read in that csv into a list of Regions, we need to use LINQ to add a join to the region on
school.state to equal region.state.
Once we have properly joined that information in LINQ, we can add Region = Regions.region or whatever we choose to call it to the anonymous type that we are creating. That should add an additional column to joinedRanks2020.csv when we write it out.

Use CSVHelper more efficiently to read in CSV files from info directory.

CSVHelper is already loaded into this project, and it is very useful.

However, it is not being used consistently in Program.cs.
For example, there are some informational CSV files that are read in and stored as list objects so that we can use LINQ to join information properly.

Here's an example of CSV files that are read in without using CSVHelper:

scrapysharp-dt2020/Program.cs

Lines 108 to 137 in d50bb3c

    
                       var schoolsAndConferences = System.IO.File.ReadAllLines($"info{Path.DirectorySeparatorChar}SchoolStatesAndConferences.csv") 
        
                                                   .Skip(1) 
        
                                                   .Where(s => s.Length > 1) 
        
                                                   .Select( s => 
        
                                                   { 
        
                                                       var columns = s.Split(','); 
        
                                                       return new School(columns[0], columns[1], columns[2]); 
        
                                                   }) 
        
                                                   .ToList(); 
        
                       //Get position types 
        
                       var positionsAndTypes = System.IO.File.ReadAllLines($"info{Path.DirectorySeparatorChar}PositionInfo.csv") 
        
                                                   .Skip(1) 
        
                                                   .Where(s => s.Length > 1) 
        
                                                   .Select( s => 
        
                                                   { 
        
                                                       var columns = s.Split(','); 
        
                                                       return new PositionType(columns[0], columns[1], columns[2]); 
        
                                                   }) 
        
                                                   .ToList(); 
        
                       // Let's assign these ranks point values. 
        
                       var ranksToProjectedPoints = System.IO.File.ReadAllLines($"info{Path.DirectorySeparatorChar}RanksToProjectedPoints.csv") 
        
                                                   .Skip(1) 
        
                                                   .Where(s => s.Length > 1) 
        
                                                   .Select( s => 
        
                                                   { 
        
                                                       var columns = s.Split(','); 
        
                                                       return new PointProjection(columns[0], columns[1]); 
        
                                                   }) 
        
                                                   .ToList();

That is done in a hacky fashion, and one of the CSVHelper read functions should probably be used instead.
Information on CSVHelper CSV Read functions:
https://joshclose.github.io/CsvHelper/examples/reading/

As long as the output can still be joined in the LINQ statement, the refactored code would be nicer than what's there.

	// Get Schools and the States where they are located.
	List<School> schoolsAndConferences;
	using (var reader = new StreamReader($"info{Path.DirectorySeparatorChar}SchoolStatesAndConferences.csv"))
	using (var csv = new CsvReader(reader))
	{
	csv.Configuration.RegisterClassMap<SchoolCsvMap>();
	schoolsAndConferences = csv.GetRecords<School>().ToList();
	}

	var schoolsAndConferences = System.IO.File.ReadAllLines($"info{Path.DirectorySeparatorChar}SchoolStatesAndConferences.csv")
	.Skip(1)
	.Where(s => s.Length > 1)
	.Select( s =>
	{
	var columns = s.Split(',');
	return new School(columns[0], columns[1], columns[2]);
	})
	.ToList();

	//Get position types
	var positionsAndTypes = System.IO.File.ReadAllLines($"info{Path.DirectorySeparatorChar}PositionInfo.csv")
	.Skip(1)
	.Where(s => s.Length > 1)
	.Select( s =>
	{
	var columns = s.Split(',');
	return new PositionType(columns[0], columns[1], columns[2]);
	})
	.ToList();
	// Let's assign these ranks point values.
	var ranksToProjectedPoints = System.IO.File.ReadAllLines($"info{Path.DirectorySeparatorChar}RanksToProjectedPoints.csv")
	.Skip(1)
	.Where(s => s.Length > 1)
	.Select( s =>
	{
	var columns = s.Split(',');
	return new PointProjection(columns[0], columns[1]);
	})
	.ToList();

leagify / scrapysharp-dt2020 Goto Github PK

scrapysharp-dt2020's Introduction

leagify

scrapysharp-dt2020's People

Contributors

Stargazers

Watchers

Forkers

scrapysharp-dt2020's Issues

Recommend Projects

Recommend Topics

Recommend Org