Issues with projection sources about ffanalytics HOT 33 CLOSED

Hi! Yes, we review these. ESPN changed their site, so we are downloading their projections manually. We can add these to our to-do list to look into. People are welcome to make pull requests to address any bugs: https://fantasyfootballanalytics.net/about-the-site/faq#shareScripts
Thanks!

from ffanalytics.

I'm relatively new to R as well, so my approach may not be most efficient, but ...

I cut and paste the html_sites code from the package in to my script, made the edits, and then ran the code. I believe this causes my html_sites definition to supersede the html_sites from the package, which fixes the error.

Hopefully the change will be incorporated into the package. I'd do it myself, but I don't understand how to make changes, commit to repo, etc.

I went ahead and tried it, and this is what worked for me:

1. I downloaded the raw code from here
2. then I opened source_configs.R and changed the following in the Yahoo list:
   • league_id to your new id
   • yahoo_qry[status] from "A" to "ALL"
   • and min_week/max_week to 0/17
3. ran source_configs excluding the last line
4. then ran scrape_data
   and it included yahoo projections!

from ffanalytics.

Hey @gheemony . You should probably open this issue on the ffespn issue tracker instead. I only posted it here to see if could help out the ffanalytics community get espn projections in the meantime. That being said - I'm happy to look into it!

from ffanalytics.

Adding this here because it's relevant.

the only sources I'm able to pull are CBS, FantasyPros, FantasySharks, FFToday, FleaFlicker, NumberFire, & FantasyFootballNerd

My own priority is to get Yahoo functioning, which I haven't seen mentioned as a problem up until this point.
I did notice in the source_configs.R file that min_week & max_week are both set to -1. I changed them to 0 and 17 but that didn't seem to fix it (although I'm not that skilled in all this stuff).

If I run the following code:
my_scrape <- scrape_data(season = 2020, week = 1)

I only get projections from the sources listed above, and I get this:

Warning messages:
1: In .Primitive("as.double")(x, ...) : NAs introduced by coercion
2: `funs()` is deprecated as of dplyr 0.8.0.
Please use a list of either functions or lambdas: 

  # Simple named list: 
  list(mean = mean, median = median)

  # Auto named with `tibble::lst()`: 
  tibble::lst(mean, median)

  # Using lambdas
  list(~ mean(., trim = .2), ~ median(., na.rm = TRUE))

when I run this code:
scrape_yahoo <- scrape_source(src = "Yahoo", season = 2020, week = 1, pos = "QB")

I get this error (regardless of the source specified):
Error: $ operator is invalid for atomic vectors

Here's my session info:

R version 4.0.2 (2020-06-22)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 18363)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] ffanalytics_2.0.2.0001 Hmisc_4.4-1            Formula_1.2-3          survival_3.1-12        lattice_0.20-41        glue_1.4.2             janitor_2.0.1         
 [8] readxl_1.3.1           httr_1.4.2             rvest_0.3.6            xml2_1.3.2             forcats_0.5.0          stringr_1.4.0          dplyr_1.0.2           
[15] purrr_0.3.4            readr_1.3.1            tidyr_1.1.2            tibble_3.0.3           ggplot2_3.3.2          tidyverse_1.3.0       

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.5          lubridate_1.7.9     png_0.1-7           digest_0.6.25       assertthat_0.2.1    R6_2.4.1            cellranger_1.1.0    backports_1.1.9    
 [9] reprex_0.3.0        pillar_1.4.6        rlang_0.4.7         curl_4.3            data.table_1.13.0   rstudioapi_0.11     blob_1.2.1          rpart_4.1-15       
[17] Matrix_1.2-18       checkmate_2.0.0     splines_4.0.2       selectr_0.4-2       foreign_0.8-80      htmlwidgets_1.5.1   munsell_0.5.0       tinytex_0.25       
[25] broom_0.7.0         compiler_4.0.2      modelr_0.1.8        xfun_0.17           pkgconfig_2.0.3     base64enc_0.1-3     htmltools_0.5.0     nnet_7.3-14        
[33] tidyselect_1.1.0    htmlTable_2.0.1     gridExtra_2.3       fansi_0.4.1         crayon_1.3.4        dbplyr_1.4.4        withr_2.2.0         grid_4.0.2         
[41] jsonlite_1.7.1      gtable_0.3.0        lifecycle_0.2.0     DBI_1.1.0           magrittr_1.5        scales_1.1.1        cli_2.0.2           stringi_1.5.3      
[49] fs_1.5.0            latticeExtra_0.6-29 snakecase_0.11.0    ellipsis_0.3.1      generics_0.0.2      vctrs_0.3.4         RColorBrewer_1.1-2  tools_4.0.2        
[57] hms_0.5.3           jpeg_0.1-8.1        colorspace_1.4-1    cluster_2.1.0       knitr_1.29          haven_2.3.1 

You can get Yahoo to work by changing the league_id to a known league (I created a new one this morning) and changing "Status=A" to "Status=All".

from ffanalytics.

Are these issues being reviewed by anyone?

from ffanalytics.

@MrDAndersen I'll second this issue.

from ffanalytics.

Thanks! I'm brushing up on my R, but hopefully I can make some PRs in the not too distant future :) Would it be possible to add some form of license to the repo? I'm not sure what use cases are allowed (Github's TOS is pretty limited, even for personal use). Also, would it be worth adding a CSV of the manual scrape to the library for the time being?

from ffanalytics.

ESPN has an API but it is in JSON. I just haven't had the time to fudge around besides extracting the data. 1 thing is you have to know the column mapping. Like 22= rushing attempts.

Also my CBS data doesn't have player information. Anyone else having this problem?

from ffanalytics.

from ffanalytics.

https://fantasy.espn.com/apis/v3/games/ffl/seasons/2019/segments/0/leagues/8557?view=kona_player_info

8557 is replaced with your league id....if it isn't public you'll have issues. I found this one to work...thanks whomever they are lol.

Kona_player_info is the data for projections it seems.

I'd you Google ESPN fantasy API V3 you'll get more info. This is apparently undocumented. There is a history link too for prior years I've read.

from ffanalytics.

I made an attempt...I got the data in a table.

One thing i noticed is that if a player did not have any data for a stat..ex Receptions..they would not have the key in their stats. This was a paid for WR or a RB lol.

If you to my ffanalytics fork on my account under test/ESPN.R you'll see the R code i came up with. It probably is crude but you get stats at least.

from ffanalytics.

Note that for FantasyFootballNerd you will need a valid API key. You can get a free one by registering on the site, but that only gives you the to 15 at each position. Further, FantasyFootballNerd is an aggregation of other (unspecified) sites so by using that source with others you may be getting some sources included twice.

from ffanalytics.

FYI: CBS week 1 projection scrape is not generating any MFL ID matches.

from ffanalytics.

FYI: Week 1 projections for FFToday are available, but error when attempting to scrape.

from ffanalytics.

What error do you get? I pulled it just fine.

Yahoo I get IDs for QB, WR but not RB.

I do know the split cols function for CBS needs changed to get the player info. Player needs to be PLAYER and it didn't split the column.

from ffanalytics.

Current errors/issues with Week 1:

• FFToday: scrape_data causes this error:

Error in out[j + k, ] : subscript out of bounds In addition: Warning message: In .Primitive("as.double")(x, ...) : NAs introduced by coercion Called from: utils::head(out[j + k, ], i - 1) Error during wrapup: arguments imply differing number of rows: 1, 576, 3798, 5388, 10217, 164

• ID Matching That doesn't work
  QB: NumberFire Rookies
  RB: CBS, NumberFire Rookies, a small number of NFL, Sharks and Yahoo RBs
  WR: NumberFire Rookies, a small number of NFL, Sharks and Yahoo WRs
  TE: CBS, NumberFire Rookies, a small number of NFL TEs
  DST: Yahoo

The CBS projections are very bad (suspended Kareem Hunt is RB4), so I may not incorporate them until they get their act together

from ffanalytics.

I am getting weekly data using this repo and mine for FFToday.

As for the ID the only major one I have issues with is CBS. Number fire when use add_player_info(), it shows Kyler Murrary still there.

CBS I think is last years week 1 data lol. CBS is being garbage

from ffanalytics.

Posting this here in case anyone is interested in helping with the project:
https://fantasyfootballanalytics.net/2019/09/call-for-help-lead-r-shiny-developer.html

from ffanalytics.

Hoping to re-up this issue; as of today, five sources are busted - ESPN, Yahoo, NFL, FleaFlicker, FantasyData.

from ffanalytics.

ESPN has a new API so wont happen.

NFL I think closed their public feeds.

The current working ones have ID column issues for rookies on 2 sources

I have code to work around it to marxh the names which gets the relevant ones

from ffanalytics.

If it's of interest, here's a script to pull in ESPN projections from the new API. Obviously not a drop-in replacement and I don't have enough knowledge of the inner workings of this package to do it myself, but I thought it might be helpful (no worries if not, of course). League ID 8557 is a public league, any public league ID should work there. Hopefully the column names I chose are intuitive.

# Load packages, installing if necessary
if(require(jsonlite)==F){install.packages('jsonlite'); library(jsonlite)} else {library(jsonlite)}
if(require(tidyverse)==F){install.packages('tidyverse'); library(tidyverse)} else {library(tidyverse)}

# Grab the JSON from a known public league; can be any, since grabbing stats not points
json_file <- "https://fantasy.espn.com/apis/v3/games/ffl/seasons/2020/segments/0/leagues/8557?view=kona_player_info"
json_data <- fromJSON(paste(readLines(json_file), collapse=""))

# Empty object to append player rows to
projections = c()

# Loop through all players (no positional separation, do that elsewhere)
for(p in json_data$players){
  player_df = data.frame(
    id   = p$player$id,
    name = p$player$fullName,
    
    # Only grab values if they exists.
    pass_comp  = ifelse(!is.null(p$player$stats[[3]]$stats$`1`),p$player$stats[[3]]$stats$`1`,NA),
    pass_att   = ifelse(!is.null(p$player$stats[[3]]$stats$`0`),p$player$stats[[3]]$stats$`0`,NA),
    pass_yds   = ifelse(!is.null(p$player$stats[[3]]$stats$`3`),p$player$stats[[3]]$stats$`3`,NA),
    pass_td    = ifelse(!is.null(p$player$stats[[3]]$stats$`4`),p$player$stats[[3]]$stats$`4`,NA),
    pass_2pt   = ifelse(!is.null(p$player$stats[[3]]$stats$`19`),p$player$stats[[3]]$stats$`19`,NA),
    pass_int   = ifelse(!is.null(p$player$stats[[3]]$stats$`20`),p$player$stats[[3]]$stats$`20`,NA),
    
    rush_att   = ifelse(!is.null(p$player$stats[[3]]$stats$`23`),p$player$stats[[3]]$stats$`23`,NA),
    rush_yds   = ifelse(!is.null(p$player$stats[[3]]$stats$`24`),p$player$stats[[3]]$stats$`24`,NA),
    rush_td    = ifelse(!is.null(p$player$stats[[3]]$stats$`25`),p$player$stats[[3]]$stats$`25`,NA),
    rush_2pt   = ifelse(!is.null(p$player$stats[[3]]$stats$`26`),p$player$stats[[3]]$stats$`26`,NA),
    
    rec        = ifelse(!is.null(p$player$stats[[3]]$stats$`53`),p$player$stats[[3]]$stats$`53`,NA),
    rec_tgt    = ifelse(!is.null(p$player$stats[[3]]$stats$`58`),p$player$stats[[3]]$stats$`58`,NA),
    rec_yds    = ifelse(!is.null(p$player$stats[[3]]$stats$`42`),p$player$stats[[3]]$stats$`42`,NA),
    rec_td     = ifelse(!is.null(p$player$stats[[3]]$stats$`43`),p$player$stats[[3]]$stats$`43`,NA),
    rec_2pt    = ifelse(!is.null(p$player$stats[[3]]$stats$`44`),p$player$stats[[3]]$stats$`44`,NA),
    
    fumbles    = ifelse(!is.null(p$player$stats[[3]]$stats$`72`),p$player$stats[[3]]$stats$`72`,NA),
    
    fg0039     = ifelse(!is.null(p$player$stats[[3]]$stats$`80`),p$player$stats[[3]]$stats$`80`,NA),
    fgmiss0039 = ifelse(!is.null(p$player$stats[[3]]$stats$`82`),p$player$stats[[3]]$stats$`82`,NA),
    fg4049     = ifelse(!is.null(p$player$stats[[3]]$stats$`77`),p$player$stats[[3]]$stats$`77`,NA),
    fgmiss4049 = ifelse(!is.null(p$player$stats[[3]]$stats$`79`),p$player$stats[[3]]$stats$`79`,NA),
    fg50       = ifelse(!is.null(p$player$stats[[3]]$stats$`74`),p$player$stats[[3]]$stats$`74`,NA),
    fgmiss50   = ifelse(!is.null(p$player$stats[[3]]$stats$`76`),p$player$stats[[3]]$stats$`76`,NA),
    xpt        = ifelse(!is.null(p$player$stats[[3]]$stats$`86`),p$player$stats[[3]]$stats$`86`,NA),
    xptmiss    = ifelse(!is.null(p$player$stats[[3]]$stats$`88`),p$player$stats[[3]]$stats$`88`,NA),
    
    sacks      = ifelse(!is.null(p$player$stats[[3]]$stats$`99`),p$player$stats[[3]]$stats$`99`,NA),
    fforced    = ifelse(!is.null(p$player$stats[[3]]$stats$`106`),p$player$stats[[3]]$stats$`106`,NA),
    frecovered = ifelse(!is.null(p$player$stats[[3]]$stats$`96`),p$player$stats[[3]]$stats$`96`,NA),
    def_int    = ifelse(!is.null(p$player$stats[[3]]$stats$`95`),p$player$stats[[3]]$stats$`95`,NA),
    safeties   = ifelse(!is.null(p$player$stats[[3]]$stats$`98`),p$player$stats[[3]]$stats$`98`,NA),
    blocks     = ifelse(!is.null(p$player$stats[[3]]$stats$`97`),p$player$stats[[3]]$stats$`97`,NA),
    
    block_td   = ifelse(!is.null(p$player$stats[[3]]$stats$`93`),p$player$stats[[3]]$stats$`93`,NA),
    kret_td    = ifelse(!is.null(p$player$stats[[3]]$stats$`101`),p$player$stats[[3]]$stats$`101`,NA),
    pret_td    = ifelse(!is.null(p$player$stats[[3]]$stats$`102`),p$player$stats[[3]]$stats$`102`,NA),
    fumret_td  = ifelse(!is.null(p$player$stats[[3]]$stats$`103`),p$player$stats[[3]]$stats$`103`,NA),
    intret_td  = ifelse(!is.null(p$player$stats[[3]]$stats$`104`),p$player$stats[[3]]$stats$`104`,NA),
    
    pa0        = ifelse(!is.null(p$player$stats[[3]]$stats$`89`),p$player$stats[[3]]$stats$`89`,NA),
    pa16       = ifelse(!is.null(p$player$stats[[3]]$stats$`90`),p$player$stats[[3]]$stats$`90`,NA),
    pa713      = ifelse(!is.null(p$player$stats[[3]]$stats$`91`),p$player$stats[[3]]$stats$`91`,NA),
    pa1417     = ifelse(!is.null(p$player$stats[[3]]$stats$`92`),p$player$stats[[3]]$stats$`92`,NA),
    pa1720     = ifelse(!is.null(p$player$stats[[3]]$stats$`121`),p$player$stats[[3]]$stats$`121`,NA),
    pa2127     = ifelse(!is.null(p$player$stats[[3]]$stats$`122`),p$player$stats[[3]]$stats$`122`,NA),
    pa2834     = ifelse(!is.null(p$player$stats[[3]]$stats$`123`),p$player$stats[[3]]$stats$`123`,NA),
    pa3545     = ifelse(!is.null(p$player$stats[[3]]$stats$`124`),p$player$stats[[3]]$stats$`124`,NA),
    pa45plus   = ifelse(!is.null(p$player$stats[[3]]$stats$`125`),p$player$stats[[3]]$stats$`125`,NA)
  )
  
  projections <- bind_rows(projections, player_df)
  
}

from ffanalytics.

Yeah that helps alot lol. I never had the time to sort out all the stat columns because they are assholes and used numbers lol.

The projection sources.R file is what makes up the main code that defines all the columns and the url type stuff. The rda file in there works but when I run the R file to make it, it fails with CBS DST

I manually edited my data set from the rda file till I can figure what is different

from ffanalytics.

Ugh, ESPN paginated this, so what I posted doesn't work any more (well, actually, it works but only through Geronimo Allison, alphabetically by last name)

from ffanalytics.

if this helps anyone, I have a separate package, ffespn to gather ESPN projections. The column names might not match up exactly to ffanalytics but it should definitely help you start gathering data.

To the ffanalytics team, I really love this package so thanks for the hard work! I've just been having trouble getting my ESPN projections to work this year so I decided to write my own. Feel free to steal my code if it helps. I've only tested the code on Week 0.

Example:

remotes::install_github("dfs-with-r/ffespn")
library(ffespn)
qb <- ffespn_projections(2020, 0, "QB")

from ffanalytics.

from ffanalytics.

Been keeping tabs on this as I was excited by johnchoiniere's workaround, but then destroyed by the change a few days ago with the 50 record limit. Some jedi wizard figured it out in this repo: https://github.com/cwendt94/espn-api/blob/master/espn_api)/football/league.py#L250-L251 , and was expounded on in this reddit post: https://www.reddit.com/r/fantasyfootball/comments/im6mui/espn_api_question/?sort=new .

Here's the relevant bit:

Basically, you need to include a x-fantasy-filter with a value that is a JSON object that looks like this:

{
"players": {
"limit": 1500,
"sortDraftRanks": {
"sortPriority": 100,
"sortAsc": true,
"value": "STANDARD"
}
}
}

Super obvious, right...? Note the 1500 value for limit is what gets you the full list of players (typically somewhere around 1000). You could also use a smaller limit and add an offset, if you wanted to mess around with pagination.

For R users, define your filters as a nested list, then use jsonlite::toJSON() when you add the header to your request.

from ffanalytics.

@Jeffery-777 There are a number of reasons for why you might be getting that error. I'd recommend the following steps:

1. Update your R installation to the latest version
2. Update the packages that ffanalytics originally asked you to update (but before installing ffanalytics)
3. Install ffanalytics

from ffanalytics.

from ffanalytics.

Is anyone able to get projections from NFL.com?

from ffanalytics.

if this helps anyone, I have a separate package, ffespn to gather ESPN projections. The column names might not match up exactly to ffanalytics but it should definitely help you start gathering data.

To the ffanalytics team, I really love this package so thanks for the hard work! I've just been having trouble getting my ESPN projections to work this year so I decided to write my own. Feel free to steal my code if it helps. I've only tested the code on Week 0.

Example:

remotes::install_github("dfs-with-r/ffespn")
library(ffespn)
qb <- ffespn_projections(2020, 0, "QB")

Appears that this package doesn't work with week 1. :-(

from ffanalytics.

Adding this here because it's relevant.

the only sources I'm able to pull are CBS, FantasyPros, FantasySharks, FFToday, FleaFlicker, NumberFire, & FantasyFootballNerd

My own priority is to get Yahoo functioning, which I haven't seen mentioned as a problem up until this point.
I did notice in the source_configs.R file that min_week & max_week are both set to -1. I changed them to 0 and 17 but that didn't seem to fix it (although I'm not that skilled in all this stuff).

If I run the following code:
my_scrape <- scrape_data(season = 2020, week = 1)

I only get projections from the sources listed above, and I get this:

Warning messages:
1: In .Primitive("as.double")(x, ...) : NAs introduced by coercion
2: `funs()` is deprecated as of dplyr 0.8.0.
Please use a list of either functions or lambdas: 

  # Simple named list: 
  list(mean = mean, median = median)

  # Auto named with `tibble::lst()`: 
  tibble::lst(mean, median)

  # Using lambdas
  list(~ mean(., trim = .2), ~ median(., na.rm = TRUE))

when I run this code:
scrape_yahoo <- scrape_source(src = "Yahoo", season = 2020, week = 1, pos = "QB")

I get this error (regardless of the source specified):
Error: $ operator is invalid for atomic vectors

Here's my session info:

R version 4.0.2 (2020-06-22)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 18363)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] ffanalytics_2.0.2.0001 Hmisc_4.4-1            Formula_1.2-3          survival_3.1-12        lattice_0.20-41        glue_1.4.2             janitor_2.0.1         
 [8] readxl_1.3.1           httr_1.4.2             rvest_0.3.6            xml2_1.3.2             forcats_0.5.0          stringr_1.4.0          dplyr_1.0.2           
[15] purrr_0.3.4            readr_1.3.1            tidyr_1.1.2            tibble_3.0.3           ggplot2_3.3.2          tidyverse_1.3.0       

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.5          lubridate_1.7.9     png_0.1-7           digest_0.6.25       assertthat_0.2.1    R6_2.4.1            cellranger_1.1.0    backports_1.1.9    
 [9] reprex_0.3.0        pillar_1.4.6        rlang_0.4.7         curl_4.3            data.table_1.13.0   rstudioapi_0.11     blob_1.2.1          rpart_4.1-15       
[17] Matrix_1.2-18       checkmate_2.0.0     splines_4.0.2       selectr_0.4-2       foreign_0.8-80      htmlwidgets_1.5.1   munsell_0.5.0       tinytex_0.25       
[25] broom_0.7.0         compiler_4.0.2      modelr_0.1.8        xfun_0.17           pkgconfig_2.0.3     base64enc_0.1-3     htmltools_0.5.0     nnet_7.3-14        
[33] tidyselect_1.1.0    htmlTable_2.0.1     gridExtra_2.3       fansi_0.4.1         crayon_1.3.4        dbplyr_1.4.4        withr_2.2.0         grid_4.0.2         
[41] jsonlite_1.7.1      gtable_0.3.0        lifecycle_0.2.0     DBI_1.1.0           magrittr_1.5        scales_1.1.1        cli_2.0.2           stringi_1.5.3      
[49] fs_1.5.0            latticeExtra_0.6-29 snakecase_0.11.0    ellipsis_0.3.1      generics_0.0.2      vctrs_0.3.4         RColorBrewer_1.1-2  tools_4.0.2        
[57] hms_0.5.3           jpeg_0.1-8.1        colorspace_1.4-1    cluster_2.1.0       knitr_1.29          haven_2.3.1 

from ffanalytics.

I apologize, as this is my first experience really using github like this.

You can get Yahoo to work by changing the league_id to a known league (I created a new one this morning) and changing "Status=A" to "Status=All".

OK, so I did this in the source_configs.R that I downloaded, then installed that version of the package using this command:

install.packages("~/ffanalytics", 
                 repos = NULL, 
                 type = "source")``

load the package, and get the same result. Am I doing that wrong?

from ffanalytics.

I'm relatively new to R as well, so my approach may not be most efficient, but ...

I cut and paste the html_sites code from the package in to my script, made the edits, and then ran the code. I believe this causes my html_sites definition to supersede the html_sites from the package, which fixes the error.

Hopefully the change will be incorporated into the package. I'd do it myself, but I don't understand how to make changes, commit to repo, etc.

I apologize, as this is my first experience really using github like this.

You can get Yahoo to work by changing the league_id to a known league (I created a new one this morning) and changing "Status=A" to "Status=All".

OK, so I did this in the source_configs.R that I downloaded, then installed that version of the package using this command:

install.packages("~/ffanalytics", 
                 repos = NULL, 
                 type = "source")``

load the package, and get the same result. Am I doing that wrong?

from ffanalytics.