ropensci / skynet Goto Github PK

View Code? Open in Web Editor NEW

10.0 5.0 4.0 62.54 MB

Generates networks from BTS DB1B/T-100 data

Home Page: https://docs.ropensci.org/skynet

License: Other

R 100.00%

db1b transtats bureau-of-transport-statistics bts rita t100 air-transport skynet peer-reviewed r

skynet's Introduction

skynet

Overview

The rationale behind Skynet, is to provide researchers with a unifying tool overcoming some of the challenges faced when dealing with the Bureau of Transport Statistics, DB1B and T100 data. The DB1B data consists of 2 sets of files, Coupon and Ticket. They can be both downloaded at https://www.transtats.bts.gov/Fields.asp?gnoyr_VQ=FLM and https://www.transtats.bts.gov/Fields.asp?gnoyr_VQ=FKF respectively while the T100 data can be found here https://www.transtats.bts.gov/Fields.asp?gnoyr_VQ=FIL.

Note

To comply with R syntax guidelines, we changed to a clearer function naming from version 1.2.0. Deprecated functions are still present, but will be removed for the next versions.

Note on importing from other data sources

We are constantly working on new functions that allow importing data from different data sources. However, as we can’t cover them all at least for now, in case you would like to work with a database which is not covered by skynet, simply create a data.frame with the following variables:

itin_id, mkt_id, seq_num, origin_mkt_id, origin, year, quarter, dest_mkt_id, dest, trip_break, op_carrier, distance, gateway, roundtrip, itin_yield, passengers, itin_fare, bulk_fare, distance_full

For more information on the variables, please visit https://www.transtats.bts.gov/Fields.asp?gnoyr_VQ=FLM and https://www.transtats.bts.gov/Fields.asp?gnoyr_VQ=FKF.

Skynet allows that some of this variables have a 0 or NA value, however, if you’re working with a specific dataset which doesn’t allow an easy conversion to our format, please feel free to create an issue so we can look into it. Please make sure to include at least one small example of a csv file with the data you’re trying to import.

Installation

You can install skynet from github with:

# install.packages("devtools")
devtools::install_github("FilipeamTeixeira/skynet")

Import Data

To import data, simply type import_db1b() or import_t100() including the path to your desired file.
Note: The Coupon file should take the first argument while the Ticket file should take the second argument.

 library(skynet)
 import_db1b("folder/Coupon 2016Q1.csv", "folder/Ticket 2016Q1.csv")
 import_t100("folder/T100_2016.csv")

The BTS DB1B data consists of 2 sets of files, Coupon and Ticket. They can be both downloaded at https://www.transtats.bts.gov/Fields.asp?gnoyr_VQ=FLM and https://www.transtats.bts.gov/Fields.asp?gnoyr_VQ=FKF respectively.

Despite being possible to download the complete zipped file, which includes all variables, due to its size, we recommend selecting the following set.

Coupon	Ticket
Itinerary ID	Itinerary ID
Market ID	Roundtrip
Sequence Number	Itinerary Yield
Origin City Market ID	Passengers
Origin	Itinerary Fare
Year	Bulkfare Indicator
Quarter	Distance
Destination City Market ID
Destination
Trip Break
Operating Carrier
Distance
Gateway

Since version 1.0.2 that the import method changed being the netimport() function no longer available. When importing from the prezipped DB1B file, just add the argument zip = TRUE to the import_db1b() function. This does not apply to the T100 file which can be simply imported by typing import_t100(). In order to save space, it is possible as well to import the prezipped file, and convert it to a smaller file with only the necessary variables, with the function convert_raw().

Example

To generate a directed network, please type:

library(skynet)
# For DB1B data
import_db1b("folder/Coupon_2011Q1.csv", "folder/Ticket_2011Q1.csv")
make_net_dir(OD_2011Q1, disp = TRUE, alpha = 0.05)

# For T100 data
import_t100("folder/T100_2011.csv")
make_net_dir(T100_2011Q1, disp = TRUE, alpha = 0.05)

skynet's People

Contributors

Stargazers

Watchers

Forkers

guoyu07 nemochina2008 shriyakaram gimeno65

skynet's Issues

download_t100 not reaching website

The basic idea is to use R to download a zip file generated here https://www.transtats.bts.gov/DL_SelectFields.aspx?gnoyr_VQ=FIL&QO_fu146_anzr=Nv4%20Pn44vr45.

When accessing the website above, we can select from a table, the variables we want, the year, and month (or all months), and press download.
Before, when I right-clicked on the POST line and choose "Copy as cURL", I would get a really nice set of code which I could translate into this:

httr::POST(
  url = "https://www.transtats.bts.gov/DownLoad_Table.asp",
  httr::add_headers(
    Referer = "https://www.transtats.bts.gov/DL_SelectFields.asp?Table_ID=289"
  ),
  body = list(
    UserTableName = "DB1BCoupon",
    DBShortName = "", 
    RawDataTable = "T_DB1B_COUPON",
    sqlstr = " SELECT ORIGIN_AIRPORT_ID,ORIGIN_AIRPORT_SEQ_ID,ORIGIN_CITY_MARKET_ID,DEST_AIRPORT_ID,DEST_AIRPORT_SEQ_ID,DEST_CITY_MARKET_ID FROM T_DB1B_COUPON WHERE Quarter=1 AND YEAR=2018",
    varlist = "ORIGIN_AIRPORT_ID,ORIGIN_AIRPORT_SEQ_ID,ORIGIN_CITY_MARKET_ID,DEST_AIRPORT_ID,DEST_AIRPORT_SEQ_ID,DEST_CITY_MARKET_ID",
    grouplist = "", suml = "",
    sumRegion = "", filter1 = "title=",
    filter2 = "title=", geo = "All\xa0",
    time = "Q+1", timename = "Quarter",
    GEOGRAPHY = "All", XYEAR = "2018",
    FREQUENCY = "1", 
    VarDesc = "ItinID", VarType = "Num", 
    VarDesc = "MktID", VarType = "Num", 
    VarDesc = "SeqNum", VarType = "Num", 
    VarDesc = "Coupons", VarType = "Num", 
    VarDesc = "Year", VarType = "Num", VarName = "ORIGIN_AIRPORT_ID", 
    VarDesc = "OriginAirportID", VarType = "Num", VarName = "ORIGIN_AIRPORT_SEQ_ID", 
    VarDesc = "OriginAirportSeqID", VarType = "Num", VarName = "ORIGIN_CITY_MARKET_ID", 
    VarDesc = "OriginCityMarketID", VarType = "Num", 
    VarDesc = "Quarter", VarType = "Num", 
    VarDesc = "Origin", VarType = "Char", 
    VarDesc = "OriginCountry", VarType = "Char", 
    VarDesc = "OriginStateFips", VarType = "Char", 
    VarDesc = "OriginState", VarType = "Char", 
    VarDesc = "OriginStateName", VarType = "Char", 
    VarDesc = "OriginWac", VarType = "Num", VarName = "DEST_AIRPORT_ID", 
    VarDesc = "DestAirportID", VarType = "Num", VarName = "DEST_AIRPORT_SEQ_ID", 
    VarDesc = "DestAirportSeqID", VarType = "Num", VarName = "DEST_CITY_MARKET_ID", 
    VarDesc = "DestCityMarketID", VarType = "Num", 
    VarDesc = "Dest", VarType = "Char", 
    VarDesc = "DestCountry", VarType = "Char", 
    VarDesc = "DestStateFips", VarType = "Char", 
    VarDesc = "DestState", VarType = "Char", 
    VarDesc = "DestStateName", VarType = "Char", 
    VarDesc = "DestWac", VarType = "Num", 
    VarDesc = "Break", VarType = "Char", 
    VarDesc = "CouponType", VarType = "Char", 
    VarDesc = "TkCarrier", VarType = "Char", 
    VarDesc = "OpCarrier", VarType = "Char", 
    VarDesc = "RPCarrier", VarType = "Char", 
    VarDesc = "Passengers", VarType = "Num", 
    VarDesc = "FareClass", VarType = "Char", 
    VarDesc = "Distance", VarType = "Num", 
    VarDesc = "DistanceGroup", VarType = "Num", 
    VarDesc = "Gateway", VarType = "Num", 
    VarDesc = "ItinGeoType", VarType = "Num", 
    VarDesc = "CouponGeoType", VarType = "Num"
  ), 
  encode = "form",
  query = list(
    Table_ID = "289",
    Has_Group = "0", 
    Is_Zipped = "0"
  )
) -> res

The code used to extract the cURL was this:

library(curlconverter)

straighten() %>% make_req() -> tmp

The new saved res, had binary data which I could easily use to download the zip file generated by the POST.

res
## Response [https://transtats.bts.gov/ftproot/TranStatsData/351117019_T_DB1B_COUPON.zip]
##   Date: 2018-10-14 02:18
##   Status: 200
##   Content-Type: application/x-zip-compressed
##   Size: 14.6 MB
## <BINARY BODY>

However, when I do the same now it's a mess and I can't get anything from it.

httr::VERB(
  verb = "GET", url = "https://www.transtats.bts.gov/DL_SelectFields.aspx",
  query = list(
    gnoyr_VQ = "FIL",
    QO_fu146_anzr = "Nv4+Pn44vr45' -X 'POST' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Origin: https://www.transtats.bts.gov' -H 'Cookie: _ga=GA1.2.279767402.1654087030; _ga_NQ5ZN114SB=GS1.1.1654601577.6.1.1654602173.0; _gat_gtag_UA_18660041_1=1; _gid=GA1.2.975702239.1654601578; f5avr1969959209aaaaaaaaaaaaaaaa_cspm_=DKBJFJCFEDEGJMPHAKDLLGKAOMIGFJJBLFKJJFLCOKMFKOPOIBLJNFBDAENMKNMNNBNCKLDMCJCFKOMIFAFAGDCBBEIKHALBFLPDDHNGGPNBOHCDHALHLHAFCPDCECCK; _gat=1; __utma=261918792.279767402.1654087030.1654149013.1654601578.6; __utmb=261918792.6.10.1654601578; __utmc=261918792; __utmz=261918792.1654087030.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); _ga=GA1.3.279767402.1654087030; _gat_b=1; _gid=GA1.3.975702239.1654601578; has_js=1; QSI_SI_e8TSnTOxZxScWdT_intercept=true; QSI_HistorySession=https://www.transtats.bts.gov/DL_SelectFields.aspx?gnoyr_VQ=GEE&QO_fu146_anzr=Nv4%20Pn44vr45~1653914765211|https://www.transtats.bts.gov/DL_SelectFields.aspx?gnoyr_VQ=GEE&QO_fu146_anzr=Nv4+Pn44vr45~1653918802064|https://www.transtats.bts.gov/DL_SelectFields.aspx?gnoyr_VQ=FIL&QO_fu146_anzr=Nv4%20Pn44vr45~1653926488227|https://www.transtats.bts.gov/DL_SelectFields.aspx?gnoyr_VQ=FIL&QO_fu146_anzr=Nv4+Pn44vr45~1653933301130|https://www.transtats.bts.gov/DL_SelectFields.aspx?gnoyr_VQ=FJE&QO_fu146_anzr=Nv4Pn44vr45~1653943359984|https://www.transtats.bts.gov/DL_SelectFields.aspx?gnoyr_VQ=FIL&QO_fu146_anzr=Nv4+Pn44vr45~1654090368888|https://www.transtats.bts.gov/DL_SelectFields.aspx?gnoyr_VQ=FIL&QO_fu146_anzr=Nv4%20Pn44vr45~1654092812643|https://www.transtats.bts.gov/DL_SelectFields.aspx?gnoyr_VQ=FLM&QO_fu146_anzr=b4vtv0+n0q+Qr56v0n6v10+f748rB~1654094623107|https://www.transtats.bts.gov/DL_SelectFields.aspx?gnoyr_VQ=FIL&QO_fu146_anzr=Nv4%25Pn44vr45~1654102786829|https://www.transtats.bts.gov/DL_SelectFields.aspx?gnoyr_VQ=FIL&QO_fu146_anzr=Nv4+Pn44vr45~1654601914519; f5avraaaaaaaaaaaaaaaa_session_=PBGOPMEEHCNFKKAPFNCACAKCCBFOPNPHICHCBEFGHCMKMLBFFNNFBOCOFLEKPFCNHNLDMANNOIAOFKNKGMJADPMKGDFMDGGEFHKCCGNOLACEFOLDEKIKCIOLAIIANEKM; ASPSESSIONIDQCQCSDSA=JPCOCLGDHAIJOKGLBBJNAKCN; _ga_TS9639848Y=GS1.1.1654095826.2.0.1654095826.0; ASPSESSIONIDQCSATBTA=PFAPIPGDONFFBLDGBBMOOAJO; ASP.NET_SessionId=wr043omrzfn1emifxmj1wadr; ASPSESSIONIDQCSASBTB=BMMLLHNBPOKMOHFPOHEOJOOB' -H 'Content-Length: 8825' -H 'Accept-Language: en-us' -H 'Host: www.transtats.bts.gov' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Safari/605.1.15' -H 'Referer: https://www.transtats.bts.gov/DL_SelectFields.aspx?gnoyr_VQ=FIL",
    QO_fu146_anzr = "Nv4+Pn44vr45' -H 'Accept-Encoding: gzip, deflate, br' -H 'Connection: keep-alive' --data '__EVENTTARGET=chkAllVars",
    `__EVENTARGUMENT` = "",
    `__LASTFOCUS` = "", `__VIEWSTATE` = "s5lt5rBqZX9v/JLyUuO6kxf+qIUW68llMVM3sydtYijhto3Fidu3O/WGJ1VjJcRr1BeRbJcXFN89qYPysApyatjfXjGvzeBGgpTbMcIpSxztQwdL/cWbPv4txnZ+dOJQer1kYzuvfBl08iyWWck97yWXFANUe7FQ8cv7aw8Wm/8E+7GR9RcvXPAx0sZVlUem/s36TXoDlcTU1PWWNrbpwsQKOlzxG7ZWAfsCwv6/Pdygh0lPmkOBvD+UkYBu+2jCGKht/PlZPFap6F2SeORYjnAiXG62IMWa+amyOMJZrcWoiGXjooSPdlLgHOGL29OrIBilrLeGMjMAO64FgU1T1L13E0Vwomn1OIpyeAEMZHcVm0DF9ERh5RY3V1IDeUtlmdXKqy3c1GzA2Dkdg+PlB4k0gTKgLc9tp82GfwaxgtDuCsDNtwmMcibObF0KPMeCNRov+97YEUY0SJ+6mvPT5XNLYHwO9jEd+y3l5TE6qUKKaxv+CufxMlTUGyY7ASU3MxC2XvzuwfeudWGUfKmAFC/43fhCpus2s1Ly0yL6bfjhO2S71k2f0KyINFg6aTcMxuL9vwgvNIlGWClZAmk6fmnimNp4GhzsCTUkmVBPckug8P1HG6HEWCin/AG5WXpGBlAl4/UqfTAwh/kO2MJzyrOSySRgo77LfDI87OUSZ9TEd8ZPRIvSROsZLT4h2wG4Hzhkqxrl4WIw3rt8xryG9kjoIpKd5FhjNnhZpVTLKfzuvkc3tcZfmLiy1FY49EwoudWiaNPZsIcz51Ws8ldNi4o+EyXkVCZJJFcq3mFAg7iJeKZx6m5tGbAgE5kc1sVbqWvJoQVuz6qKpGbeK+jZetVD34aBOKvmOn8FKTIN/n48aeIygD/946cgNI92r2bCU5Wjb0sE4kRWpqQvdPOtLQStK7WqHhAOZhew2oLSuAG44le+03knVl433U4IZq7dJcgkxv+uqHK0pCzVf6lAxwhhGEqSQlByAgkXiNSGImxkDASB7uOAbk6Nt9x3CVozQrzSFrTpLE/d9X34qPJMfR67NrriUvAEN0V3DDxIcTFkg0QYskF3DZLRP7zwj9h4Sj+tKixAf/jrfP0jE848t1DTiBxgAwS1CkcoBW18M7W9kDtMQzQYKstMNWdpAWvy2qoj6KbbRxFloDrR6OAZuMZphHmZhttHAxqRWwvy9AKfg+QAwdKb0ZQUx97AFAonY8Q9nvWx74ww9lzJoHBaz52JxULgBryILXXHDMY68hG3XgmUnrY8mMpZLT42QR4IiLrifRi/AhEpFQ3SELS4w6CIs6V972hOKnfbL2JvPt8K93yuTR2Oz9rG2YbGBf4jIOIj4PvjgJ7uw4B9Fx+XGAeM9Y3s5SaQh5jCNmWHa2aUieQV30gFiiT2LXTIi25SL97GIoSiJoqsMzCdk30y77vhIH7b6gKP053PRohNgP3r4WhwOzn6GAA+E+WyMoz25MJNNk3+4Z8nV4h58jeZb0CGkNvz+ORNDYEh3BTtKuqJG1TFvEWIANM+U2mLlrxZ8QmBHcadGshIH4OtXBnqThr9KDHCj0KeeET8QsvL/ofQKJkqaChrM3WLBf3LUEpA+FmAJIwk27rwz9gArBY/B+MwvPEq8Yz+cKZhE8Sa7yBCFz7wXIkqdg0szO2ZaFVv2DAPOc7c0gpbVGQPJq64d4IGZ/qbqyf63IreCeh5kClA/ZBzlR5/JsWa9tQ036+kyeptLvxaJpXP78Bm+ailFHEmvZmBtnCTDP+1jTi239XRhhg0kBvVTYHn1rLMPz4UZkYtOpUg/lxUGB+lunCNo2uaZOe2BGDD3L+hcZ4J/ToWY+t6/G8lK8LkdkPdn2xSLXSWbbVHv2PbvlxwLmmvvHmA9rsoeQg3LXnEuSvmZhmdv3mFxwA/K/xz5sa++sjeY17pwab84w9ibZ8Bs8k4LrGISybuWHNA7V2VzrcISPHGpE+jDN1623cLHCYCKUIbSuIu24zigeE3+FtSwZ9Usjp/40s39UiAJxFfYPBZZvW8BUF2JOY1kVeQ3/PSpGsaM5o8QNl02O/5lUqRNfzubcATxGuBu6AH+Dj1nfWwebh5aFwRwo7ZfRWc4SQ6LKUCc/6XZJSSlLL7uAk2laCZ+ryvxNqI5Vd1DPNI78JD9IUf0qADvzsiG8eSieCBjf5Um47fq3k19RfGW96ufeO8FOoVNaYjhdaodwTd1BWJAzl/JqBO41LTS8fIzt+YkMnjzF1U/gUTN5+Tpx1GS4fuBrNHhbi0lGXbLDTqYkL+lx2RoPmLKzKIwGw4r8fjF46SCLewQxv/hkW8d3ClcyrbFDvXWZirfNAIssf1HjF11jfdneyRfa5PDkaR6NhhByL/gBwZim6Jz+bFoCORwIeUVFfCP1GmsCFBAPbU3NzDn30GDrfxn8o8LYXmRzRHc32PUDnAkHr/AdOUKzbddptM5oalj1wvuTOLmawT8RrbZUqQptqvmq01MNA3X9EwMFxWfwjsCWrPLXgwioo1NGp5xTheU3tMp/5KY5mus4aHEsPmgeEWByGmJCfoMp+/KTtFGhiEvi/jZeo7VEBVfJiaZM4yK+uLph6oWbMDLjK7AC1bV1bkdXsW3T700gkr/EpLId2aJ3xVVywDHq4Vl+GJNtjaQSNc2UiIrEhAwSLmwxXJxK16fKD7cRwPxy4i+X2j8w3oSUnyeepP58OFtpHAMzTfnl6PreLB7Qv0iBIEB7PjT38PDqMiBK5RGbHs+MSfaqITFsB/BolvoGsMzCMY/aWdsEu6Kse9Pz+UU8bKJda/h+K4DTO7dy2OyXJG+4S/MLeYvaGQN+7BXyBY/ECdO7tGsUhtE5PbaPy/bnwg9oRSjPlGuLaC46Y2SmxuE8NtFs6ZK1I1MoLjPzGGf70u3Eou6G9n7spW9fZb2wWgsfjaBqdQLpcgIpBHwjFZCKe6R6wRsTKzAHL9EPQ04WSCr9yHJFOHeQWt7hd3Ba06bN1LkwqSyUISRcWkVKoRvImre7R9arEjRjaB1cQwA3b0ocVlGL81iwSdhrUaat0AkMo8NZst4CMzi6SrIj7yNw3HquMFp97ZHtleKp+Ap2TWEwyZYhycS0aaoLqkIYijVP/l0I1+tCQ7jvb9zq1z+JJNcnTVIuHu2W/V1fO/vloJ/L4fWMQ9+gEMcgRPwLKrdmJTCFuVUZFREAkxmjJkzkxmcQiZbIlP1ws89CCaEr7PUy93BZNxSpS4aKHRxQiy3LSpkibaZeVaxiJ4xQo7hmG6tPd1uoGkDTbn1Qkav75DiHxUkTfsLNGZkopMrWZnqP9vur+qf2DUljUbYg0q1Vw7WpImiaXP3u7UPB9/CKUENV8yNdxZaz7wvyzrdaO71r8CiipXddoJl3VydnAJj11yR26hms/68M/n/BHSGx06xiSENKW+4LiiucR4uEd8a3sxWR+wefSaWPXL7j0aDktHrJ+O0NwJoTaBL8kitE0jkULTQY0poGPkr/YPbMHCklryFlFBbDPXUZD2sWqTsvYbDw+13m1vzL2QrFSBBY95OwjEnTfRycvC8amXGs32yD6mx6CTPTnS65OTIdNxes69/lJWlbRY9zZHUaKM3OVxPhAdpQb0syL2manh8qongwLD1IFYg72ijzX5xZB/NfSupvDq3jdmzm/hN/vZhUhi8oM4DgAS7U5W2PgQh43bc+vJLylzUrRLi3u2p+qgyuZobE5LsgpJULC85wjyJnVDAHCSMSIOxammFJApQY6ZNlV+aPXM52D5rKOC+WAAD4D0PABGn8e+zIV3rvZy2Scv8BqbVtVrBWNT4OmR52/z4a8EixCU6a2SS2OZvw5Mnmow9fnooodw261ff58NX64MLuzo082u7VaGCTGMpMePjB1JqdhFq5iUdmt1nWRa74kdnzM+Sw8NZ2yfDdXN9sXTshz9fCYO9w/sfXxHMYUoytcPE4D6/Bu17eqrJv4c0BBwDjkpXAW4KMemKCE1Cph0VeaiQkaFKhxoYCyo3ATHO6B2Neck8r11QbHZes/UfNuIv4XVkI5pJQo74KwezFh1gYyrKd4TJXpFpP+eH3QY1GcxAJ6ZIf/Rj7CUU2e1+PUPNROpZpElnkavU/7ezbBwGPhDYEVALA/jFiPW0qChhlhlo7PJeJIzeXXIgBH+G7DF9Y827mE8Ifl42cd2AxZiYqKa3f0/vTcqoFLEN4gSkHwkXJeNpQHu0wuzWbLzrkPXk7sz1VAR9cgFDIouWuBcD1ru5aMbQX4yN+NDqQw0V3f0eCybatrxa2LKoR/L3wI4KxfPTspQUzEBTy9yX7bPnTMWUPxJAkAFjM/iIcrycYkgpgDpuY1ZBlOezTZm2RvM77WvolD+qceeawfy/k9M8CMPpDh4JOnoyJF/W3cG54zcqj+jVpfuhUTwfJeS14M9+ECEG+eAbOwhYZIfE6pxgl7iIu7ydy6xnUdtKr7e92BxIcV848pu6yOyDpU3ovikh5rzJScih+hXoukbkvi2azQWr9/NBP+g13GAnNCu8kuTdvI29t358QwW7XAXFZvpu/jT33bEbF5lAjfgzSmC8EW1KpVvvrecpQG8jY/40Vcrg1HKEI2v/wgRFJHMCSE5ndjWXb0O8Km3KEbethBIQ73kqH8T7SowC9YEzqb4X5p5wcRj+K9h7F+XDgAP/c4sMQoowu3AINx7nwwM7e/52To9zP/Rv3lNMOPJDMheqx3Oh0TCNFU81zNYTXq6JbQoWd5Xn7kdnITrrvMr4aKDNNqOjr1YGWEfc1avlZ1Q3BI9skychiDGQrXH6wF5Q+3maWT4XIb5RBJV9Q556qqKm2OM1sDFjWWAJMz5oaBlZx8yybn2EFsSeVxLnnkEs2I3",
    `__VIEWSTATEGENERATOR` = "82CE8F70",
    `__EVENTVALIDATION` = "P8zqeK87DbezpJjSLiKpZ8OpZeoEr3iYiM856mfjXSxKDLh9yCTKYoB6C1bfipflE3CAUBAz/vHfCJcTC6wWr6PWoOLYqs1r9V75cs/zMS68Wc0O/5tYdH3Yi+56B0ZU/sb4qkLi0uIts16qnMye/H74uOplOooYSraTsfw3cG7/ysjjPjPqZbBVtL8sKbZcVXAvO2F6wg5dcpaiONg8RB6qhSodJHULBb9LLfi3UMpRtrN1hi/1I27XJ2MDZWajQdnQcvgqFga+S14Y4uNU86BuzdasoyCMTc5SZ7nynyM0NE6xemPHyF9yyf2w7nTap46hmYWxn4SYEsor3Hclu5Tge9zrJPrABjIOh/q19dHwwe/9LZ4SGVelAqxRkZCuTAwuiqi2xLqty3m71EPGEbXKqDYG/ghH6HzXBAC2CFJNvoOiDaQBc+kt1gBGHrv6KIMGL3Morbo4hDkaZA2BqeZFQvw63mVAcXfNyimBMCiVC+t2y2BN+o69EPsIQufSA+k8W4lu0WgR2c8FdUBnD8gs//JsaYBTjj+XDGuY2q4rYSuJNVkg/dwjjrz6rieOIsyzUBlbSUX8n+msll32+VE2Cxn4G36m4YO+Fn4gKyije2trLx2im4wgW2zbtCGbIt1L/jRW7U6ZAEiCHBU1tiIi52BD2mqOuxabo++/ua+PdOz1PxUOAQY2Rw+pqmZTLdJPY2Yu2lEU2Lp5I9lEyI09IXW6cQpqqo1DyoadurgTVpdao9oCIQBwueyWjgdpK9K7ygOA2rNteAwd1U6SXQVRuqOeGbTuh+lZWxCdLwen16bibjrlzB5HMNHOnf7DfQqJsgmuAZFKriFR4CX08DBjNK9MVEVzx6a+Lzm6t8W3y8xpU31wmf+ZakgJ4WFz05WPdhnM9aUcCjBZUozO1RCA9QDrp6lCGGTdjK3D975reuwN+Lax1aryT/DjCIYs/yHk7qIAmJ4mEjQfhJQgADxOZcM7qGNHCV+9lT+5hKY8xtosIdt/8/DFRGuZlhyvL4mxrrcmMb3aaDqnbFXiPTqAMseN3Tp39yxpEBuJhlwnF9M1bGMJsTccCrPj/1zN7xrVsmCr4ie7WOSwj1Egeop/mZv2KKIEskQhvul7+z7rcZLgGOhpzFpAr3xLpdkSLi0Ktg6fFd3g6KihofBmaktDlqSYnsOYyRphl4N+QqDI5l3WOp3xDrD9jfP7+hNn2TwNsm/MdaI8hm0pthQ4YnLlj2gbtw+H9IduHlwrmpl4mp5EoeG44OCSS7kvMtWmmwIKVZf/ZnViNRNvpEkeBlMIlYi5zCdKBD4WinvIWXZIKCrug07apQ55x+NyS/ei0s8XHGOGFMulyXHPESsEbOQdSeSxZggCGusE4FtvyVZ9XbRvS5yrdmz2XXZgkUQIVLyxG0tWeJIp45qnEMuJfRvYDVRVAW7HuMfZ4hWF6Rs9sT03ijh8tXEX0CQNLWI5iHxHOj3R8uZ4RBj3yRsKHWNBpiJXDsUYbZEg2wVUzc0d7ErJUwlJ5MbEubCRxO8hE7JX29OKymmC9FjrImmNkM75Hyg5QEyVQUMe9MKi6Li4MNDDueYvPVRzs+7MoDineqHaZSBbw1MZ7ur6+QnbERBWd9lXrzxmZ2d2bsISHZieG49Y6yr38wSIX8Kdftl2CiA7zNvaWcxs0AnKYRsWLp2okRr4lA8qVp07wOiX4H2vH5wy494cY49jE8AA1Aqtg76WwIZeLFaFAxjnNXasGBWZSpbEky1H957jcYHSTiG5u3o7kwKANnWgiR9SbOvdfrsYOUxjEysFHaf4j7CVeyF1TiQrZsDb+9nQtTbVFkmimgYy/zm2sSZSqc1DaYMIfq1rykRljKkJbfsFY2JnneCQYjK4gIYzj4fw2Gzpum5Jhg1tj8W8GqMpwOpks4jAggjAIFWHNiO4z9IhBl7B1/4QmV97xn7oncH1n5py2fyf46dafVGO9LWoyujRot0Ndab6P3M9TJpPdhzaKuzpT2M4jROFllyy8EtBaBjFZeEcU8zb7YR1nebNl2Jr3LETibV2vjmcfs/dvU4XqvX9cjG8K7rVp1sX51f7nD2aOLsysHowxC27Lg1FMXY7b7irdYRONeoa4ecWNop75j/FOLmFza4+4HQUxXQf8b5JBhzEswdTX+JIgAIJTx7YmZOs30G8Ffu9fdN4ne0Wu6tdLRLhaUOaNJ69H77Eq3VvMVfU4FJKU6he+fRqIwHUMMD196d3/h2P+EG5nps9heqmlXwnG9zyYG4767E84Qm+5mNq/XOMaVoyB+v3FcFjGmNQ8MGUKnahH8dORAkOuw+17wZW4RI6JsuDxIoGbkuwwsOnK7SgXs34QWbORSbFDs0HImlEADpzIS2aGVvTQJgVKCQYiKvWXU10+Jd1ZUiMy0yHcihFRkgnpL+F+nVeJSfW/AuyHUVuj7H3ueMoGUy86NDc5Cb6FrWVsjnsIOjI+61E5t9O4okI3AwwmxdA5x4lVYJBxlyXMzprZCgaHowTK4JCQvwbAnuzAsYss/W0xXtmFNVVZCEYhGnRhi7ZiHTIuDlAi4puVYi1gfsU+JPcyAVDej13CSrSZGE8fhwhQwNb/YMobr2tDA0MiJR7VXPRbfksbbLYkA8/ukFvYzokU7fQWzcIImYdN4hUEUoSEq/d1BC4UxKFph+UCXz/guz6lFJy4m0s7SGp5Rt3XDSHZEV/flFjSWK0QsJ+NLOQ+cUmbP1Ftp1kzc7Qk4c3mWflvun+H8Y98mJqntquxAMrCbyQENnFcEMhR7hq5fA5VkEwO2Ks3JKdwR690C9E6dSeYl1Z5IlK4SzYpsCU/mv2iZ5Mv/ALdOX3t2vLTedH4pAqbO1+x8HQP0IOxJOpxpQiv8CbBpkCRNvQGhESuWGgloGXag0GRnJd1LqLak1aowvcGEJmVI3Z01Kg9OJwFyaLamVWZx9t4GfCBPkXK3Ep8sW8uvARqxolVuuK1fNeJwFAD05a8VqKnD30Fe7Bn5oI0CLwiIsB8Ou7W5yExnBHmBTyeNzMP/0qeo6PTCbH88UrD9DPPXWMFXWvR7cXpAeSRsAHjtbVPwFq3bNxEunExbP54adWifEPsQs7zmBf8xQV2W1F2Rd1wmJ7DvfhwyVyITcUGig4/WcBHkEFHtCHuQvNzWTePbP5osJgeM+pTDU=",
    txtSearch = "", cboGeography = "All",
    cboYear = "2022", cboPeriod = "All",
    chkAllVars = "on", UNIQUE_CARRIER = "on",
    UNIQUE_CARRIER_NAME = "on",
    ORIGIN_AIRPORT_ID = "on",
    ORIGIN = "on", DEST_AIRPORT_ID = "on",
    DEST = "on", MONTH = "on"
  )
) -> res

In fact, when trying to run that res, it won't get any zip file anymore, and I suspect that it's due to this __EVENTVALIDATION and/or the __VIEWSTATE.

Fix pkgdown configuration for reference

Some topics are missing from the configuration file.

 Error in check_missing_topics(rows, pkg) : 
  All topics must be included in reference index
• Missing topics: MetroFull, MetroLookup, OD_Sample, aircraft_type, airportCode, airportCodeFull, airportMaster, carriers, download_db1b, download_ontime, download_t100, download_t100_int, import_ontime, make.netInt, make_net_trip, netImport, nodeStatsMetro, skynet_example

Note that for topics you do not want to include in the index you can create an "internal" section https://pkgdown.r-lib.org/reference/build_reference.html?q=internal#missing-topics

To check all topics are listed, after editing the configuration file you can run pkgdown::check_pkgdown().

download_t100 error

Hi Filipe,

I'm trying to download some T100 segment data using the download_t100 function but I'm getting the error: No internet connection or data source broken (my code: t100_seg <- download_t100(2019, "seg"). I checked your source code in the download_t100.R file and it seems as the url: "https://www.transtats.bts.gov/DownLoad_Table.asp" is broken.

Wondering if this can be fixed or if you have an alternative solution/suggestions.

Thanks!

Download error from R CMD check

Hi Filipe,

I'm checking revdeps as part of releasing data.table. skynet is failing for me with the following error. But I see it's passing fine on CRAN. Maybe I'm just unlucky and the file it's trying to fetch is not available starting today, perhaps. When I got the error with the new version of data.table, I tried again with the current CRAN version of data.table (1.12.8) and I still see this error with skynet. So I don't think it is due to data.table, but if you don't see it, then it might be due to my computer or environment. Any ideas?

Thanks, Matt

$ Rdevel CMD check skynet_1.3.8.tar.gz 
* using log directory ‘/home/mdowle/build/revdeplib/skynet.Rcheck’
* using R Under development (unstable) (2020-07-14 r78854)
* using platform: x86_64-pc-linux-gnu (64-bit)
* using session charset: UTF-8
* checking for file ‘skynet/DESCRIPTION’ ... OK
* checking extension type ... Package
* this is package ‘skynet’ version ‘1.3.8’
* package encoding: UTF-8
* checking package namespace information ... OK
* checking package dependencies ... OK
* checking if this is a source package ... OK
* checking if there is a namespace ... OK
* checking for executable files ... OK
* checking for hidden files and directories ... OK
* checking for portable file names ... OK
* checking for sufficient/correct file permissions ... OK
* checking whether package ‘skynet’ can be installed ... OK
* checking installed package size ... OK
* checking package directory ... OK
* checking ‘build’ directory ... OK
* checking DESCRIPTION meta-information ... OK
* checking top-level files ... OK
* checking for left-over files ... OK
* checking index information ... OK
* checking package subdirectories ... OK
* checking R files for non-ASCII characters ... OK
* checking R files for syntax errors ... OK
* checking whether the package can be loaded ... OK
* checking whether the package can be loaded with stated dependencies ... OK
* checking whether the package can be unloaded cleanly ... OK
* checking whether the namespace can be loaded with stated dependencies ... OK
* checking whether the namespace can be unloaded cleanly ... OK
* checking loading without being on the library search path ... OK
* checking dependencies in R code ... NOTE
Namespace in Imports field not imported from: ‘maps’
  All declared Imports should be used.
* checking S3 generic/method consistency ... OK
* checking replacement functions ... OK
* checking foreign function calls ... OK
* checking R code for possible problems ... OK
* checking Rd files ... OK
* checking Rd metadata ... OK
* checking Rd cross-references ... OK
* checking for missing documentation entries ... OK
* checking for code/documentation mismatches ... OK
* checking Rd \usage sections ... OK
* checking Rd contents ... OK
* checking for unstated dependencies in examples ... OK
* checking contents of ‘data’ directory ... OK
* checking data for non-ASCII characters ... OK
* checking data for ASCII and uncompressed saves ... OK
* checking installed files from ‘inst/doc’ ... OK
* checking files in ‘vignettes’ ... OK
* checking examples ... OK
* checking for unstated dependencies in ‘tests’ ... OK
* checking tests ...
  Running ‘testthat.R’
 ERROR
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
  Backtrace:
   1. skynet::download_t100(2011, "mkt")
   2. httr::POST(...)
   3. httr:::request_perform(req, hu$handle$handle)
   5. httr:::request_fetch.write_memory(req$output, req$url, handle)
   6. curl::curl_fetch_memory(url, handle = handle)
  
  ══ testthat results  ═══════════════════════════════════════════════════════════
  [ OK: 70 | SKIPPED: 0 | WARNINGS: 2 | FAILED: 3 ]
  1. Error: Download Ontime (@test_download_ontime.R#6) 
  2. Error: Download T100 (@test_download_t100.R#6) 
  3. Error: Download T100 (@test_download_t100.R#12) 
  
  Error: testthat unit tests failed
  Execution halted
* checking for unstated dependencies in vignettes ... OK
* checking package vignettes in ‘inst/doc’ ... OK
* checking running R code from vignettes ...
  ‘skynet.Rmd’ using ‘UTF-8’... OK
 NONE
* checking re-building of vignette outputs ... OK
* checking PDF version of manual ... OK
* DONE

Status: 1 ERROR, 1 NOTE
See
  ‘/home/mdowle/build/revdeplib/skynet.Rcheck/00check.log’
for details.

deprecation message from tibble

`funs()` was deprecated in dplyr 0.8.0.
ℹ Please use a list of either functions or lambdas:

# Simple named list: list(mean = mean, median = median)

# Auto named with `tibble::lst()`: tibble::lst(mean, median)

# Using lambdas list(~ mean(., trim = .2), ~ median(., na.rm = TRUE))
ℹ The deprecated feature was likely used in the skynet package.

while running test_smallerfunctions.R

Tried Example - error object 'C_map_type' not found

Tried example and got this error. igraph package is loaded

network <- make.netDir(OD_Sample)
netMap(network$netDir, pct = 10)
Error in maptype(database) : object 'C_map_type' not found

igraph 1.3.0 will break `skynet` unit tests

Dear Skynet team,

I am writing you to let you know that igraph 1.3.0 will introduce a change that causes some of the skynet unit tests to fail. In particular, calling length(g) on an igraph graph g in earlier versions returned 10 unconditionally, which was considered a bug by many (see igraph/rigraph#427 ). From igraph 1.3.0, length(g) will be consistent with the [[ indexing operator; since [[ indexes into the adjacency list of the graph, length(g) will return the number of vertices in the graph.

Your unit tests check whether length(g) is 10 and I'm not sure what's the purpose of that. If you would like to test the number of vertices, use vcount(g) instead and compare it with the expected vertex count. If you would like to know whether you are working with an igraph graph, use class(g) instead and check whether the result is "igraph".

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.