wcrasta / swa-scraper Goto Github PK

The form for retrieving prices for flights from British Airways looks pretty similar to the SWA form, with the main difference being the passenger boxes which are split into four: adults, young adults (12-15), children (2-11) and infants (under 2). We'll either need to only retrieve prices for adults or introduce more command line args...
For the time being, we should implement the code for adults only and then we have the option of extending at a later date.

https://www.britishairways.com/travel/fx/public/en_gb

Was working fine, monitoring every 30m until early this AM...See Attached.
Scraper-Log.txt

Anyone else having this issue?

https://www.aerlingus.com/html/planPage/planPage.html

https://www.thomascookairlines.com/en/book-plan/flight/timetable.jsp

Hello, cool idea for a project!

I gave this a try by running with one of the example commands you provide, but got the following error:

$ python3 app.py --company Southwest --depart HOU --arrive MDW --departure-date 05/12/2018 --return-date 05/14/2018 --passengers 2 --max-price 215 --interval 30             
[2018-04-08 22:29:06] Submitting form...
Traceback (most recent call last):
  File "app.py", line 17, in <module>
    main()
  File "app.py", line 12, in main
    real_total = swaScraper.scrape(args)
  File "/home/dev/Documents/src/SWA-Scraper/swaScraper.py", line 78, in scrape
    wait.until(EC.element_to_be_clickable((By.ID, "faresOutbound")))
  File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/support/wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 
Screenshot: available via screen

(I actually tried various things to get it to output a screenshot, but couldn't get it to do so -- selenium/phantom is not something I'm terribly familiar with)

Here's my phantomjs version:

$ /usr/local/bin/phantomjs --version
2.1.1

If you have any ideas, please let me know! thanks.

https://www.lufthansa.com/online/portal/lh/uk/booking2011?nodeid=457726854&l=en&cid=18002

https://www.aa.com/booking/find-flights?tripType=roundTrip

https://fly10.emirates.com/CAB/IBE/SearchAvailability.aspx

Nevermind didn't realize it was command line arguments sorry.

Related to my other request of adding points support -- it would also be great (for points or dollars) to watch a specific FLIGHT as opposed to just any flight that day or during a specific time.

Basically, if the price of a flight drops with SWA, you can fairly easily rebook to get the discount and get the difference refunded or back as a credit. So, this tool would be great to be able to monitor a specific flight that you've already booked - so that if that specific flight drops in price, you can make a quick adjustment.

Again, thank you for your time and this great tool!

So I originally started this due to a user error in entering departure dates, I didn't realize I had to enter the "0" before the single digit month and it was causing me issues.

But I thought since I can't delete the issue I would offer three enhancements that I coded up tonight once I got it working.

1. I created an xpath to search for a specific flight number
2. I added the ability to search based on points
3. I use a script to run this via cron because I've had problems with long running python scripts crashing, so I made the interval optional

If any of these might be useful to others I'd be happy to do a pull request

https://www.klm.com/travel/gb_en/plan_and_book/booking/book_a_flight/index.htm

I was wondering if its possible to use some of your code. A license will quickly let me know!

Thanks!

A number of major airlines have options for young adults (sometimes referred to as teens), children and infants. We should consider adding flags in for these so that we can further extend the software to include other airlines.

Airlines Supporting these Types:

• Aer Lingus
• American Airlines
• British Airways
• easyJet
• Emirates
• Qatar Airways
• Virgin Atlantic

I think the code should be refactored to include a variable called south_west_url. That would reduce the need for commenting in the code.

https://www.virginamerica.com/book/

https://www.qatarairways.com/en-gb/book.html#beTId_flight|

https://www.virginatlantic.com/air-shopping/searchFlights.action

Great work with putting this together!

I'm having a hard time figuring out how to monitor flights that aren't non-stop. How do we specify the "flightnum" variable for those flights?

Thanks.

https://www.easyjet.com/us?link_megadrop

Currently a bunch of the files are named in a way that isn't really descriptive. These should be updated to improve readability.

I saw this on reddit and wanted to give it a try, so I installed what was needed and ran it as below.

python3 app.py --depart MCO --arrive PDX --departure-date 07/17 --passengers 1 --desired-total 250 --interval 30 --one-way

However I get this issue when it runs. Does Selenium require Chrome or something?

Traceback (most recent call last):
  File "app.py", line 204, in <module>
    main()
  File "app.py", line 24, in main
    scrape(args)
  File "app.py", line 92, in scrape
    browser = webdriver.PhantomJS()
  File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/phantomjs/webdriver.py", line 52, in __init__
    self.service.start()
  File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/common/service.py", line 74, in start
    stdout=self.log_file, stderr=self.log_file)
  File "/usr/lib/python3.4/subprocess.py", line 859, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.4/subprocess.py", line 1457, in _execute_child
    raise child_exception_type(errno_num, err_msg)
OSError: [Errno 8] Exec format error
Exception ignored in: <bound method Service.__del__ of <selenium.webdriver.phantomjs.service.Service object at 0x76a9d550>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/common/service.py", line 173, in __del__
    self.stop()
  File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/common/service.py", line 145, in stop
    if self.process is None:
AttributeError: 'Service' object has no attribute 'process'

Any chance you could modify this to work with points? I noticed there is a line to specify USD or POINTS for the URL -- but switching over to POINTS causes it to not work. (Code further down appears to assume dollars, of course.)

It would be great to be able to monitor points drops with this. Thanks!