zeshuaro / linkedrw Goto Github PK

LinkedRW (version 1.2.0) crashes when trying to scrape data from LinkedIn.
See last message, I also tried to increase the timeout to 30.

[1018/205149.931:INFO:CONSOLE(1)] "document.registerElement is deprecated and will be removed in M80, around February 2020. Please use window.customElements.define instead. See https://www.chromestatus.com/features/4642138092470272 and https://developers.google.com/web/updates/2019/07/web-components-time-to-upgrade for more details.", source: https://static-exp1.licdn.com/sc/h/br/brvu2i6m20qsgbxiefnnmzycz (1)
Traceback (most recent call last):
File "C:\Users\Koen\AppData\Local\Programs\Python\Python38-32\Scripts\linkedrw-script.py", line 11, in
load_entry_point('linkedrw==1.2.0', 'console_scripts', 'linkedrw')()
File "C:\Users\Koen\AppData\Local\Programs\Python\Python38-32\lib\site-packages\linkedrw\main.py", line 37, in main
args.method(**vars(args))
File "C:\Users\Koen\AppData\Local\Programs\Python\Python38-32\lib\site-packages\linkedrw\main.py", line 73, in run
profile = scrape(driver.lower(), driver_path, email, password, output_dir, timeout)
File "C:\Users\Koen\AppData\Local\Programs\Python\Python38-32\lib\site-packages\linkedrw\scraper\scrape.py", line 72, in scrape
elem = WebDriverWait(driver, timeout).until(ec.presence_of_element_located(
File "C:\Users\Koen\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

removed.

What happened

I ran pip install --user linkedrw because I have use sudo without --user.
I installed geckodriver because I use Firefox.
linkedrw was not in my path but I was able run python3 -m linkedrw

You might want to expand on the installation instructions if I did it incorrectly.

Did you keep your browser window on top when linkedrw is running

Yes.

Did you try increasing the timeout by using -t/--timeout option

Not necessary as it failed looking for chromedriver.

Which web driver did you use

geckodriver 0.24.0

Logs

[2019-06-26 21:57:25] NOTICE: Scraping LinkedIn profile
[2019-06-26 21:57:25] NOTICE: Please keep the browser window on top
Traceback (most recent call last):
File "/home/dlewis/.local/lib/python3.7/site-packages/selenium/webdriver/common/service.py", line 76, in start
stdin=PIPE)
File "/usr/lib/python3.7/subprocess.py", line 775, in init
restore_signals, start_new_session)
File "/usr/lib/python3.7/subprocess.py", line 1522, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'chromedriver': 'chromedriver'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/dlewis/.local/lib/python3.7/site-packages/linkedrw/main.py", line 3, in
main()
File "/home/dlewis/.local/lib/python3.7/site-packages/linkedrw/main.py", line 36, in main
args.method(**vars(args))
File "/home/dlewis/.local/lib/python3.7/site-packages/linkedrw/main.py", line 70, in run
profile = scrape(driver.lower(), email, password, output_dir, timeout)
File "/home/dlewis/.local/lib/python3.7/site-packages/linkedrw/scraper/scrape.py", line 20, in scrape
driver = webdriver.Chrome()
File "/home/dlewis/.local/lib/python3.7/site-packages/selenium/webdriver/chrome/webdriver.py", line 73, in init
self.service.start()
File "/home/dlewis/.local/lib/python3.7/site-packages/selenium/webdriver/common/service.py", line 83, in start
os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home