vanyasem / vk-scraper Goto Github PK

View Code? Open in Web Editor NEW

54.0 7.0 16.0 78 KB

Scrape VK media

Home Page: https://pypi.org/project/VK-Scraper/

License: GNU General Public License v3.0

Python 100.00%

vk vkontakte python scraper api vk-api vkontakte-api scrape downloader

vk-scraper's Introduction

VK Scraper

vk-scraper is a command-line application written in Python that scrapes and downloads VK user's / community's data. Use responsibly.

To get a better understanding of how it works, head to the docs.

Features

Scrape photos
Scrape videos (both uploaded and external)
Scrape saved photos
Scrape stories

Install

Arch GNU/Linux

For the stable version (vk-scraper):

git clone https://aur.archlinux.org/vk-scraper.git vk-scraper

For the git version (vk-scraper-git):

git clone https://aur.archlinux.org/vk-scraper-git.git vk-scraper

Then build & install:

cd vk-scraper 
makepkg -sic

Or use an AUR helper of your choice.

Other distros

For the stable version:

$ pip3 install vk-scraper --upgrade --user

For the git version:

$ pip3 install git+https://github.com/vanyasem/VK-Scraper.git --upgrade --user

Usage

To scrape media:

vk-scraper <username/community> -u <your username> -p <your password>

By default, downloaded media will be placed in <current working directory>/<username>.

To specify multiple users/communities, pass a comma separated list of users:

vk-scraper username1,community1,username2,username3,community2

You can also supply a file containing a list of users/communities:

vk-scraper -f scrape_list.txt

$ cat vk_users.txt
username1
community1
username2
username3
community2
...

Usernames may be separated by newlines, commas, semicolons, or whitespace.

Arguments

--help -h             Show help message and exit

--login-user  -u      Your VK username

--login-pass  -p      Your VK password

--filename    -f      Path to a file containing a list of users/communities to scrape

--destination -d      Specify destination folder. By default, media will
                      be downloaded to <current working directory>/<username>

--retain-username -n  Creates a subdirectory for each scraped name when the flag is set

--media-types -t      Specify media types to scrape. Enter as space separated values.
                      Valid values are image, saved, video, story, wall, or none
                      (defaults to image)

--latest              Scrape only new media since the last scrape. Uses the last modified
                      time of the latest media item in the destination directory for comparasion

--quiet       -q      Be quiet while scraping

--maximum     -m      Maximum number of items to scrape

--offset      -o      Offset from which the scrape starts. 0 is from the oldest. (Defaults to 0)

Contribution

Check open issues, or open a new one to start a discussion around your idea or a bug you found
Fork the repository and make your changes
Send a pull request

Futurelog

Scrape by hashtag
Scrape by location
Save metadata to a file (likes, comments, etc)
Sort photos by their albums

vk-scraper's People

Contributors

Stargazers

Watchers

Forkers

alextavgen tomydmg bardimnik github-userx o7n wrightway12 imcowboysibs cndctr tokanawa syedpeer baronrustamov arjung24 gsamonin d34m0nn0n3 rsxrwscjpzdzwpxaujrr azthec

vk-scraper's Issues

ERROR: This video was marked as adult content

Hi
I am unable to download video tagged with adult

Searching -204491214 for videos: 2 videos [00:01, 1.43 videos/s] Downloading: 0%| | 0/2 [00:00<?, ?it/s] ERROR: This video was marked as adult content. Embedding adult videos on external websites is prohibited. Downloading: 50%|████████████████████████████████████ | 1/2 [00:02<00:02, 2.95s/it] ERROR: This video was marked as adult content. Embedding adult videos on external websites is prohibited.

Does anyone know how to make it work.

Stories are duplicated sometimes

When downloading user's profile several times a day, the name of the mp4 file on VK servers changes and thus detects as new media and gets downloaded again with a different name resulting in a lot of duplicates

Scraping user private videos is broken

Regression introduced in d327791

Scraping Communities Data?

Hello-- the app description states "vk-scraper is a command-line application written in Python that scrapes and downloads VK user's / community's data. Use responsibly" but when I add the name of the community to the command as 'VK-Scraper @vk_args.txt [community_name]' or the list file as 'VK-Scraper @vk_args.txt -f vk_users.txt' I get the error: 'Error getting user details for [community_name]' Is the feature not working or am I doing something incorrectly?

Outdated API

vk-scraper uses outdated VK API (thought still available to use for backwards compatibility). Should be addressed in future releases

IndexError: list index out of range

Hi,

I am getting an IndexError issue using the latest version.
Python 3.6.6 via pyenv, Debian 10

Command: vk-scraper user -u "[email protected]" -p "pass"

Traceback (most recent call last):
  File "/home/sideloading/.local/bin/vk-scraper", line 11, in <module>
    load_entry_point('VK-Scraper==2.0.3', 'console_scripts', 'vk-scraper')()
  File "/home/sideloading/.local/lib/python3.6/site-packages/vk_scraper/app.py", line 492, in main
    scraper.scrape()
  File "/home/sideloading/.local/lib/python3.6/site-packages/vk_scraper/app.py", line 155, in scrape
    self.login()
  File "/home/sideloading/.local/lib/python3.6/site-packages/vk_scraper/app.py", line 92, in login
    self.vk_session.auth()
  File "/home/sideloading/.local/lib/python3.6/site-packages/vk_api/vk_api.py", line 177, in auth
    self._auth_cookies(reauth=reauth)
  File "/home/sideloading/.local/lib/python3.6/site-packages/vk_api/vk_api.py", line 208, in _auth_cookies
    self._api_login()
  File "/home/sideloading/.local/lib/python3.6/site-packages/vk_api/vk_api.py", line 439, in _api_login
    params = response.url.split('#', 1)[1].split('&')
IndexError: list index out of range