python3webspider / douyin Goto Github PK

View Code? Open in Web Editor NEW

643.0 33.0 262.0 588 KB

API of DouYin for Humans used to Crawl Popular Videos and Musics

License: MIT License

Python 100.00%

douyin spider crawler videos

douyin's Introduction

注：项目已不可用...

DouYin

API of DouYin App for humans.

Installation

pip3 install douyin

Usage

Here is the sample code:

import douyin
from douyin.structures import Topic, Music

# define file handler and specify folder
video_file_handler = douyin.handlers.VideoFileHandler(folder='./videos')
music_file_handler = douyin.handlers.MusicFileHandler(folder='./musics')
# define mongodb handler
mongo_handler = douyin.handlers.MongoHandler()
# define downloader
downloader = douyin.downloaders.VideoDownloader([mongo_handler, video_file_handler, music_file_handler])

for result in douyin.hot.trend():
    for item in result.data:
        # download videos of topic/music for 100 max per
        downloader.download(item.videos(max=100))

then you can get:

Item <Topic: <1565818716518401, panama>>
Processing <Video: <6616517521098935565, 真香#panama>> ...
Processing <Video: <6500385230921141518, 哈哈哈哈哈>> ...
...
Processing <Video: <6479958542747962637, 👅ก่อนกินข้>> ...
Processing <Video: <6473811426107460878, 😁>> ...
0%|                                                      | 0/10 [00:00<?, ?it/s]
Processing 1-10 of files
Processing <Video: <6616517521098935565, 真香#panama>> ...
Saving <Video: <6616517521098935565, 真香#panama>> to mongodb...
Processing <Video: <6500385230921141518, 哈哈哈哈哈>> ...
Saving <Video: <6500385230921141518, 哈哈哈哈哈>> to mongodb...
Processing <Video: <6562690160868199693, 皇城相府版C哩C哩跨>> ...
....
Downloading <Video: <6580510322468064526, 第二遍 后面的小哥哥>> ...
Saved <Video: <6479958542747962637, 👅ก่อนกินข้>> to mongodb successfully
Downloading <Video: <6479958542747962637, 👅ก่อนกินข้>> ...
Saved <Video: <6473811426107460878, 😁>> to mongodb successfully
Downloading <Video: <6473811426107460878, 😁>> ...
Downloaded file to ./videos/6580510322468064526.mp4
10%|████▌                                         | 1/10 [00:01<00:16,  1.84s/it]
Downloaded file to ./videos/6516746291806997763.mp4
20%|█████████▏                                    | 2/10 [00:01<00:10,  1.33s/it]
Downloaded file to ./videos/6600742831352974596.mp4
40%|██████████████████▍                           | 4/10 [00:02<00:05,  1.03it/s]
Downloaded file to ./videos/6484393014599879950.mp4
50%|███████████████████████                       | 5/10 [00:02<00:04,  1.15it/s]
Downloaded file to ./videos/6616517521098935565.mp4
60%|███████████████████████████▌                  | 6/10 [00:03<00:03,  1.27it/s]
Downloaded file to ./videos/6479958542747962637.mp4
70%|████████████████████████████████▏             | 7/10 [00:03<00:01,  1.68it/s]
Downloaded file to ./videos/6472305134377372941.mp4
80%|████████████████████████████████████▊         | 8/10 [00:03<00:00,  2.05it/s]
Downloaded file to ./videos/6562690160868199693.mp4
90%|█████████████████████████████████████████▍    | 9/10 [00:04<00:00,  2.27it/s]
Downloaded file to ./videos/6500385230921141518.mp4
100%|█████████████████████████████████████████████| 10/10 [00:04<00:00,  2.33it/s]

Examples

See examples

douyin's People

Contributors

Stargazers

Watchers

Forkers

sangensong zaiwang a1exw0ng wyl-brucelong xiaosimao amoyyean sysdxj henryhaohao miaotiaotech carter115 zloveh zixu4728 jeremydong ted-ch waitingfy ssmail johnwongchn gokey codenewer feizhang365 gpsbird czar10 moearly spidermatrix a515151 gnolmon zorrock linwei-yoh likeucode jdc08161063 1997lw bubueric longjunzhang wgwjifeng chaya0101 pol9111 shaohef little1tow visense qq703048949 hacksman kixiang hongkongdavid hanmichael jwangxin w3cui jackzzhu lxlpp123 buptbf fightyang daydayup2016 npubird ape-scott mohammedgomaa jianjie97 achi-migo mathchan acewjh edgarcai qgtiger ieyer dtzfast trackson yanqinghou hehan502 yonghuming saoinformaticsteam igraph2017 kimliu2 lduml hdzhangjl helianghit hejinpeng123 jack0707 danceiny shorion youmuyou allenzzz weiling103 kingsun0 calemitty visionandy zhanghongji heyuanlong zzzz123321 lanxuewsr gregoriusxu tianshl holysll starorm lambertcsy honglianggong1990 legendtianjin xrr8417403 jstwatchig mrroche 1060460048 leiray vvmagic chooinc

douyin's Issues

get hot video result多久更新一次？

请问 hot_video_url 多久更新一次呀？另外 hot_trend_url 这个是失效了吗？没内容了

Exception ??

Exception <class 'requests.exceptions.ConnectionError'> occurred, retrying...

视频下载数，不是10的倍数，就不会下载？

downloader.download(item.videos(max=2))

# downloaders/base.py
        if isinstance(inputs, types.GeneratorType):
            temps = []
            for result in inputs:
                print('Processing', result, '...')
                temps.append(result)
                if len(temps) == self.batch:
                    self.process_items(temps)
                    temps = []

有没有示例的代码啊？

哈喽，各位大佬，
请教下有没有实际能用的？
可以
[email protected]

Python 3.7.2 has issue of "Exception <class 'requests.exceptions.ConnectionError'> occurred, retrying..."?

D:\Program Files (x86)\Python\Python37-32\apps\DouYin-master\examples>python save_hot_videos_and_musics_to_file.py
Exception <class 'requests.exceptions.ConnectionError'> occurred, retrying...
Exception <class 'requests.exceptions.ConnectionError'> occurred, retrying...
Exception <class 'requests.exceptions.ConnectionError'> occurred, retrying...
Exception <class 'requests.exceptions.ConnectionError'> occurred, retrying...
Exception <class 'requests.exceptions.ConnectionError'> occurred, retrying...
Exception <class 'requests.exceptions.ConnectionError'> occurred, retrying...
Exception <class 'requests.exceptions.ConnectionError'> occurred, retrying...
Exception <class 'requests.exceptions.ConnectionError'> occurred, retrying...
Exception <class 'requests.exceptions.ConnectionError'> occurred, retrying...
Exception <class 'requests.exceptions.ConnectionError'> occurred, retrying...
0it [00:00, ?it/s]
Exception <class 'requests.exceptions.ConnectionError'> occurred, retrying...
Exception <class 'requests.exceptions.ConnectionError'> occurred, retrying...
Exception <class 'requests.exceptions.ConnectionError'> occurred, retrying...
Exception <class 'requests.exceptions.ConnectionError'> occurred, retrying...

import douyin
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2881, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
import douyin
File "C:\Program Files\JetBrains\PyCharm Community Edition 181.3986.3\helpers\pydev_pydev_bundle\pydev_import_hook.py", line 20, in do_import
module = self.system_import(name, *args, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\douyin_init.py", line 2, in
import douyin.handlers
File "C:\Program Files\JetBrains\PyCharm Community Edition 181.3986.3\helpers\pydev_pydev_bundle\pydev_import_hook.py", line 20, in do_import
module = self.system_import(name, *args, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\douyin\handlers_init.py", line 2, in
from douyin.handlers.file import FileHandler
File "C:\Program Files\JetBrains\PyCharm Community Edition 181.3986.3\helpers\pydev_pydev_bundle\pydev_import_hook.py", line 20, in do_import
module = self._system_import(name, *args, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\douyin\handlers\file.py", line 5, in
import aiohttp
File "C:\Program Files\JetBrains\PyCharm Community Edition 181.3986.3\helpers\pydev_pydev_bundle\pydev_import_hook.py", line 20, in do_import
module = self.system_import(name, *args, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\aiohttp_init.py", line 6, in
from .client import * # noqa
File "C:\Program Files\JetBrains\PyCharm Community Edition 181.3986.3\helpers\pydev_pydev_bundle\pydev_import_hook.py", line 20, in do_import
module = self._system_import(name, *args, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\aiohttp\client.py", line 18, in
from . import client_exceptions, client_reqrep
File "C:\Program Files\JetBrains\PyCharm Community Edition 181.3986.3\helpers\pydev_pydev_bundle\pydev_import_hook.py", line 20, in do_import
module = self._system_import(name, *args, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\aiohttp\client_reqrep.py", line 17, in
from . import hdrs, helpers, http, multipart, payload
File "C:\Program Files\JetBrains\PyCharm Community Edition 181.3986.3\helpers\pydev_pydev_bundle\pydev_import_hook.py", line 20, in do_import
module = self._system_import(name, *args, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\aiohttp\helpers.py", line 166, in
@attr.s(frozen=True, slots=True)
TypeError: attributes() got an unexpected keyword argument 'frozen'