Code Monkey home page Code Monkey logo

templelv / 91porn_spider Goto Github PK

View Code? Open in Web Editor NEW
174.0 174.0 33.0 68 KB

91视频网站爬虫工具,可以批量或单独爬取视频。 不带参数运行程序时,进入日常爬取模式,固定每天8点爬取24小时内发布的30个评分最高的视频,评分由关键字、视频时长、作者分三项评分组成(score下的两个txt定义了关键词评分和作者评分,分数范围[-∞,100])。每周六9点会爬取本周评分最高的30个最热视频并把当周的视频整理到一个文件夹下。程序有去重机制不会重复下载同一个视频。

Go 99.82% Shell 0.18%

91porn_spider's People

Contributors

ltpjob avatar templelv avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

91porn_spider's Issues

go build 报错

在根目录下,go build 报错如下:

go: github.com/PuerkitoBio/[email protected]: Get "https://proxy.golang.org/github.com/%21puerkito%21bio/goquery/@v/v1.6.1.mod": dial tcp 172.217.160.81:443: connectex: A connection attempt failed because the connected par ty did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

咋解决呀?

jieba包好像有问题

麻烦大佬帮忙看下哈。或者直接帮忙打个包release出来。谢谢啦

github.com/yanyiwu/gojieba

In file included from vendor/github.com/yanyiwu/gojieba/deps/cppjieba/Unicode.hpp:9,
from vendor/github.com/yanyiwu/gojieba/deps/cppjieba/DictTrie.hpp:15,
from vendor/github.com/yanyiwu/gojieba/deps/cppjieba/QuerySegment.hpp:8,
from vendor/github.com/yanyiwu/gojieba/deps/cppjieba/Jieba.hpp:4,
from jieba.cpp:5:
vendor/github.com/yanyiwu/gojieba/deps/limonp/LocalVector.hpp: In instantiation of ‘void limonp::LocalVector::reserve(size_t) [with T = std::pair<long unsigned int, const cppjieba::DictUnit*>; size_t = long unsigned int]’:
vendor/github.com/yanyiwu/gojieba/deps/limonp/LocalVector.hpp:83:7: required from ‘void limonp::LocalVector::push_back(const T&) [with T = std::pair<long unsigned int, const cppjieba::DictUnit*>]’
vendor/github.com/yanyiwu/gojieba/deps/cppjieba/Trie.hpp:99:81: required from here
vendor/github.com/yanyiwu/gojieba/deps/limonp/LocalVector.hpp:95:11: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<long unsigned int, const cppjieba::DictUnit*>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
memcpy(ptr_, old, sizeof(T) * capacity_);

报错:* DlAddr not set

采用第一种方式:
context deadline exceeded
采用第二种方式报错如下:
********************* DlAddr not set!
context deadline exceeded
[原创] ********************* DlAddr not set!
context deadline exceeded

ERROR: could not unmarshal event: unknown PrivateNetworkRequestPolicy value

D:\91porn_spider>spider91 -c -u "http://91porn.com/v.php?category=rf&viewtype=basic&page=2" -p "http://127.0.0.1:8000"
http://91porn.com/v.php?category=rf&viewtype=basic&page=2 Crawl done!
DownladMany:len([]*VideoInfo)=24
2022/11/30 17:40:20 ERROR: could not unmarshal event: unknown PrivateNetworkRequestPolicy value

DlAddr https://www.91porn.com/view_video.php?viewkey=438060464 context deadline exceeded
[付费] [原创] DlAddr not set!

D:\91porn_spider>spider91 -c -u "http://91porn.com/view_video.php?viewkey=8cd0148b3fe08d4a4c2f" -p "http://127.0.0.1:8000"
context deadline exceeded

如果使用小火箭shadowsocksR,是不是不用配置代理了

PS F:\91porn_spider-master> ./spider91 -now 3 -n 100
2021/10/31 02:10:27 Start 3 days Download, count 100!!
wrong own value format! []
Crawl http://91porn.com/v.php?next=watch&page=1 context deadline exceeded
Crawl http://91porn.com/v.php?next=watch&page=1 page load error net::ERR_PROXY_CONNECTION_FAILED
Crawl http://91porn.com/v.php?next=watch&page=1 page load error net::ERR_PROXY_CONNECTION_FAILED
Crawl http://91porn.com/v.php?next=watch&page=1 page load error net::ERR_PROXY_CONNECTION_FAILED
Crawl http://91porn.com/v.php?next=watch&page=1 page load error net::ERR_PROXY_CONNECTION_FAILED
Crawl http://91porn.com/v.php?next=watch&page=1 page load error net::ERR_PROXY_CONNECTION_FAILED

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.