Code Monkey home page Code Monkey logo

andatoshiki / toshiki-proxypool Goto Github PK

View Code? Open in Web Editor NEW
37.0 2.0 18.0 134.74 MB

🧸 A magical scraper that automatically crawl SS, SSR, vmess, trojan node info from web, Telegram Channels, and localize/integrate into a web panel for convenient usages

Home Page: https://proxypool.toshiki.dev

License: GNU General Public License v3.0

Dockerfile 0.33% Makefile 1.63% Go 76.71% HTML 20.15% JavaScript 0.89% Shell 0.29%
clash node proxypool ssr ssrs trojan vmess vpn web

toshiki-proxypool's Introduction

logo

俊樹のProxypool

The all-in-one crawler/scraper for crunching free proxy nodes of SS, SSR, vmess, trojan!


wakatime status chat on telegram go built stats Github Actions release codeql docker build github issues FOSSA Status codecov

Read Before You Proceed

Note: Note that this project was originally released by sansui233 licensed under GNU for project derivational development and distribution; As required by original developers, project content contained under proxypool and its affiliates MUST be copyleft yet pointing the credit of the original developer. Hence this project and its assets in contained is a new branch & version published and secondarily developed from the original project, please understand and ensure copywrites are pointing the original author for your own development.

Toshiki's proxypool contains a complete suite of SS, SSR, vmess, trojan node crawler written in GoLang and complied into a compiled into a Linux image for VPS deployment, you can also try deployment for Heroku, completely free to use, see the next section for further info.

What does it support?

  • 🧩 Extensible: Support SS, SSR, vmess, trojan & a variable types of nodes.
  • 💬 Telegram: Telegram channel sniffer & crawling.
  • 🎲 Featured: Support address subscription sniffer, analysis & crawling.
  • 🌐 Web: Open network vague crawling.
  • 🕒 Cron: Timed crawling & updates supported via cron job.
  • 🎨 Customizable: Customizable crawl source via configuration yaml file.
  • 🎀 Automated: Automatically detect uptime and availability of nodes.
  • 🎯 Convenient: Serve direct config files for clash, surge for importation.
  • 🔮 Subscription: Provide direct SS, SSR, vmess, sip002 subscription.
  • 🎠 Deployable: Easy deployment, VPS/Heroku/Local/Docker deployment supported.
  • 🍔 Customizable: Allowed to whitelist the countries to your demand.
  • ...

Why does toshiki-proxypool exists?

  • The original author is no longer maintaining the original project, thus this project exists.

  • Issues function will be disabled in this repository for preventing further affairs and issues from happening.

Installation

Click the button below and fill in the basic information

其中 DOMAIN 需要填写为你需要绑定的域名,CONFIG_FILE 需要填写你的配置文件路径。

heroku app域名为appname.herokuapp.com。项目内配置文件为./config/config.yaml

配置文件模板见 config/config.yaml 文件,可选项区域均可不填。完整配置选项请查看配置文件说明

Deploy

因为爬虫程序需要持续运行,所以至少选择 $7/月 的配置 免费配置长时间无人访问会被heroku强制停止

从源码编译

需要安装Golang

$ go get -u -v github.com/andatoshiki/toshiki-proxypool

运行

$ go run main.go -c ./config/config.yaml

编译

make

下载预编译程序

从这里下载预编译好的程序 release

use docker

docker pull lukemin/toshiki-proxypool

download config.yaml and source.yaml to /path/to/config

wget https://raw.githubusercontent.com/andatoshiki/toshiki-proxypool/master/config/config.yaml wget https://raw.githubusercontent.com/andatoshiki/toshiki-proxypool/master/config/source.yaml

docker run -d --restart=always \
  --name=proxypool \
  -p 8080:8080 \
  -v /path/to/config:/toshiki-proxypool-src/config \
  lukemin/toshiki-proxypool \
  -c config/config.yaml

使用 -p 参数映射配置文件里的端口 使用 -v 参数指定配置文件夹位置(配置文件要自行下载放到目录,方便修改) 使用 -c 参数指定配置文件路径,支持http链接

使用

运行该程序需要具有访问完整互联网的能力。

修改配置文件

首先修改 config.yaml 中的必要配置信息。带有默认值的字段均可不填写。完整的配置选项见配置文件说明

启动程序

使用 -c 参数指定配置文件路径,支持http链接

proxypool -c ./config/config.yaml

如果需要部署到VPS,更多细节请查看wiki

Clash配置文件

远程部署时Clash配置文件访问:https://domain/clash/config

本地运行时Clash配置文件访问:http://127.0.0.1:[端口]/clash/localconfig

本地检查节点可用性

此项非必须。为了提高实际可用性,可选择增加一个本地服务器,检测远程proxypool节点在本地的可用性并提供配置,见proxypoolCheck

screenshot

License

FOSSA Status

FOSSA Status

toshiki-proxypool's People

Contributors

andatoshiki avatar chenskiro avatar dependabot[bot] avatar fossabot avatar lukemin avatar toshikijp avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

toshiki-proxypool's Issues

Fixed internal/cloudflare/cahche.go `not enough arguments in call to api.ZoneDetails` issue with `Error: internal/cloudflare/cache.go:24:31: not enough arguments in call to api.ZoneDetails` error outputs, uncommented `cache.go` file, see the following commits for details. (Reference from @one-pieces's commit on Sep 9, 2021)

Fixed internal/cloudflare/cahche.go's not enough arguments in call to api.ZoneDetails issue with Error: internal/cloudflare/cache.go:24:31: not enough arguments in call to api.ZoneDetails error outputs, uncommented cache.go file, see the following commits for details. (Reference from @one-pieces's commit on Sep 9, 2021)
56e6e6a

📌 Potential to-do lists after 06/22

TODO

This files contains the current to-do list for future and also some memos in case I forget, will be synced into issue#24 every time I update this file. Note that this has not been fully formatted but with simple markdown listings (since all these were just a rush through of my memory on what I did/bugs/failures I encounter in a short time period in case I forget)

self-hosted workflows issues and errors explained:

  • Docker image build and deploy with ghcr.io registries, add personal access token according to https://github.com/skills/publish-packages documentation
  • Difference between self make build and GitHub action auto build and release, how it works out.
  • Go-Bindata command bash not found, make command incompatibility
  • Only bind html or bind geoip database+html
  • go report card cli, also usage of command go fmt for linting and formatting go source codes, refer to https://stackoverflow.com/questions/47735678/goland-how-to-use-gofmt
  • dockerfile compilation errors behind it, not using the original forked "proxypool" default repository name in dockerfile but change to GitHub repository name in dockerfile in regard to ghcr.io
  • Explain what the fuck is ghcr.io
  • Failures on github dependabot package updating due to careless git-lfs tracking on all files ended with extension .go
  • How to add a fucking GitHub personal access token as a repository secret in a repo for workflow to properly run
  • Bindata command line .sh shell script completion
  • GitHub default git push default is 25mb, but when it encounters files larger than 25mb, use either git lfs or increase the post buffer size with git config --global http.postBuffer 524288000 command, refer more at https://www.codegrepper.com/code-examples/shell/git+increase+buffer+size
  • (Optional): CodeQl code security checking via workflows on GitHub for both javascript and golang
  • Update packages manually, refer to this article on medium https://khorlee.medium.com/dependency-management-in-go-lang-using-go-commands-7900a7b2f760
  • running make command on win based systems either via mingw32 or chocolatey, never tested on the other ones.
  • Add heroku deployment option via app.json+heroku deploy button on README
  • Add fly.io deployment option via fy.toml, refer to https://github.com/ssrlive/proxypool for newly implemented options.
  • cloudflare/cache.go commented some lines due to not enough arguments, refer more at https://github.com/andatoshiki/toshiki-proxypool/runs/6932178272?check_suite_focus=true, solution found in https://github.com/ssrlive/proxypool
  • golang.org/x/net and other pkgs require go version <=1.17.x, incompatible with greater versioned workflow runners, that causes error on building outputs, refer the issue here https://github.com/andatoshiki/toshiki-proxypool/runs/6930875206?check_suite_focus=true
  • add bt/aapanel panel installation guide for Chinese users
  • Building from source
  • Go dev environment setup and running locally
  • Static html files and resources with Jsdelivrs CDN's might block the viewing speed of mainland Chinese user's page loading speed the fuck out, switch from cdn.jsdelivr... to fastly.jsdelivr...
  • Explain the detailed reasons on why the first few workflows failed that was automatically triggered by dependabot, and view it's original workflow files for further debugging details
  • Conventional commit via husky, not yet implemented, video on bilibili detailly explained
  • PR templates, issue templates, discussions...
  • Changelog automatic generations by conventional commits
    • Manually updated Changelog of v0.0.1
  • Code of Conduct, contributing.md... in .github folder
  • Documentations via docusaurus hosted in a subfolder of that doc site
  • Reconstruct better looking READMEs
  • Code owners, and it's original owners, full reasons on why this toshiki-proxypool project exists.
  • Thanks to @xiaofei-ya's Note.md note file in the root folder of the repo, without that shit, my dev speed will be 10 times slower along with thousands of tabs on stackoverflow in my browser
  • i18n language localizations and etc (Simplified Chinese, Traditional Chinese, English, Japanese), the program itself with Chinese commenting is fine, but the web UI
  • Rewrite web UI, publish a new release without my own comments/links on, based on the original proxypool UI, rewrite using materialize framework (dark theme switch included)
  • Also unifies the badge/shield styles based on shields.io, add extra shields
  • Add repo star sparklines graphs in regard to https://github.com/antonmedv/spark
  • Redesign logo for proxypool
  • Configure markdown, javascript, go... lint workflow action, currently unknown on how to work out
  • Get some sleep you idiot
  • Rewrite the docs both in GitHub wikis, docsify and docusaurus for backup in case accessible domain blockages or CDN issues, REMEMBER TO BACKUP THE IMAGES!
  • How to make the docker image/ghcr package public on GitHub
    • Linking to repo
    • Change visibility
    • And a lot of other shits I don't know, but yes, sleep rn (update 6/17/2022)
  • FOSSA and codecov code analysis badge added
    • Add a coverage.txt file to the root of project for making codecov's API able to fetch the coverage report file.
  • Added country node speed white listing feature which is available to set through under config.yml
  • #37
  • heroku alterbative-koyeb deployment
  • #41

[BUG]docker镜像不能下载,源码编译也不行。

  • [Y] I'm using the newest release version, and bug isn't solved
    我使用的是最新的release版本,Bug仍未解决
  • [Y] I've tested on the newest master, and bug isn't solved
    我测试的是最新master分支,Bug仍未解决
  • [Y] I've searched the issues and there's not a similar one
    我已经查看过其他issue,没有类似的情况

Bug Version出现Bug的版本:v0.x.x
Environment操作环境:heroku/ubuntu/mac

Description 问题描述

docker pull docker.pkg.github.com/andatoshiki/toshiki-proxypool/proxypool:latest
Error response from daemon: Head "https://docker.pkg.github.com/v2/andatoshiki/toshiki-proxypool/proxypool/manifests/latest": no basic auth credentials

docker login https://docker.pkg.github.com
Username:****
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Docker pull docker.pkg.github.com/andatoshiki/toshiki-proxypool/proxypool:latest
Error response from daemon: manifest unknown

源码编译:
root@instance-20210810-1659:/home/ubuntu# go get -u -v github.com/andatoshiki/toshiki-proxypool
go: go.mod file not found in current directory or any parent directory.
'go get' is no longer supported outside a module.
To build and install a command, use 'go install' with a version,
like 'go install example.com/cmd@latest'
For more information, see https://golang.org/doc/go-get-install-deprecation
or run 'go help get' or 'go help install'.
root@instance-20210810-1659:/home/ubuntu# go version
go version go1.19.1 linux/amd64

To Reproduce 复现步骤

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.