jagrit007 / telegram-clonebot Goto Github PK

[USE ALPHA BRANCH] Simple Bot to clone Google Drive Files (or Folders) to your Team Drive[or Normal Drive]. P.S This is not a Mirror Bot. Enjoy ✌🏻

License: MIT License

Python 99.97% Shell 0.03%

team-drive google-drive autorclone gdrive-folder bot telegram telegram-bot python gclone clonebot

telegram-clonebot's Introduction

Why?

For all my friends using my TDs who now need to store everything in it instead of their Drive. Need help?

Guide:

YouTube Guide: Google Drive Clone Bot Set-Up Tutorial | Telegram Bot Setup Guide
- Follow the above guide for Heroku.
- If you wish to run on a VPS, Do all the stuff I did on the VPS Terminal ;)
- Wish to run anywhere else? Follow the guide till the part where I download ZIP Archive from Repl.it. Use that zip on any device you'd like to run the bot on.
- Don't forget to install requirements.txt
```
pip3 install -r requirements.txt
```
Adding Service Accounts to Google Group/TeamDrive

Setting up config file (present in bot/config.py)

BOT_TOKEN : The telegram bot token that you get from @BotFather
GDRIVE_FOLDER_ID : This is the folder ID of the Google Drive Folder to which you want to clone.
OWNER_ID : The Telegram user ID (not username) of the owner of the bot (if you do not have that, send /id to @kelverbot )
AUTHORISED_USERS : The Telegram user IDs (not username) of people you wish to allow for bot access.It can also be group chat id. Write like: [123456, 4030394, -1003823820]
IS_TEAM_DRIVE : (Optional field) Set to True if GDRIVE_FOLDER_ID is from a Team Drive else False or Leave it empty.
USE_SERVICE_ACCOUNTS: (Optional field) (Leave empty if unsure) Whether to use service accounts or not. For this to work see "Using service accounts" section below.
INDEX_URL : (Optional field) Refer to https://github.com/maple3142/GDIndex/ The URL should not have any trailing '/'

Getting Google OAuth API credential file

Visit the Google Cloud Console
Go to the OAuth Consent tab, fill it, and save.
Go to the Credentials tab and click Create Credentials -> OAuth Client ID
Choose Other and Create.
Use the download button to download your credentials.
Move that file to the root of clone-bot, and rename it to credentials.json
Visit Google API page
Search for Drive and enable it if it is disabled
Finally, run the script to generate token file (token.pickle) for Google Drive:

pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib
python3 generate_drive_token.py

Running

To run this bot (locally) (suggested)

python3 -m bot

Deploying to Heroku (Optional) (Not Suitable for very big Clones!)

Please know that after using this button, your work isn't done. You gotta clone heroku app and add credentials.json and token.pickle (By now you would know how to make it.) and this is the perfect time to generate service accounts if you wish to use them. After it's all done, Push changes to Heroku (Step1-2 only).

Tip: Instead of using Termux or local machine, use repl.it, atleast it won't throw any errors in installing Python requirements. From repl.it you could push to a private GitHub repo and attach that to Heroku.

Using service accounts for uploading to avoid user rate limit

For Service Account to work, you must set USE_SERVICE_ACCOUNTS=True in config file or environment variables Many thanks to AutoRClone for the scripts

Generating service accounts

Step 1. Generate service accounts What is service account

Let us create only the service accounts that we need. Warning: abuse of this feature is not the aim of autorclone and we do NOT recommend that you make a lot of projects, just one project and 100 sa allow you plenty of use, its also possible that overabuse might get your projects banned by google.

Note: 1 service account can copy around 750gb a day, 1 project makes 100 service accounts so thats 75tb a day, for most users this should easily suffice.

python3 gen_sa_accounts.py --quick-setup 1 --new-only

A folder named accounts will be created which will contain keys for the service accounts created

NOTE: If you have created SAs in past from this script, you can also just re download the keys by running:

python3 gen_sa_accounts.py --download-keys project_id

Add all the service accounts to the Team Drive or folder

Run:

python3 add_to_team_drive.py -d SharedTeamDriveSrcID

Credits

telegram-clonebot's People

Contributors

Stargazers

Watchers

Forkers

begulatuk xeniel009 herrysis404 cvatricks dakshvv luna009 savmuv omnirepo snk10401 gokhanboss haxshaan bhuvi1242 parammarok infernoscorpion sivagurus cdfxscrq mayursuccess screwd-kid waga43 sesprie likerplayer spxleechers lucifer-00007 baiguke2019 fahdiiig bhayanakaatma1 ksmaheshkumar masterteddyau zero-debugs realtec7 xtivat0r fristyflakes trancanh1102 adhilnagaker obada8 docmjha adisubagja syamsulqadrinurhaidir17 phoenix9head otakuhp meichuanyi itsmasterwayne ronioncloud caffirana diansteve sravanjarvis winkxx yashbarod tarak829 afjal-sharif adithyannellickal askfriends whenugopal bc1804 rocky2242 ch3kuthan ankit-sinha-18 genbogo reejit hramose heavenjasmin angelminati kiprox 547555909 tototto09889 bollyverse fxsihad birrul maddkidd hrd2020github pras92 hfk1991 rzdys vivektp maxxrider noobpeoples syafiqarzaqi shuxiaokai baipiaoheroku bhumik777 sageofd6path kamileecher11 rinrinx2 cyrus-op peics ekal8 htaylight htet09 linwathan1 skueletor darkhacker12121 jpeiterson annihilatorrrr abhint gautam-1834 gamesdrive gautam1834 just5ky nmalik23 vari98

telegram-clonebot's Issues

Error while using Service Accounts

Logs while USE_SERVICE_ACCOUNTS is true

2020-10-09T12:00:29.359005+00:00 app[worker.1]: 2020-10-09 12:00:29,358 - bot - INFO - Authorizing with 0.json service account
2020-10-09T12:00:29.608422+00:00 app[worker.1]: 2020-10-09 12:00:29,608 - bot - INFO - File ID: 10MJ2vwavNELczT2nMvjKghcFft8tTfdC
2020-10-09T12:00:32.639252+00:00 app[worker.1]: 2020-10-09 12:00:32,638 - bot - ERROR - Message to edit not found

Message from telegram (Clone Unsuccessful)

HttpError 404 when requesting https://www.googleapis.com/drive/v3/files/10MJ2vwavNELczT2nMvjKghcFft8tTfdC?supportsAllDrives=true&fields=name%2Cid%2CmimeType%2Csize&alt=json returned "File not found: 10MJ2vwavNELczT2nMvjKghcFft8tTfdC."

Logs while USE_SERVICE_ACCOUNTS is false (Clone Successful)

2020-10-09T12:06:40.531246+00:00 app[worker.1]: 2020-10-09 12:06:40,530 - bot - ERROR - Can't parse entities: can't find end of the entity starting at byte offset 248
2020-10-09T12:06:45.553824+00:00 app[worker.1]: 2020-10-09 12:06:45,553 - bot - ERROR - Can't parse entities: can't find end of the entity starting at byte offset 325
2020-10-09T12:06:50.578583+00:00 app[worker.1]: 2020-10-09 12:06:50,578 - bot - ERROR - Can't parse entities: can't find end of the entity starting at byte offset 256
2020-10-09T12:06:55.601809+00:00 app[worker.1]: 2020-10-09 12:06:55,601 - bot - ERROR - Can't parse entities: can't find end of the entity starting at byte offset 310

2020-10-09T12:08:35.554561+00:00 app[worker.1]: 2020-10-09 12:08:35,554 - googleapiclient.http - WARNING - Encountered 403 Forbidden with reason "userRateLimitExceeded"
2020-10-09T12:08:37.441123+00:00 app[worker.1]: 2020-10-09 12:08:37,440 - bot - ERROR - Message to edit not found

Need Some Improvements

Currently when i run clone command, it stucks after about 1.3or 1.4TB every time and i cant tell if the bot got reset or it just stuck itself. There is no any /status command which can tell cloning is running or got cancelled.

Also there should be option to select howmany operations can run simultaneously like if there are small files than we can take a much as possible otherwise very few like we do in rclone command on pc.

So here is some suggestions;

Add /status command which will tell what operation is running or tells that no any operation is running.
Should show what files its checking or skipping so that we can be assure the bot is working, otherwise it looks like it got stuck
Bot status which will tell how long bot is running and some more info u can gather
Add option to skip some folders by using their folder-id. Like i am cloning a huge folder and i know i have already cloned some sub-folders and dont want want to check their files to save some time.
Ability to run multiple clone operations

[Feature Request] Bot should ask source and destination folder id every time we run clone

hi..
first of all thanks for the bot.

but i want to ask why we need to put folder id in the config file?
bcoz we will not copy just one folder but we will use this bot to manage our tds, lets say about 10 tds, and there are so many folders inside we want to clone, if we put folder id in config file it means we will be able to copy that folder only??

you need to make it variable, and it must ask folder id from user in telegram instead.

Username and buttons

New Features Request

Add Support Of Buttons For Drive And index link
2.And add username support of the uploader

Heroku ERROR

this is my heroku logs:

2021-04-27T18:27:17.132784+00:00 app[worker.1]: 2021-04-27 18:27:17,131 - telegram.utils.promise - ERROR - An uncaught error was raised while running the promise
2021-04-27T18:27:17.132798+00:00 app[worker.1]: Traceback (most recent call last):
2021-04-27T18:27:17.132809+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/telegram/utils/promise.py", line 57, in run
2021-04-27T18:27:17.132811+00:00 app[worker.1]: self._result = self.pooled_function(*self.args, **self.kwargs)
2021-04-27T18:27:17.132811+00:00 app[worker.1]: File "/app/bot/decorators.py", line 7, in wrapper
2021-04-27T18:27:17.132812+00:00 app[worker.1]: if args[0].effective_message.from_user.id in AUTHORISED_USERS or args[0].message.chat_id in AUTHORISED_USERS:
2021-04-27T18:27:17.132812+00:00 app[worker.1]: TypeError: argument of type 'int' is not iterable

Feature Request : Delete Files/Folder

Like a reference : https://github.com/magneto261290/magneto-python-aria/commit/fb75f71325f8c9c27ba815f3436f7ec158db5ce3

Error 400: invalid_request

i am getting this error in the time using this command "python3 generate_drive_token.py"

Possible improvements.

I think you need a script to add them to google groups, when you are copying from public td you need to join the group to clone.

HI! error

When i run the /clone ID
The bot was cloning but sende massive mesagges wit (HI!) too much..

HttpError 404 when requesting

Error when i can clone

HttpError 404 when requesting https://www.googleapis.com/drive/v3/files/1VwHyP_L8OXSYJVZMaX3iW34b7q5uHCzp?supportsAllDrives=true&fields=name%2Cid%2Csize&alt=json returned "File not found: 1VwHyP_L8OXSYJVZMaX3iW34b7q5uHCzp."

Al parecer no encuentra la carpeta destino, y el link me manda a que debo iniciar sesion para que funcione

{
"error": {
"errors": [
{
"domain": "usageLimits",
"reason": "dailyLimitExceededUnreg",
"message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup.",
"extendedHelp": "https://code.google.com/apis/console"
}
],
"code": 403,
"message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup."
}
}

Timeout Error :(

Tried Coping 8TB and it throws Timeout Error.

Logs :-

2020-07-29T12:39:14.692732+00:00 app[worker.1]: An uncaught error was raised while running the promise
2020-07-29T12:39:14.692742+00:00 app[worker.1]: Traceback (most recent call last):
2020-07-29T12:39:14.692742+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.7/site-packages/telegram/utils/promise.py", line 56, in run
2020-07-29T12:39:14.692743+00:00 app[worker.1]: self._result = self.pooled_function(*self.args, **self.kwargs)
2020-07-29T12:39:14.692743+00:00 app[worker.1]: File "clonerbot.py", line 158, in clone
2020-07-29T12:39:14.692744+00:00 app[worker.1]: message.edit_text(to_send)
2020-07-29T12:39:14.692744+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.7/site-packages/telegram/message.py", line 876, in edit_text
2020-07-29T12:39:14.692745+00:00 app[worker.1]: chat_id=self.chat_id, message_id=self.message_id, *args, **kwargs)
2020-07-29T12:39:14.692745+00:00 app[worker.1]: File "", line 2, in edit_message_text
2020-07-29T12:39:14.692745+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.7/site-packages/telegram/bot.py", line 67, in decorator
2020-07-29T12:39:14.692746+00:00 app[worker.1]: result = func(*args, **kwargs)
2020-07-29T12:39:14.692746+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.7/site-packages/telegram/bot.py", line 1898, in edit_message_text
2020-07-29T12:39:14.692746+00:00 app[worker.1]: return self._message(url, data, timeout=timeout, reply_markup=reply_markup, **kwargs)
2020-07-29T12:39:14.692747+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.7/site-packages/telegram/bot.py", line 175, in _message
2020-07-29T12:39:14.692747+00:00 app[worker.1]: result = self._request.post(url, data, timeout=timeout)
2020-07-29T12:39:14.692747+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.7/site-packages/telegram/utils/request.py", line 333, in post
2020-07-29T12:39:14.692747+00:00 app[worker.1]: **urlopen_kwargs)
2020-07-29T12:39:14.692748+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.7/site-packages/telegram/utils/request.py", line 237, in _request_wrapper
2020-07-29T12:39:14.692748+00:00 app[worker.1]: message = self._parse(resp.data)
2020-07-29T12:39:14.692748+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.7/site-packages/telegram/utils/request.py", line 195, in _parse
2020-07-29T12:39:14.692756+00:00 app[worker.1]: raise RetryAfter(retry_after)
2020-07-29T12:39:14.692756+00:00 app[worker.1]: telegram.error.RetryAfter: Flood control exceeded. Retry in 9 seconds

Youtube-dl issue

The bot was initially was giving error for not able to found youtube-dl so i changed the requirement.txt and made it to point to original youtube-dl but because of that my app get suspended.
Please update the proper youtube-dl in requirement.

file error not found

2021-11-12T23:31:24.542476+00:00 heroku[worker.1]: Starting process with command python3 -m bot
2021-11-12T23:31:25.172484+00:00 heroku[worker.1]: State changed from starting to up
2021-11-12T23:31:26.003644+00:00 app[worker.1]: 2021-11-12 23:31:26,003 - bot - INFO - Bot Started!
2021-11-12T23:34:00.811086+00:00 app[worker.1]: 2021-11-12 23:34:00,810 - bot - INFO - Authorizing with 0.json service account
2021-11-12T23:34:00.812033+00:00 app[worker.1]: 2021-11-12 23:34:00,811 - telegram.utils.promise - ERROR - An uncaught error was raised while running the promise
2021-11-12T23:34:00.812034+00:00 app[worker.1]: Traceback (most recent call last):
2021-11-12T23:34:00.812035+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/telegram/utils/promise.py", line 57, in run
2021-11-12T23:34:00.812037+00:00 app[worker.1]: self._result = self.pooled_function(*self.args, **self.kwargs)
2021-11-12T23:34:00.812038+00:00 app[worker.1]: File "/app/bot/decorators.py", line 8, in wrapper
2021-11-12T23:34:00.812038+00:00 app[worker.1]: return func(*args, **kwargs)
2021-11-12T23:34:00.812039+00:00 app[worker.1]: File "/app/bot/main.py", line 54, in cloneNode
2021-11-12T23:34:00.812039+00:00 app[worker.1]: gd = GoogleDriveHelper(GFolder_ID=DESTINATION_ID)
2021-11-12T23:34:00.812039+00:00 app[worker.1]: File "/app/bot/gDrive.py", line 44, in init
2021-11-12T23:34:00.812039+00:00 app[worker.1]: self.__service = self.authorize()
2021-11-12T23:34:00.812040+00:00 app[worker.1]: File "/app/bot/gDrive.py", line 284, in authorize
2021-11-12T23:34:00.812040+00:00 app[worker.1]: credentials = service_account.Credentials.from_service_account_file(
2021-11-12T23:34:00.812041+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/google/oauth2/service_account.py", line 238, in from_service_account_file
2021-11-12T23:34:00.812041+00:00 app[worker.1]: info, signer = _service_account_info.from_filename(
2021-11-12T23:34:00.812041+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/google/auth/_service_account_info.py", line 72, in from_filename
2021-11-12T23:34:00.812042+00:00 app[worker.1]: with io.open(filename, "r", encoding="utf-8") as json_file:
2021-11-12T23:34:00.812042+00:00 app[worker.1]: FileNotFoundError: [Errno 2] No such file or directory: 'accounts/0.json'

this is log in heroku i followed the video to the letter cannot get past this stage what have i done wrong

Get rid of /clone

Can we get rid of /clone command? The bot doesn't do anything rather cloning then why we need to type /clone every time. That's kinda annoying.

feature add

Telehack

Let's make it happen

cloneBot needs

Hi, I'm trying to make the bot and I can't.
I would like to know if you provide the service of carrying it out or of assistance,
I need this bot to make copies of my content systematically, I am very interested

skip files you already have on Team Drive at the time of copying

would it be possible to add this skill? skip files that are already on the team drive at the time of copying

After following the video i still got Error on lines

 python3 -m bot
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 185, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/lib/python3.8/runpy.py", line 144, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "/usr/lib/python3.8/runpy.py", line 111, in _get_module_details
import(pkg_name)
File "/home/runner/Telegram-CloneBot/bot/init.py", line 4, in
import telegram.ext as tg
ModuleNotFoundError: No module named 'telegram'
 python3 -m bot
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 185, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/lib/python3.8/runpy.py", line 144, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "/usr/lib/python3.8/runpy.py", line 111, in _get_module_details
import(pkg_name)
File "/home/runner/Telegram-CloneBot/bot/init.py", line 4, in
import telegram.ext as tg
ModuleNotFoundError: No module named 'telegram'

Encountered 403 error

2022-01-23T15:35:45.165691+00:00 app[worker.1]: 2022-01-23 15:35:45,165 - googleapiclient.http - WARNING - Encountered 403 Forbidden with reason "teamDrivesSharingRestrictionNotAllowed" 2022-01-23T15:35:51.889428+00:00 app[worker.1]: 2022-01-23 15:35:51,889 - googleapiclient.http - WARNING - Encountered 403 Forbidden with reason "teamDrivesSharingRestrictionNotAllowed" 2022-01-23T15:35:58.507440+00:00 app[worker.1]: 2022-01-23 15:35:58,507 - googleapiclient.http - WARNING - Encountered 403 Forbidden with reason "teamDrivesSharingRestrictionNotAllowed" 2022-01-23T15:36:05.078522+00:00 app[worker.1]: 2022-01-23 15:36:05,078 - googleapiclient.http - WARNING - Encountered 403 Forbidden with reason "teamDrivesSharingRestrictionNotAllowed" 2022-01-23T15:36:11.670738+00:00 app[worker.1]: 2022-01-23 15:36:11,670 - googleapiclient.http - WARNING - Encountered 403 Forbidden with reason "teamDrivesSharingRestrictionNotAllowed" 2022-01-23T15:36:18.317766+00:00 app[worker.1]: 2022-01-23 15:36:18,317 - googleapiclient.http - WARNING - Encountered 403 Forbidden with reason "teamDrivesSharingRestrictionNotAllowed" 2022-01-23T15:36:24.910884+00:00 app[worker.1]: 2022-01-23 15:36:24,910 - googleapiclient.http - WARNING - Encountered 403 Forbidden with reason "teamDrivesSharingRestrictionNotAllowed" 2022-01-23T15:36:31.497982+00:00 app[worker.1]: 2022-01-23 15:36:31,497 - googleapiclient.http - WARNING - Encountered 403 Forbidden with reason "teamDrivesSharingRestrictionNotAllowed" 2022-01-23T15:36:31.505162+00:00 app[worker.1]: 2022-01-23 15:36:31,504 - bot - INFO - Total Attempts: 15 2022-01-23T15:36:31.505392+00:00 app[worker.1]: 2022-01-23 15:36:31,505 - bot - ERROR - <HttpError 403 when requesting https://www.googleapis.com/drive/v3/files/1PcrIlzJ07zylQZOnPzxeiI2Yz-2LtPSS/copy?supportsAllDrives=true&alt=json returned "Sharing restrictions cannot be set on a shared drive item.">

Can i know where is the problem is?