Comments (9)
No, the default is 100 MB, not KB.
I made a small callback progressbar function and it downloads the file in 100MB chunks.
Also you can see the python process taking 100 MB in the RAM and then releasing the 100 MB every chunk cycle.
Unfortunately I never did a PR, but there is always a first time ^^
I think so, yes. Because from what I've seen, the more you switch between writing and downloading, the more time you need because it needs to restart the download process.
But I could also be wrong and maybe it gets worse.
from pydrive2.
@KengoSawa2 that can be a good PR, happy to help with it! thanks 🙏
There is a workaround though, that you could use for now. That's how we do this in DVC (the primary reason we forked and support PyDrive2 at all):
from pydrive2.
Closed by #50 Thanks @TA40 !
from pydrive2.
@KengoSawa2 also, consider creating a separate feature request issue for this.
from pydrive2.
I think default is 100KB, not 100MB.
@TA40 I think it should be very easy to add:
pretty much add it here:
https://github.com/iterative/PyDrive2/blob/master/pydrive2/files.py#L311
and pass it then to the downloader:
https://github.com/iterative/PyDrive2/blob/master/pydrive2/files.py#L331
unless I'm missing something.
Would be great to add a test as well for this.
Feel free to create a PR and we'll release a new version very quick.
Btw, out of curiosity - do you expect a better performance with a large buffer size?
from pydrive2.
@TA40 hmm 🤔
Yes, you are right, it's 100Mb, sorry.
https://github.com/googleapis/google-api-python-client/blob/master/googleapiclient/http.py#L72
https://github.com/googleapis/google-api-python-client/blob/master/googleapiclient/http.py#L665
https://github.com/googleapis/google-api-python-client/blob/master/googleapiclient/http.py#L673
Because from what I've seen, the more you switch between writing and downloading, the more time you need because it needs to restart the download process.
that's probably true. 200MB can do some improvement in this case. Depending on the network speed/file system performance, I would expect though that OS buffers can mitigate this to some extent. I would test the network performance with some external tool and see if there are spikes.
Btw, curious what is your use case for this? E.g. if you need to download multiple files you can also do it in parallel that should also keep the network loaded.
from pydrive2.
I think... I did it? #50
I'm not really sure.
Btw, curious what is your use case for this?
I need to download multiple files to reupload them to another cloud service. And I want to do it as fast as possible.
E.g. if you need to download multiple files you can also do it in parallel that should also keep the network loaded.
I don't like downloading multiple files at once because it messes around with the disk i/o and RAM and I am running multiple scripts at once.
from pydrive2.
Hello
Thank you for development and contributions of PyDrive2:)
I want to upload a huge zip file by PyDrive2.
When I looked into the API, GetContents() has a callback for progress.
But UpLoad() does not.
Are you planning to add a callback to show the upload progress?
Progress is a very important feature as I plan to upload a zip close to 750GB of single file
If the community isn't interested in implementing it, I'd consider implementing it myself. Please let us know your ideas and opinions.
Thank you
from pydrive2.
Thank you for your kind response.
I raised the new issue.
I also understand your workaround.
I refer to your workaround so you don't have to rush to officially add features.
Thank you :)
from pydrive2.
Related Issues (20)
- Cannot install PyDrive2 (Raspberry PI3A+)
- error: from pydrive2.auth import GoogleAuth HOT 7
- How to distinguish between same-named files and directories in Google Drive
- How to distinguish between same-named files and directories in Google Drive HOT 1
- A simple way of initializing a pydrive.GoogleDrive object fails when pydrive is replaced by pydrive2 HOT 1
- Cant copy the file... using service account HOT 3
- Fail to authenticate with pydrive2 to google drive HOT 4
- Access blocked: This application request is invalid HOT 1
- Is there a Google Drive file sharing feature? HOT 3
- Unable to list files with service account auth for GDriveFilesystem HOT 3
- How do I write to specific tab? HOT 1
- fetch a specific folder id by folder name HOT 4
- Missing 1.17.1 release? HOT 1
- auth: LocalWebserverAuth: fail if in non-interactive mode
- ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2427) HOT 2
- Review and fix issues with find
- Create Succesfull Authentication using Google OAuth client ID for a remote server accessible via SSH in a local machine HOT 1
- AssertionError on using fsspec with service account HOT 1
- [Q]Set Disable Download HOT 1
- Would you be interested to add type hints?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pydrive2.