Comments (6)
@gcilinskas
As @erikn69 pointed out, extensive object caching breaks concurrency. The code of this adapter assumes that it's operating alone on a selected drive.
One way to partially work around this would be to disable all caching in the code (which would slow everything down considerably)...but that's only half of the fix - there's still the issue of mixing write, read, delete, create operations and you'd still have race conditions to solve.
Perhaps you can just change the approach and not work on the same folder from multiple threads/jobs/queues? Or just not delete the folder after you're done with it (I assume you use one job to fill the folder and another to consume what's in it and then delete the folder)?
from flysystem-google-drive-ext.
Yeah, manual human interaction with gdrive is possible on our side so removal of folders might be at random times. We are fetching data from api and saving its assets to gdrive in structured folders hierarchy.
clearCache method helped for now, performance for gdrive upload is not crucial for us. Regarding to racing conditions we did some workarounds with sleep functions, currently seems to work
We don't have options to change approaches in usage of gdrive folders
from flysystem-google-drive-ext.
Ok. So I guess you're doing something similar to:
lRetry:
try {
... small set of operations on the drive or a complete set with error mitigation
} catch(\Throwable $e) {
sleep(rand()%4);
$adapter->clearCache();
if($retry-->0)
goto lRetry;
}
from flysystem-google-drive-ext.
This package is using a little cache trick, if you delete a folder in Google drive manually, that folder still remains in cache even though it no longer exists in google drive (if you are using a long running script, because on every new instance it starts the connection and cache), maybe the function Storage::disk('google')->getDriver()->getAdapter()->clearCache();
before createDir
will be useful to you (maybe with catch, only when there is these exception).
flysystem-google-drive-ext/src/GoogleDriveAdapter.php
Lines 257 to 263 in aa0b678
@masbug is this a bug?, maybe i can make a fix
from flysystem-google-drive-ext.
Storage::disk('google')->getDriver()->getAdapter()->clearCache();
This one helped, thanks!
Just for information, Since I use queued jobs, I had to call this clearCache command not only once before creating directories, but also in every job class that was using google drive upload
from flysystem-google-drive-ext.
On limitation's section
Using display paths as identifiers for folders and files requires them to be unique. Unfortunately Google Drive allows users to create files and folders with same (displayed) names. In such cases when unique path cannot be determined this adapter chooses the oldest (first) instance
Concurrent use of same Google Drive might lead to unexpected problems due to heavy caching of file/folder identifiers and file objects.
So, for concurrent threads of queued jobs you have to do this workaround with cache, otherwise you get those errors, or when create a directory duplicate(created at the same time with same name) and you put a unique file or folder inside, the adapter will be able to reach it properly because it chooses the oldest (first), so, create directory, uncache the folder, then put file inside
I make a PR with uncacheFolder
function, if it is merged it be available on the next release
$disk=\Storage::disk('google');
$disk->createDir('test/testa');
//manually delete without adapter or other job create it first
$dir=$disk->getDriver()->getAdapter()->getMetadata('test/testa');
$disk->getDriver()->getAdapter()->getService()->files->delete($dir['id']);
//uncache
$disk->getDriver()->getAdapter()->uncacheFolder('test/testa');
// refresh id and put file
$disk->put('test/testa/file.txt', 'Contents1');
from flysystem-google-drive-ext.
Related Issues (20)
- creating a new folder in my drive HOT 2
- Backup failed because: Unable to write file at location: HOT 11
- Invalid value at 'payload' (Map) ? HOT 6
- Error in masbug because upgrade system Google API HOT 5
- Duplicate root directories created when instantiating the adapter HOT 3
- 401 Request had invalid authentication credentials (laravel jobs) HOT 4
- Problem with spatie laravel backup - League\Flysystem\UnableToReadFile HOT 8
- Don't list files and Directories in shared folder HOT 1
- Laravel 10 support HOT 2
- Can't display paths, just IDs HOT 1
- How to move app to live HOT 1
- Argument #1 ($value) must be of type Countable|array, null given in GoogleDriveAdapter.php:1037 HOT 3
- Fails to report delete error
- Moving folders will fall into the trash HOT 3
- Allowed memory size issue HOT 1
- When uploading a file to a shared folder, the file doesn't end up in the desired destination. HOT 10
- PHP Warning: Invalid argument supplied for foreach() in vendor/masbug/flysystem-google-drive-ext/src/GoogleDriveAdapter.php on line 1104 HOT 19
- listContents returns empty array HOT 3
- Move Shared Drive file to a folder HOT 2
- cannot list directory shortcuts HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flysystem-google-drive-ext.