Comments (4)
For good measure: It's in a branch where I'm experimenting with namespaces, so that may be related.
from ucs.
Not related to namespaces, but fs-cache GC - when run with a cache-size of eg. 1k, it will GC a lot, and eventually it can't keep up with the clients -- half closed connections and open file-handles all over.
go run ./cmd/ucs -quota 100k -verbose
go run ./cmd/ucs-bender #Possibly in separate terminal window
# Now observe the server's epic slowness
(Edited to clarify it's about the cache GC, not Go GC + example)
from ucs.
Thinking it over, it should have some way of ensuring we'll do a GC on/after an upload. Current logic is to check if (quota < size + newly-uploaded-item) { gc() }
, which means we'll do a GC on most uploads when we're hovering around the quota.
- Run GC periodically (either timer or every X requests/connections)
- Over-GC (say, 10% or just remove the oldest entry from every bucket)
- Have a lock/queue, so we don't over-schedule GC's (we can upload small items a lot faster than we can GC them)
- More fine-grained locking so GC doesn't block new uploads (goes with locked GC'ing, so we don't have something looking at files while another is deleting them...)
from ucs.
After some thinking / digging it turned out to be caused by a missing Close()
call when GC'ing.
Bumping the last comment to a new issue, as it really should be fixed too - see #13
from ucs.
Related Issues (19)
- File-system cache HOT 1
- Add namespacing HOT 5
- Be more intelligent about when to GC HOT 1
- ucs-cli
- GC: Remove files in pairs
- go run ./cmd/ucs puts cache-directory in a temporary location HOT 1
- Graceful shutdown HOT 1
- Filesystem-cache reports having a quota of zero in /metrics HOT 1
- Replace PutObject with composed struct
- Caching large projects hang forever HOT 2
- When a fresh project is simultaneously built multiple times, metrics get off
- Admin interface
- Client library HOT 1
- Fuzz the parser HOT 1
- Bittorrent cache backend
- Cache size quotas HOT 1
- Benchmark utility HOT 1
- HTTP Backend
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ucs.