Comments (9)
The defaults for the sqlite storage aren't optimal. You can tune those (and get 10-20x improvement), but additionally you can set sqlite storage to be entirely in memory. I'll provide the details soon.
from torrent.
Sorry for the delay on this: Here are the defaults I use in https://www.coveapp.info/ for a squirrel.Cache, the implementation behind the direct sqlite storage:
cacheOpts.SetAutoVacuum = g.Some("full")
cacheOpts.SetJournalMode = "wal"
cacheOpts.SetSynchronous = 0
cacheOpts.Path = "squirrel2.db"
cacheOpts.Capacity = 9 << 30
cacheOpts.MmapSizeOk = true
cacheOpts.MmapSize = 64 << 20
cacheOpts.CacheSize = g.Some[int64](-32 << 20)
cacheOpts.SetLockingMode = "normal"
cacheOpts.JournalSizeLimit.Set(1 << 30)
cacheOpts.MaxPageCount.Set(15 << 30 >> 12)
This essentially says: Allow concurrent reads and a single writer (with decreased transaction overhead), don't bother to flush to disk on writes (it's a cache), store all the data in a file called squirrel2.db, limit the file to 9 GiB. Memory map the first 64 MiB of data (I think). Keep 32 MiB of the database in memory at most. Allow regular transactions. Don't let the journal get over 1 GiB in size. If the file gets over 15 GiB, return an error.
Many of these settings should be the default. Take a look at squirrel.NewCacheOpts, there's plenty of stuff in there, including exclusive mode, and memory mode which will give you even better performance.
from torrent.
if torrent file size like 5gb is it store whole into memory?🤔
if store data (~20mb chunk) into disk then I can get the best speed ?
from torrent.
Yes, download speeds are improved, but not suficient .
I noticed that Blot Storage has fast download speed. Is there bolddb has size limit, or is it possible to set limit?
from torrent.
No, the bolt DB implementation provided in anacrolix/torrent doesn't include size limits, or any cache eviction.
from torrent.
Did you want to try with https://github.com/anacrolix/possum?
The Go interface is here: https://pkg.go.dev/github.com/anacrolix/possum/go.
You can use the resource.Provider interface in https://pkg.go.dev/github.com/anacrolix/possum/[email protected]/resource with http://localhost:6060/github.com/anacrolix/[email protected]/storage#NewResourcePieces. It does require that you compile a Rust library.
In my testing it's not currently faster than using squirrel, but it is heading that way.
I'm not sure in general why you're not happy with the other storage backends, I've not seen them be bottlenecks before, so if you have more information you could share, please do (torrent/magnet link for example).
from torrent.
I'm working on a project that user can upload torrent file into their cloud storage like one drive,pcloud,storj,wasabi.. etc. I made an api that that require file reader to upload file . In my case I have got good speed except sqlite and filecache(maybe) .
from torrent.
Can you just pick one storage backend and go with that? Any reason you need the sqlite or filecache ones?
from torrent.
because cheap vps have low storage, so I'm trying to upload it like streaming without saving the file to disk.
from torrent.
Related Issues (20)
- Slow downloads while multiple large torrents are being downloaded. HOT 25
- panic: net/http: internal error: connCount underflow HOT 9
- listen tcp6 :42069: socket: address family not supported by protocol HOT 3
- ICE Servers config doesn't support authentication for TURN servers HOT 1
- hign cpu usage of sha1.blockAMD64 HOT 8
- Cannot download file from a not completed peer HOT 5
- Can't build [email protected] with go1.20 HOT 2
- Cannot use download subcmd to download what i seed torrent HOT 6
- Ignore malformed (optional) fields in metainfo.Load
- ratelimireader.go panic HOT 2
- Does repo support to build private dht bootstrap server? HOT 1
- Error tracking HOT 1
- Add mention of Torrent-WebDAV HOT 1
- missing package 'main' when trying to build it as Go library HOT 1
- Error while trying to use CGO to make it a C library, any suggestions? HOT 7
- how to ban client ? HOT 2
- Missing projects using this library HOT 9
- Memory stays high after dropping torrent HOT 10
- Small torrents with sparse seeding not completing HOT 3
- Unable to selectively download files within a torrent HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from torrent.