nyaxt / otaru Goto Github PK

Hard to Understand

It should have at least minimum documentation on its architecture and basic code flow

Vague Layering and Violations

• Internal members of CachedBlobEntry shouldn't be touched casually.
• invalidateCacheBlob takes an instance of CachedBlobStore. srsly?

CachedBlobEntryManager entry writeback scheduling

We currently casually block on Open, but we should avoid it by scheduling write back better.

CachedBlobEntry.mu held too long

We should avoid scheduling backend IO while holding mu.
Maybe introduce more "-ing" states.

make chmod +x file work

This indicator would be useful to diagnose otaru bottleneck.
If sync writeback is happening too often, it indicates that cachedblobstore is not keeping up with new writes.

assetfs is broken somehow

/scripts/app.js 404

inodedb.Sync() takes too long time if cachedblobstore is full and busy flushing out dirty blob caches.
This might be cause of unknown EIOs and pauses during rsyncs.

Currently we:

lock {
  serialize+gzip
  write blob
}

This should be:

lock {
  serialize+gzip
}
write blob

The "Mem Allocated" count seems to increase over time.

Looks like this is the root cause of #39.

We currently never explicitly GC type OpenFile struct.
This is problematic because:

• The FileWriteCache for the OpenFiles may not be ever flushed at exit.
• The OpenFile eventually takes up system memory, causing OOM.

We regularly get some, so need to write retries:
https://circleci.com/gh/nyaxt/otaru/55

Would be useful for debugging #39

Now that we have optimized inodedb, gcloud/datastore/dbtransactionlogio isn't able to keep up with persisting inodedb txs.

https://circleci.com/gh/nyaxt/otaru/64

No HTTP req is sent when loglevel is changed from webui.

The CachedBlobEntry returned from the manager may be closed before be.OpenHandle is called.

This can lead to broken cache -> data corruption