Comments (3)
I would also consider adding some jitter here :) Also I believe the ultimate solution is to move the dataItems delivery to a separate/background job (#877)
from ao.
Yes, we should have exponential backoff, but then also a robust way to ensure the item makes it through the upload eventually if it fails.
from ao.
In uploader.rs, add exponential backoff to the loop that retries the upload, if it reaches some specified number of retries, say 10, add it to a persistent queue of items that need to be tried again later. And have a background process pulling from this queue and uploading.
uploader.rs will need access to store.rs, so store.rs will become a dependency of uploader.rs, new methods will need to be added to store.rs to persist the list of items that need to be retried. Perhaps they can be stored in rocksdb with a different key
from ao.
Related Issues (20)
- chg(CU): Result Storage - SQLite is great at storing small amounts of data but may not be great at storing large datasets, consider a different storage strategy
- chg(mu): refactor spawn message for .receive
- feat(cu): Handle READ Messages HOT 2
- feat(dev-cli): Lua Bindings for C++ Physics Library and Compilation to WASM32 HOT 1
- bug(su): no longer able to run the su service in docker HOT 2
- bug(mu): procs.json is being emptied out HOT 1
- Standardize ETH address format case HOT 6
- feat(mu): Utilize Litestream to replicate and restore SQLite to s3
- fix(mu): Move procs.json to a sqlite table
- bug(connect): `message` creating invalid DataItems when used in Deno - possibly `createDataItemSigner` HOT 7
- chore(su): unit tests failing to build HOT 1
- feat(dev-cli): Ability to build 32bit and 64bit modules.
- bug(mu): message loss from intermittent fetch failure
- chore(cu): map `signature`, `data`, and `anchor` fields in `loadProcess`
- feat(mu): add metric to track size of trace database
- feat(dev-cli): Auto-set Memory-Limit flag based on module requirements during `ao publish`
- bug(su): processes with a lot of large messages killing the machine on get_messages
- feat(cu): utilize cache on dry-run if underlying state was recently evaluated.
- perf(su): implement SU datastore to run purely on rocksdb
- bug(su): su returns null data for strings with invalid utf8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ao.