Use an exponential backoff on SU failed data item uploads about ao HOT 3 CLOSED

I would also consider adding some jitter here :) Also I believe the ultimate solution is to move the dataItems delivery to a separate/background job (#877)

from ao.

Yes, we should have exponential backoff, but then also a robust way to ensure the item makes it through the upload eventually if it fails.

from ao.

In uploader.rs, add exponential backoff to the loop that retries the upload, if it reaches some specified number of retries, say 10, add it to a persistent queue of items that need to be tried again later. And have a background process pulling from this queue and uploading.

uploader.rs will need access to store.rs, so store.rs will become a dependency of uploader.rs, new methods will need to be added to store.rs to persist the list of items that need to be retried. Perhaps they can be stored in rocksdb with a different key

from ao.