Twitter: aborts if media download yields "403 Forbidden", e.g. removed by copyright claim about timeliner HOT 4 OPEN

I dunno, this is a pickle. The obvious error is not being able to continue after 403. My data retrieval process just aborts.

But, what should we do about it? Sure, continue after the error. But, personally, I am not fan of losing any information. In this case the information is:

there once was an attachment, but we didn't manage to fetch it in time because it was later taken down because of a DMCA complaint

I'd prefer this to be stored in the data model. I haven't researched Timeliner's data model, something like attachment: {id: '987654321', permanentFetchFailureReason: '403 not found - Twitter or the author removed it?'} ?

Things to think about:

• there's a distinction between transient errors and "likely fetch will never work again" (like in this case)
• I guess Timeliner already supports transient errors (since re-running my retrieval always ended up fetching this errored attachment) and
• Does Timeliner mind on "force full refresh" where an attachment was managed to be fetched before (we already have a copy of that attachment stored) and now it's unavailable doing "full refresh"? Obviously it should just shrug and carry on instead of aborting.
• if we're doing "full refresh" and we have a permanentFetchFailureReason, should we still re-try fetching it? I guess chances are kinda slim that the likes of Twitter restore content. But then again, re-trying probably isn't expensive (if we re-try, maybe not pollute the log about errors we think are really likely to surface from re-try attempts).

from timeliner.

...if we're doing "full refresh" and we have a permanentFetchFailureReason, should we still re-try fetching it?

I concur with your conclusion that rechecking is inexpensive, and so worth trying.

from timeliner.

Ah, oops. Not something I anticipated or encountered. How do you think we should handle this?

from timeliner.

403 is usually permanent, or something has to be changed on the server to remove that error.

Perhaps Timeliner should simply continue to the next item after seeing a 403. Log the 403, but continue on, since there's nothing we can do about it. This should probably be the behavior no matter what mode it's running in.

But I also agree that simply trying once or twice more before continuing on wouldn't be a bad idea, in case it was a fluke.

from timeliner.