Internal server error for a NYTimes URL about translation-server HOT 10 CLOSED

I wouldn't use NYT for CI, since they rate-limit. Use something that will work reliably.

Is that okay, or is it best to just fast-forward modules/translators since the other two should be updated in lock-step with zotero/translation-server?

Yeah, definitely don't update the others. Just use git pull origin master in the translators submodule.

where'd you see the TypeError: Cannot read property 'replace' of null log?

It's just in the stdout from the server, which includes Zotero debug output (with lines beginning with, e.g., (3)(+0000010):).

from translation-server.

@dongbohu how do we see the logs for the translation-server process that is managed with supervisor?

from translation-server.

@dhimmel: According to /etc/supervisor/conf.d/translation-server.conf:

• standard output is in /var/log/supervisor/translation-server.log
• standard error is in /var/log/supervisor/translation-server.err

from translation-server.

Works for us.

https://zbib.org/?q=https://nyti.ms/1NuB0WJ

from translation-server.

Now failing after a few tries from AWS. Still working for a local install. Possible they're rate-limiting.

TypeError: Cannot read property 'replace' of null

    TypeError: Cannot read property 'replace' of null
        at addHighwireMetadata (eval at <anonymous> (/var/task/src/translation/sandboxManager.js:70:4), <anonymous>:471:59)
        at completeItem (eval at <anonymous> (/var/task/src/translation/sandboxManager.js:70:4), <anonymous>:214:2)
        at eval (eval at <anonymous> (/var/task/src/translation/sandboxManager.js:70:4), <anonymous>:343:4)
        at /var/task/modules/zotero/chrome/content/zotero/xpcom/translation/translate.js:384:8
        at Zotero.Translate.Import._runHandler (/var/task/modules/zotero/chrome/content/zotero/xpcom/translation/translate.js:1128:32)
        at run (/var/task/modules/zotero/chrome/content/zotero/xpcom/translation/translate.js:188:23)
        at /var/task/modules/zotero/chrome/content/zotero/xpcom/translation/translate.js:250:6
        at new Promise (<anonymous>)
        at Object._itemDone (/var/task/modules/zotero/chrome/content/zotero/xpcom/translation/translate.js:248:11)
        at Object._itemDone (/var/task/src/translation/sandboxManager.js:96:17)

from translation-server.

I suspect they're serving something different in the cases where it's failing, but this looks like a bug in the Embedded Metadata translator — if I skip the call to addHighwireMetadata() it succeeds. We'll investigate.

from translation-server.

They're definitely also rate-limiting, though. After enough requests the server starts getting a 403 from nytimes.com, and then it returns 500. So it's possible that's all you're seeing.

from translation-server.

OK, rate-limiting and page differences aside, there was a regression in the Embedded Metadata translator from a couple weeks ago. I've pushed a fix, so if you were seeing the Cannot read property 'replace' of null error above, pull the latest translators and it should be fixed. If you're getting a 403, nothing we can do about that.

Thanks for reporting.

from translation-server.

Thanks a lot @dstillman for zotero/translators@0d435d8! Our CI builds are now back to 🟢.

pull the latest translators and it should be fixed

I did this using git submodule update --remote --merge which updated the submodules beyond the commits specified by zotero/translation-server currently for these submodules:

Submodule path 'modules/translate': merged in 'a9308c0e8632846ca2dc069a1b72db0a33f99ca6'
Submodule path 'modules/translators': merged in '0d435d8a952639d4e7489263b3a40c89377ecd31'
Submodule path 'modules/zotero-schema': merged in '97e0a8efa2cb2cf6c9853ceca334ec56180a9df0'

Is that okay, or is it best to just fast-forward modules/translators since the other two should be updated in lock-step with zotero/translation-server?

from translation-server.

BTW we actually still get some CI failures:

But since it only happened in the later jobs, I bet it's rate limiting like @dstillman mentioned. One reason we should look into caching.

If you're getting a 403, nothing we can do about that.

I'm not actually sure how to see that. In the stderr logs for translation-server, this is what the failure looks like:

2021-11-07 10:31:22,151: 
  InternalServerError: An error occurred retrieving the document
      at Object.throw (/home/translate/translation-server/node_modules/koa/lib/context.js:97:11)
      at module.exports.WebSession.handleURL (/home/translate/translation-server/src/webSession.js:219:19)
      at <anonymous>
      at process._tickDomainCallback (internal/process/next_tick.js:228:7)

@dstillman where'd you see the TypeError: Cannot read property 'replace' of null log?

from translation-server.