Code Monkey home page Code Monkey logo

Comments (15)

zedeus avatar zedeus commented on May 10, 2024 1

Alright, I've forked httpbeast and fixed the bugs. The illegal storage access segfaults should no longer happen, same with the index out of bounds errors.

from nitter.

Perflyst avatar Perflyst commented on May 10, 2024

deleted cache.db and the tmp/ folder

I am hosting also a public instance and my cache.db is 676K while tmp is 1.7G. Was really cache.db the bigger file in size on your side?

Isn't nitter caching DNS queries' result?

Why should it? This is the job of the operating system. Please install something like dnsmasq for caching.

During our tests, we've noticed that the Nitter instance started consuming all its reserved RAM (150MB)

I am not sure if I can reproduce this, even if I could 150MB is nothing.

Please modify your dockerfile and build via nimble build only without -d:release to get more debug logs.

from nitter.

Perflyst avatar Perflyst commented on May 10, 2024

Seems like the DNS queries to api.twitter.com are another bug, #56

from nitter.

avatar commented on May 10, 2024

I am hosting also a public instance and my cache.db is 676K while tmp is 1.7G. Was really cache.db the bigger file in size on your side?

We've basically removed all caching related data. The tmp folder is way bigger in size, but isn't cache.db storing data about what is stored in the tmp folder? If it isn't the case, removing cache.db makes no sense, indeed.

Why should it? This is the job of the operating system. Please install something like dnsmasq for caching.

Understood ; we have a local cache, but we forced the container to use Quad9 for testing / debugging purposes.

Please modify your dockerfile and build via nimble build only without -d:release to get more debug logs.

Debug mode enabled. We'll keep an eye on those logs and post here if we get errors.
For now we are getting a 504 Gateway Time-out from nginx for some of the requests, but no logs on Nitter about it.

from nitter.

Perflyst avatar Perflyst commented on May 10, 2024

Probably the response from Nitter takes too long (should be logged in nginx error log) which means nginx returns 504. Try to adjust the timeouts:

proxy_connect_timeout       500;
proxy_send_timeout          500;
proxy_read_timeout          500;
send_timeout                500;

from nitter.

avatar commented on May 10, 2024

Probably the response from Nitter takes too long (should be logged in nginx error log) which means nginx returns 504. Try to adjust the timeouts

Nitter shouldn't ever take more than 60 seconds to answer, unless the instance is under heavy traffic.

First segfault after 5 minutes running. Tried to access https://nitter.42l.fr/GretaThunberg 3 minutes before the actual crash.

DEBUG UPDATE profile SET username = ?, fullname = ?, location = ?, website = ?, bio = ?, userpic = ?, banner = ?, following = ?, followers = ?, tweets = ?, likes = ?, media = ?, verified = ?, protected = ?, joinDate = ?, updated = ? WHERE id = ? <- 'GretaThunberg', 'Greta Thunberg', 'Sverige', 'youtu.be/H2QxFM9y0tY', '16 year old climate and environmental activist with Asperger’s  #climatestrike #fridaysforfuture', 'https://pbs.twimg.com/profile_images/1103201826449604608/c0t5HuM-.jpg', 'https://pbs.twimg.com/profile_banners/1006419421244678144/1559499911/1500x500', '1,758', '2,730,019', '5,860', '15,826', '337', 1, 0, 1528758840, 1570367817, 84
Traceback (most recent call last)
/nitter/src/nitter.nim(28) nitter
/tmp/nimble_271/githubcom_dom96jester_0.4.3/jester.nim(482) serve
/tmp/nimble_271/githubcom_dom96httpbeast_0.2.2/src/httpbeast.nim(423) run
/tmp/nimble_271/githubcom_dom96httpbeast_0.2.2/src/httpbeast.nim(287) eventLoop
/tmp/nimble_271/githubcom_dom96httpbeast_0.2.2/src/httpbeast.nim(166) processEvents
/nim/lib/pure/asyncdispatch.nim(1562) poll
/nim/lib/pure/asyncdispatch.nim(1328) runOnce
/nim/lib/pure/asyncdispatch.nim(210) processPendingCallbacks
/nim/lib/pure/asyncmacro.nim(34) showTimelineNimAsyncContinue
/nitter/src/routes/router_utils.nim(8) showTimelineIter
/tmp/nimble_271/githubcom_dom96jester_0.4.3/jester/request.nim(48) path
/tmp/nimble_271/githubcom_dom96httpbeast_0.2.2/src/httpbeast.nim(350) path
SIGSEGV: Illegal storage access. (Attempt to read from nil?)

from nitter.

Perflyst avatar Perflyst commented on May 10, 2024

Is there anything further logged in dmesg? dmesg -T

from nitter.

zedeus avatar zedeus commented on May 10, 2024

Hi, sorry for the trouble and thank you for the detailed report. The segfault crashes are due to a bug in the web server I'm using called httpbeast. I'm switching to one which doesn't have this issue, but due to another bug it's been taking some time. The huge amount of DNS queries is due to #56, a bug I didn't notice till now.

is it possible to set a maximum cache size?

This might be added once I do a rewrite of the caching code, it's not currently possible. You could use a cron job to simply clean the directory once it reaches a certain size. It's just a cache of images to reduce amount of requests, it doesn't matter too much.

isn't cache.db storing data about what is stored in the tmp folder?

This is not currently the case, it's just a very simple file cache that checks whether it's already downloaded. With the rewrite it'll be done properly.

from nitter.

avatar commented on May 10, 2024

Is there anything further logged in dmesg? dmesg -T

Well done. Nitter gets killed by quickly reaching the container RAM limit (150MB).

[Sun Oct  6 14:45:34 2019] Task in /docker/*** killed as a result of limit of /docker/***
[Sun Oct  6 14:45:34 2019] memory: usage 153528kB, limit 153600kB, failcnt 167064
[Sun Oct  6 14:45:34 2019] memory+swap: usage 307120kB, limit 307200kB, failcnt 2
[Sun Oct  6 14:45:34 2019] kmem: usage 2248kB, limit 9007199254740988kB, failcnt 0
[Sun Oct  6 14:45:34 2019] Memory cgroup stats for /docker/***: cache:0KB rss:151484KB rss_huge:20480KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:153252KB inactive_anon:75752KB active_anon:75528KB inactive_file:0KB active_file:0KB unevictable:0KB
[Sun Oct  6 14:45:34 2019] Tasks state (memory values in pages): ***
[Sun Oct  6 14:45:34 2019] Memory cgroup out of memory: Kill process 8807 (nitter) score 1019 or sacrifice child
[Sun Oct  6 14:45:34 2019] Killed process 8807 (nitter) total-vm:363628kB, anon-rss:148580kB, file-rss:8368kB, shmem-rss:0kB
[Sun Oct  6 14:45:34 2019] oom_reaper: reaped process 8807 (nitter), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

So as far as I can guess, the segfault isn't triggerred by Nitter itself but the kernel's OOM killer.

The segfault crashes are due to a bug in the web server I'm using called httpbeast. I'm switching to one which doesn't have this issue, but due to another bug it's been taking some time.

So httpbeast is taking too much RAM for no reason, causing the segfault? Or maybe 150MB RAM is just not enough for Nitter...

Thanks for your helpful answers!

from nitter.

zedeus avatar zedeus commented on May 10, 2024

So httpbeast is taking too much RAM for no reason, causing the segfault? Or maybe 150MB RAM is just not enough for Nitter...

That should be unrelated. WIth the latest commit httpbeast is no longer used, so the path segfaults should be gone. There is a known bug with asynchttpserver crashing when requests get cancelled, for example when skipping to the end of an unbuffered video, but this should happen far less often and will get fixed.

The "tons of requests" bug may have increased RAM usage rapidly. I'm not quite sure about the realistic max RAM usage for nitter as the frequent crashes made it hard to see long-term usage. Running ~100 tests across 8 threads and playing videos simultaneously I could get it up to around 200 MB, so 150 MB isn't enough under heavy load. Increasing it to at least 250 would be a good starting point.

If you see any more index out of bounds errors please open a new issue.

from nitter.

Perflyst avatar Perflyst commented on May 10, 2024 
nitter@nitter:~/nitter$ ./nitter
INFO Jester is making jokes at http://0.0.0.0:8080
DEBUG GET /
DEBUG PRAGMA foreign_keys = ON
DEBUG SELECT id, replaceTwitter, replaceYouTube, mp4Playback, hlsPlayback, proxyVideos, muteVideos, autoplayGifs, hideTweetStats, hideBanner, stickyProfile FROM prefs WHERE id = ? <- '84'
DEBUG   200 OK {"content-type": @["text/html;charset=utf-8"]}
/home/nitter/nitter/src/nitter.nim(28) nitter
/home/nitter/.nimble/pkgs/jester-0.4.3/jester.nim(499) serve
/home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncdispatch.nim(1863) runForever
/home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncdispatch.nim(1562) poll
/home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncdispatch.nim(1328) runOnce
/home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncdispatch.nim(210) processPendingCallbacks
/home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncmacro.nim(34) sendNimAsyncContinue
/home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncnet.nim(445) sendIter
/home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncfutures.nim(374) read
[[reraised from:
/home/nitter/nitter/src/nitter.nim(28) nitter
/home/nitter/.nimble/pkgs/jester-0.4.3/jester.nim(499) serve
/home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncdispatch.nim(1863) runForever
/home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncdispatch.nim(1562) poll
/home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncdispatch.nim(1328) runOnce
/home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncdispatch.nim(210) processPendingCallbacks
/home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncfutures.nim(422) asyncCheckCallback
]]
Error: unhandled exception: Bad file descriptor
Async traceback:
  /home/nitter/nitter/src/nitter.nim(28)                                         nitter
  /home/nitter/.nimble/pkgs/jester-0.4.3/jester.nim(499)                         serve
  /home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncdispatch.nim(1863) runForever
  /home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncdispatch.nim(1562) poll
  /home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncdispatch.nim(1328) runOnce
  /home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncdispatch.nim(210)  processPendingCallbacks
  /home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncmacro.nim(34)      sendNimAsyncContinue
  /home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncnet.nim(445)       sendIter
  /home/nitter/.choosenim/toolchains/nim-0.20.2/lib/pure/asyncfutures.nim(374)   read
Exception message: Bad file descriptor
Exception type: [OSError]

now it is broken completely

from nitter.

zedeus avatar zedeus commented on May 10, 2024

Didn't happen locally without nginx but I could reproduce when running it on my server. Will have to investigate, reverted to httpbeast for now.

from nitter.

Perflyst avatar Perflyst commented on May 10, 2024

Illegal storage access occured again:

Oct 19 11:00:05 nitter nitter[12636]: DEBUG POST /enablehls
Oct 19 11:00:05 nitter nitter[12636]: DEBUG PRAGMA foreign_keys = ON
Oct 19 11:00:05 nitter nitter[12636]: DEBUG INSERT INTO prefs (replaceTwitter, replaceYouTube, mp4Playback, hlsPlayback, proxyVideos, muteVideos, autoplayGifs, hideTweetStats, hideBanner, stickyProfile) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) <- 'nitter.snopyta.org', 'invidio.us', 1, 1, 1, 0, 1, 0, 0, 1
Oct 19 11:00:05 nitter nitter[12636]: DEBUG   303 See Other {"set-cookie": @["preferences=227; Expires=Tue, 13 Oct 2020 11:00:04 GMT; Secure; HttpOnly; SameSite=Lax"], "location": @["/AbiRatchford?max_position=1167536069752975360"]}
Oct 19 11:00:05 nitter nitter[12636]: DEBUG GET /AbiRatchford
Oct 19 11:00:05 nitter nitter[12636]: DEBUG PRAGMA foreign_keys = ON
Oct 19 11:00:05 nitter nitter[12636]: DEBUG SELECT id, replaceTwitter, replaceYouTube, mp4Playback, hlsPlayback, proxyVideos, muteVideos, autoplayGifs, hideTweetStats, hideBanner, stickyProfile FROM prefs WHERE id = ? <- '227'
Oct 19 11:00:05 nitter nitter[12636]: DEBUG PRAGMA foreign_keys = ON
Oct 19 11:00:05 nitter nitter[12636]: DEBUG SELECT id, username, fullname, location, website, bio, userpic, banner, following, followers, tweets, likes, media, verified, protected, joinDate, updated FROM profile WHERE lower(username) = ? <- 'abiratchford'
Oct 19 11:00:05 nitter nitter[12636]: DEBUG PRAGMA foreign_keys = ON
Oct 19 11:00:06 nitter nitter[12636]: DEBUG SELECT id, videoId, contentId, duTraceback (most recent call last)
Oct 19 11:00:06 nitter nitter[12636]: /home/nitter/nitter/src/nitter.nim(28) nitter
Oct 19 11:00:06 nitter nitter[12636]: /home/nitter/.nimble/pkgs/jester-0.4.3/jester.nim(482) serve
Oct 19 11:00:06 nitter nitter[12636]: /home/nitter/.nimble/pkgs/httpbeast-0.2.2/httpbeast.nim(423) run
Oct 19 11:00:06 nitter nitter[12636]: /home/nitter/.nimble/pkgs/httpbeast-0.2.2/httpbeast.nim(287) eventLoop
Oct 19 11:00:06 nitter nitter[12636]: /home/nitter/.nimble/pkgs/httpbeast-0.2.2/httpbeast.nim(166) processEvents
Oct 19 11:00:06 nitter nitter[12636]: /home/nitter/.choosenim/toolchains/nim-1.0.0/lib/pure/asyncdispatch.nim(1569) poll
Oct 19 11:00:06 nitter nitter[12636]: /home/nitter/.choosenim/toolchains/nim-1.0.0/lib/pure/asyncdispatch.nim(1335) runOnce
Oct 19 11:00:06 nitter nitter[12636]: /home/nitter/.choosenim/toolchains/nim-1.0.0/lib/pure/asyncdispatch.nim(210) processPendingCallbacks
Oct 19 11:00:06 nitter nitter[12636]: /home/nitter/.choosenim/toolchains/nim-1.0.0/lib/pure/asyncmacro.nim(34) showTimelineNimAsyncContinue
Oct 19 11:00:06 nitter nitter[12636]: /home/nitter/nitter/src/routes/router_utils.nim(8) showTimelineIter
Oct 19 11:00:06 nitter nitter[12636]: /home/nitter/.nimble/pkgs/jester-0.4.3/jester/request.nim(48) path
Oct 19 11:00:06 nitter nitter[12636]: /home/nitter/.nimble/pkgs/httpbeast-0.2.2/httpbeast.nim(350) path
Oct 19 11:00:06 nitter nitter[12636]: SIGSEGV: Illegal storage access. (Attempt to read from nil?)
Oct 19 11:00:06 nitter nitter[12636]: rationMs, url, thumb, views, available, reason, playbackType FROM video WHERE videoId = ? <- 1165376110227410944

from nitter.

zedeus avatar zedeus commented on May 10, 2024

It's not using my fork for some reason, try this:
nimble install https://github.com/zedeus/httpbeast@"#fix-crash"

from nitter.

Perflyst avatar Perflyst commented on May 10, 2024

On normal build

Info: Dependency on https://github.com/zedeus/httpbeast@#fix-crash already satisfied
Verifying dependencies for httpbeast@#fix-crash

With nimble install:
Prompt: httpbeast@#fix-crash already exists.
I selected to overwrite it, lets see if that fixed it.

from nitter.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.