Turbo daemon uses 100% CPU even when no tasks are running about turbo HOT 9 OPEN

Thanks for the very thorough explanation! This should be a pretty easy fix. We currently filter out change events for the root .git folder, but we can probably extend that to be any nested .git folder too.

from turbo.

Hi @AdiRishi, thanks for the issue. Could you share the output of turbo daemon logs? You can also access the log file directly by running turbo daemon status and going to the log file path. If you're not comfortable sharing it here, you can also send it to me at [email protected]

from turbo.

Logs seem pretty empty 🙃

from turbo.

Hmm very interesting. How large of a repository are you running inside of? And do you see any logs after a significant amount of time, say 10 minutes?

from turbo.

Hmm very interesting. How large of a repository are you running inside of? And do you see any logs after a significant amount of time, say 10 minutes?

I'd say it's a mid-size repository. Around 21 sub-projects in total, it's a mix of around 6 webapps, 7 cloudflare workers and then more utility libaries / configs etc. Nothing crazy.

I'll run some further debugging and get the information you want. I'll also try to continue to re-clone and see if I can reporduce the issues on other systems. I'll get back to you on this.

from turbo.

Gotcha. Any chance you could run the daemon directly by stopping it (turbo daemon stop) then doing turbo daemon -vvv? This will run it in full verbosity. Hopefully that should give a better idea of where the daemon is stalling.

from turbo.

Alright, I have some very interesting discoveries to go through.

First off I want to start with the fact that when I re-clone this repository into a different location, and run the daemon from it, this behavior does not occur.

Next I tried to run turbo daemon -vvv on a different turborepo repository which doesn't exhibit this issue. Here was the output, seems fairly normal. The logs stopped after a few seconds.
arishi-monorepo-daemon-logs.txt

I then ran turbo daemon -vvv on the problem repository, and the logs wouldn't stop. I've captured around 1 minute of logs in this file. The full logfile is around 25MB so I had to gzip it 😅
bad-monorepo-daemon-logs.txt.gz

I've captured both logfiles using a command like this on mac pnpm turbo daemon -vvv &> daemon-logs.txt.

Root Cause of Bug

Looking through the bad logs I realised there were mentions of a .git folder in workers/turborepo-remote-cache. This was confusing since I didn't think I had git submodules. I went into the directory, and sure enough, there is an inner git repository here with unstaged changes 🙃 . I think around 1 month ago I was updating my local copy of this worker and I accidentally left the git repository cloned and forgot to remove the .git folder.
So it seems like having this unstaged change causes whatever the turbo daemon is doing to spin in an infinite loop.

I confirmed this by removing the unstaged changes and deleting the .git folder in the worker/turborepo-remote-cache folder and everything is back to normal 🎉

Still, a very very odd manifestation, definitely does indicate a subtle bug in turbo haha.
I'm happy to help with more debugging if it will be helpful to fix the underlying bug :)

from turbo.