Comments (9)
emdioh
commented on September 17, 2024
which commit did you use for the build?
What happens before the crash? do you have some logs or other info?
…On Sun, Oct 14, 2018 at 7:09 AM Brandon Clifford ***@***.***> wrote:
Haven't been able to reproduce this on my own, but the same stack happens
occasionally during a DoTCP call.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x9abe1b]
goroutine 14743 [running]:github.com/miekg/dns.(*SessionUDP).RemoteAddr(...)
***@***.***/udp.go:36github.com/miekg/dns.(*response).RemoteAddr(0xc00010d100, 0xc0001a2c00, 0xc00004ae72)
***@***.***/server.go:773 +0x5b
main.(*DNSHandler).do(0xc0001bc660, 0xb8e740, 0x3, 0xc675a0, 0xc00010d100, 0xc0002d22d0)
/go/grimd/handler.go:75 +0x2801
created by main.(*DNSHandler).DoTCP
/go/grimd/handler.go:237 +0x72
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#47>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/ACw8mGkt8BHQPFjQUX_jcGGnJcJeuplnks5ukscdgaJpZM4Xa9rZ>
.
from grimd.
brandon099
commented on September 17, 2024
Good point -- I am not running master currently because of the issue you're fixing in #46 so I'm running your race branch, so I could have put it there, but the stack is only showing code not modified by recent PR's so I thought it might be an original bug. I didn't have debug logging on previously, so I've turned it on now. I'll post back when I've got some additional logging, because nothing is getting logged before the panic.
from grimd.
brandon099
commented on September 17, 2024
Here are some debug logs.
- There's a look up for
ichnaea.netflix.comand is found in the cache under the key16388b3f593f450c0ad4cb124eaf7e3b. - Grimd expires the cache entry and the key is no longer in the cache.
- It then stores a new entry for
ichnaea.netflix.comwith TTL of 1 second. - Then, when it's looked up again, the cache returns the old key, instead of the new one, causing the panic.
Hope this helps! Let me know if additional information is needed.
20:28:19.220 INFO handler.go:80 ▶ 12bba 10.0.1.28 lookup ichnaea.netflix.com IN A
20:28:19.220 DEBU cache.go:205 ▶ 12bbb KeyGen: ichnaea.netflix.com IN A 16388b3f593f450c0ad4cb124eaf7e3b
20:28:19.220 DEBU cache.go:120 ▶ 12bbc Cache: Key expired 16388b3f593f450c0ad4cb124eaf7e3b
20:28:19.221 DEBU cache.go:120 ▶ 12bbd Cache: Key expired 16388b3f593f450c0ad4cb124eaf7e3b
20:28:19.221 DEBU cache.go:120 ▶ 12bbe Cache: Key expired 16388b3f593f450c0ad4cb124eaf7e3b
20:28:19.221 DEBU cache.go:120 ▶ 12bbf Cache: Key expired 16388b3f593f450c0ad4cb124eaf7e3b
20:28:19.221 DEBU cache.go:120 ▶ 12bc0 Cache: Key expired 16388b3f593f450c0ad4cb124eaf7e3b
20:28:19.221 DEBU cache.go:120 ▶ 12bc1 Cache: Key expired 16388b3f593f450c0ad4cb124eaf7e3b
20:28:19.221 DEBU cache.go:120 ▶ 12bc2 Cache: Key expired 16388b3f593f450c0ad4cb124eaf7e3b
20:28:19.221 DEBU cache.go:120 ▶ 12bc3 Cache: Key expired 16388b3f593f450c0ad4cb124eaf7e3b
20:28:19.222 DEBU cache.go:120 ▶ 12bc4 Cache: Key expired 16388b3f593f450c0ad4cb124eaf7e3b
20:28:19.222 DEBU cache.go:120 ▶ 12bc5 Cache: Key expired 16388b3f593f450c0ad4cb124eaf7e3b
20:28:19.222 DEBU cache.go:130 ▶ 12bc6 Cache: Cannot find key 16388b3f593f450c0ad4cb124eaf7e3b
20:28:19.222 DEBU handler.go:102 ▶ 12bc7 ichnaea.netflix.com IN A didn't hit cache
20:28:19.222 DEBU handler.go:172 ▶ 12bc8 ichnaea.netflix.com not found in blocklist
20:28:19.339 DEBU resolver.go:57 ▶ 12bc9 ichnaea.netflix.com resolv on 127.0.0.1:5353 (udp)
20:28:19.340 DEBU handler.go:211 ▶ 12bca Answer 0 - ichnaea.netflix.com. 711 IN CNAME ichnaea.geo.netflix.com.
20:28:19.340 DEBU handler.go:211 ▶ 12bcb Answer 1 - ichnaea.geo.netflix.com. 711 IN CNAME ichnaea.us-west-2.prodaa.netflix.com.
20:28:19.340 DEBU handler.go:211 ▶ 12bcc Answer 2 - ichnaea.us-west-2.prodaa.netflix.com. 1 IN A 54.186.225.211
20:28:19.341 DEBU handler.go:211 ▶ 12bcd Answer 3 - ichnaea.us-west-2.prodaa.netflix.com. 1 IN A 35.163.6.39
20:28:19.341 DEBU handler.go:211 ▶ 12bce Answer 4 - ichnaea.us-west-2.prodaa.netflix.com. 1 IN A 35.164.67.16
20:28:19.341 DEBU handler.go:211 ▶ 12bcf Answer 5 - ichnaea.us-west-2.prodaa.netflix.com. 1 IN A 52.34.133.109
20:28:19.341 DEBU handler.go:211 ▶ 12bd0 Answer 6 - ichnaea.us-west-2.prodaa.netflix.com. 1 IN A 52.34.255.169
20:28:19.342 DEBU handler.go:211 ▶ 12bd1 Answer 7 - ichnaea.us-west-2.prodaa.netflix.com. 1 IN A 52.41.75.150
20:28:19.342 DEBU handler.go:211 ▶ 12bd2 Answer 8 - ichnaea.us-west-2.prodaa.netflix.com. 1 IN A 54.148.89.232
20:28:19.342 DEBU handler.go:211 ▶ 12bd3 Answer 9 - ichnaea.us-west-2.prodaa.netflix.com. 1 IN A 54.148.125.37
20:28:19.342 DEBU handler.go:230 ▶ 12bd4 insert ichnaea.netflix.com IN A into cache with ttl 1
20:28:19.343 DEBU resolver.go:57 ▶ 12bd5 api-global.netflix.com resolv on 127.0.0.1:5353 (udp)
20:28:19.343 DEBU handler.go:211 ▶ 12bd6 Answer 0 - api-global.netflix.com. 703 IN CNAME api-global.geo.netflix.com.
20:28:19.343 DEBU handler.go:211 ▶ 12bd7 Answer 1 - api-global.geo.netflix.com. 156 IN CNAME api-global.us-west-2.prodaa.netflix.com.
20:28:19.344 DEBU handler.go:211 ▶ 12bd8 Answer 2 - api-global.us-west-2.prodaa.netflix.com. 31 IN A 52.34.187.217
20:28:19.344 DEBU handler.go:211 ▶ 12bd9 Answer 3 - api-global.us-west-2.prodaa.netflix.com. 31 IN A 52.34.218.190
20:28:19.344 DEBU handler.go:211 ▶ 12bda Answer 4 - api-global.us-west-2.prodaa.netflix.com. 31 IN A 52.35.117.77
20:28:19.345 DEBU handler.go:211 ▶ 12bdb Answer 5 - api-global.us-west-2.prodaa.netflix.com. 31 IN A 52.35.167.163
20:28:19.345 DEBU handler.go:211 ▶ 12bdc Answer 6 - api-global.us-west-2.prodaa.netflix.com. 31 IN A 52.27.18.203
20:28:19.345 DEBU handler.go:211 ▶ 12bdd Answer 7 - api-global.us-west-2.prodaa.netflix.com. 31 IN A 52.32.62.213
20:28:19.345 DEBU handler.go:211 ▶ 12bde Answer 8 - api-global.us-west-2.prodaa.netflix.com. 31 IN A 52.33.176.57
20:28:19.346 DEBU handler.go:211 ▶ 12bdf Answer 9 - api-global.us-west-2.prodaa.netflix.com. 31 IN A 52.34.76.132
20:28:19.346 DEBU handler.go:230 ▶ 12be0 insert api-global.netflix.com IN A into cache with ttl 31
20:28:19.346 INFO handler.go:80 ▶ 12be1 10.0.1.28 lookup ichnaea.netflix.com IN A
20:28:19.347 DEBU cache.go:205 ▶ 12be2 KeyGen: ichnaea.netflix.com IN A 16388b3f593f450c0ad4cb124eaf7e3b
20:28:19.347 DEBU handler.go:112 ▶ 12be3 ichnaea.netflix.com IN A hit cache
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x9abe1b]
goroutine 21942 [running]:
github.com/miekg/dns.(*SessionUDP).RemoteAddr(...)
/go/pkg/mod/github.com/miekg/[email protected]/udp.go:36
github.com/miekg/dns.(*response).RemoteAddr(0xc0000d9a80, 0xc00019b380, 0xc000350e72)
/go/pkg/mod/github.com/miekg/[email protected]/server.go:773 +0x5b
main.(*DNSHandler).do(0xc00019bbf0, 0xb8e740, 0x3, 0xc675a0, 0xc0000d9a80, 0xc0009d7e60)
/go/grimd/handler.go:75 +0x2801
created by main.(*DNSHandler).DoTCP
/go/grimd/handler.go:237 +0x72
from grimd.
emdioh
commented on September 17, 2024
Thanks, this was helpful.
I'll work on a repro test (and hopefully on a fix :) tonight.
from grimd.
emdioh
commented on September 17, 2024
I don't think that the cache is the issue here. The key is always the same, derived from the queried domain name so it will always return the same domain.
From the stack trace things seem to happen at the beginning of the query handling, when recovering the DNS ip address for logging and caching, on line 75.
There is something strange happens, because it looks like the RemoteAddr at go/pkg/mod/github.com/miekg/[email protected]/server.go:773 is calling w.udpSession when we have a TCP session, and UDP crashes.
One thing that it's strange for me is that my lines for the same calls in miekg/[email protected] are different: at line 773 I have LocalAddr(), not RemoteAddr().
Do you mind enabling the core dumps and sharing a dump?
from grimd.
brandon099
commented on September 17, 2024
How do I do that? I'm running it like this: GOTRACEBACK=crash ./grimd and there aren't any core dumps being generated. Also, my ulimit is set to unlimited. I got a core dump of a running instance but it was 1.2gb and didn't have any backtrace or crash info. Must be doing something wrong, but would love to supply a core dump if I can. Thoughts?
from grimd.
emdioh
commented on September 17, 2024
I just tried by injecting a nil reference and I git a core (and a way longer backtrace).
It printed 'Aborted (core dumped)' but with no core until I ran 'ulimit -c unlimited' from the shell were I ran grimd (before starting it of course)
from grimd.
photostorm
commented on September 17, 2024
Have you tried v1.0.14 release? It looks like it used to call w.udpSession.RemoteAddr() when w.tcp is nil. I seen that flip to nil when the connection get interrupt during your request. It looks like now it just call a panic which it should recover from.
The change made in v1.0.14:
miekg/dns@008c8ca#diff-34c6b408d72845d076d47126c29948d1
from grimd.
brandon099
commented on September 17, 2024
Yes, it does look like that change to miekg/[email protected] has done the trick for this specific issue. I was trying to figure out how it suddenly got better as I've been trying to reproduce this issue. There does seem to be another issue that crashes grimd even less frequently, but it appears to be a different issue, so I'll troubleshoot that a bit more, and open a new issue if needed. Thanks for your help @emdioh and @photostorm
from grimd.
Related Issues (20)
- Update Go 1.19 and fix current CI issues HOT 1
- .
- Dashboard isn't working both on reverse proxy and direct ip access
- No recursion/flattening for CNAME resolution for custom records
- Only single custom record returned per host HOT 1
- panic: send on closed channel HOT 3
- Custom dns entries HOT 1
- Grimd + dnsmasq backend HOT 6
- "DNS over HTTPS" support for Android 9 Private DNS HOT 2
- Race error HOT 1
- nameserver tls upstream
- Client list api to view all the clients using DNS server HOT 2
- Can I define custom domains on my lan using grimd? HOT 1
- Block via http api does not persist HOT 3
- Security vulnerability: gin-gonic/gin: v1.7.0 HOT 2
- Whether the compile parameter should enable `CGO` HOT 2
- [Improve] Update project ignore list and document configuration HOT 1
- [Fix] allow the project to complete the build in the docker HOT 5
- [Feat] Automatically replace the default listening address in the docker with `0.0.0.0` HOT 1
- [Feat] Built-in management interface HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from grimd.