OS Version: Mac OS Catalina
Go Version: go1.16.4 darwin/amd64

Hello, following https://docs.solo.io/gloo-mesh/latest/setup/istio_fips/ to verify fips and after running go get github.com/rsc/goversion:

running into

➜ goversion -crypto /tmp/pilot-discovery                     

/tmp/pilot-discovery unknown Go version

I'd like to better understand how the module info is inserted by the linker and how it is found by this tool.

What I understand so far: The binary is read from disk, format-specific (ELF, Mach-O, PE) libraries are used to extract certain sections (e.g. the read-only data), a magic sequence of bytes is searched for indicating the start and end of the module information.

How I've tried to understand it:

• Spent a couple hours doing a shallow read of cmd/link, mostly cmd/link/internal/ld and cmd/link/internal/sym. I grokked that there's a Moduledata symbol for each object file. Couldn't find where module info was written to the final binary though. Most of the module-related code I found seemed related to shared libraries (I only care about static binaries for now).
• Disassembled some go binaries and used nm/objdump/od to poke around at the symbol table and try to discover more info. Was able to find the go runtime version by referencing the _runtime.buildVersion in the symbol table (mostly thanks to this article). I found a few promising symbols (runtime.findmoduledatap, runtime.firstmoduledata, runtime.lastmoduledatap). I even succeeded in finding portions of the module info text (i.e. the h1 hash).
• Searched for the magic module info start/stop values (defined here) by converting the binary to hex and searching (using both little- and big-endian).

I'm clearly out of my depth here, but I'd like to understand how this works, specifically:

• Where in the linker is this module info inserted?
• Where are the magic values coming from?

This is all in the context of doing security audits on dependencies (go list -m all leaves a lot to be desired for this use case). I'll feel more comfortable if I can explain how this works better :)

goversion fails to find modules table under Windows 64-bit: goversion searches it in .text while it is in .rdata.

I don't know if this is due to the way GO111MODULE=on go build works, but when I run version.ReadExe(os.Args[0]) from inside of a Go test, I do not get a module list. It does find the Version.Release, but Version.ModuleInfo is empty.

I am trying to use gops which uses goversion. I stepped through the code all the way to here:
https://github.com/rsc/goversion/blob/master/version/read.go#L142

and I get back err "address not mapped". isGo is true here:
https://github.com/rsc/goversion/blob/master/version/read.go#L52

Someone on my team uses the GoLand IDE on MacOS, which uses github.com/google/gops to list processes to attach a debugger to, which in turn uses goversion to identify which processes are go programs. They recently ran into an issue where GoLand no longer shows our own builds as a process to attach to, and determined that it was due to an incompatibility with goversion.

The incompatibility only exists with some of our builds. On these files we get errors like this:

implausible string size 13669722811215734528 for runtime.buildVersion

I stepped through this issue with a debugger, and it turns out some (but not all) of our builds include a "symbolic debugging entry" for _runtime.buildVersion, which happens to be found earlier in the file than the symbol entry which actually points to its value.

I don't know much about Mach-O, but I was looking at this include file:

https://opensource.apple.com/source/xnu/xnu-201/EXTERNAL_HEADERS/mach-o/nlist.h

The entry that confuses goversion has a type of 0x20 and a value of 0. According to nlist.h:

/*
 * The n_type field really contains three fields:
 *	unsigned char N_STAB:3,
 *		      N_PEXT:1,
 *		      N_TYPE:3,
 *		      N_EXT:1;
 * which are used via the following masks.
 */
#define	N_STAB	0xe0  /* if any of these bits set, a symbolic debugging entry */
#define	N_PEXT	0x10  /* private external symbol bit */
#define	N_TYPE	0x0e  /* mask for the type bits */
#define	N_EXT	0x01  /* external symbol bit, set for external symbols */

/*
 * Only symbolic debugging entries have some of the N_STAB bits set and if any
 * of these bits are set then it is a symbolic debugging entry (a stab).  In
 * which case then the values of the n_type field (the entire field) are given
 * in <mach-o/stab.h>
 */

Note that the value of 32 is one of those N_STAB bits.

If I modify goversion to ignore symbols where (type & 0xe0) != 0, then goversion can read the version of our builds again (I will follow up with a PR for this shortly).

Unfortunately I have not been able to come up with a way to reproduce these builds (without sharing code that I do not have permission to share). The only hints I have are that it only started happening after we migrated our code to a go module and/or after we upgraded from go 1.13 to 1.14 (1.14.1, to be precise). I've built a few targets from our repo, and it seems like only the relatively larger builds have these symbols, but I have too few data points so far to conclude that it has anything to do with size or complexity.

With the latest changes it is able to detect the version of most binaries, I found some where it didn't work:
One is containerd, the other one where it fails to detect is runc. This might be related to -buildmode=pie?

Golang version: 1.10.1 (fails with 1.10 as well, but works with 1.9.4)
Steps to reproduce:

go get github.com/containerd/containerd
cd ~/go/src/github.com/containerd/containerd
git co v1.0.3
make
goversion bin/containerd

goversion bin/containerd    
bin/containerd unknown Go version

goversion -d bin/containerd
data @0xba81b0: 31ed4989d15e4889e24883e4f050544c
data: 31ed4989d15e4889e24883e4f050544c8d05cad5bf00488d0d53d5bf00488d3d
no match (0) 0xba81b0 [e9 1ff00 ff00 ff00 2ff00 cc cc cc] 31ed4989d15e4889e24883e4f050544c8d05cad5bf00488d0d53d5bf00488d3d
no match (1) 0xba81b0 [48 8d 74 24 8 48 8b 3c 24 48 8d 5 1ff00 ff00 ff00 2ff00 ff e0] 31ed4989d15e4889e24883e4f050544c8d05cad5bf00488d0d53d5bf00488d3d
no match (2) 0xba81b0 [48 8d 74 24 8 48 8b 3c 24 b8 1ff00 ff00 ff00 ff00 ff e0] 31ed4989d15e4889e24883e4f050544c8d05cad5bf00488d0d53d5bf00488d3d
no match (3) 0xba81b0 [48 8b 3c 24 48 8d 74 24 8 e9 1ff00 ff00 ff00 2ff00 cc cc] 31ed4989d15e4889e24883e4f050544c8d05cad5bf00488d0d53d5bf00488d3d
match (4) 0xba81b0+15 [4c 8d 5 ff00 ff00 ff00 ff00 48 8d d ff00 ff00 ff00 ff00 48 8d 3d 1ff00 ff00 ff00 2ff00 ff 15] 4c8d05cad5bf00488d0d53d5bf00488d3d2c8c0500ff15
reload @0xc00e00: e90b000000cccccccccccccccccccccc4889f84889f34883ec274883e4f04889
no match (5) 0xc00e00 [54 49 c7 c0 ff00 ff00 ff00 ff00 48 c7 c1 ff00 ff00 ff00 ff00 48 c7 c7 1ff00 ff00 ff00 ff00] e90b000000cccccccccccccccccccccc4889f84889f34883ec274883e4f04889
no match (6) 0xc00e00 [48 8d 5 1ff00 ff00 ff00 2ff00 ff e0] e90b000000cccccccccccccccccccccc4889f84889f34883ec274883e4f04889
no match (7) 0xc00e00 [b8 1ff00 ff00 ff00 ff00 ff e0] e90b000000cccccccccccccccccccccc4889f84889f34883ec274883e4f04889
no match (8) 0xc00e00 [48 89 f8 48 89 f3 48 83 ec 27 48 83 e4 f0 48 89 44 24 10 48 89 5c 24 18] e90b000000cccccccccccccccccccccc4889f84889f34883ec274883e4f04889
bin/containerd unknown Go version

Using the latest goversion, and a binary cross-compiled from MacOS using "GOOS=freebsd GOARCH=amd64 GO111MODULE=on go build", I get this:

icsil1noteb238:conode jallen$ goversion -d ./conode-v3.0.0-check/conode.FreeBSD.amd64 
data @0x458680: 488d7708488b3fe9d4c6ffffcccccccc
data: 488d7708488b3fe9d4c6ffffcccccccc488b7c24088b7424108b5424144c8b54
no match (0) 0x458680 [e9 1ff00 ff00 ff00 2ff00 cc cc cc] 488d7708488b3fe9d4c6ffffcccccccc488b7c24088b7424108b5424144c8b54
no match (1) 0x458680 [48 8d 74 24 8 48 8b 3c 24 48 8d 5 1ff00 ff00 ff00 2ff00 ff e0] 488d7708488b3fe9d4c6ffffcccccccc488b7c24088b7424108b5424144c8b54
no match (2) 0x458680 [48 8d 74 24 8 48 8b 3c 24 b8 1ff00 ff00 ff00 ff00 ff e0] 488d7708488b3fe9d4c6ffffcccccccc488b7c24088b7424108b5424144c8b54
no match (3) 0x458680 [48 8b 3c 24 48 8d 74 24 8 e9 1ff00 ff00 ff00 2ff00 cc cc] 488d7708488b3fe9d4c6ffffcccccccc488b7c24088b7424108b5424144c8b54
no match (4) 0x458680 [4c 8d 5 ff00 ff00 ff00 ff00 48 8d d ff00 ff00 ff00 ff00 48 8d 3d 1ff00 ff00 ff00 2ff00 ff 15] 488d7708488b3fe9d4c6ffffcccccccc488b7c24088b7424108b5424144c8b54
no match (5) 0x458680 [54 49 c7 c0 ff00 ff00 ff00 ff00 48 c7 c1 ff00 ff00 ff00 ff00 48 c7 c7 1ff00 ff00 ff00 ff00] 488d7708488b3fe9d4c6ffffcccccccc488b7c24088b7424108b5424144c8b54
no match (6) 0x458680 [48 8d 5 1ff00 ff00 ff00 2ff00 ff e0] 488d7708488b3fe9d4c6ffffcccccccc488b7c24088b7424108b5424144c8b54
no match (7) 0x458680 [b8 1ff00 ff00 ff00 ff00 ff e0] 488d7708488b3fe9d4c6ffffcccccccc488b7c24088b7424108b5424144c8b54
no match (8) 0x458680 [e9 1ff00 ff00 ff00 2ff00 cc cc cc] 488d7708488b3fe9d4c6ffffcccccccc488b7c24088b7424108b5424144c8b54
no match (9) 0x458680 [48 89 f8 48 89 f3 48 83 ec 27 48 83 e4 f0 48 89 44 24 10 48 89 5c 24 18] 488d7708488b3fe9d4c6ffffcccccccc488b7c24088b7424108b5424144c8b54
./conode-v3.0.0-check/conode.FreeBSD.amd64 unknown Go version

Same problem with linux/arm:

icsil1noteb238:conode jallen$ goversion ./conode-v3.0.0-check/conode*
./conode-v3.0.0-check/conode.Darwin.x86_64 go1.11.5
./conode-v3.0.0-check/conode.FreeBSD.amd64 unknown Go version
./conode-v3.0.0-check/conode.Linux.armv7l unknown Go version
./conode-v3.0.0-check/conode.Linux.x86_64 go1.11.5
./conode-v3.0.0-check/conode.exe go1.11.5

go 1.16 crypto/internal/boring/sig.FIPSOnly
go 1.17 crypto/internal/boring/sig.FIPSOnly.abi0

This check does not pass when fipsonly is set
https://github.com/rsc/goversion/blob/master/version/read.go#L63

rsc.io/[email protected] fails to extract the module info in this scenario:

$ KURED_VERSION=1.9.1
$ GOBIN=$PWD
$ go install -ldflags "-X main.version=${KURED_VERSION}" \
    "github.com/weaveworks/kured/cmd/kured@${KURED_VERSION}"
$ goversion -m kured
kured go1.16.13
$

Obviously the built-in go version cmd in 1.16.13 can successfully extract the module info:

$ go version -m kured  | head -5
kured: go1.16.13
        path    github.com/weaveworks/kured/cmd/kured
        mod     github.com/weaveworks/kured     v0.0.0-20220106181305-2b36eab0f829      h1:gkiQx0LbROxAqRgcqLqO8dnKWWmbuHpOwwbOl9Numcc=
        dep     github.com/MakeNowJust/heredoc  v0.0.0-20170808103936-bb23615498cd      h1:sjQovDkwrZp8u+gxLtPgKGjk5hCxuy2hrRejBTA9xFU=
        dep     github.com/PuerkitoBio/purell   v1.1.1  h1:WEQqlqaGbrPkxLJWfBwQmfEAE1Z7ONdDLqrN38tNFfI=

And similarly if I use Filo Sottile's version that rewrites atop the internal/version pkg (#17) then that can also read it:

$ goversion -m kured  | head -5
kured go1.16.13
        path  github.com/weaveworks/kured/cmd/kured
        mod   github.com/weaveworks/kured                       v0.0.0-20220106181305-2b36eab0f829
        dep   github.com/MakeNowJust/heredoc                    v0.0.0-20170808103936-bb23615498cd
        dep   github.com/PuerkitoBio/purell                     v1.1.1

This worked on go1.13:

docker run -it --entrypoint=bash golang:1.13
$ go get -d rsc.io/goversion
$ go install -ldflags="-s -w -buildid=" rsc.io/goversion
$ goversion `which goversion`

/go/bin/goversion go1.13.15

But on 1.14+, the same steps result in an error about there being no symbol section:

docker run -it --entrypoint=bash golang:1.14
$ go get -d rsc.io/goversion
$ go install -ldflags="-s -w -buildid=" rsc.io/goversion
$ goversion `which goversion`

/go/bin/goversion: no symbol section

On 1.15:

docker run -it --entrypoint=bash golang:1.15
$ go get -d rsc.io/goversion
$ go install -ldflags="-s -w -buildid=" rsc.io/goversion
$ goversion `which goversion`

/go/bin/goversion: no symbol section

The application doesn't correctly identify executable files on "windows".

info.Mode()&0111 == 0 is always true on "windows" machines, even if the file is actually executable.

I've opened a PR (goversion: improve portability), using GOEXE and file extension comparison, to fix the problem. I hope it's helpful.

The most recent tag for goversion package is 1.2.0 created on Apr 04, 2018(years ago). Can the latest master branch be tagged with a new tag?

I am trying to call "go run github.com/rsc/goversion -crypto /tmp/myfile" I get:
/tmp/myfile unknown Go version. This file has been compiled recently.

When I use an older version this /tmp/myfile it works.

If I run "go version" I get: go version go1.16.4 linux/amd64, (Ubuntu 7.4.0 and Centos), this seems to the latest version.

Can you please check.

I was trying to use goversion as a library in a vgo-managed project, and it chooses v1.0.0, which does not have the Version.ModuleInfo, which is exactly what I wanted to use it for.

goversion compiled with go-1.10

goversion -d /usr/bin/goversion                                                                                                                                                                                           :(
no match (0) 0x453a60 [48 8d 74 24 8 48 8b 3c 24 48 8d 5 1ff00 ff00 ff00 2ff00 ff e0] e96bc2ffffcccccccccccccccccccccc8b7c2408b8e70000000f05c3cccccccc
no match (1) 0x453a60 [48 8d 74 24 8 48 8b 3c 24 b8 1ff00 ff00 ff00 ff00 ff e0] e96bc2ffffcccccccccccccccccccccc8b7c2408b8e70000000f05c3cccccccc
no match (2) 0x453a60 [4c 8d 5 ff00 ff00 ff00 ff00 48 8d d ff00 ff00 ff00 ff00 48 8d 3d 1ff00 ff00 ff00 2ff00 ff 15] e96bc2ffffcccccccccccccccccccccc8b7c2408b8e70000000f05c3cccccccc
no match (3) 0x453a60 [54 49 c7 c0 ff00 ff00 ff00 ff00 48 c7 c1 ff00 ff00 ff00 ff00 48 c7 c7 1ff00 ff00 ff00 ff00] e96bc2ffffcccccccccccccccccccccc8b7c2408b8e70000000f05c3cccccccc
no match (4) 0x453a60 [48 8d 5 1ff00 ff00 ff00 2ff00 ff e0] e96bc2ffffcccccccccccccccccccccc8b7c2408b8e70000000f05c3cccccccc
no match (5) 0x453a60 [b8 1ff00 ff00 ff00 ff00 ff e0] e96bc2ffffcccccccccccccccccccccc8b7c2408b8e70000000f05c3cccccccc
no match (6) 0x453a60 [48 89 f8 48 89 f3 48 83 ec 27 48 83 e4 f0 48 89 44 24 10 48 89 5c 24 18] e96bc2ffffcccccccccccccccccccccc8b7c2408b8e70000000f05c3cccccccc
/usr/bin/goversion unknown Go version

with latest master. It works with go-1.9 and previous version of golang