I have downloaded gau, with the command specified in the installation GO111MODULE=on go get -u -v github.com/lc/gau . But I am not able to locate where is my gau saved. As I downloaded it in Desktop once, once in home. Tried to run from both locations but it says gau: No such file or directory. And on giving command like ./gau it says ./gau: No such file or directory. Any solution please.

runtime: g1: frame.sp=0xc0001dd5c0 top=0xc0001ddfe0
stack=[0xc0001dc000-0xc0001de000] n=7 max=2147483647
fatal error: traceback did not unwind completely

runtime stack:
runtime.throw(0x858e48, 0x23)
/usr/lib/go/src/runtime/panic.go:1116 +0x72
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000000180, 0x0, 0x0, 0x7fffffff, 0x867c90, 0x7fff0dc14ef8, 0x0, ...)
/usr/lib/go/src/runtime/traceback.go:559 +0x1a8a
runtime.copystack(0xc000000180, 0x2000)
/usr/lib/go/src/runtime/stack.go:910 +0x287
runtime.newstack()
/usr/lib/go/src/runtime/stack.go:1076 +0x28d
runtime.morestack()
/usr/lib/go/src/runtime/asm_amd64.s:449 +0x8f

goroutine 1 [copystack]:
runtime.heapBitsSetType(0xc000011590, 0x8, 0x8, 0x7c27a0)
/usr/lib/go/src/runtime/mbitmap.go:911 +0xaa5 fp=0xc0001dd3e0 sp=0xc0001dd3d8 pc=0x418c65
runtime.mallocgc(0x8, 0x7c27a0, 0x1, 0x400000900)
/usr/lib/go/src/runtime/malloc.go:1090 +0x5a5 fp=0xc0001dd480 sp=0xc0001dd3e0 pc=0x40f2e5
runtime.growslice(0x7c27a0, 0x0, 0x0, 0x0, 0x1, 0xc0000721e0, 0x0, 0xb0fec0)
/usr/lib/go/src/runtime/slice.go:230 +0x1e9 fp=0xc0001dd4e8 sp=0xc0001dd480 pc=0x44dee9
runtime.doaddtimer(0xc00002e000, 0xc0000721e8)
/usr/lib/go/src/runtime/time.go:280 +0x131 fp=0xc0001dd540 sp=0xc0001dd4e8 pc=0x455f71
runtime.addtimer(0xc0000721e8)
/usr/lib/go/src/runtime/time.go:260 +0xad fp=0xc0001dd578 sp=0xc0001dd540 pc=0x455ded
time.startTimer(0xc0000721e8)
/usr/lib/go/src/runtime/time.go:207 +0x2b fp=0xc0001dd590 sp=0xc0001dd578 pc=0x468d6b
time.AfterFunc(0x37e11b553, 0xc000053400, 0xb0f9e0)
/usr/lib/go/src/time/sleep.go:162 +0x91 fp=0xc0001dd5c0 sp=0xc0001dd590 pc=0x4d7e51
what can i do to slove that

In the current version of gau, using the --o option will output to the specified outfile only, suppressing results output to stdout.

I think it might be nice to have --o operate independently without silencing stdout.

The would be useful in certain terminal logging setups. My existing 'workaround' had been omit --o and instead tee to an outfile.

Thanks!

Hey there,
I really love this tool and appreciate your hard work, kindly please develop 32-bit binary too, will be really helpful.
Thanks and Regards.

failed to fetch wayback pagination: Get "https://web.archive.org/cdx/search/cdx?url=*.example.com/*&output=json&collapse=urlkey&fl=original&page=0&showNumPages=true": dial tcp: i/o timeout
failed to fetch otx results page 0: Get "https://otx.alienvault.com/api/v1/indicators/domain/example.com/url_list?limit=200&page=0": dial tcp: i/o timeout

OS: Ubuntu

It was working fine before but now it's not giving any output.
Tried reinstalling but still the same, If similar happend with anyone else then plz share what worked for you

when running gau in zsh
fatal: not a git repository (or any of the parent directories): .git

ratx@Kali:/usr/bin/gau$ printf example.com | gau
bash: gau: command not found

-b jpg wont catch JPG endpoint

Hey Corben! I'd like to request a new feature - stdout to JSON file to a specific directory. I need it for another script and I can use other methods to convert the output to JSON but it'd cool if you add it to your tool straightaway.

Thanks and have a nice day. Max (@w2w).

First of all thanks for creating this awesome tool, it helps greatly in daily usage to find multiple vulnerabilities. Coming to the point, gau breaks certain times, there are multiple cases I guess, and not all times these cases appears to be faulty:

1. When response contains ? mark sign.
2. When response length is long.

I have two sites which demonstrate this, but I don't want to mark them here, to know this one I can mail you if you wish or I can DM you, as you prefer. One more thing I have compared results with waybackurls and found that it is providing the appropriate results in format.

panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/lc/gau/providers.NewCommonProvider(0xc00008e360, 0x0, 0x0, 0x0, 0x0)
/root/go/pkg/mod/github.com/lc/[email protected]/providers/common.go:45 +0x2ac
main.run(0xc00008e360, 0xc000206000, 0x369, 0x400)
/root/go/pkg/mod/github.com/lc/[email protected]/main.go:35 +0x98f
main.main()
/root/go/pkg/mod/github.com/lc/[email protected]/main.go:166 +0x7e5

It would be great if you can add --exclude function to gau so that we can exclude unwanted files url.
Thank you

Every time I try to run cat http-300terra.txt | ./gau
It gives me this error.
WARN[0018] error instantiating commoncrawl: commoncrawl.apiResult: decode slice: expect [ or n, but found , error found in #0 byte of ...||..., bigger context ...||...

Hi,
unfortunately I am seeing the same exact behaviour described previously also after updating to the latest new version.

Feel free to DM on Twitter and I will give you the exact URL I am trying or I will help you to debug this.

Thanks again in advance.

When I'm Running Command gau Then I'm Receiving This Error.
$gau
WARN[0001] error instantiating commoncrawl: API responded with non-200 status code

I had an error like this .

mv: cannot overwrite non-directory '/usr/bin/gau' with directory 'gau'
when excute
mv gau /usr/bin/gau

When I try to use this tool
It showing error like this

fatal: pathspec 'video.uber.com' did not match any files

hi
when i try to install this tools . i'll get 403 error

root@localhost:/home/Download# go install github.com/lc/gau/v2/cmd/gau@latest
go: downloading github.com/klauspost/compress v1.13.4
/root/go/pkg/mod/github.com/valyala/[email protected]/compress.go:10:2: github.com/klauspost/[email protected]: reading https://proxy.golang.org/github.com/klauspost/compress/@v/v1.13.4.zip: 403 Forbidden
/root/go/pkg/mod/github.com/valyala/[email protected]/compress.go:11:2: github.com/klauspost/[email protected]: reading https://proxy.golang.org/github.com/klauspost/compress/@v/v1.13.4.zip: 403 Forbidden
/root/go/pkg/mod/github.com/valyala/[email protected]/compress.go:12:2: github.com/klauspost/[email protected]: reading https://proxy.golang.org/github.com/klauspost/compress/@v/v1.13.4.zip: 403 Forbidden

itry it with 2 different ip address but i still getting this error

how can i fix it??

Nice work @lc.

It seems that gau returns duplicate URLs in the output with no way to remove without piping to another command (the dupes usually shouldn't be a big deal, but I encountered a case where robots.txt appeared a very large number of times in the output). Could it be helpful to have a flag to optionally remove duplicates for the user automatically?

I tired the below. The output is showing on the console but it's not really creating and dumping into the file.
root@kali:# gau --version
gau version: 1.0.3
root@kali:#
root@kali:# gau target.com -subs -o target.txt
root@kali:# ls -l
total 0
root@kali:~#

Hello!

Awesome tool, much faster than my bash script... but it appears to have an issue with invalid characters

2020/03/15 01:35:37 Could not decode json: invalid character '<' looking for beginning of value

This is separate issue, but entire script stops when a timeout occurs... specifically seeing it with alienvault.

Thanks!

GO111MODULE=on go get -u -v github.com/lc/gau after this when i try to run
gau: command not found?

Hello,

I just have a simple question. It mentions -providers flag to fetch urls from waybackurls. Is there any other provider we can add?

-providers wayback

I got this error any solution???

goroutine 1 [running]:
github.com/lc/gau/v2/pkg/providers/commoncrawl.New(0xc0001b2000, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...})
	/root/go/pkg/mod/github.com/lc/gau/[email protected]/pkg/providers/commoncrawl/commoncrawl.go:42 +0x1b0
github.com/lc/gau/v2/runner.(*Runner).Init(0xc000074690, 0xc0001b2000, 0x81a513)
	/root/go/pkg/mod/github.com/lc/gau/[email protected]/runner/runner.go:41 +0x585
main.main()
	/root/go/pkg/mod/github.com/lc/gau/[email protected]/cmd/gau/main.go:35 +0x23f

Hello!
I see a lot of bug bounty writeups where people use gau along with waybackurls.

• Does it make sense? My first thought was that gau already includes all the urls of waybackurls...

Hey I am not able to save requests in my file. gau -o urls.txt redacted.com

Hi there,

Since the update 2.0.4 I'm seeing a weird fatal error.

gau example.com
FATA[0005] error instantiating client: empty response, check your proxy configuration

This is my .gau.toml file, located in the home directory:

threads = 2
verbose = false
retries = 5
subdomains = false
providers = ["gau","commoncrawl","otx","urlscan"]
blacklist = ["ttf","woff","svg","png","jpg"]
json = false

[urlscan]
  apikey = "redacted"

[filters]
  from = ""
  to = ""
  matchstatuscodes = []
  matchmimetypes = []
  filterstatuscodes = []
  filtermimetypes = ["image/png", "image/jpg", "image/svg+xml"]

I'm not using any proxy. Am I doing something wrong here?

gau not work as u see in the pic
i even tried with sudo or by root account

panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/lc/gau/providers.NewCommonProvider(0xc00008e360, 0x0, 0x0, 0x0, 0x0)
/root/go/pkg/mod/github.com/lc/[email protected]/providers/common.go:45 +0x2ac
main.run(0xc00008e360, 0xc000206000, 0x369, 0x400)
/root/go/pkg/mod/github.com/lc/[email protected]/main.go:35 +0x98f
main.main()
/root/go/pkg/mod/github.com/lc/[email protected]/main.go:166 +0x7e5

Here is the request I used:

./gau -providers common -o test.txt www.hackerone.com

Resulted in the following error:

Error: common is not a valid provider.

panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/lc/gau/providers.NewCommonProvider(0xc0000c2300, 0x0, 0x0, 0x0, 0x0)
/home/dexos/go/pkg/mod/github.com/lc/[email protected]/providers/common.go:45 +0x2ac
main.run(0xc0000c2300, 0xc000286000, 0x1740, 0x1800)
/home/dexos/go/pkg/mod/github.com/lc/[email protected]/main.go:35 +0x98f
main.main()
/home/dexos/go/pkg/mod/github.com/lc/[email protected]/main.go:166 +0x7e5

when i try to install it with go build i got this error

command-line-arguments

/usr/bin/ld: /root/.cache/go-build/45/45e7b436531a3ff0bbfde65724b3c47cf223bf26529f52a761b9d83143178b3c-d(go.o): in function github_0com_1modern_x2dgo_1reflect2.UnsafeSliceType.UnsafeMakeSlice': /root/go/pkg/mod/github.com/modern-go/[email protected]/unsafe_slice.go:78: undefined reference to reflect.unsafe_NewArray'
/usr/bin/ld: /root/.cache/go-build/45/45e7b436531a3ff0bbfde65724b3c47cf223bf26529f52a761b9d83143178b3c-d(go.o): in function github_0com_1modern_x2dgo_1reflect2.unsafeType.UnsafeNew': /root/go/pkg/mod/github.com/modern-go/[email protected]/unsafe_type.go:48: undefined reference to reflect.unsafe_New'
/usr/bin/ld: /root/go/pkg/mod/github.com/modern-go/[email protected]/unsafe_type.go:48: undefined reference to reflect.unsafe_New' /usr/bin/ld: /root/go/pkg/mod/github.com/modern-go/[email protected]/unsafe_type.go:48: undefined reference to reflect.unsafe_New'
/usr/bin/ld: /root/go/pkg/mod/github.com/modern-go/[email protected]/unsafe_type.go:48: undefined reference to reflect.unsafe_New' /usr/bin/ld: /root/go/pkg/mod/github.com/modern-go/[email protected]/unsafe_type.go:48: undefined reference to reflect.unsafe_New'
/usr/bin/ld: /root/.cache/go-build/45/45e7b436531a3ff0bbfde65724b3c47cf223bf26529f52a761b9d83143178b3c-d(go.o):/root/go/pkg/mod/github.com/modern-go/[email protected]/unsafe_type.go:48: more undefined references to `reflect.unsafe_New' follow
collect2: error: ld returned 1 exit status

Hi,

First, thank for the great tools, it helps me found several bugs, including my seconds first bug too :)

I notices that a url that i can find on WaybackMachine, but cannot find it in gau output (i tried waybackurls, not found too). The sample url is from a H1 private program, i will send you an email with the sample url instead.

I Tried it by clone , gau_1.0.3_linux_amd64.tar.gz (extraction) , and by using go.

but when i try gau -h it shows
gau: command not found

root@kali:# export PATH=$PATH:/usr/local/go/bin
root@kali:# go get -u -v github.com/lc/gau
github.com/lc/gau (download)
github.com/json-iterator/go (download)
github.com/modern-go/concurrent (download)
github.com/modern-go/reflect2 (download)
root@kali:~# gau -h
bash: gau: command not found

You should add a -b flag to blacklist specific file types, for example.(jpg|jpeg|gif|css|tif|tiff|png|ttf|woff|woff2|ico) etc etc

~ @m0chan

The output of gau almost always includes URLs such as https://example.com:443/somepath. I think it would be only beneficial to remove these ports from the output.

I would be happy to implement this if it's a feature that'd be wanted.

Thanks!

Hi,

Thanks for the tool! the new blacklisted extension filtering is awesome too but found that it's case sensitive which is the nature of Go obviously. I've come across URL cases where it ends with an upper case extension such as .PNG so even if you blacklist png like gau -b png,svg it won't work and will still be outputted.

Similarly, if someone mistakenly uses an uppercase letter, say gau -b pNg,svg it will not work either since both extensions don't match.

Regards.

As gau binary conflict with the gau git inbuild alias
gau stands for git add --update (Also: "git add -u")

Been getting a bunch of errors that look like this. If it helps, i'm trying to run this in parallel... Any idea?

       /root/go/src/github.com/lc/gau/gau.go:132 +0x1e7
main.Run(0x7fff09682390, 0x17)
        /root/go/src/github.com/lc/gau/gau.go:62 +0xb8
main.main()
        /root/go/src/github.com/lc/gau/gau.go:53 +0x1b9
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x661a27]

goroutine 1 [running]:
main.getWaybackUrls(0x7ffeccdf538d, 0x1a, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/go/src/github.com/lc/gau/gau.go:132 +0x1e7
main.Run(0x7ffeccdf538d, 0x1a)
        /root/go/src/github.com/lc/gau/gau.go:62 +0xb8
main.main()
        /root/go/src/github.com/lc/gau/gau.go:53 +0x1b9
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x661a27]

Error: error getting current commoncrawl url: could not unmarshal json from CC: invalid character '<' looking for beginning of value
Error: error getting current commoncrawl url: could not unmarshal json from CC: invalid character '<' looking for beginning of value
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x661a27]

goroutine 1 [running]:
main.getWaybackUrls(0x7ffc73e6b38e, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/go/src/github.com/lc/gau/gau.go:132 +0x1e7
main.Run(0x7ffc73e6b38e, 0x19)
        /root/go/src/github.com/lc/gau/gau.go:62 +0xb8
main.main()
        /root/go/src/github.com/lc/gau/gau.go:53 +0x1b9
Error: error getting current commoncrawl url: could not unmarshal json from CC: invalid character '<' looking for beginning of value
Error: error getting current commoncrawl url: could not unmarshal json from CC: invalid character '<' looking for beginning of value
Error: error getting current commoncrawl url: could not unmarshal json from CC: invalid character '<' looking for beginning of value
Error: error getting current commoncrawl url: could not unmarshal json from CC: invalid character '<' looking for beginning of value

failed to decode otx resuts for page 0: invalid character '<' looking for beginning of value

am getting that error above and it stops.

When doing echo 'example.com' | gau you will get urls from any domain containing 'example.com',
like for example: evil.com/?a=example.com and this it's a little annoying, is there a way to only get urls from the domain we provide?

Hi,
unfortunately I am seeing the same exact behaviour described here (#1) also after updating to the new version.

Feel free to DM on Twitter and I will give you the exact URL I am trying.

Thanks a lot, your tool is great!

i have problem conecting the proxy with burp suite during the crawl ! on windows

Started getting this after updating to 2.04 and even after 2.05.
gau example.com
FATA[0000] error instantiating client: API responded with non-200 status code