As a user looking to fingerprint things quickly, it'd be handy if ident was smart enough to do an initial check against a single (or small set) of urls, then run follow-on checks for stuff that is specific to a given product.

Note that this increases the importance and difficulty of getting the initial checks right, but extends the usefulness of ident by significantly speeding it up in the general case. Some tech may need to be revisited to see if we can address fingerprinting it with just the base url.

Lots of devices have unique certificates, it'd be handy to check for these.

Running the latest docker image one host doesn't fingerprint the server as nginx.

How to reproduce:
docker pull intrigueio/intrigue-ident && docker run -t intrigueio/intrigue-ident --url http://server.amconstruct1.com:8080

Output:

Checking URL: http://server.amconstruct1.com:8080
Fingerprint:

curl --head http://server.amconstruct1.com:8080

HTTP/1.1 302 Found
cache-control: max-age=0, private, must-revalidate
connection: close
content-length: 11
date: Thu, 12 Mar 2020 14:51:32 GMT
location: http://survey-smiles.com
server: nginx
set-cookie: sid=f6e916ac-6470-11ea-9d4c-692b809f72b1; path=/; domain=.amconstruct1.com; expires=Tue, 30 Mar 2088 18:05:39 GMT; max-age=2147483647; HttpOnly

Expected output(example domain "DISTINCTIVEINDUSTRIES.com" was used, it also reports simple "server: nginx" ):

Checking URL: http://DISTINCTIVEINDUSTRIES.com
Fingerprint:
 - Nginx Nginx   - Nginx (CPE: cpe:2.3:a:nginx:nginx::) (Tags: ["Web Server"]) (Hide: false)

curl --head http://DISTINCTIVEINDUSTRIES.com

HTTP/1.1 200 OK
Server: nginx
Date: Thu, 12 Mar 2020 14:51:15 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Vary: Accept-Encoding
Expires: Wed, 17 Aug 2005 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: 2863bf481cda1bd44e4fb03af25b5843=f454d5edd431232714564a97f54a7b86; path=/; HttpOnly
Last-Modified: Thu, 12 Mar 2020 14:51:15 GMT

Ran build on the latest Dockerfile --

Step 9/10 : RUN bundle install
 ---> Running in f9e1e186a768
You must use Bundler 2 or greater with this lockfile.
The command '/bin/sh -c bundle install' returned a non-zero code: 20

This is on the latest master (2.0.7):

 /app/intrigue-ident/lib/ident.rb:103:in `require_relative': cannot load such file -- /app/intrigue-ident/checks/pop3/base (LoadError)
        from /app/intrigue-ident/lib/ident.rb:103:in `<top (required)>'
        from ./util/ident.rb:2:in `require_relative'
        from ./util/ident.rb:2:in `<main>'
 1

Hello, devs!

I'm running ident like this:

docker run --cpu-shares 1024 --rm  -v dockerresults:/dockerresults intrigueio/intrigue-ident --threads 10 --debug --file /dockerresults/whatweb699507042838/whatwebhttpx.txt --json

inside the file there are urls, 1 url per line.

I'm getting this error message:

./util/ident.rb:122:in `write_simple_csv': wrong number of arguments (given 1, expected 2) (ArgumentError)
	from ./util/ident.rb:213:in `check_uris_from_file'
	from ./util/ident.rb:117:in `main'
	from ./util/ident.rb:275:in `<main>'
	

Could you help me please, what am I doing wrong?

As a user, it'd be super handy if you'd fingerprint each endpoint as i hit it during a redirect chain. This would allow me to identify more assets. Note that this behavior should be optional, as it's not always intuitive that you'd combine endpoints like this.

Trying to run the docker image and give it a list of URLs to scan with browser enabled:
docker run -v /root/urls:/mnt/localurls -t intrigueio/intrigue-ident --file /mnt/localurls/targeturls.txt --browser true

Get an error:
EXCEPTION! undefined local variable or method enable_browser' for main:Object Traceback (most recent call last): from ./util/ident.rb:291:in

It seems to me, that the variable enable_browser is not defined in function check_file_urls in util/ident.rb, thus the error.

Trying to run docker image for a single url with browser enabled:
docker run -t intrigueio/intrigue-ident --browser true --url https://google.com

Get an error:
Checking URL: https://google.com
Traceback (most recent call last):
from ./util/ident.rb:291:in <main>' from ./util/ident.rb:275:in main'
from ./util/ident.rb:156:in check_single_url' from /home/ident/lib/ident.rb:192:in generate_http_requests_and_check'
from /home/ident/lib/ident.rb:192:in each' from /home/ident/lib/ident.rb:218:in block in generate_http_requests_and_check'
/home/ident/lib/http/browser.rb:17:in `ident_create_browser_session': uninitialized constant Intrigue::Config (NameError)

Today, in order to scan anything other than a URL, you have to enter a --ip option and (optionally) specify the port.

URIs are a much more consistent way to specify input...

Rather than doing something like:
./util/ident.rb --ip 1.1.1.1 --port 25

it'd be better if you could do something like:
./util/ident.rb --uri smtp:1.1.1.1:25

Address this for all supported protocols

via @bcoles

Using JSON files or any other non language specific data storage format to hold your fingerprints will make it a lot easier for other languages to use your fingerprints. You have done some amazing work here I and I'm sure others would love to use your fingerprints in our personal tools.

Your open redirect check has this regex in it:

/\?url=http/i

But that is used by quite a few other things, on my site, most of the social media links match it:

<a title="Share on Twitter" id="twitter-share" href="https://twitter.com/share?text=DigiNinja&amp;url=https://digi.ninja/index.php" rel="noopener" target="_blank"><i class="fa fa-twitter"></i></a>

I think it is too vague a check to determine open redirect based off that. It also took me a while to track down why you thought there was an open redirect, maybe giving the reason for the decision would be good as it would help determine whether it is a false positive or not.

As a user it'd be handy if this method was supported.

Intrigue ident misidentifies Apache Tomcat as "Apache Coyote 1.1". In fact, the "Apache-Coyote/1.1" "Server" header is only sent back from versions of Apache Tomcat from 4.1.x to 8.0.x (see https://tomcat.apache.org/tomcat-7.0-doc/security-howto.html).

I get a result such as the following:

cpe: "cpe:2.3:a:apache:coyote:1.1:"
hide: false
inference: true
issues: null
match_details: "Apache coyote application server - server header"
match_type: "content_headers"
method: "ident"
product: "Coyote"
tags: ["Application Server"]
0: "Application Server"
tasks: null
type: "fingerprint"
update: null
vendor: "Apache"
version: "1.1"

the cli's export output needs to be reworked.

the json / csv options are commented out, and dropping an output.csv in the working directory isn't desirable, and no csv is generated if you're targeting a single URL/IP.

via @bcoles

As a user, it'd be handy to have a content check store a checksum per endpoint as this can then be referred to in the future.

As surfaced in the comments on #20, Webresource.asmx is being requested as a first-round resource. Figure out which check is requesting it, and make it a follow-on check to ASP.net.

Checking URL: http://server.amconstruct1.com:8080
Getting http://server.amconstruct1.com:8080
Getting http://server.amconstruct1.com:8080/api
Getting http://server.amconstruct1.com:8080/doesntexist-123
Getting http://server.amconstruct1.com:8080/WebResource.asmx
Getting http://server.amconstruct1.com:8080/admin
Getting http://server.amconstruct1.com:8080/error.json
Ran 576 checks against base URL
Fingerprint:
 - Nginx Nginx   - Nginx (no version) (CPE: cpe:2.3:a:nginx:nginx::) (Tags: ["Web Server"]) (Hide: false)

See #8 as well, but this sort of verbosity around content checks is unnecessary. Save it for the JSON/CSV output. For the CLI, only print stuff that would be useful to a tester.

Specifically:

• remove location header
• remove security header info if they don't print something that can be actioned
• remove anything that's 'false'
• remove email addresses output if there are none.

Checking URL: http://127.0.0.1/
Checking... http://127.0.0.1/
Fingerprint:
 - Apache HTTP Server 2.4.41  - Apache web server - server header - with versions (CPE: cpe:2.3:a:apache:http_server:2.4.41:) (Tags: ["Web Server"])
Content Checks:
 - Access-Control-Allow-Origin Header: false
 - P3P Header: false
 - X-Frame-Options Header: false
 - X-XSS-Protection Header: false
 - Google Analytics Account Detected: false
 - Location Header: 
 - Directory Listing Detected: false
 - Form Detected: false
 - File Upload Form Detected: false
 - Email Addresses Detected: []
 - Authentication - HTTP: false
 - Authentication - Session Identifier: false````

via @bcoles 

The security header content checks currently only check for the existence of the header, which is next to useless. It'd be far better if they actually verified / checked the value/content of the header.

For example:

{
        :type => "content",
        :name => "Access-Control-Allow-Origin Header",
        :match_type => :content_headers,
        :dynamic_result => lambda { |d|
          return true if _first_header_match d, /^Access-Control-Allow-Origin:.*/i;
        false
        },
        :dynamic_hide => lambda { |d| false },
        :dynamic_issue => lambda { |d| false },
        :paths => ["#{url}"]
      },

and

{
        :type => "content",
        :name => "X-Frame-Options Header",
        :match_type => :content_headers,
        :dynamic_result => lambda { |d|
          return true if _first_header_match d, /^x-frame-options:.*/i;
        false
        },
        :dynamic_hide => lambda { |d| false },
        :dynamic_issue => lambda { |d| false },
        :paths => ["#{url}"]
      },```

but the header could be `X-Frame-Options: kill all humans` .. 

in addition, many applications will not respond with a CORS header unless the client request contains an `Origin` header

via @bcoles