disclose / diosts Goto Github PK

21.0 5.0 4.0 62 KB

A Go scraper that validates security.txt files and outputs them in the disclose.io JSON format.

License: MIT License

Go 100.00%

security-txt scraper golang json

diosts's Introduction

diosts

The disclose.io security.txt scraper (diosts) takes a list of domains as the input, retrieves and validates the security.txt if available and outputs it in the disclose.io JSON format.

Installation

Prerequisites: a working Golang installation >= 1.13

go get github.com/disclose/diosts/cmd/diosts

Usage

cat domains.txt | ~/go/bin/diosts -t <threads> 2>diosts.log >securitytxt.json

This wil try and scrape the security.txt from the domains listed in domains.txt, with <threads> parallel threads (defaults to 8). Logging (with information on each of the domains in the input) will be written to diosts.log (because it's output to stderr) and a JSON array of retrieved security.txt information in disclose.io format will be written to securitytxt.json.

For each input, the following URIs are tried, in order:

https://<domain>/.well-known/security.txt
https://<domain>/security.txt
http://<domain>/.well-known/security.txt
http://<domain>/security.txt

Any non-fatal violations of the security.txt specification will be logged.

Build

Note: building is not necessary if you use the installation instructions, Go will take care of this for you.

git clone https://github.com/disclose/diosts
cd diosts
go build ./cmd/diosts

Notes

Redirects

According to the specifications, a redirect should be followed when retrieving security.txt. However:

When retrieving the file and any resources referenced in the file, researchers should record any redirects since they can lead to a different domain or IP address controlled by an attacker. Further inspections of such redirects is recommended before using the information contained within the file.

At this point, we blindly accept redirects within the same organization (e.g., google.com to www.google.com is accepted). Any other redirect is logged as an error, to be dealt with later.

Canonical

A security.txt should contain a Canonical field with a URL pointing to the canonical version of the security.txt. We should check if we retrieved the security.txt from the canonical URL and if not, do so.

Program name

Currently, we use the input domain name as program name. This might or might not be correct, especially with redirects and canonical URL entries. To be discussed later.

diosts's People

Contributors

Stargazers

Watchers

Forkers

ravkishu rajeev2510 ethicalsecurity-agency chriss-0x01

diosts's Issues

Missing license and file headers

We should decide on and add a LICENSE and add a simple copyright/license header in each file.

panic: runtime error: index out of range

Conditions:
Running the script using the following list of domains on Ubuntu 18.04 LTS (go version go1.10.4 linux/amd64)

facebook.com
google.com
youtube.com
twitter.com
instagram.com
linkedin.com
microsoft.com
apple.com
wikipedia.org
plus.google.com

Result:

panic: runtime error: index out of range

goroutine 7 [running]:
github.com/disclose/diosts/internal/pkg/discloseio.FromSecurityTxt(0xc4201a8700, 0x7e9d16)
	/root/go/src/github.com/disclose/diosts/internal/pkg/discloseio/discloseio.go:71 +0x3a1
github.com/disclose/diosts/internal/app/run.(*Writer).Start.func1(0xc42010e300)
	/root/go/src/github.com/disclose/diosts/internal/app/run/writer.go:50 +0x195
created by github.com/disclose/diosts/internal/app/run.(*Writer).Start
	/root/go/src/github.com/disclose/diosts/internal/app/run/writer.go:34 +0x5c

Expected:
Normal completion.

Allow for comparison and automatic PR against diodb

The main function of diosts is to provide authoritative updates to diodb based on data it parses from security.txt files.

For each URL processed by diosts it would be ideal if diosts (or another small service that consumes diosts output, and potential the output of other similar scrapers) could:

Check if the insertion of a new object into diodb is appropriate,
Check if the updating of a key pair for an existing diodb object is appropriate, and
Formulate and push a PR to the diodb repo for review on a per object basis, vs in a batch.

Rename the "program_name" field to "security_txt_domain"

Can we rename the "program_name" field to "security_txt_domain" denoting the domain the security.txt was retrieved from?

"program_name" in diodb is tied to the company, organization, or business unit responsible for the policy and intake channel which is a looser coupling than the data returned by diosts.

Looking at the merged data, I'm thinking that we should consider outputting the diosts data to a separate data store to allow for the differences between security.txt rendering of this information and that which we collect in the diodb.

Update the scanner to reflect Expiry and time standard changes in the securitytxt.org standard

The securitytxt.org standard has recently been updated to make "Expiry" a mandatory field, and the format adjusted to ISO 8601. Details are on the front page of the securuitytxt.org website.