The releases for v0.15.0 are now no longer single-file binary releases, but instead come in tarballs. These tarballs should additionally include a README and the LICENSE.

The Docker image currently takes quite a long time to build (~20 minutes in GitHub Actions). A large part of that time is spent running cargo, fetching the crates.io index and downloading packages:

Is it possible and feasible to expose the contents of the GitHub Actions cache to docker build to speed it up?

Currently the report command only supports the default --format=human format. Nosey Parker should support the other formats available in the CLI also.

The scan command currently is able to automatically clone Git repositories when invoked with the --git-url, --github-user, or --github-org arguments. This runs sequentially, and when you cast a large net (e.g., end up indirectly specifying 1000 repositories), cloning the input repositories takes the majority of the total time.

It doesn't appear that cloning a single git repo at a time is either network, CPU, or memory-bound on any system I've used. It seems that the remote server that we are cloning the repo from is the bottleneck.

It would be better if Nosey Parker could clone Git repositories in parallel — maybe a limit of 4 at a time by default.

Nosey Parker currently bundles Vectorscan 5.4.8, released in September 2022.

Vectorscan have put out three new releases in the meantime: 5.4.9, 5.4.10 and 5.4.10.1. These include additional features ported from Hyperscan, performance improvements, and bug fixes.

Nosey Parker should be updated to use the latest version. While addressing this issue, we should also take the opportunity to address #71.

If Nosey Parker could output findings in SARIF format, it could be more easily integrated into other tools, like GitHub Code Scanning.

This would probably be exposed as a --format=sarif option for the report (and possibly summarize) commands.

Some references:

• https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning
• https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/sarif-support-for-code-scanning
• https://sarifweb.azurewebsites.net/

The Docker Image CI workflow fails sporadically, e.g., https://github.com/praetorian-inc/noseyparker/actions/runs/3943453279.

Every time I have seen this workflow fail, it is with an exit code 132 when trying to run the Docker image. Re-running the workflow once or twice has always fixed it.

Why is this happening, and how do we fix it?

noseyparker report currently limits the number of displayed matches per finding to 3 in the default human format. This is to prevent reporting thousands of matches in degenerate cases (which do exist!). However, it would be better if this setting could be controlled at runtime with a new CLI option (perhaps --max-matches=N).

The current mechanism for selecting rules to be used when scanning is very simplistic: you can use all the default rules, and you can specify additional YAML-format rules to load with the --rules FILE_OR_DIR option.

What are the problems with this?

• There is no way to disable a particular rule at scan time
• There is no way to enable only a particular rule at scan time
• Some of the default rules are much noisier than others (The Generic * rules in particular), and result in the largest proportion of reported findings

Let's improve the rule selection mechanism. I'm thinking that this would involve a new "ruleset" mechanism, which is an explicitly-specified set of available rules. Perhaps a YAML list of rule names to enable. Or perhaps gitignore format.

We will also want a new rules list CLI command, which will print out the set of selected rules according to some ruleset.

SARIF support was recently added (#33, #4), adding a new output format to Nosey Parker's report command. This support is preliminary, but good enough that viewers like the VSCode SARIF plugin can do something useful with the output in some cases.

However, I want Nosey Parker to do something useful in all cases. The end goal is that Nosey Parker's SARIF output is complete enough that common viewers can usefully render all findings.

Viewers of particular interest:

• GitHub Code Analysis (so that SARIF output can be automatically shown in pull requests)
• VSCode SARIF Viewer
• The sarif-fmt command-line program

Rough edges and opportunities for improvement:

1. Findings in blobs from Git repositories don't have useful location information associated with them.
2. Nosey Parker rules don't have a stable and machine-friendly ID associated with them, just a name.
3. Nosey Parker rules don't have a long description, severity, or precision associated with them.
4. Currently, the VSCode SARIF Viewer's functionality to annotate findings as false positives crashes with Nosey Parker-generated output, probably due to some missing field.
5. The location info in SARIF results is for the entire regex match rather than just the match group.

Vectorscan is now vendored in the Nosey Parker source tree (#41), making it simpler to build and distribute the noseyparker binary. However, there are still a couple complications beyond simply doing cargo build -r.

1. The build environment needs Python to be available: this is for the Vectorscan code itself (see vectorscan-sys/vectorscan/CMakeLists.txt), I think for formatting a timestamp for reproducible builds, and looks like it could easily be stubbed out.

2. The build environment needs libclang to be available, for the use of bindgen in vectorscan-sys/build.rs. This is necessary to run bindgen, but that could be done once and the result committed, with the invocation of bindgen put behind a feature flag.

If these two items were addressed, then someone trying to build Nosey Parker from source would only need the usual Rust toolchain stuff and cmake.

Hi 👋

A great option in secret scanner is to be able to scan a range of commits, for example by adding an option to scan.

In my case, we use scanners for very large repositories. Once reported, in futures runs there will be no need to scan previously scanned commits. Only new commits are relevant. It saves a lot of time in large repositories.

Gitleaks has this feature , and Trufflehog too.

For example a since_commits option, scanning between a specific commit and HEAD. And why not a until_commits option.

Do you see any blocking issues for this enhancement?

😄

Truffle Hog has a GitHub Action: https://github.com/marketplace/actions/trufflehog-oss
GitLeaks has a GitHub Action: https://github.com/gitleaks/gitleaks-action

Why not Nosey Parker?

I could help debug on these. would be very nice to have these.

Hello,
Since my company is actually working with Bitbucket, I would be glad to work in order to add the same features for Github to Bitbucket. For example listing Bitbucket project repositories, listing user repositories, and ensure that each feature of noseyparker is working on it.

I could take inspiration from the existing code for Github and make a Bitbucket version.
Why not GitLab too in the future?
😄

Using musl instead of glibc when building Nosey Parker results in a significant drop in scan performance, presumably due to the allocator implementation in musl not supporting threaded workloads very well (see here).

It may be possible to sidestep this by using a different global allocator in Nosey Parker. In particular, it appears that jemalloc does not build with musl. But mimalloc does build there, and there is a Rust crate for it already.

Is it easy to switch Nosey Parker to use mimalloc as its global allocator?

How does switching impact performance of native-code builds? How does it affect performance of Docker-based builds, particularly the Alpine-based build in #77?

Currently, the scan command runs in two main phases: input enumeration and content scanning. Each of these phases runs in parallel (but not concurrently; the input enumeration phase completes entirely before the content scanning phase completes).

However, within the input enumeration phase, when a Git repository is discovered on the filesystem, that repository is enumerated sequentially, by a single thread. This becomes noticeable when you are scanning just a single huge repository, such as the Linux kernel, which has over a million commits, several million objects, and can take over a hundred GB of space when uncompressed.

It would be better if Nosey Parker did not have this sequential bottleneck, and was instead able to enumerate a single Git repository in parallel, using all available cores.

The implementation of this will be a bit tricky, requiring rework of the parallelism mechanism in the input enumerator code. That currently uses the ignore crate to do parallel filesystem walking, but that does not seem to expose its thread pool. We would want the proposed parallel Git enumerator to not oversubscribe the system running scan; the total number of enumeration threads should be controllable.

Additionally complicated will be figuring out how to build up the Git metadata graph that is being added in #66 (to address #16): the core graph data structure there is not designed for out-of-the-box mutation from many threads.

Nosey Parker should keep track of which inputs have already been scanned, and avoid rescanning them if possible on future scanning runs.

Currently, noseyparker scan -d DATASTORE INPUT will completely enumerate and scan INPUT from scratch. Nosey Parker is fast, but for large repositories (like the LInux kernel, with 100+GB of blobs), it still takes a couple minutes. However, simply enumerating contents goes pretty quickly, especially in Git repositories (e.g., the Linux kernel repo can be enumerated in 13-25 seconds, depending on filesystem cache). If Nosey Parker kept track of which blobs it had scanned and with which set of rules, it could avoid re-scanning things.

Caching is tricky to get right. The information about which inputs have already been scanned should probably be persisted in the datastore sqlite database. An entry would be a (blob id, ruleset id, nosey parker version id), indicated that the particular blob had been scanned with a particular set of rules and Nosey Parker version.

If the context size for reported findings in Nosey Parker becomes runtime-configurable, that parameter would also need to be taken into account for caching.

The cache could be a fixed-size LRU cache, 128MB of entries for example, loaded into a fast in-memory structure at enumeration time, and then updated in bulk at the end of scanning. (Some implementation like this may be necessary to avoid tanking Nosey Parker speed.)

One complication: a Nosey Parker datastore sqlite database is currently a very simple, totally denormalized schema with a single table. There is also no such thing currently as a ruleset id; that notion would need to be added (perhaps sha512 of all the loaded rules).

          @bradlarsen Can we please provide a `musl` binary as well?

I get the following GLIB error when trying to run noseyparker on Ubuntu 20:

❯ noseyparker
noseyparker: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by noseyparker)
noseyparker: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by noseyparker)
noseyparker: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by noseyparker)
noseyparker: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by noseyparker)

I seem to have GLIB 2.31:

❯ ldd --version
ldd (Ubuntu GLIBC 2.31-0ubuntu9.9) 2.31

Originally posted by @dufferzafar in #28 (comment)

vectorscan is currently vendored, built from source as part of Nosey Parker. For the sake of expediency I implemented that by adding a copy of all vectorscan 5.4.8 in a subdirectory, and then modifying those in-place.

It would be better for maintainability if the vendored build process instead extracted a pristine shasum-verified release tarball and applied patchfiles to that prior to building. This will make it clearer in the future what modifications were made to the Vectorscan sources.

Additionally, being able to get rid of the enormous vectorscan and boost source trees from this repository would make continued development of Nosey Parker a bit nicer, as things like fzf-based fuzzy search and whole-source-tree textual search wouldn't get confounded by all the stuff in there (which we mostly don't care about).

Use case

You are reviewing a bunch of findings from Nosey Parker and want to focus on particular categories of findings, eliminate findings from certain files or repos, etc.

Existing limitations

At present, the only way to do this is to write some postprocessing script. This has a couple downsides:

• If you postprocess the human-readable text-based report, you have to do grungy parsing
• If you postprocess the machine-readable JSON report, you afterwards have to write your own logic to assemble that into a human-readable report from the filtered result
• Any postprocessing you do is strictly "post"; the filtered results cannot be fed back into Nosey Parker. (Example use case: you filter results and want the noseyparker summarize output only on the filtered results.)

Proposal

The report and summarize commands should have an additional --filter CMD option that behaves as follows:

• CMD is the name of a program that takes a JSON array of findings on stdin and emits a JSON array of findings on stdout
• If CMD returns a non-zero exit code, prints unintelligible output, or emits findings that did not appear in the original input, noseyparker exits with an error
• When --filter CMD is given to either the report or summarize commands, the filter program is run on Nosey Parker's findings as soon as they are loaded, but prior to downstream processing within noseyparker

Currently you have to explicitly specify the datastore to most Nosey Parker commands. For example, to scan something, you have to say noseyparker scan -d DATASTORE SOME_INPUT, and to report findings, you have to say noseyparker report -d DATASTORE.

It can be a hassle to keep specifying the datastore over and over. Additionally, the -d/--datastore argument must be given after the subcommand name, which complicates command-line editing. For example, to scan and then report, you first run scan, then go to your previous command in shell history to change the subcommand to report, but you have to edit within the line, rather than being able to move the cursor to scan, deleting until end of line, and typing report.

You can kind of work around this hassle at present by setting the NP_DATASTORE=DATASTORE environment variable, which lets you get away with saying just noseyparker scan SOME_INPUT and noseyparker report.

It would be nicer if Nosey Parker would choose a default datastore name if it is not explicitly specified. The proposal is this:

• Use a default value of datastore.np for the datastore
• To avoid a UX hazard of accidentally polluting an existing datastore, modify the scan command so that if no datastore is explicitly given via -d/--datastore or the NP_DATASTORE environment variable — i.e., the default datastore path is used — and the datastore already exists, exit with an error

The prebuilt Docker images for releases do not include all the metadata one would like. See here, for example: https://github.com/praetorian-inc/noseyparker/pkgs/container/noseyparker/137062442?tag=v0.15.0. No labels!

I would be HAPPY to test azure devops support (via ssh would be REALLY nice!). Thanks for a better than the steaming pile o crap that is here for password scanners. Its not perfect, but its a WHOLE lot better than everything else!

          It seems that it is related to the scan :

I will create an issue for this later 😄

Originally posted by @Coruscant11 in #29 (comment)

This feature was demoed at Black Hat EU 2022 from the internal version of Nosey Parker, but is not yet implemented here.

The listed repos should include user gists and repo wikis.

This should exist as both a new github list [--github-user USER | --github-org ORG]... command, and as new --github-user USER, --github-org ORGarguments forscan`.

Hi,

Thank you for your tool which works very well. However, I have a question about the best way to ignore certain strings.

For example, I would like to be able to ignore some AWS keys :

AKIA111111111EXAMPLE
AKIA111111112EXAMPLE

So I've created a yaml file with this pattern (~/tmp/nosey/aws.yml) :

rules:
- name: AWS API Key Key
  pattern: '\b(?:A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16})\b'
  references:
  - https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html
  - https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
  - https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html
  - https://docs.aws.amazon.com/accounts/latest/reference/credentials-access-keys-best-practices.html

  examples:
  - 'A3T0ABCDEFGHIJKLMNOP'
  - AKIADEADBEEFDEADBEEF'

  negative_examples:
  - 'AKIA111111111EXAMPLE'
  - 'AKIA222222222EXAMPLE'
  - 'AKIAI44QH8DHBEXAMPLE'
  - '======================'
  - '//////////////////////'
  - '++++++++++++++++++++++'

Then :

noseyparker scan -r ~/tmp/nosey/ --datastore ~/gitlab-dump.db ~/gitlab-dump/

But these keys are still detected by the tool.
I guess I'm doing something wrong but I don't see how to solve my issue.

Thanks in advance for your help !

noseyparker scan currently always does an initial enumeration of the filesystem inputs. The only user-facing reason for doing this currently is to show a progress bar when scanning. This is detrimental in a couple cases:

1. When running without a terminal (like with output directed to a file), the progress bars are not shown, and the initial filesystem enumeration is not useful.
2. When scanning large inputs from slow filesystems (like old magnetic disks or Docker bind mounts on macOS), simply enumerating the inputs can be as slow as actually scanning everything!

It would be useful to automatically avoid enumeration when progress bars are not displayed. It would also be beneficial to have an explicit control to avoid enumeration, to help in cases of slow I/O.

This shouldn't work anyway, but I suspect you may want to handle the exception more gracefully

Specifying a datastore that doesn't exist when using the report command causes a full stack trace, appears to just be an uncaught exception

14:51:44 › noseyparker report -d /tmp/no-such-file
Error: Failed to open database at "/tmp/no-such-file/datastore.db"

Caused by:
    0: unable to open database file: /tmp/no-such-file/datastore.db
    1: Error code 14: Unable to open the database file

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
   1: noseyparker::datastore::Datastore::open
   2: noseyparker::cmd_report::run
   3: noseyparker::main
   4: std::sys_common::backtrace::__rust_begin_short_backtrace
   5: std::rt::lang_start::{{closure}}
   6: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/e75aab045fc476f176a58c408f6b06f0e275c6e1/library/core/src/ops/function.rs:609:13
   7: std::panicking::try::do_call
             at /rustc/e75aab045fc476f176a58c408f6b06f0e275c6e1/library/std/src/panicking.rs:483:40
   8: std::panicking::try
             at /rustc/e75aab045fc476f176a58c408f6b06f0e275c6e1/library/std/src/panicking.rs:447:19
   9: std::panic::catch_unwind
             at /rustc/e75aab045fc476f176a58c408f6b06f0e275c6e1/library/std/src/panic.rs:137:14
  10: std::rt::lang_start_internal::{{closure}}
             at /rustc/e75aab045fc476f176a58c408f6b06f0e275c6e1/library/std/src/rt.rs:148:48
  11: std::panicking::try::do_call
             at /rustc/e75aab045fc476f176a58c408f6b06f0e275c6e1/library/std/src/panicking.rs:483:40
  12: std::panicking::try
             at /rustc/e75aab045fc476f176a58c408f6b06f0e275c6e1/library/std/src/panicking.rs:447:19
  13: std::panic::catch_unwind
             at /rustc/e75aab045fc476f176a58c408f6b06f0e275c6e1/library/std/src/panic.rs:137:14
  14: std::rt::lang_start_internal
             at /rustc/e75aab045fc476f176a58c408f6b06f0e275c6e1/library/std/src/rt.rs:148:20
  15: main
  16: __libc_start_main
             at ./csu/../csu/libc-start.c:308:16
  17: _start

Obviously I don't feel too strongly about this being "fixed" but FYI. A pretty error is probably preferred?

Great tool by the way, thanks for publishing

The new Release Build job in GitHub Actions is intended to produce a release-mode binary for noseyparker for x86_64 Linux.

It doesn't quite work at present as desired, because Hyperscan does not in fact get statically linked:

$ ldd /scan/noseyparker
        libhs.so.5 => not found
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000004001f27000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000000400200e000)
        /lib64/ld-linux-x86-64.so.2 (0x0000004000000000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x0000004002236000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x0000004002256000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x000000400225b000)

See the reference to libhs.so.5 there. What this means is that someone who downloads a prebuilt noseyparker binary will also have to install Hyperscan.

It would also be really nice for ease-of-distribution if the binary could be linked statically, not requiring any dynamic libraries whatsoever, but that's probably a bigger job.

I'd love to use Nosey Parker for defensive purposes, and being able to filter results to only show secrets which are confirmed valid is essential.

It should be an optional feature, likely occurring during the report command to keep scans performant.

Each rule could have optional section listing URL + parameters / headers to specify what network request to make in order to verify the credential. The slack rule https://github.com/praetorian-inc/noseyparker/blob/74098e8881bd68a6368032b78b56ddad5ecbbad4/crates/noseyparker/data/default/rules/slack.yml could have a section which specifies curl https://slack.com/api/auth.test -H "Authorization: Bearer $(1)" as the request to make. With $(1) as a placeholder for the first capture group from the regex.

Many project re-invent their own request / response matching format, very curious if you might be able to integrate an existing system like https://hurl.dev with a flexible way to test a response to confirm a valid secret.

The prebuilt releases could include man pages generated from the CLI documentation.

See clap_mangen. This would possibly need to be called from the build.rs script. Then, in the scripts/create_release.zsh script, we would want to copy those generated man pages into somewhere like release/share/man/1/.

Something like scan -d DATASTORE --git-url https://github.com/python/cpython should work. This would automatically clone the requested repo and then scan it.

A new clones subdirectory within the datastore can be used for storing automatically cloned repos.

An automatically cloned repo should be cloned using the --mirror option, which pulls down additional objects compared to a regular git clone invocation.

If a Git URL is requested that was already automatically cloned, the existing clone state should be reused and updated if possible.

Probably the correct default is to not use color output when writing to a file instead of a terminal.

This affects both report and summarize commands.

Currently, Nosey Parker comes in source form, and has an automatically built Docker image available.

I'd like to provide a new distribution mechanism: a prebuilt binary distributed as a single file for popular platforms. To start with:

• GNU x86_64 Linux
• macOS x86_64
• macOS aarch64

The biggest obstacle to this currently is the use of Hyperscan in Rust:

• Hyperscan proper only builds on x86, but the Vectorscan fork of that project builds on other platforms (see #5)
• When installed from system package managers, static linking against Hyperscan/Vectorscan doesn't work as expected (there is still a dynamic dependency on libhs; see #22)
• Building Hyperscan or Vectorscan from source is nontrivial, requiring several additional dependencies (ragel, cmake, boost, ...)

A few commands enumerate GitHub repositories:

• scan --github-user=USER
• scan --github-org=ORG
• github repos list --user=USER
• github repos list --org=ORG

These currently do not offer any filtering mechanism on the set of resulting repo URLs. It would be useful to have an option to ignore forked repos.

The Docker images that are currently build in GitHub Actions are built only for x86_64. When someone runs one of these on a different architecture, such as ARM, some kind of binary translation or runtime emulation is performed to get the image running.

It would be better if instead of relying on emulation, Nosey Parker provided multi-architecture Docker images.

Hello I'm probably doing something wrong here, but when I run NoseyParker I'm getting errors related to SSL certificates. When I search for solutions, most mention corporate firewalls or importing CA certificates - both of which seem too involved given I'm running this on a home network and it's trying to connect to api.github.com - any help/suggestions would be appreciated.

There are certain rule types in particular where it would be helpful to have an expressive postprocessing/filtering mechanism. For example:

• Matches from the JWT rule could be postprocessed to decode the JWT and only keep those that have no expiration time
• Matches from PEM-encoded keys could be postprocessed to filter out matches where the payload doesn't PEM-decode

Hyperscan only supports x86. It has been forked, however, to support ARM and other architectures as well: https://github.com/vectorcamp/vectorscan

I experimented in a local copy on an M1 MacBook Pro, and was able to get Nosey Parker building and running there using vectorscan instead of hyperscan. The build process for that experiment was rather manual:

• Install boost, ragel, cmake, etc in order to build vectorscan from source
• Build vectorscan
• Ensure vectorscan's test suite passed
• export HYPERSCAN_ROOT=$PATH_TO_VECTORSCAN_BUILD (causes the hyperscan-sys crate to use vectorscan instead)
• cargo build --release for Nosey Parker

After this process, I was able to run Nosey Parker on the M1 MacBook Pro, and it seemed to behave as expected. The resulting binary was also statically linked against vectorscan, and didn't have a dynamic dependency on libhs.

I'd like to figure out how to streamline this build process so that nothing more than a cargo build in Nosey Parker is required.

The prebuilt releases of Nosey Parker should also include pregenerated shell completion scripts (the support for which was added in #76).

It would be useful for troubleshooting if Nosey Parker's -V command, used to show version info, would include more detail than just the declared version number, such as Git commit hash.

A library that would be helpful for implementing this: https://docs.rs/vergen/latest/vergen/

It would also be useful to add a top-level version command that would emit more information than the simple -V option.

The example snippets and usage instructions in the README are out-of-date now. They could also stand to be generally improved.

Some ideal properties of the example usage snippets:

• they can be automatically regenerated from a script
• they include short animation screen captures, in addition to static terminal command/output text

Useful references:

• Make Your CLI Demos a Breeze with Zero Stress and Zero Mistakes
• https://github.com/paxtonhare/demo-magic
• https://github.com/faressoft/terminalizer

Looks like it might have gotten forgotten. The readme and help mentions it, but attempting to use it produces:

Error: The `report` command currently only supports the `human` output format.
Support for other formats is coming soon.

Like #15, this was also demoed at Black Hat EU 2022.

When a match is found within a blob in a Git repository, detailed provenance information should be reported, including:

• The commit(s) that first introduced the match, along with author, date, and commit message
• The path(s) that the introducing blob appeared as
• The repository origin URL, if available

With all this information, it is possible to generate permalinks to GitHub for matches.

Compiling the rules into a Vectorscan database currently is very expensive in debug builds, where I see it taking some 6 seconds. In release builds, the compilation is much faster (perhaps 1/3 of a second). The slow compilation in debug builds is especially irksome, because that's the development configuration where you'd like to be able to quickly iterate.

It would be ideal if, instead of compiling the rules database from scratch each run, Nosey Parker could load a precompiled rules database.

Last I looked at Vectorscan, there were APIs for serialization of databases, but the serialized format did not sound to be machine-independent. So it would be difficult to precompile that database and include it as an asset in the release binary.

An alternative would be to have Nosey Parker optionally (and by default) look in a special location at runtime for a machine-specific precompiled rule file that it could load, and generate that if needed. Though this approach is fraught with its own perils, such as: what if the notable location ends up in a network-mounted home directory shared across systems?

Currently, Nosey Parker rules are just a bag of rules, undifferentiated from each other in terms of severity or the kind of thing they detect.

As noted by @CameronLonsdale in #52 (comment):

Another useful category would be whether the finding is for a "secret" like an API key, or something more informational like an s3 bucket name (https://github.com/praetorian-inc/noseyparker/blob/main/crates/noseyparker/data/default/rules/aws.yml#L120-L121)

This is related to #51: if we had useful metadata attached to each rule, we could use that for filtering.

I propose adding a category field to each rule. This field will have a list of values of notable types. I'm not quite sure what the taxonomy should be, but some ideas:

• secret: indicating that the rule detects things that are in fact secrets (e.g., GitHub Personal Access Token). These are the most severe, as they can give an attacker nearly immediate access to additional resources.
• identifier: indicating that the rule detects things that are not secrets but could be used by an attacker to enumerate additional resources (e.g., AWS API Key, S3 buckets).
• hashed, encrypted: indicating that the rule detects things that are hashed or encrypted payloads (e.g., bcrypt hashes). Things like password hashes probably shouldn't be leaked, as an attacker could use them to brute force credentials.
• test: indicating that the rule detects things that are explicitly used for test deployments (e.g., stripe test keys)
• api: indicating that the rule detects things that are API tokens

The scan command currently has a --ignore FILENAME option, which allows one to specify a gitignore-style rules files for paths to ignore when scanning. Those ignore rules are only applied to plain files that are scanned, and not blobs found within Git repositories. Those rules should also apply to Git blobs.

This is probably dependent on #16 being completed first.

The Docker workflow added in #11 builds and publishes a Docker image for Nosey Parker. That workflow runs totally independent of the other workflows, including the testing workflow.

Let's make the Docker workflows run only if tests are first successful.

References:

• https://stackoverflow.com/a/68078768
• https://docs.github.com/en/actions/using-workflows/reusing-workflows

report --format json and report --format jsonl emit output in a currently unspecified JSON/JSON Lines format. It would be helpful for downstream users if these formats were documented with a schema.