renovatebot / osv-offline Goto Github PK

View Code? Open in Web Editor NEW

10.0 3.0 5.0 3.37 MB

A collection of packages for using GitHub security advisories in Node.js.

License: MIT License

TypeScript 92.59% Dockerfile 1.75% JavaScript 5.66%

ghsa github-security-advisories open-source-vulnerabilities osv

osv-offline's Introduction

osv-offline

A collection of packages for using Open Source Vulnerabilities in Node.js.

Packages

Name	Version
`@renovatebot/osv-offline`
`@renovatebot/osv-offline-db`
`@renovatebot/osv-offline-updater`

License

All packages in this repository are licensed under the MIT license.

Vulnerabilities are provided from the following upstream data sources and licenses:

osv-offline's People

Stargazers

Watchers

Forkers

aboodvcg churro gabriel-ladzaretti alanwang611 jporzucek

osv-offline's Issues

Migrate to github/advisory-database

https://github.com/github/advisory-database/

Use `ghcr.io/containerbase/devcontainer` image

Add useful information to `throw`s

See renovatebot/renovate#22502

osvVulnerabilityAlerts - Connect Timeout Error behind HTTP proxy

Issue created following the discussion renovatebot/renovate#20876

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

35.1.1

If you're self-hosting Renovate, select which platform you are using.

GitLab self-hosted

If you're self-hosting Renovate, tell us what version of the platform you run.

15.8.2

Was this something which used to work for you, and then stopped?

It used to work, and then stopped

Wanted end result.

After upgrading to renovate >= v35.x , osvVulnerabilityAlerts feature doesn't work anymore and there is only a warn log (even with trace log level): Unable to read vulnerability information.
It work correctly on version <= 34.160.0.
I'm behinh an HTTP proxy.

What you tried so far.

With adding debug logs, I see there is an error when calling fetch nodejs api: throw ConnectTimeoutError: Connect Timeout Error.
I'm behind an HTTP proxy with these env vars:

export HTTP_PROXY=http://[REDACTED]:3131
export HTTPS_PROXY=http://[REDACTED]:3131
export http_proxy=http://[REDACTED]:3131
export https_proxy=http://[REDACTED]:3131

All works with my HTTP proxy with renovate < v35.x.

I see that renovatebot/osv-offline use Octokit api to download osv database.
There is a new fetch api in Node18 and Octokit use this native api when available:
octokit/request.js@d000a0a

And Node have been upgrade from v16 to Node v18 in Renovate Docker image since v35:
renovatebot/docker-renovate@f367ca3

I think my problem is because fetch native api is now used in Renovate Docker image >= v35 and don't work by default with HTTP proxy env vars (nodejs/undici#1650).
It seems to be possible to configure Octokit used by renovatebot/osv-offline (https://github.com/renovatebot/osv-offline/blob/main/packages/osv-offline/src/lib/download.ts#L44) with an HttpProxyAgent :
https://github.com/octokit/core.js/blob/main/test/agent-proxy/agent-proxy-test.test.ts#L62

Relevant debug logs

Logs

TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11413:11)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at tryDownloadDb (/opt/buildpack/tools/renovate/35.0.1/node_modules/@renovatebot/osv-offline/dist/lib/download.js:40:26)
    at OsvOffline.initialize (/opt/buildpack/tools/renovate/35.0.1/node_modules/@renovatebot/osv-offline/dist/lib/osv-offline.js:13:25)
    at Function.create (/opt/buildpack/tools/renovate/35.0.1/node_modules/@renovatebot/osv-offline/dist/lib/osv-offline.js:25:9)
    at Vulnerabilities.initialize (/opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/workers/repository/process/vulnerabilities.ts:44:23)
    at Function.create (/opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/workers/repository/process/vulnerabilities.ts:49:5)
    at fetchVulnerabilities (/opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/workers/repository/process/extract-update.ts:177:31)
    at lookup (/opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/workers/repository/process/extract-update.ts:189:3)
    at extractDependencies (/opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/workers/repository/process/index.ts:131:31)
    at Object.renovateRepository (/opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/workers/repository/index.ts:56:9)
    at attributes.repository (/opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/workers/global/index.ts:181:11)
    at start (/opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/workers/global/index.ts:166:7)
    at /opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/renovate.ts:18:22 {
  cause: ConnectTimeoutError: Connect Timeout Error
      at onConnectTimeout (node:internal/deps/undici/undici:8380:28)
      at node:internal/deps/undici/undici:8338:50
      at Immediate._onImmediate (node:internal/deps/undici/undici:8369:13)
      at processImmediate (node:internal/timers:476:21) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}

RequestError [HttpError]: fetch failed
    at /opt/buildpack/tools/renovate/35.0.1/node_modules/@octokit/request/dist-src/fetch-wrapper.js:96:15
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at tryDownloadDb (/opt/buildpack/tools/renovate/35.0.1/node_modules/@renovatebot/osv-offline/dist/lib/download.js:40:26)
    at OsvOffline.initialize (/opt/buildpack/tools/renovate/35.0.1/node_modules/@renovatebot/osv-offline/dist/lib/osv-offline.js:13:25)
    at Function.create (/opt/buildpack/tools/renovate/35.0.1/node_modules/@renovatebot/osv-offline/dist/lib/osv-offline.js:25:9)
    at Vulnerabilities.initialize (/opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/workers/repository/process/vulnerabilities.ts:44:23)
    at Function.create (/opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/workers/repository/process/vulnerabilities.ts:49:5)
    at fetchVulnerabilities (/opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/workers/repository/process/extract-update.ts:177:31)
    at lookup (/opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/workers/repository/process/extract-update.ts:189:3)
    at extractDependencies (/opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/workers/repository/process/index.ts:131:31)
    at Object.renovateRepository (/opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/workers/repository/index.ts:56:9)
    at attributes.repository (/opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/workers/global/index.ts:181:11)
    at start (/opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/workers/global/index.ts:166:7)
    at /opt/buildpack/tools/renovate/35.0.1/node_modules/renovate/lib/renovate.ts:18:22 {
  status: 500,
  request: {
    method: 'GET',
    url: 'https://api.github.com/repos/renovatebot/osv-offline/releases',
    headers: {
      accept: 'application/vnd.github.v3+json',
      'user-agent': 'octokit-rest.js/19.0.7 octokit-core.js/4.2.0 Node.js/18.15.0 (linux; x64)',
      authorization: 'token [REDACTED]'
    },
    request: { hook: [Function: bound bound register] }
  }
}

Publish `osv-offline` with bundled databases

Make use of npm's prePublishOnly scripts to publish to the npm registry with a copy of the OSV databases. That way, they can be used as a fallback of last resort if there are network connectivity issues, GitHub rate limiting, or for use in an airgapped environment.

Switch to `pnpm` like out other repos

Update `NPM_TOKEN` secret

The current NPM_TOKEN secret allows publishing in the @jamiemagee npm package scope. After moving the packages to the @renovatebot scope, the secret needs to be updated.

Document real offline mode

For security reasons, we don't have access to Internet on our server where run Renovate bot.
I achieve running OSV scanning with Renovate using this procedure:

export RENOVATE_OSV_VULNERABILITY_ALERTS="true"
export OSV_OFFLINE_ROOT_DIR="/tmp"
export OSV_OFFLINE_DISABLE_DOWNLOAD="true"
curl https://<internal proxy>/osv-offline.zip -o ${OSV_OFFLINE_ROOT_DIR}/osv-offline.zip
unzip ${OSV_OFFLINE_ROOT_DIR}/osv-offline.zip -d ${OSV_OFFLINE_ROOT_DIR}
renovate

I found no documentation about this. Is there an other procedure?

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Pending Approval

These branches will be created by Renovate only once you click their checkbox below.

chore(deps): update linters (major) (@typescript-eslint/eslint-plugin, @typescript-eslint/parser, eslint-plugin-jest)
chore(deps): update mcr.microsoft.com/vscode/devcontainers/typescript-node docker tag to v22
fix(deps): update dependency got to v14
🔐 Create all pending approval PRs at once 🔐

Pending Status Checks

These updates await pending status checks. To force their creation now, click the checkbox below.

chore(deps): update dependency @types/node to v20.12.12
chore(deps): update dependency npm-run-all2 to v6.2.0

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

chore(deps): lock file maintenance

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

Detected dependencies

dockerfile

.devcontainer/Dockerfile

mcr.microsoft.com/vscode/devcontainers/typescript-node 20@sha256:ac09ab0d7edb7cecbeed8ad2dd90bb61aac9e9d6ce9d0c1d026889696de814eb

github-actions

.github/workflows/build.yml

actions/checkout v4.1.6@a5ac7e51b41094c92402da3b24376905380afc29

actions/setup-node v4.0.2@60edb5dd545a775178f52524783378180af0d1f8

.github/workflows/release.yml

actions/checkout v4.1.6@a5ac7e51b41094c92402da3b24376905380afc29

actions/setup-node v4.0.2@60edb5dd545a775178f52524783378180af0d1f8

.github/workflows/update.yml

actions/checkout v4.1.6@a5ac7e51b41094c92402da3b24376905380afc29

actions/setup-node v4.0.2@60edb5dd545a775178f52524783378180af0d1f8

npm

package.json

@qiwi/multi-semantic-release 7.1.1

@tsconfig/node18 18.2.4

@tsconfig/strictest 1.0.2

@types/eslint 8.56.10

@types/jest 29.5.12

@typescript-eslint/eslint-plugin 6.21.0

@typescript-eslint/parser 6.21.0

cross-env 7.0.3

eslint-config-prettier 9.1.0

eslint-plugin-import 2.29.1

eslint-plugin-jest 27.9.0

eslint-plugin-promise 6.1.1

expect-more-jest 5.5.0

jest 29.7.0

jest-extended 4.0.2

npm-run-all2 6.1.2

ts-jest 29.1.2

packages/osv-offline-db/package.json

@seald-io/nedb ^4.0.4

@tsconfig/node18 18.2.4

@tsconfig/strictest 1.0.2

@types/node 20.12.8

prettier 3.2.5

ts-node 10.9.2

typescript 5.4.4

packages/osv-offline-updater/package.json

@octokit/rest 20.1.1

@seald-io/nedb 4.0.4

adm-zip 0.5.12

fs-extra 11.2.0

got 11.8.6

luxon 3.4.4

signale 1.4.0

@tsconfig/node18 18.2.4

@tsconfig/strictest 1.0.2

@types/adm-zip 0.5.5

@types/fs-extra 11.0.4

@types/luxon 3.4.2

@types/node 20.12.8

@types/signale 1.4.7

prettier 3.2.5

ts-node 10.9.2

typescript 5.4.4

packages/osv-offline/package.json

@octokit/rest ^20.1.1

adm-zip ~0.5.12

fs-extra ^11.2.0

got ^11.8.6

luxon ^3.4.4

node-fetch ^2.7.0

@types/adm-zip 0.5.5

@types/fs-extra 11.0.4

@types/luxon 3.4.2

@types/node 20.12.8

@types/node-fetch 2.6.11

prettier 3.2.5

ts-node 10.9.2

typescript 5.4.4

Check this box to trigger a request for Renovate to run again on this repository

Upload database action failing

The Upload database has been failing since 8e99a29 on the the 1st April. The error is:

/home/runner/work/osv-offline/osv-offline/node_modules/ts-node/dist/util.js:62
    return value.replace(backslashRegExp, directorySeparator);
                 ^

TypeError: value.replace is not a function
    at normalizeSlashes (/home/runner/work/osv-offline/osv-offline/node_modules/ts-node/dist/util.js:62:18)
    at Object.getExtendsConfigPath (/home/runner/work/osv-offline/osv-offline/node_modules/ts-node/dist/ts-internals.js:24:54)
    at readConfig (/home/runner/work/osv-offline/osv-offline/node_modules/ts-node/dist/configuration.js:127:64)
    at findAndReadConfig (/home/runner/work/osv-offline/osv-offline/node_modules/ts-node/dist/configuration.js:50:84)
    at phase3 (/home/runner/work/osv-offline/osv-offline/node_modules/ts-node/dist/bin.js:254:67)
    at bootstrap (/home/runner/work/osv-offline/osv-offline/node_modules/ts-node/dist/bin.js:47:30)
    at main (/home/runner/work/osv-offline/osv-offline/node_modules/ts-node/dist/bin.js:33:12)
    at Object.<anonymous> (/home/runner/work/osv-offline/osv-offline/node_modules/ts-node/dist/bin.js:579:5)
    at Module._compile (node:internal/modules/cjs/loader:1254:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)

and is due to ts-node's lack of support for TypeScript 5.0's extends array.

This was resolved in TypeStrong/ts-node#1958 but is not yet available in a release. The two options are to:

Rollback to a single extends
Use ts-node from GitHub instead of npm

Implement OSV schema v1.5.0

See https://github.com/ossf/osv-schema/releases/tag/v1.5.0