vulsio / goval-dictionary Goto Github PK

View Code? Open in Web Editor NEW

90.0 90.0 59.0 1.83 MB

Build a local copy of OVAL. Server mode for easy querying.

License: Apache License 2.0

Go 92.78% Makefile 3.88% Dockerfile 0.20% Python 3.15%

goval-dictionary's People

Stargazers

Watchers

Forkers

pombredanne sadayuki-matsuno ymomoi yezhiyuan mai346 cstackpole tsu-root knqyf263 milk iwamatsu ibuystuff valimail iveskins sudnonk dmp42 directionless shigechika masahiro331 j4ckzh0u wagdez ldelossa joycedjamila pkrzyzan3356 shopper29 segatomo 5l1v3r1 hubspot mainek00n mrquiet vincemacbuche cybersift normation medasz dogasantos etzhangsx idanski yuanbing1113 tramora romav1 tttfrfr2 amankumarchagti evict wagde-orca martin-koerner tokyowizard maito1201 isabella232 sthagen ottijp awilson-urbane yusuke-koyoshi infinipoint sahar042 wxr343 erdalga smileqiny ksawai996 gandikotaajay

goval-dictionary's Issues

security in docker : use an applicative user instead of root to run goval-dictionary

Hi,
Currently (in version 0.2.11 at least) the process runs under root :

    1 root      0:00 goval-dictionary server -bind=0.0.0.0 -dbpath /tmp/oval.sqlite3

It seems better to create and use an applicative user instead.

Failed to HMGet. Some fields do not exist. continue scanning on redis-backend

What did you do? (required. The issue will be closed when not provided.)

If the definition body corresponding to the definitionID held in the secondary index(Hash) is empty, it should be deleted from the secondary index.

Implement the TODO.
https://github.com/vulsio/goval-dictionary/blob/master/db/redis.go#L191
https://github.com/vulsio/goval-dictionary/blob/master/db/redis.go#L231

Feature Request: Added "Accept-Encoding" header to HTTP GET request

SUSE and Debian's OVAL can be gzipped on the server side by giving "Accept-Encoding: gzip, deflate" header.

OVAL entries are not found for Amazon Linux 1 and 2

I scanned amazon 2 (Karoo) , and execute vuls report.
Then it says

[May 26 18:01:42] ERROR [localhost] Failed to fill with OVAL:
    github.com/future-architect/vuls/report.FillCveInfo
        /root/go/src/github.com/future-architect/vuls/report/report.go:162
  - OVAL entries of amazon 2 (Karoo) are not found. Fetch OVAL before reporting. For details, see `https://github.com/kotakanbe/goval-dictionary#usage`:
    github.com/future-architect/vuls/report.FillWithOval
        /root/go/src/github.com/future-architect/vuls/report/report.go:332

This is the same in Amazon Linux 2015.9.

I'm using goval-directory which I cloned at 2020/05/18, and go version is 1.14.2.

I think the problem is in database.
Because func getAmazonLinux1or2 returns "1" or "2", but in roots table, the os_version column is "0".

# sqlite3 oval.sqlite3
sqlite> .schema roots
CREATE TABLE "roots" ("id" integer primary key autoincrement,"family" varchar(255),"os_version" varchar(255),"timestamp" datetime );
sqlite> select * from roots;
~~~snip~~~
8|redhat|8|2020-05-18 19:35:51.733595417+09:00
13|amazon|0|2020-05-26 17:37:23.183687508+09:00

I tried goval-directory fetch-amazon 1 and goval-directory fetch-amazon 2, but that column still remains "0".
After changing os_version manually to "2", vuls report works fine.

Trying to build, receiving error about labstack/echo/engine/standard

I'm attempting to build this manually as I'm running it in a Docker Container which runs Alpine Linux.

When I attempt a go get or the build, I get:

alpine:/src/github.com/kotakanbe/goval-dictionary-0.1.0# go get
package github.com/labstack/echo/engine/standard: cannot find package "github.com/labstack/echo/engine/standard" in any of:
	/usr/lib/go/src/github.com/labstack/echo/engine/standard (from $GOROOT)
	/src/github.com/labstack/echo/engine/standard (from $GOPATH)

I attempted to download the previous version of labstack/echo with no success as well.

MySQL Error

Hi,

Got this on storing the data in a standard MariaDB 10.2 server:
goval-dictionary fetch-redhat -dbtype mysql -dbpath "vuls:test@tcp(localhost:3306)/ovalDict?parseTime=true" 6
INFO[01-30|15:13:02] Fetching... URL=https://www.redhat.com/security/data/oval/com.redhat.rhsa-RHEL6.xml.bz2
INFO[01-30|15:13:02] Fetched... URL=https://www.redhat.com/security/data/oval/com.redhat.rhsa-RHEL6.xml.bz2
INFO[01-30|15:13:02] Finished fetching OVAL definitions
INFO[01-30|15:13:03] Fetched URL=https://www.redhat.com/security/data/oval/com.redhat.rhsa-RHEL6.xml.bz2 OVAL definitions=1075
INFO[01-30|15:13:03] Refreshing... Family=redhat Version=6
EROR[01-30|15:13:06] Failed to insert oval err="Failed to insert. cve: &models.Root{\n ID: 0x2,\n Family: "redhat",\n OSVersion: "6",\n Definitions: []models.Definition{...},\n Timestamp: 2019-01-30 15:13:03 Local,\n}, err: Error 1366: Incorrect string value: '\xC4\x8D of ...' for column 'description' at row 1"

/Theo

refreshing OVAL definitions taking longer

hi,

while updating the new oval data: the refreshing taking too long. Is this an issue? How long does it take to refresh the definitions normally?

Travis CI integration

#32

Support Amazon 22

Amazon released a new amazon linux version (amazon 22)
and they have ALAS
https://alas.aws.amazon.com/alas2022.html
need to fetch it as amazon 1 and amazon 2

Arch linux support

Hello how hard will be to modify a current fetch for arch?

docker volume oddity

Hi. I'm not sure I've wholly diagnosed this correctly. but the docker images seem to have some odd files around the mounts. These end up confusing docker-compose.

Specifically, you've got the line VOLUME [$WORKDIR, $LOGDIR] in the dockerfile, and see a /[vuls directory. When I try to use docker-compose, it complains about [/vuls not being absolute.

~~I suspect this means something that did the build/upload to dockerhub is out of date.~~ This looks like it's mis-specified

$ docker --version
Docker version 18.09.3, build 774a1f4

$ docker run --rm -it     --entrypoint ""   vuls/goval-dictionary  ls /
[      dev    home   media  proc   run    srv    tmp    var
bin    etc    lib    mnt    root   sbin   sys    usr    vuls

$ docker run --rm -it     --entrypoint ""   vuls/goval-dictionary  ls '/['
vuls,

I see this in all the vuls images I've looked at

Panic if unable to create the log file

panic: open /var/log/vuls/goval-dictionary.log: permission denied

goroutine 1 [running]:
github.com/inconshreveable/log15.must(...)
        /home/ubuntu/go/pkg/mod/github.com/inconshreveable/[email protected]/handler.go:340
github.com/inconshreveable/log15.muster.FileHandler(0xc000091680, 0x22, 0xc0cde0, 0xb5ecf0, 0x22, 0x0)
        /home/ubuntu/go/pkg/mod/github.com/inconshreveable/[email protected]/handler.go:348 +0x8a
github.com/kotakanbe/goval-dictionary/util.SetLogger(0xb3649e, 0xd, 0xc000000000)
        /home/ubuntu/go/src/github.com/kotakanbe/goval-dictionary/util/util.go:62 +0x227
github.com/kotakanbe/goval-dictionary/commands.(*FetchDebianCmd).Execute(0xc000257450, 0xc184e0, 0xc000092000, 0xc00027c540, 0x0, 0x0, 0x0, 0x0)
        /home/ubuntu/go/src/github.com/kotakanbe/goval-dictionary/commands/fetch-debian.go:97 +0x10a
github.com/google/subcommands.(*Commander).Execute(0xc0000a8000, 0xc184e0, 0xc000092000, 0x0, 0x0, 0x0, 0xc0002c6580)
        /home/ubuntu/go/pkg/mod/github.com/google/[email protected]/subcommands.go:142 +0x2f9
github.com/google/subcommands.Execute(...)
        /home/ubuntu/go/pkg/mod/github.com/google/[email protected]/subcommands.go:420
main.main()
        /home/ubuntu/go/src/github.com/kotakanbe/goval-dictionary/main.go:52 +0x463```

Change in SUSE OVAL layout breaks parsing for some advisories

This is in turn causing a lot of SUSE definitions to be tied to an entry in the 'roots' table that is labeled suse.linux.enterprise.server but has a version of '' (empty string).

I've included the section of a sample advisory (oval:org.opensuse.security:def:20149939) taken from suse.linux.enterprise.server.12.xml. The working one is from 2018/11/19, and the broken one from 2019/05/22.

I think the format change happened in the last couple months but I don't have other copies of the OVAL to properly bisect it :(

BTW, I will mention in passing that considering -LTSS to be the same as regular SUSE does not seem correct. They have different patched version numbers. This is what is doing it: https://github.com/kotakanbe/goval-dictionary/blob/fd8ff5a6343912117d1b7db16fbd5fa1f4116c3a/models/suse.go#L71

Working - 2018/11/19

  <criteria operator="OR">
    <criteria operator="AND">
      <criterion test_ref="oval:org.opensuse.security:tst:2009118803" comment="SUSE Linux Enterprise Server 12 SP1-LTSS is installed"/>
        <criterion test_ref="oval:org.opensuse.security:tst:2009237410" comment="binutils-2.31-9.26 is installed"/>
    </criteria>
    <criteria operator="AND">
      <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server 12 SP2 is installed"/>
        <criterion test_ref="oval:org.opensuse.security:tst:2009186658" comment="binutils-2.29.1-9.20 is installed"/>
    </criteria>
    <criteria operator="AND">
      <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server 12 SP2-BCL is installed"/>
        <criterion test_ref="oval:org.opensuse.security:tst:2009237410" comment="binutils-2.31-9.26 is installed"/>
    </criteria>
    <criteria operator="AND">
      <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server 12 SP2-ESPOS is installed"/>
        <criterion test_ref="oval:org.opensuse.security:tst:2009237410" comment="binutils-2.31-9.26 is installed"/>
    </criteria>
    <criteria operator="AND">
      <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server 12 SP2-LTSS is installed"/>
        <criterion test_ref="oval:org.opensuse.security:tst:2009237410" comment="binutils-2.31-9.26 is installed"/>
    </criteria>
    <criteria operator="AND">
      <criterion test_ref="oval:org.opensuse.security:tst:2009158922" comment="SUSE Linux Enterprise Server 12 SP3 is installed"/>
        <criterion test_ref="oval:org.opensuse.security:tst:2009237410" comment="binutils-2.31-9.26 is installed"/>
    </criteria>
    <criteria operator="AND">
      <criterion test_ref="oval:org.opensuse.security:tst:2009116126" comment="SUSE Linux Enterprise Server 12-LTSS is installed"/>
        <criterion test_ref="oval:org.opensuse.security:tst:2009237410" comment="binutils-2.31-9.26 is installed"/>
    </criteria>
    <criteria operator="AND">
      <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server for Raspberry Pi 12 SP2 is installed"/>
        <criterion test_ref="oval:org.opensuse.security:tst:2009186658" comment="binutils-2.29.1-9.20 is installed"/>
    </criteria>
    <criteria operator="AND">
      <criterion test_ref="oval:org.opensuse.security:tst:2009118803" comment="SUSE Linux Enterprise Server for SAP Applications 12 SP1 is installed"/>
        <criterion test_ref="oval:org.opensuse.security:tst:2009237410" comment="binutils-2.31-9.26 is installed"/>
    </criteria>
    <criteria operator="AND">
      <criterion test_ref="oval:org.opensuse.security:tst:2009118803" comment="SUSE Linux Enterprise Server for SAP Applications 12 SP1-LTSS is installed"/>
        <criterion test_ref="oval:org.opensuse.security:tst:2009237410" comment="binutils-2.31-9.26 is installed"/>
    </criteria>
    <criteria operator="AND">
      <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server for SAP Applications 12 SP2 is installed"/>
        <criterion test_ref="oval:org.opensuse.security:tst:2009186658" comment="binutils-2.29.1-9.20 is installed"/>
    </criteria>
    <criteria operator="AND">
      <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server for SAP Applications 12 SP2-BCL is installed"/>
        <criterion test_ref="oval:org.opensuse.security:tst:2009237410" comment="binutils-2.31-9.26 is installed"/>
    </criteria>
    <criteria operator="AND">
      <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server for SAP Applications 12 SP2-ESPOS is installed"/>
        <criterion test_ref="oval:org.opensuse.security:tst:2009237410" comment="binutils-2.31-9.26 is installed"/>
    </criteria>
    <criteria operator="AND">
      <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server for SAP Applications 12 SP2-LTSS is installed"/>
        <criterion test_ref="oval:org.opensuse.security:tst:2009237410" comment="binutils-2.31-9.26 is installed"/>
    </criteria>
    <criteria operator="AND">
      <criterion test_ref="oval:org.opensuse.security:tst:2009158922" comment="SUSE Linux Enterprise Server for SAP Applications 12 SP3 is installed"/>
        <criterion test_ref="oval:org.opensuse.security:tst:2009237410" comment="binutils-2.31-9.26 is installed"/>
    </criteria>
    <criteria operator="AND">
      <criterion test_ref="oval:org.opensuse.security:tst:2009116126" comment="SUSE Linux Enterprise Server for SAP Applications 12-LTSS is installed"/>
        <criterion test_ref="oval:org.opensuse.security:tst:2009237410" comment="binutils-2.31-9.26 is installed"/>
    </criteria>
  </criteria>

Broken - 2019/05/22

        <criteria operator="OR">
                <criteria operator="AND">
                        <criteria operator="OR">
                                <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server 12 SP2 is installed"/>
                                <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server for Raspberry Pi 12 SP2 is installed"/>
                                <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server for SAP Applications 12 SP2 is installed"/>
                        </criteria>
                        <criterion test_ref="oval:org.opensuse.security:tst:2009186658" comment="binutils-2.29.1-9.20 is installed"/>
                </criteria>
                <criteria operator="AND">
                        <criteria operator="OR">
                                <criterion test_ref="oval:org.opensuse.security:tst:2009118803" comment="SUSE Linux Enterprise Server 12 SP1-LTSS is installed"/>
                                <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server 12 SP2-BCL is installed"/>
                                <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server 12 SP2-ESPOS is installed"/>
                                <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server 12 SP2-LTSS is installed"/>
                                <criterion test_ref="oval:org.opensuse.security:tst:2009158922" comment="SUSE Linux Enterprise Server 12 SP3 is installed"/>
                                <criterion test_ref="oval:org.opensuse.security:tst:2009158922" comment="SUSE Linux Enterprise Server 12 SP3-TERADATA is installed"/>
                                <criterion test_ref="oval:org.opensuse.security:tst:2009240280" comment="SUSE Linux Enterprise Server 12 SP4 is installed"/>
                                <criterion test_ref="oval:org.opensuse.security:tst:2009116126" comment="SUSE Linux Enterprise Server 12-LTSS is installed"/>
                                <criterion test_ref="oval:org.opensuse.security:tst:2009118803" comment="SUSE Linux Enterprise Server for SAP Applications 12 SP1 is installed"/>
                                <criterion test_ref="oval:org.opensuse.security:tst:2009118803" comment="SUSE Linux Enterprise Server for SAP Applications 12 SP1-LTSS is installed"/>
                                <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server for SAP Applications 12 SP2-BCL is installed"/>
                                <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server for SAP Applications 12 SP2-ESPOS is installed"/>
                                <criterion test_ref="oval:org.opensuse.security:tst:2009128018" comment="SUSE Linux Enterprise Server for SAP Applications 12 SP2-LTSS is installed"/>
                                <criterion test_ref="oval:org.opensuse.security:tst:2009158922" comment="SUSE Linux Enterprise Server for SAP Applications 12 SP3 is installed"/>
                                <criterion test_ref="oval:org.opensuse.security:tst:2009158922" comment="SUSE Linux Enterprise Server for SAP Applications 12 SP3-TERADATA is installed"/>
                                <criterion test_ref="oval:org.opensuse.security:tst:2009116126" comment="SUSE Linux Enterprise Server for SAP Applications 12-LTSS is installed"/>
                        </criteria>
                        <criterion test_ref="oval:org.opensuse.security:tst:2009237410" comment="binutils-2.31-9.26 is installed"/>
                </criteria>
        </criteria>

I uploaded the full OVAL files here if you need them: https://drive.google.com/open?id=1SqwUS2bHQ_9yE6hPbFatllV834K3pJxP

Thanks!

Redis driver should not rely on `references` to get cve ID

Redis driver currently rely on def.References to get CVE IDs.
However, this field is not mandatory in the spec (See 4.3.4 MetadataType of the oval spec. Only title and description are mandatory, reference has a cardinality of 0..*.

The main issue with relying on it to get CVE IDs is that some Debian OVAL files do not provide it, instead they use debian/moreinfo field to provide description (which is not in spec at all, I agree).

So, relying on the reference fields currently make fetch-debian 7 8 to fail silently as no CVE are found, while it works perfectly with rdb based backends (took me a few hours of debug btw, but it led me here ).

Could you at least fallback to def.Debian.CveID in case def.References is empty?

I understand redis driver is trying to be distribution agnostic, while rdb drivers are per-distribution, but it looks like the model is not generic enough to provide such level of abstraction, so I see no other way of doing this until there is a generic cveid in model.

fetch-ubuntu fails for versions 12 and 19

What did you do?

There are two issues with the fetch-ubuntu functionality. The first is that version 12 is no longer supported, and so the download fails. The second is that the config is storing an old release name for 19.

# checked help doc for ubuntu
goval-dictionary fetch-ubuntu --help

# ran suggested command from help doc
goval-dictionary 12 14 16 18 19 20

What did you expect to happen?

I expected all of them to be downloaded correctly

What happened instead?

Installation for 12 and 19 failed.
12 failed because it's no longer supported
19 failed because the release name in the config file is no longer accurate

Current Output

Please re-run the command using -debug and provide the output below.

➜  goval-dictionary git:(update-fetch-versions) ./goval-dictionary fetch-ubuntu 12 -debug
EROR[06-11|01:17:18] Failed to create log directory           err="mkdir /var/log/vuls: permission denied"
WARN[06-11|01:17:18] Skip unknown ubuntu.                     version=-debug
INFO[06-11|01:17:18] Fetching...                              URL=https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.precise.cve.oval.xml.bz2
INFO[06-11|01:17:19] Finished fetching OVAL definitions
EROR[06-11|01:17:19] Failed to fetch files                    err="Failed to fetch. err: [aborting: could not write to output stream: Expected HTTP Status 200, received: \"404 Not Found\"]"


➜  goval-dictionary git:(update-fetch-versions) ./goval-dictionary fetch-ubuntu 19 -debug
EROR[06-11|01:17:24] Failed to create log directory           err="mkdir /var/log/vuls: permission denied"
WARN[06-11|01:17:24] Skip unknown ubuntu.                     version=-debug
INFO[06-11|01:17:24] Fetching...                              URL=https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.disco.cve.oval.xml.bz2
INFO[06-11|01:17:24] Finished fetching OVAL definitions
EROR[06-11|01:17:24] Failed to fetch files                    err="Failed to fetch. err: [aborting: could not write to output stream: Expected HTTP Status 200, received: \"404 Not Found\"]"
➜  goval-dictionary git:(update-fetch-versions)

Steps to reproduce the behaviour

git clone [email protected]:kotakanbe/goval-dictionary.git
make build
./goval-dictionary fetch-ubuntu 12 19

Configuration (MUST fill this out):

n/a

Go version (go version):

go version go1.14.4 darwin/amd64

Go environment (go env):

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/redacted/Library/Caches/go-build"
GOENV="/Users/redacted/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/redacted/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/96/8g5pq9753glcvlyk5jcy0rk00000gq/T/go-build168065861=/tmp/go-build -gno-record-gcc-switches -fno-common"
➜  test git:(init) ✗

False Positive for mysql-5.7

What did you do? (required. The issue will be closed when not provided.)

wget https://people.canonical.com/~ubuntu-security/oval/oci.com.ubuntu.bionic.cve.oval.xml.bz2 bunzip2 oci.com.ubuntu.bionic.cve.oval.xml.bz2
opened oci.com.ubuntu.bionic.cve.oval.xml file
and examined CVE-2020-14828

What did you expect to happen?

I expected that we ignore this CVE for version 5.7.32,

What happened instead?

but we have it in the oval db
it's a False Positive
you can see there that mysql-5.7 is not vulnerable for this CVE
https://ubuntu.com/security/CVE-2020-14828

I think we have an unhandled edge case in the reFixed regexp
https://github.com/kotakanbe/goval-dictionary/blob/af24f653208df1f222cc9368eaf2a021c1edb2ae/models/ubuntu.go#L150
var reFixed = regexp.MustCompile(^(.+) package in .+ has been fixed $note: '([^\s]+).*'$.$)

in the regular cases we have this entry: (for example for CVE-2020-14827)
<criterion test_ref="oval:com.ubuntu.bionic:tst:2020146720000010" comment="mysql-5.7 package in bionic was vulnerable but has been fixed (note: '5.7.32-0ubuntu0.18.04.1')." />

but in the problematic cve (CVE-2020-14828) we see
<criterion test_ref="oval:com.ubuntu.bionic:tst:2020147730000010" comment="mysql-5.7 package in bionic, is related to the CVE in some way and has been fixed (note: '8.0 only')." />
Maybe we need to discard criterions which has 'only' in the note

CentOS lookups return ‘ Unsupport family’ error

I've encountered what seems to be a bug with CentOS when goval-dictionary is operated in server mode (using MySQL as a database).

The following request returns null

curl http://127.0.0.1:1324/packs/centos/7.7.1908/kernel
null

Compare this to the same request, but using redhat instead of centos

curl http://127.0.0.1:1324/packs/redhat/7.7.1908/kernel16:20:15
[
  {
    "ID": 158,
    "DefinitionID": "oval:com.redhat.rhsa:def:20140678",
    ...

Upon looking at the logs, I noticed that anytime I make a request using centos, I get the following error.

EROR[01-28|16:23:19] Failed to get by CveID.                  err="Unsupport family: redhat"

The reason this error seems to be occuring is because when checking if ovalMap has a key of family:
https://github.com/kotakanbe/goval-dictionary/blob/9dfb9028794e2316c3ca45be18199e261c592d5b/db/rdb/rdb.go#L200
It is failing since ovalMap["redhat"] does not exist (ovalMap["centos"] exists instead).

https://github.com/kotakanbe/goval-dictionary/blob/9dfb9028794e2316c3ca45be18199e261c592d5b/db/rdb/rdb.go#L82-L83
Changing the above to the following seems to have fixed the issue for me, however, I'm not sure if this change breaks anything else.

case c.RedHat, c.CentOS:
		ovalMap[c.RedHat] = NewRedHat()

Re-issuing the same HTTP request with the above patch applied produces the expected output.

curl http://127.0.0.1:1324/packs/centos/7.7.1908/kernel
[
  {
    "ID": 158,
    "DefinitionID": "oval:com.redhat.rhsa:def:20140678",
    ...

I would gladly submit a PR, assuming my reading of how to fix this bug is correct.

fetch debian 8 is not working

What did you do? (required. The issue will be closed when not provided.)

goval-dictionary fetch debian 8

What did you expect to happen?

I expect that the db will have CVEs in the cves table

What happened instead?

the cves table has rows but does not have any cve_id

Current Output

Please re-run the command using -debug and provide the output below.

[devenv3] ~/g/s/g/k/goval-dictionary ❯❯❯ ./goval-dictionary fetch debian 8 --debug                                               master ✱ ◼
INFO[11-23|23:05:02] Fetching...                              URL=https://www.debian.org/security/oval/oval-definitions-jessie.xml
INFO[11-23|23:05:03] Fetched...                               URL=https://www.debian.org/security/oval/oval-definitions-jessie.xml
INFO[11-23|23:05:03] Finished fetching OVAL definitions
INFO[11-23|23:05:04] Fetched                                  URL=https://www.debian.org/security/oval/oval-definitions-jessie.xml OVAL definitions=4518
INFO[11-23|23:05:04] Refreshing...                            Family=debian Version=8
INFO[11-23|23:05:04] Inserting new Definitions...
4518 / 4518 [------------------------------------------------------------------------------------------------------------] 100.00% 12898 p/s
INFO[11-23|23:05:04] Finish                                   Updated=4518

Steps to reproduce the behaviour

goval-dictionary fetch debian 8

Configuration (MUST fill this out):

no configuration

panic occurs in select command

What did you do? (required. The issue will be closed when not provided.)

I ran select to check the CVE for the Package. At that time, I referred to README.md, but an error occurred.

What did you expect to happen?

The CVE information about the following packages is displayed.

models.Definition{
    ID:           0x3ff9,
    RootID:       0x1,
    DefinitionID: "oval:org.debian:def:20206504",
    Title:        "CVE-2020-6504",
    Description:  "Insufficient policy enforcement in notifications in Google Chrome prior to 74.0.3729.108 allowed a remote attacker to bypass notification restrictions via a crafted HTML page.",
    Advisory:     models.Advisory{
      ID:              0x0,
      DefinitionID:    0x0,
      Severity:        "",
      Cves:            []models.Cve{},
      Bugzillas:       []models.Bugzilla{},
      AffectedCPEList: []models.Cpe{},
      Issued:          1-01-01 00:00:00 UTC,
      Updated:         1-01-01 00:00:00 UTC,
    },
    Debian: models.Debian{
      ID:           0x3ff9,
      DefinitionID: 0x3ff9,
      CveID:        "CVE-2020-6504",
      MoreInfo:     "",
      Date:         2020-06-25 00:00:00 UTC,
    },
    AffectedPacks: []models.Package{
      models.Package{
        ID:           0x3ff9,
        DefinitionID: 0x3ff9,
        Name:         "chromium",
        Version:      "76.0.3809.100-1~deb10u1",
        Arch:         "",
        NotFixedYet:  false,
      },
    },
    References: []models.Reference{
      models.Reference{
        ID:           0x3d67,
        DefinitionID: 0x3ff9,
        Source:       "CVE",
        RefID:        "CVE-2020-6504",
        RefURL:       "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-6504",
      },
    },

What happened instead?

panic occurred and execution failed.

Current Output

goval-dictionary select -by-package debian 10 chromium
panic: runtime error: index out of range [3] with length 3

goroutine 1 [running]:
github.com/kotakanbe/goval-dictionary/commands.(*SelectCmd).Execute(0xc0002bf630, 0xbf2f00, 0xc000024080, 0xc0002e2660, 0x0, 0x0, 0x0, 0x1040800)
        /home/mainek00n/go/1.14.3/src/github.com/kotakanbe/goval-dictionary/commands/select.go:113 +0x96c
github.com/google/subcommands.(*Commander).Execute(0xc000116000, 0xbf2f00, 0xc000024080, 0x0, 0x0, 0x0, 0xc000326600)
        /home/mainek00n/go/1.14.3/pkg/mod/github.com/google/[email protected]/subcommands.go:142 +0x2f9
github.com/google/subcommands.Execute(...)
        /home/mainek00n/go/1.14.3/pkg/mod/github.com/google/[email protected]/subcommands.go:420
main.main()
        /home/mainek00n/go/1.14.3/src/github.com/kotakanbe/goval-dictionary/main.go:55 +0x461

Steps to reproduce the behaviour

Execute the following command with reference to README.md.

$ goval-dictionary select -by-package debian 10 chromium

Configuration (MUST fill this out):

Go version (go version):

$go version
go version go1.14.3 linux/amd64

Go environment (go env):

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/mainek00n/.cache/go-build"
GOENV="/home/mainek00n/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/mainek00n/go/1.14.3"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/mainek00n/.goenv/versions/1.14.3"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/mainek00n/.goenv/versions/1.14.3/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/mainek00n/github/github.com/MaineK00n/goval-dictionary/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build176697113=/tmp/go-build -gno-record-gcc-switches"

goval-dictionary environment:

$ goval-dictionary -v
goval-dictionary v0.2.8 71b5469

OVAL Fetching for SLES 15 does not work

What did you do? (required. The issue will be closed when not provided.)

goval-dictionary fetch-suse -suse-enterprise-server 15

What did you expect to happen?

Like this output for OpenSuSE:
goval-dictionary fetch-suse -opensuse 13.2
INFO[07-10|06:42:54] Fetching... URL=http://ftp.suse.com/pub/projects/security/oval/opensuse.13.2.xml
INFO[07-10|06:42:55] Fetched... URL=http://ftp.suse.com/pub/projects/security/oval/opensuse.13.2.xml
INFO[07-10|06:42:55] Finished fetching OVAL definitions
INFO[07-10|06:42:56] Fetched URL=http://ftp.suse.com/pub/projects/security/oval/opensuse.13.2.xml OVAL definitions=5169
INFO[07-10|06:42:57] Refreshing... Family=opensuse Version=

What happened instead?

goval-dictionary fetch-suse -suse-enterprise-server 15
INFO[07-10|06:43:24] Fetching... URL=http://ftp.suse.com/pub/projects/security/oval/suse.linux.enterprise.server.15.xml
INFO[07-10|06:43:25] Fetched... URL=http://ftp.suse.com/pub/projects/security/oval/suse.linux.enterprise.server.15.xml
INFO[07-10|06:43:25] Finished fetching OVAL definitions
INFO[07-10|06:43:28] Fetched URL=http://ftp.suse.com/pub/projects/security/oval/suse.linux.enterprise.server.15.xml OVAL definitions=7185
EROR[07-10|06:43:28] Failed to insert oval err="Unsupport family: opensuse"

Please re-run the command using -debug and provide the output below.

goval-dictionary fetch-suse -debug -suse-enterprise-server 15
INFO[07-10|06:57:14] Fetching... URL=http://ftp.suse.com/pub/projects/security/oval/suse.linux.enterprise.server.15.xml
INFO[07-10|06:57:15] Fetched... URL=http://ftp.suse.com/pub/projects/security/oval/suse.linux.enterprise.server.15.xml
INFO[07-10|06:57:15] Finished fetching OVAL definitions
INFO[07-10|06:57:18] Fetched URL=http://ftp.suse.com/pub/projects/security/oval/suse.linux.enterprise.server.15.xml OVAL definitions=7185
EROR[07-10|06:57:18] Failed to insert oval err="Unsupport family: opensuse"

Testing with wget works, file can be downloaded:
wget http://ftp.suse.com/pub/projects/security/oval/suse.linux.enterprise.server.15.xml

Steps to reproduce the behaviour

Try goval-dictionary fetch-suse -suse-enterprise-server 15

Configuration (MUST fill this out):

Go version (go version):
go1.14.4 linux/amd64
Go environment (go env):
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/vuls/.cache/go-build"
GOENV="/home/vuls/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/vuls/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/vuls/go/src/github.com/kotakanbe/goval-dictionary/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build528403967=/tmp/go-build -gno-record-gcc-switches"
Vuls environment:
I can not provide Vuls environment because I am currently try to install Vul :-). I am in section Installation -> Install Manually -> Deploy goval-dictionary.

Because we are using SLES for our SAP Environment I need SLES OVAL for testing and evaluation. Therefore it make no sense to go any further until this bug is fixed.

Hash : ____

To check the commit hash of HEAD
$ vuls -v

$ cd $GOPATH/src/github.com/future-architect/vuls
$ git rev-parse --short HEAD

config.toml:
command:

Mysql backend not working

Hi,

I'm trying to use goval with mysql backend, however, importing oval from, for ex, alpine fails with:

Error 1292: Incorrect datetime value: '0000-00-00' for column 'issued' at row 1"

Has anyone tested Mysql backend recently ?

MySQL database backend write errors?

What did you do? (required. The issue will be closed when not provided.)

I'm running the following command to populate the oval database:

goval-dictionary fetch-ubuntu --dbtype=mysql --dbpath="write:pass@/oval?charset=utf8mb4&parseTime=true&loc=America%2FNew_York" 18

I've already tried without the pareTime options as well.
MySQL information: 5.7.30-0ubuntu0.18.04.1
charset is utf8mb4
I have strict mode disabled.

What did you expect to happen?

I expect to be able to pull OVAL data when I run vuls.

What happened instead?

I'm getting a few different warnings and a fatal error.

Current Output

[Jun 26 20:47:19]  INFO [localhost] Validating config...
INFO[0000] -cvedb-type: mysql, -cvedb-url: user:pass@/cve?charset=utf8mb4, -cvedb-path:  
[Jun 26 20:47:19]  INFO [localhost] Loaded: /usr/share/vuls-data/results/2020-06-26T19:48:21Z
[Jun 26 20:47:19]  INFO [localhost] Validating db config...
INFO[0000] -cvedb-type: mysql, -cvedb-url: user:pass@/cve?charset=utf8mb4, -cvedb-path:  
INFO[0000] -ovaldb-type: mysql, -ovaldb-url: user:pass@/oval?charset=utf8mb4, -ovaldb-path:  
INFO[0000] -gostdb-type: sqlite3, -gostdb-url: , -gostdb-path: /usr/share/vuls-data/gost.sqlite3 
INFO[0000] -exploitdb-type: mysql, -exploitdb-url: user:pass@/exploitdb?charset=utf8mb4, -exploitdb-path:  
[Jun 26 20:47:19]  WARN [localhost] --gostdb-path=/usr/share/vuls-data/gost.sqlite3 file not found. Vuls can detect `patch-not-released-CVE-ID` using gost if the scan target server is Debian, RHEL or CentOS, For details, see `https://github.com/knqyf263/gost#fetch-redhat`
INFO[06-26|20:47:19] Opening Database.                        db=mysql
INFO[06-26|20:47:19] Migrating DB.                            db=mysql
[Jun 26 20:47:19]  INFO [localhost] localhost: 0 CVEs are detected with Library
[Jun 26 20:47:19]  WARN [localhost] OVAL for ubuntu 18.04 is old, last modified is 0001-01-01 00:00:00 +0000 UTC. It's recommended to update OVAL to improve scanning accuracy. How to update OVAL database, see https://github.com/kotakanbe/goval-dictionary#usage
[Jun 26 20:47:19]  WARN [localhost] Unable to detect vulns of running kernel because the version of the runnning kernel is unknown. server: localhost
[Jun 26 20:47:19]  WARN [localhost] The OVAL name of the running kernel image {Release:4.15.0-106-generic Version: RebootRequired:false} is not found. So vulns of `linux` wll be detected. server: localhost
[Jun 26 20:47:19] ERROR [localhost] Failed to fill with OVAL: Failed to get ubuntu OVAL info by package: oval.request{packName:"apparmor", versionRelease:"2.12-4ubuntu5.1", newVersionRelease:"", arch:"", binaryPackNames:[]string(nil), isSrcPack:false}, err: sql: Scan error on column index 4, name "date": unsupported Scan, storing driver.Value type []uint8 into type *time.Time

Please re-run the command using -debug and provide the output below.

goval-dictionary fetch-ubuntu --debug --dbtype=mysql --dbpath="write:pass@/oval?charset=utf8mb4&parseTime=true&loc=America%2FNew_York" 18
INFO[06-26|20:51:18] Fetching...                              URL=https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.bionic.cve.oval.xml.bz2
INFO[06-26|20:51:21] Fetched...                               URL=https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.bionic.cve.oval.xml.bz2
INFO[06-26|20:51:21] Finished fetching OVAL definitions 
INFO[06-26|20:51:23] Fetched                                  URL=https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.bionic.cve.oval.xml.bz2 OVAL definitions=10360
DBUG[06-26|20:51:24] in Ubuntu 
INFO[06-26|20:51:24] Skip (Same Timestamp)                    Family=ubuntu Version=18

Steps to reproduce the behaviour

Run the command to fill the database

Configuration (MUST fill this out):

Go version (go version): go version go1.13 linux/amd64
Go environment (go env):

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build463003783=/tmp/go-build -gno-record-gcc-switches"

Vuls environment:
vuls v0.10.0 build-20200625_154321_89f49b0
config.toml:

[cveDict]
type = "mysql"
url  = "read:pass@/cve?charset=utf8mb4"

[ovalDict]
type = "mysql"
url  = "read:pass@/oval?charset=utf8mb4"

[exploit]
type = "mysql"
url  = "read:pass@/exploitdb?charset=utf8mb4"

[servers]

[servers.localhost]
host = "localhost"
port = "local"
scanMode = [ "fast" ]

command:

goval-dictionary fetch-ubuntu --dbtype=mysql --dbpath="write:pass@/oval?charset=utf8mb4&parseTime=true&loc=America%2FNew_York" 18

Notes

I've tried with the sqlite3 backend it works as intended. I'm hoping to get the MySQL version fixed.

-by-cveid option is not working

What did you do? (required. The issue will be closed when not provided.)

I searched CVE-2020-8169 on ubuntu 20 with -by-cveid.
However, it was not displayed, so when I searched curl with -by-package, CVE-2020-8169 seems to exist.
Is -by-cveid working?

What did you expect to happen?

Use the following command, you can search for OVAL from CVE.

$ goval-dictionary select -by-cveid ubuntu 20 CVE-2020-8169

What happened instead?

Current Output

$ goval-dictionary select -by-cveid ubuntu 20 CVE-2020-8169

$ goval-dictionary select -by-package ubuntu 20 curl x86_64
CRIT[07-30|15:11:20]
                Usage:
                select OVAL by package name
                ./goval-dictionary select -by-package RedHat 7 java-1.7.0-openjdk x86_64

                select OVAL by CVE-ID
                ./goval-dictionary select -by-cveid RedHat 7 CVE-2015-1111

------------------
[]models.Definition{
  models.Definition{
    ID:           0x50bb,
    RootID:       0x2,
    DefinitionID: "oval:com.ubuntu.focal:def:201938230000000",
    Title:        "CVE-2019-3823 on Ubuntu 20.04 (focal) - low.",
    Description:  "libcurl versions from 7.34.0 to before 7.64.0 are vulnerable to a heap out-of-bounds read in the code handling the end-of-response for SMTP. If the buffer passed to `smtp_endofresp()` isn't NUL terminated and contains no character ending the parsed number, and `len` is set to 5, then the `strtol()` call reads beyond the allocated buffer. The read contents will not be returned to the caller.",
    Advisory:     models.Advisory{
      ID:              0x1084,
      DefinitionID:    0x50bb,
      Severity:        "Low",
      Cves:            []models.Cve{},
      Bugzillas:       []models.Bugzilla{},
      AffectedCPEList: []models.Cpe{},
      Issued:          1-01-01 00:00:00 UTC,
      Updated:         1-01-01 00:00:00 UTC,
    },
    Debian: models.Debian{
      ID:           0x50ba,
      DefinitionID: 0x50bb,
      CveID:        "CVE-2019-3823",
      MoreInfo:     "",
      Date:         1-01-01 00:00:00 UTC,
    },
    AffectedPacks: []models.Package{
      models.Package{
        ID:           0x5683,
        DefinitionID: 0x50bb,
        Name:         "curl",
        Version:      "7.64.0-1ubuntu1",
        Arch:         "",
        NotFixedYet:  false,
      },
    },
    References: []models.Reference{
      models.Reference{
        ID:           0x8751,
        DefinitionID: 0x50bb,
        Source:       "CVE",
        RefID:        "CVE-2019-3823",
        RefURL:       "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-3823",
      },
      models.Reference{
        ID:           0x8752,
        DefinitionID: 0x50bb,
        Source:       "Ref",
        RefID:        "",
        RefURL:       "http://people.canonical.com/~ubuntu-security/cve/2019/CVE-2019-3823.html",
      },
      models.Reference{
        ID:           0x8753,
        DefinitionID: 0x50bb,
        Source:       "Ref",
        RefID:        "",
        RefURL:       "https://curl.haxx.se/docs/CVE-2019-3823.html",
      },
      models.Reference{
        ID:           0x8754,
        DefinitionID: 0x50bb,
        Source:       "Ref",
        RefID:        "",
        RefURL:       "https://usn.ubuntu.com/usn/usn-3882-1",
      },
    },
  },
  models.Definition{
    ID:           0x55ac,
    RootID:       0x2,
    DefinitionID: "oval:com.ubuntu.focal:def:201686250000000",
    Title:        "CVE-2016-8625 on Ubuntu 20.04 (focal) - low.",
    Description:  "curl before version 7.51.0 uses outdated IDNA 2003 standard to handle International Domain Names and this may lead users to potentially and unknowingly issue network transfer requests to the wrong host.",
    Advisory:     models.Advisory{
      ID:              0x1575,
      DefinitionID:    0x55ac,
      Severity:        "Low",
      Cves:            []models.Cve{},
      Bugzillas:       []models.Bugzilla{},
      AffectedCPEList: []models.Cpe{},
      Issued:          1-01-01 00:00:00 UTC,
      Updated:         1-01-01 00:00:00 UTC,
    },
    Debian: models.Debian{
      ID:           0x55ab,
      DefinitionID: 0x55ac,
      CveID:        "CVE-2016-8625",
      MoreInfo:     "",
      Date:         1-01-01 00:00:00 UTC,
    },
    AffectedPacks: []models.Package{
      models.Package{
        ID:           0x5d4c,
        DefinitionID: 0x55ac,
        Name:         "curl",
        Version:      "7.55.1-1ubuntu2.1",
        Arch:         "",
        NotFixedYet:  false,
      },
    },
    References: []models.Reference{
      models.Reference{
        ID:           0x9d21,
        DefinitionID: 0x55ac,
        Source:       "CVE",
        RefID:        "CVE-2016-8625",
        RefURL:       "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-8625",
      },
      models.Reference{
        ID:           0x9d22,
        DefinitionID: 0x55ac,
        Source:       "Ref",
        RefID:        "",
        RefURL:       "http://people.canonical.com/~ubuntu-security/cve/2016/CVE-2016-8625.html",
      },
      models.Reference{
        ID:           0x9d23,
        DefinitionID: 0x55ac,
        Source:       "Ref",
        RefID:        "",
        RefURL:       "https://curl.haxx.se/docs/adv_20161102K.html",
      },
    },
  },
  models.Definition{
    ID:           0x56ff,
    RootID:       0x2,
    DefinitionID: "oval:com.ubuntu.focal:def:202081690000000",
    Title:        "CVE-2020-8169 on Ubuntu 20.04 (focal) - medium.",
    Description:  "Partial password leak over DNS on HTTP redirect",
    Advisory:     models.Advisory{
      ID:              0x16c8,
      DefinitionID:    0x56ff,
      Severity:        "Medium",
      Cves:            []models.Cve{},
      Bugzillas:       []models.Bugzilla{},
      AffectedCPEList: []models.Cpe{},
      Issued:          1-01-01 00:00:00 UTC,
      Updated:         1-01-01 00:00:00 UTC,
    },
    Debian: models.Debian{
      ID:           0x56fe,
      DefinitionID: 0x56ff,
      CveID:        "CVE-2020-8169",
      MoreInfo:     "",
      Date:         1-01-01 00:00:00 UTC,
    },
    AffectedPacks: []models.Package{
      models.Package{
        ID:           0x69b6,
        DefinitionID: 0x56ff,
        Name:         "curl",
        Version:      "7.68.0-1ubuntu2.1",
        Arch:         "",
        NotFixedYet:  false,
      },
    },
    References: []models.Reference{
      models.Reference{
        ID:           0xa52f,
        DefinitionID: 0x56ff,
        Source:       "CVE",
        RefID:        "CVE-2020-8169",
        RefURL:       "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8169",
      },
      models.Reference{
        ID:           0xa530,
        DefinitionID: 0x56ff,
        Source:       "Ref",
        RefID:        "",
        RefURL:       "http://people.canonical.com/~ubuntu-security/cve/2020/CVE-2020-8169.html",
      },
      models.Reference{
        ID:           0xa531,
        DefinitionID: 0x56ff,
        Source:       "Ref",
        RefID:        "",
        RefURL:       "https://usn.ubuntu.com/usn/usn-4402-1",
      },
    },
  },
  models.Definition{
    ID:           0x5700,
    RootID:       0x2,
    DefinitionID: "oval:com.ubuntu.focal:def:202081770000000",
    Title:        "CVE-2020-8177 on Ubuntu 20.04 (focal) - medium.",
    Description:  "curl overwrite local file with -J",
    Advisory:     models.Advisory{
      ID:              0x16c9,
      DefinitionID:    0x5700,
      Severity:        "Medium",
      Cves:            []models.Cve{},
      Bugzillas:       []models.Bugzilla{},
      AffectedCPEList: []models.Cpe{},
      Issued:          1-01-01 00:00:00 UTC,
      Updated:         1-01-01 00:00:00 UTC,
    },
    Debian: models.Debian{
      ID:           0x56ff,
      DefinitionID: 0x5700,
      CveID:        "CVE-2020-8177",
      MoreInfo:     "",
      Date:         1-01-01 00:00:00 UTC,
    },
    AffectedPacks: []models.Package{
      models.Package{
        ID:           0x69b7,
        DefinitionID: 0x5700,
        Name:         "curl",
        Version:      "7.68.0-1ubuntu2.1",
        Arch:         "",
        NotFixedYet:  false,
      },
    },
    References: []models.Reference{
      models.Reference{
        ID:           0xa532,
        DefinitionID: 0x5700,
        Source:       "CVE",
        RefID:        "CVE-2020-8177",
        RefURL:       "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8177",
      },
      models.Reference{
        ID:           0xa533,
        DefinitionID: 0x5700,
        Source:       "Ref",
        RefID:        "",
        RefURL:       "http://people.canonical.com/~ubuntu-security/cve/2020/CVE-2020-8177.html",
      },
      models.Reference{
        ID:           0xa534,
        DefinitionID: 0x5700,
        Source:       "Ref",
        RefID:        "",
        RefURL:       "https://usn.ubuntu.com/usn/usn-4402-1",
      },
    },
  },
}

Steps to reproduce the behaviour

goval-dictionary fetch-ubuntu 20
goval-dictionary select -by-cveid ubuntu 20 CVE-2020-8169
goval-dictionary select -by-package ubuntu 20 curl x86_64

Configuration (MUST fill this out):

Go version (go version):

$ go version
go version go1.14.3 linux/amd64

Go environment (go env):

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/mainek00n/.cache/go-build"
GOENV="/home/mainek00n/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/mainek00n/go/1.14.3"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/mainek00n/.goenv/versions/1.14.3"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/mainek00n/.goenv/versions/1.14.3/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/mainek00n/github/github.com/MaineK00n/vuls/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build532110654=/tmp/go-build -gno-record-gcc-switches"

goval-dictionary version

$ goval-dictionary -v
goval-dictionary v0.2.8 8afbb01

can not fetch opensuse-leap and suse-enterprise-server ovals

I tried to fetch oval db for opensuse-leap and -suse-enterprise-server
~/g/s/g/k/goval-dictionary ❯❯❯ ./goval-dictionary fetch-suse -debug -suse-enterprise-server 15
INFO[05-17|18:05:51] Fetching... URL=http://ftp.suse.com/pub/projects/security/oval/suse.linux.enterprise.server.15.xml
INFO[05-17|18:06:08] Fetched... URL=http://ftp.suse.com/pub/projects/security/oval/suse.linux.enterprise.server.15.xml
INFO[05-17|18:06:08] Finished fetching OVAL definitions
INFO[05-17|18:06:11] Fetched URL=http://ftp.suse.com/pub/projects/security/oval/suse.linux.enterprise.server.15.xml OVAL definitions=7019
EROR[05-17|18:06:11] Failed to insert oval err="Unsupport family: opensuse"

~/g/s/g/k/goval-dictionary ❯❯❯ ./goval-dictionary fetch-suse -debug -opensuse-leap 15.1
INFO[05-17|18:06:36] Fetching... URL=http://ftp.suse.com/pub/projects/security/oval/opensuse.leap.15.1.xml
INFO[05-17|18:06:50] Fetched... URL=http://ftp.suse.com/pub/projects/security/oval/opensuse.leap.15.1.xml
INFO[05-17|18:06:50] Finished fetching OVAL definitions
INFO[05-17|18:06:51] Fetched URL=http://ftp.suse.com/pub/projects/security/oval/opensuse.leap.15.1.xml OVAL definitions=1738
EROR[05-17|18:06:51] Failed to insert oval err="Unsupport family: opensuse"

Support Red Hat OVAL v2

https://www.redhat.com/security/data/oval/v2/

Still Failing on Ubuntu 20

Hey Guys,

I am running the latest docker images, but still getting the below error when running against a Ubuntu 20 Server. Any help appreciated

[May 24 11:03:57]  INFO [localhost] Loaded: /vuls/results/2020-05-24T10:42:27Z
[May 24 11:03:57]  INFO [localhost] Validating db config...
INFO[0000] -cvedb-type: sqlite3, -cvedb-url: , -cvedb-path: /vuls/cve.sqlite3
INFO[0000] -ovaldb-type: sqlite3, -ovaldb-url: , -ovaldb-path: /vuls/oval.sqlite3
INFO[0000] -gostdb-type: sqlite3, -gostdb-url: , -gostdb-path: /vuls/gost.sqlite3
INFO[0000] -exploitdb-type: sqlite3, -exploitdb-url: , -exploitdb-path: /vuls/go-exploitdb.sqlite3
INFO[05-24|11:03:57] Opening DB.                              db=sqlite3
INFO[05-24|11:03:57] Migrating DB.                            db=sqlite3
INFO[05-24|11:03:57] Opening Database.                        db=sqlite3
INFO[05-24|11:03:57] Migrating DB.                            db=sqlite3
[May 24 11:03:57]  INFO [localhost] [Reboot Required] Docker-Secure: 0 CVEs are detected with Library
[May 24 11:03:57]  INFO [localhost] OVAL is fresh: ubuntu 20.04
[May 24 11:03:57] ERROR [localhost] Failed to fill with OVAL:
    github.com/future-architect/vuls/report.FillCveInfo
        /go/src/github.com/future-architect/vuls/report/report.go:162
  - Ubuntu 20.04 is not support for now

Fetch oval for Ubuntu does not complete

future-architect/vuls#688

Support patch of SUSE Linux

https://en.opensuse.org/SDB:Zypper_usage_11.3#Listing_needed_patches
see Listing needed patches section.

SUSE OVAL
http://ftp.suse.com/pub/projects/security/oval/

# zypper lp
Reading installed packages...
Patches

Repository       | Name               | Version | Category    | Status
-----------------+--------------------+---------+-------------+-------
Updates for 11.1 | KDE4-fixes         | 38      | recommended | Needed
Updates for 11.1 | MozillaFirefox     | 50      | recommended | Needed
Updates for 11.1 | NetworkManager-kde | 49      | recommended | Needed

Parallel Download

This source code is likely to be helpful.
https://github.com/Code-Hex/pget

fetch alpine 3.12 fails

What did you do? (required. The issue will be closed when not provided.)

tried to fetch alpine 3.12
docker run --rm -v $PWD:/vuls vuls/goval-dictionary fetch-alpine 3.12

What did you expect to happen?

CVES will be fetched

What happened instead?

docker run --rm -v $PWD:/vuls vuls/goval-dictionary fetch-alpine 3.12
t=2020-06-15T21:45:04+0000 lvl=info msg="Fetching... " URL=https://raw.githubusercontent.com/alpinelinux/alpine-secdb/master/v3.12/community.yaml
t=2020-06-15T21:45:04+0000 lvl=info msg="Fetching... " URL=https://raw.githubusercontent.com/alpinelinux/alpine-secdb/master/v3.12/main.yaml
t=2020-06-15T21:45:05+0000 lvl=info msg="Finished fetching OVAL definitions"
t=2020-06-15T21:45:05+0000 lvl=eror msg="Failed to fetch files" err="Failed to fetch. err: [HTTP error. errs: [], url: https://raw.githubusercontent.com/alpinelinux/alpine-secdb/master/v3.12/main.yaml HTTP error. errs: [], url: https://raw.githubusercontent.com/alpinelinux/alpine-secdb/master/v3.12/community.yaml]"

Current Output

I think goval uses a deprecated repository of alpine-secdb as stated in:
https://github.com/alpinelinux/alpine-secdb
This repository is deprecated.

Please use the up-to-date data from https://secdb.alpinelinux.org/

Scripts to generate the data is managed in https://gitlab.alpinelinux.org/alpine/infra/docker/secdb

Searching CVE by cURL is not working

What did you do? (required. The issue will be closed when not provided.)

I did a search with cURL according to the README, but I could not get the output like the README.

What did you expect to happen?

$ curl http://127.0.0.1:1324/cves/ubuntu/16/CVE-2017-15400 | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1237  100  1237    0     0  81365      0 --:--:-- --:--:-- --:--:-- 82466
[
  {
    "ID": 10582,
    "DefinitionID": "oval:com.ubuntu.xenial:def:201715400000",
    "Title": "CVE-2017-15400 on Ubuntu 16.04 LTS (xenial) - medium.",
    "Description": "Insufficient restriction of IPP filters in CUPS in Google Chrome OS prior to 62.0.3202.74 allowed a remote attacker to execute a command with the same privileges as the cups daemon via a crafted PPD file, aka a printer zeroconfig CRLF issue.",
    "Advisory": {
      "ID": 10575,
      "Severity": "Medium",
      "Cves": null,
      "Bugzillas": null,
      "AffectedCPEList": null,
      "Issued": "0001-01-01T00:00:00Z",
      "Updated": "0001-01-01T00:00:00Z"
    },
    "Debian": {
      "ID": 9330,
      "CveID": "CVE-2017-15400",
      "MoreInfo": "",
      "Date": "0001-01-01T00:00:00Z"
    },
    "AffectedPacks": [
      {
        "ID": 16117,
        "Name": "cups",
        "Version": "",
        "NotFixedYet": true
      }
    ],
    "References": [
      {
        "ID": 48602,
        "Source": "CVE",
        "RefID": "CVE-2017-15400",
        "RefURL": "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-15400"
      },
      {
        "ID": 48603,
        "Source": "Ref",
        "RefID": "",
        "RefURL": "http://people.canonical.com/~ubuntu-security/cve/2017/CVE-2017-15400.html"
      },
      {
        "ID": 48604,
        "Source": "Ref",
        "RefID": "",
        "RefURL": "https://chromereleases.googleblog.com/2017/10/stable-channel-update-for-chrome-os_27.html"
      },
      {
        "ID": 48605,
        "Source": "Bug",
        "RefID": "",
        "RefURL": "https://bugs.chromium.org/p/chromium/issues/detail?id=777215"
      }
    ]
  }
]

What happened instead?

Current Output

// cURL client
$ curl http://127.0.0.1:1324/cves/ubuntu/16/CVE-2017-15400
{"message":"Not Found"}

// server
$ goval-dictionary server -bind=127.0.0.1 -port=1324
INFO[07-31|15:56:06] Starting HTTP Server...
INFO[07-31|15:56:06] Listening...                             URL=127.0.0.1:1324

   ____    __
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v3.3.10-dev
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\
⇨ http server started on 127.0.0.1:1324
{"time":"2020-07-31T15:56:14.0532225+09:00","id":"","remote_ip":"127.0.0.1","host":"127.0.0.1:1324","method":"GET","uri":"/cves/ubuntu/16/CVE-2017-15400","user_agent":"curl/7.68.0","status":404,"error":"","latency":156900,"latency_human":"156.9µs","bytes_in":0,"bytes_out":24}

Please re-run the command using -debug and provide the output below.

Steps to reproduce the behaviour

goval-dictionary fetch-ubuntu 16
goval-dictionary server -bind=127.0.0.1 -port=1324
curl http://127.0.0.1:1324/cves/ubuntu/16/CVE-2017-15400

Configuration (MUST fill this out):

Go version (go version):

$ go version
go version go1.14.3 linux/amd64

Go environment (go env):

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/mainek00n/.cache/go-build"
GOENV="/home/mainek00n/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/mainek00n/go/1.14.3"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/mainek00n/.goenv/versions/1.14.3"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/mainek00n/.goenv/versions/1.14.3/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/mainek00n/github/github.com/MaineK00n/vuls/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build532110654=/tmp/go-build -gno-record-gcc-switches"

goval-dictionary environment:

To check the commit hash of HEAD
$goval-dictionary -v
goval-dictionary v0.2.8 8afbb01

Goval-Dictionary Fetch Ubuntu returns an error upon completion for deleting OVALS 18

What did you do? (required. The issue will be closed when not provided.)

Attempted a goval-dictionary fetch ubunto command to update internal vuls database

What did you expect to happen?

I expected the command to run successfully without errors to complete internal database updates.

What happened instead?

I received the following error on several runs after updating to the latest release. For each run, the number that cannot be converted is different upon each failure.

lvl=eror msg="Failed to insert OVAL" err="Failed to delete: cannot convert 14905759 to Text" failed to run commands: exit status 1

Current Output
Please re-run the command using -debug and provide the output below.
INFO[08-11|16:21:13] Fetching... URL=https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml.bz2
INFO[08-11|16:21:13] Fetching... URL=https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.bionic.cve.oval.xml.bz2
INFO[08-11|16:21:19] Fetched... URL=https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.bionic.cve.oval.xml.bz2
INFO[08-11|16:21:19] Fetched... URL=https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml.bz2
INFO[08-11|16:21:19] Finished fetching OVAL definitions
INFO[08-11|16:21:20] Fetched URL=https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.bionic.cve.oval.xml.bz2 OVAL definitions=13372
DBUG[08-11|16:21:22] in Ubuntu
INFO[08-11|16:21:22] Refreshing... Family=ubuntu Version=18
EROR[08-11|16:21:22] Failed to insert OVAL err="Failed to delete: cannot convert 14905759 to Text"
Failed to delete: cannot convert 14905759 to Text

Steps to reproduce the behaviour

Run goval-dictionary command below on Ubuntu 16.04.7 LTS

$~# /home/ubuntu/go/bin/goval-dictionary fetch ubuntu --debug --dbtype postgres --dbpath=postgresql://vulsclient:xxxxxxxx@XXXXXXX-prime-production-us-west-2-db-vuls.cidujwijlfwi.us-west-2.rds.amazonaws.com:5432/ovals 16 18

Configuration (MUST fill this out):

Go version (go version):
1.16.2
Go environment (go env): N/A
goval-dictionary environment: N/A

Hash : ____

To check the commit hash of HEAD
$ goval-dictionary -v
goval-dictionary v0.3.5 c70b5d8

Behavior of GetByPackName in RDB and Redis

What did you do? (required. The issue will be closed when not provided.)

I would like to fix the architecture around the key of Redis so that the search results of RDB and Redis are the same.

Some definition in OVAL does not have <cve></cve> (⇔ Advisory.Cves = []).
To explain briefly, RDB and Redis retrieve OVALJSON from package names as shown below.
In Redis, the OVALJSON of Advisory.Cves = [] will not appear in the search results.

Redis: PackageName → CVEID → OVALJSON
RDB: PackageName → DifinitionID → OVALJSON

What did you expect to happen?

The same response is obtained by GetByPackName in RDB and Redis.

What happened instead?

The response of GetByPackName in RDB and Redis may be different.

Steps to reproduce the behaviour

$ goval-dictionary fetch oracle
$ docker run --rm --name goval-test -d -p 127.0.0.1:6379:6379 redis
$ goval-dictionary fetch oracle --dbtype redis --dbpath "redis://127.0.0.1:6379/0"

$ goval-dictionary server --port 1324
$ goval-dictionary server --port 1325 --dbtype redis --dbpath "redis://127.0.0.1:6379/0"

$ curl http://127.0.0.1:1324/packs/oracle/5/nspr/x86_64 | jq . > sqlite3.json
$ curl http://127.0.0.1:1325/packs/oracle/5/nspr/x86_64 | jq . > redis.json

$ diff <(grep "DefinitionID" sqlite3.json | sort) <(grep "DefinitionID" redis.json | sort)
8d7
<     "DefinitionID": "oval:com.oracle.elsa:def:20111282",
10d8
<     "DefinitionID": "oval:com.oracle.elsa:def:20130214",

Configuration (MUST fill this out):

Go version (go version):
1.16.2
Go environment (go env):
goval-dictionary environment:

Hash : ____

To check the commit hash of HEAD
$ goval-dictionary -v
goval-dictionary v0.3.5 c70b5d8

Reference

#139

Differences in the output of GetByPackName and GetByCveID in RDB and Redis

What did you do? (required. The issue will be closed when not provided.)

There is a difference in the output of GetByPackName and GetByCveID in RDB and Redis.

What did you expect to happen?

In RDB and Redis, this is equivalent to the output of GetByPackName, GetByCveID.

What happened instead?

sqlite3_ruby-libs.json.txt
sqlite3_CVE-2019-2683.json.txt
redis_ruby-libs.json.txt
redis_CVE-2019-2683.json.txt

Steps to reproduce the behaviour

packs/redhat/8/ruby-libs
cves/redhat/8/CVE-2019-2683

$ goval-dictionary fetch-redhat 8
$ goval-dictionary fetch-redhat --dbtype redis --dbpath "redis://127.0.0.1:6379/0" 8
$ goval-dictionary server --port 1322 --dbpath (pwd)/oval.sqlite3
$ goval-dictionary server --port 1323 --dbtype redis --dbpath "redis://127.0.0.1:6379/0"
$ curl http://127.0.0.1:1322/packs/redhat/8/ruby-libs | jq . > sqlite3.json
$ curl http://127.0.0.1:1323/packs/redhat/8/ruby-libs | jq . > redis.json
$ grep "DefinitionID" sqlite3.json
$ grep "DefinitionID" redis.json

Configuration (MUST fill this out):

Go version (go version):
1.16.2
Go environment (go env):
goval-dictionary environment:

Hash : ____

To check the commit hash of HEAD
$ goval-dictionary -v
goval-dictionary v0.3.5 0a1e75f

Server mode for SUSE Linux

On the server mode page, it is said that server mode is supported for SLES(SUSE Linux), but no command is present on the page.
https://vuls.io/docs/en/usage-server.html
please check the one-liner scan distributions.

update failed for mysql backend

It seems that it is treated as an error when SQL mode as follows

sql mode

ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

ref. Server System Variables (sql mode)
ref. MySQL 5.7.8以降で古いアプリが動かない場合の対処（sql_mode）

Debian 9(Stretch)

command

goval-dictionary fetch-debian -dbtype mysql -dbpath "mysql connection string" 9

Error Message

Error 1406: Data too long for column 'more_info' at row 1

Ubuntu 18(Bionic)

command

goval-dictionary fetch-ubuntu -dbtype mysql -dbpath "mysql connection string" 18

Error Message

Error 1292: Incorrect datetime value: '0000-00-00' for column 'issued' at row 1

got a fatal message when exec make

I got a following fatal message when exec make command, but build succeeded.

$ make install
fatal: No names found, cannot describe anything.

this occurs on following line at GNUmakefile.

VERSION := $(shell git describe --tags --abbrev=0)

Fetch suse over HTTPS

Hello,
I am using a Forward Proxy to acces internet. The FP doesn't allow HTTP request and fetch-suse is the only one using it.
The const URL is define in https://github.com/kotakanbe/goval-dictionary/blob/master/fetcher/suse.go (l.13) :

const t = "http://ftp.suse.com/pub/projects/security/oval/%s.%s.xml"

Is it possible to update it for HTTPS please ?

alpine sec db deprecated

https://secdb.alpinelinux.org/

Ubuntu 21.04(Hirsute Hippo)

What did you do? (required. The issue will be closed when not provided.)

% goval-dictionary fetch-ubuntu 21

What did you expect to happen?

Fetching...                              URL=https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.hirsute.cve.oval.xml.bz2

What happened instead?

Current Output

WARN[04-28|11:39:38] Skip unknown ubuntu.                     version=21

Please re-run the command using -debug and provide the output below.

Steps to reproduce the behaviour

Configuration (MUST fill this out):

Go version (go version):

go version go1.16.3 linux/amd64

Go environment (go env):
Vuls environment:

vuls-v0.15.11-build-20210423_164043_c36e645

config.toml:
command:

Why fetch data for all supported versions of ubuntu?

It says to run the following command to fetch data for all versions of Ubuntu.

goval-dictionary fetch-ubuntu 12 14 16

Why is that? Why not just fetch the data for the specific edition of Ubuntu one is running?

TIA

fatal error: concurrent map writes

What did you do? (required. The issue will be closed when not provided.)

$ goval-dictionary v0.4.0 047e4db
$ make diff-server-rdb

INFO[09-14|09:13:14] start server mode test(mode: cveid, os: amazon, arch: , release: ['1', '2'])
INFO[09-14|09:13:14] check the communication with the server
INFO[09-14|09:13:14] communication with the server has been confirmed
fatal error: concurrent map writes

goroutine 4457 [running]:
runtime.throw({0xd8337a, 0x10})
        /usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc0000bd1d0 sp=0xc0000bd1a0 pc=0x442651
runtime.mapassign_faststr(0xcaf300, 0xc0002351a0, {0xd79879, 0x6})
        /usr/local/go/src/runtime/map_faststr.go:294 +0x38b fp=0xc0000bd238 sp=0xc0000bd1d0 pc=0x421aab
github.com/kotakanbe/goval-dictionary/db/rdb.(*Driver).NewOvalDB(0xc00071baf0, {0xc00003e72a, 0x786e42})
        /home/ubuntu/go/src/github.com/kotakanbe/goval-dictionary/db/rdb/rdb.go:103 +0x274 fp=0xc0000bd300 sp=0xc0000bd238 pc=0x9e1a14
github.com/kotakanbe/goval-dictionary/db/rdb.NewRDB({0xc00003e72a, 0x6}, {0xd7a782, 0x7}, {0x7ffd5bd89db2, 0x56}, 0xa0)
        /home/ubuntu/go/src/github.com/kotakanbe/goval-dictionary/db/rdb/rdb.go:55 +0xd9 fp=0xc0000bd3a0 sp=0xc0000bd300 pc=0x9e1539
github.com/kotakanbe/goval-dictionary/db.NewDB({0xc00003e72a, 0xd7b950}, {0xd7a782, 0x4}, {0x7ffd5bd89db2, 0x8}, 0x8)
        /home/ubuntu/go/src/github.com/kotakanbe/goval-dictionary/db/db.go:32 +0xa5 fp=0xc0000bd3f8 sp=0xc0000bd3a0 pc=0xae6785
github.com/kotakanbe/goval-dictionary/server.getByCveID.func1({0xea3e70, 0xc0001c8fa0})
        /home/ubuntu/go/src/github.com/kotakanbe/goval-dictionary/server/server.go:120 +0x334 fp=0xc0000bd6b0 sp=0xc0000bd3f8 pc=0xb6c834
github.com/labstack/echo/v4.(*Echo).add.func1({0xea3e70, 0xc0001c8fa0})
        /home/ubuntu/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:522 +0x51 fp=0xc0000bd6e0 sp=0xc0000bd6b0 pc=0xb5cbf1
github.com/labstack/echo/v4/middleware.LoggerWithConfig.func2.1({0xea3e70, 0xc0001c8fa0})
        /home/ubuntu/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/logger.go:117 +0xe2 fp=0xc0000bd860 sp=0xc0000bd6e0 pc=0xb690a2
github.com/labstack/echo/v4/middleware.RecoverWithConfig.func1.1({0xea3e70, 0xc0001c8fa0})
        /home/ubuntu/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/recover.go:98 +0xfc fp=0xc0000bd8e0 sp=0xc0000bd860 pc=0xb6a6fc
github.com/labstack/echo/v4/middleware.LoggerWithConfig.func2.1({0xea3e70, 0xc0001c8fa0})
        /home/ubuntu/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/logger.go:117 +0xe2 fp=0xc0000bda60 sp=0xc0000bd8e0 pc=0xb690a2
github.com/labstack/echo/v4.(*Echo).ServeHTTP(0xc000294000, {0xe83f80, 0xc00037a0e0}, 0xc000a90500)
        /home/ubuntu/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:633 +0x3bc fp=0xc0000bdac0 sp=0xc0000bda60 pc=0xb5d9fc
net/http.serverHandler.ServeHTTP({0xc0003d6300}, {0xe83f80, 0xc00037a0e0}, 0xc000a90500)
        /usr/local/go/src/net/http/server.go:2878 +0x43b fp=0xc0000bdb80 sp=0xc0000bdac0 pc=0x71db5b
net/http.(*conn).serve(0xc0001f5180, {0xe88e20, 0xc00027d290})
        /usr/local/go/src/net/http/server.go:1929 +0xb08 fp=0xc0000bdfb8 sp=0xc0000bdb80 pc=0x7196c8
net/http.(*Server).Serve·dwrap·82()
        /usr/local/go/src/net/http/server.go:3033 +0x2e fp=0xc0000bdfe0 sp=0xc0000bdfb8 pc=0x71e4ae
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000bdfe8 sp=0xc0000bdfe0 pc=0x4720c1
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:3033 +0x4e8

Debian zero day vulns ignored?

In the oval definition for Debian Stretch from today:

    <definition class="vulnerability" id="oval:org.debian:def:20181000001" version="1">
      <metadata>
        <title>CVE-2018-1000001</title>
        <affected family="unix">
          <platform>Debian GNU/Linux 9</platform>
          <product>glibc</product>
        </affected>
        <reference ref_id="CVE-2018-1000001" ref_url="http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1000001" sour$
        <description>In glibc 2.26 and earlier there is confusion in the usage of getcwd() by realpath() which can be used$
        <debian>
          <date>2018-11-06</date>
          <moreinfo></moreinfo>
        </debian>
      </metadata>
      <criteria comment="Release section" operator="AND">
        <criterion comment="Debian 9 is installed" test_ref="oval:org.debian.oval:tst:1"/>
        <criteria comment="Architecture section" operator="OR">
          <criteria comment="Architecture independent section" operator="AND">
            <criterion comment="all architecture" test_ref="oval:org.debian.oval:tst:2"/>
            <criterion comment="glibc DPKG is earlier than 0" test_ref="oval:org.debian.oval:tst:16495"/>
          </criteria>
        </criteria>
      </criteria>
    </definition>

Notice the glibc DPKG is earlier than 0 - which I assume means "all versions are vulnerable".

This record is not being inserted in the database, and the CVE does not appear in the response when looked-up:

curl "http://127.0.0.1:1324/packs/debian/9/glibc" | jq .

panic: runtime error: invalid memory address or nil pointer dereference while fetching

go build ; and ./goval-dictionary fetch-redhat -debug 7                                                                                                                                                                                                                                                                                                       INFO[07-03|10:31:56] Fetching...                              URL=https://www.redhat.com/security/data/oval/com.redhat.rhsa-RHEL7.xml.bz2
INFO[07-03|10:31:57] Finished to fetch OVAL definitions.
INFO[07-03|10:31:58] Fetched                                  URL=https://www.redhat.com/security/data/oval/com.redhat.rhsa-RHEL7.xml.bz2 OVAL definitions=659
INFO[07-03|10:31:58] Skip (Same Timestamp)                    Family=redhat Version=7
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x423ba6d]

goroutine 1 [running]:
database/sql.(*Tx).rollback(0x0, 0x4672f00, 0xc420c17801, 0x5f799a0)
        /Users/kanbe/.goenv/versions/1.10.1/src/database/sql/sql.go:1944 +0x2d
database/sql.(*Tx).Rollback(0x0, 0x5f79908, 0x0)
        /Users/kanbe/.goenv/versions/1.10.1/src/database/sql/sql.go:1963 +0x30
github.com/kotakanbe/goval-dictionary/vendor/github.com/jinzhu/gorm.(*DB).Rollback(0xc420867cb0, 0x45bf840)
        /Users/kanbe/go/src/github.com/kotakanbe/goval-dictionary/vendor/github.com/jinzhu/gorm/main.go:505 +0xa4
github.com/kotakanbe/goval-dictionary/db/rdb.(*Driver).InsertFetchMeta(0xc42034e1b0, 0x0, 0xc420022fca, 0x1d, 0x0, 0xed2ca4b32, 0x0, 0x0, 0x0)
        /Users/kanbe/go/src/github.com/kotakanbe/goval-dictionary/db/rdb/rdb.go:225 +0x24f
github.com/kotakanbe/goval-dictionary/commands.(*FetchRedHatCmd).Execute(0xc420298a00, 0x4705400, 0xc420024098, 0xc420288cc0, 0x0, 0x0, 0x0, 0x0)
        /Users/kanbe/go/src/github.com/kotakanbe/goval-dictionary/commands/fetch-redhat.go:166 +0xc33
github.com/kotakanbe/goval-dictionary/vendor/github.com/google/subcommands.(*Commander).Execute(0xc4200b2000, 0x4705400, 0xc420024098, 0x0, 0x0, 0x0, 0x0)
        /Users/kanbe/go/src/github.com/kotakanbe/goval-dictionary/vendor/github.com/google/subcommands/subcommands.go:141 +0x29f
github.com/kotakanbe/goval-dictionary/vendor/github.com/google/subcommands.Execute(0x4705400, 0xc420024098, 0x0, 0x0, 0x0, 0xc4202b65a8)
        /Users/kanbe/go/src/github.com/kotakanbe/goval-dictionary/vendor/github.com/google/subcommands/subcommands.go:371 +0x5f
main.main()
        /Users/kanbe/go/src/github.com/kotakanbe/goval-dictionary/main.go:52 +0x348

ubuntu 14.04 packages table 'name' field "The 'xxxx' " ?

echo/engine/standard package no longer exists

This error is received:
package github.com/labstack/echo/engine/standard: cannot find package "github.com/labstack/echo/engine/standard" in any of: ....

when you attempt install via:
go get github.com/kotakanbe/goval-dictionary

seems to be because the echo interface has changed - its a simple fix, similar to this:
maddevsio/openfreecabs-web@0010d44

OVAL entries of centos 7.9.2009 are not found

What did you do? (required. The issue will be closed when not provided.)

Run a command:
docker run --rm -it -v /opt/vuls/.ssh:/root/.ssh:ro -v $PWD:/vuls -v $PWD/vuls-log:/var/log/vuls -v /etc/localtime:/etc/localtime:ro vuls/vuls report -format-list -config=./config.toml

What did you expect to happen?

See a report

What happened instead?

[Jul 20 15:06:33]  INFO [localhost] vuls-v0.15.13-build-20210719_203317_9077a83
[Jul 20 15:06:33]  INFO [localhost] Validating config...
[Jul 20 15:06:33]  INFO [localhost] cveDict.type=sqlite3, cveDict.url=, cveDict.SQLite3Path=/vuls/cve.sqlite3
[Jul 20 15:06:33]  INFO [localhost] ovalDict.type=sqlite3, ovalDict.url=, ovalDict.SQLite3Path=/vuls/oval.sqlite3
[Jul 20 15:06:33]  INFO [localhost] gost.type=sqlite3, gost.url=, gost.SQLite3Path=/vuls/gost.sqlite3
[Jul 20 15:06:33]  INFO [localhost] exploit.type=sqlite3, exploit.url=, exploit.SQLite3Path=/vuls/go-exploitdb.sqlite3
[Jul 20 15:06:33]  INFO [localhost] metasploit.type=sqlite3, metasploit.url=, metasploit.SQLite3Path=/vuls/go-msfdb.sqlite3
[Jul 20 15:06:33]  WARN [localhost] metasploit.SQLite3Path=/vuls/go-msfdb.sqlite3 file not found
[Jul 20 15:06:33]  INFO [localhost] Loaded: /vuls/results/2021-07-20T12:44:31Z
[Jul 20 15:06:33] DEBUG [localhost] dev-rancher1 (centos7.9.2009): config.ServerInfo{
  ServerName:         "xxxx",
  User:               "xxx",
  Host:               "xxx.xxx.xxx.xxx",
  JumpServer:         []string{},
  Port:               "22",
  SSHConfigPath:      "",
  KeyPath:            "",
  CpeNames:           []string{},
  ScanMode:           []string{},
  ScanModules:        []string{},
  OwaspDCXMLPath:     "",
  ContainersOnly:     false,
  ContainersIncluded: []string{},
  ContainersExcluded: []string{},
  ContainerType:      "",
  Containers:         map[string]config.ContainerSetting{},
  IgnoreCves:         []string{},
  IgnorePkgsRegexp:   []string{},
  GitHubRepos:        map[string]config.GitHubConf{},
  UUIDs:              map[string]string{},
  Memo:               "",
  Enablerepo:         []string{},
  Optional:           map[string]interface {}{},
  Lockfiles:          []string{},
  FindLock:           false,
  Type:               "",
  IgnoredJSONKeys:    []string{},
  IPv4Addrs:          []string{},
  IPv6Addrs:          []string{},
  IPSIdentifiers:     map[string]string{},
  WordPress:          &config.WordPressConf{
    OSUser:  "",
    DocRoot: "",
    CmdPath: "",
  },
  PortScan: &config.PortScanConf{
    IsUseExternalScanner: false,
    ScannerBinPath:       "",
    HasPrivileged:        false,
    ScanTechniques:       []string{},
    SourcePort:           "",
  },
  LogMsgAnsiColor: "\x1b[32m",
  Container:       config.Container{
    ContainerID: "",
    Name:        "",
    Image:       "",
  },
  Distro: config.Distro{
    Family:  "",
    Release: "",
  },
  Mode: config.ScanMode{
    flag: 0x01,
  },
  Module: config.ScanModule{
    flag: 0x0f,
  },
}
[Jul 20 15:06:33] DEBUG [localhost] Check if oval fetched: centos 7.9.2009
[Jul 20 15:06:33]  INFO [localhost] OVAL centos 7.9.2009 found. defs: 0
[Jul 20 15:06:33] ERROR [localhost] Failed to detect Pkg CVE:
    github.com/future-architect/vuls/detector.Detect
        /go/src/github.com/future-architect/vuls/detector/detector.go:65
  - Failed to detect CVE with OVAL:
    github.com/future-architect/vuls/detector.DetectPkgCves
        /go/src/github.com/future-architect/vuls/detector/detector.go:179
  - OVAL entries of centos 7.9.2009 are not found. Fetch OVAL before reporting. For details, see `https://github.com/kotakanbe/goval-dictionary#usage`:
    github.com/future-architect/vuls/detector.detectPkgsCvesWithOval
        /go/src/github.com/future-architect/vuls/detector/detector.go:355

Steps to reproduce the behaviour

Run vuls on CentOS Linux release 7.9.2009 (Core)

Configuration (MUST fill this out):

[servers]

[servers.x]
host            = "xxx.xxx.xxx.xxx"
port            = "22"
user            = "XXX"
# if ssh config file exists in .ssh, path to ssh config file in docker
#sshConfigPath   = "/root/.ssh/config"
# path to ssh private key in docker
kaeyPath         = "/root/.ssh/id_rsa"


[cveDict]
type =  "sqlite3"
SQLite3Path = "/vuls/cve.sqlite3"

[gost]
type =  "sqlite3"
SQLite3Path = "/vuls/gost.sqlite3"

[exploit]
type =  "sqlite3"
SQLite3Path = "/vuls/go-exploitdb.sqlite3"

[metasploit]
type =  "sqlite3"
SQLite3Path = "/opt/vuls/go-msfdb.sqlite3"

docker:

vuls/gost                latest              31b243ff7316        16 hours ago        48.3MB
vuls/go-cve-dictionary   latest              e8fbfffd081f        16 hours ago        25.8MB
vuls/vuls                latest              334fdfc00169        17 hours ago        78.2MB
vuls/go-exploitdb        latest              6bc7343b1629        3 weeks ago         28.1MB

goval-dictionary environment:

Hash : goval-dictionary v0.3.5 c70b5d8

To check the commit hash of HEAD
$ goval-dictionary -v

vulsio / goval-dictionary Goto Github PK

goval-dictionary's People

Stargazers

Watchers

Forkers

goval-dictionary's Issues

What did you do? (required. The issue will be closed when not provided.)

What did you do?

What did you expect to happen?

What happened instead?

Steps to reproduce the behaviour

Configuration (MUST fill this out):

What did you do? (required. The issue will be closed when not provided.)

What did you expect to happen?

What happened instead?

What did you do? (required. The issue will be closed when not provided.)

What did you expect to happen?

What happened instead?

Steps to reproduce the behaviour

Configuration (MUST fill this out):

What did you do? (required. The issue will be closed when not provided.)

What did you expect to happen?

What happened instead?

Steps to reproduce the behaviour

Configuration (MUST fill this out):

What did you do? (required. The issue will be closed when not provided.)

What did you expect to happen?

What happened instead?

Steps to reproduce the behaviour

Configuration (MUST fill this out):

What did you do? (required. The issue will be closed when not provided.)

What did you expect to happen?

What happened instead?

Steps to reproduce the behaviour

Configuration (MUST fill this out):

What did you do? (required. The issue will be closed when not provided.)

What did you expect to happen?

What happened instead?

Steps to reproduce the behaviour

Configuration (MUST fill this out):

What did you do? (required. The issue will be closed when not provided.)

What did you expect to happen?

What happened instead?

What did you do? (required. The issue will be closed when not provided.)

What did you expect to happen?

What happened instead?

Steps to reproduce the behaviour

Configuration (MUST fill this out):

What did you do? (required. The issue will be closed when not provided.)

What did you expect to happen?

What happened instead?

Steps to reproduce the behaviour

Configuration (MUST fill this out):

What did you do? (required. The issue will be closed when not provided.)

What did you expect to happen?

What happened instead?

Steps to reproduce the behaviour

Configuration (MUST fill this out):

Reference

What did you do? (required. The issue will be closed when not provided.)

What did you expect to happen?

What happened instead?

Steps to reproduce the behaviour

Configuration (MUST fill this out):

Debian 9(Stretch)

Ubuntu 18(Bionic)

What did you do? (required. The issue will be closed when not provided.)

What did you expect to happen?

What happened instead?

Steps to reproduce the behaviour

Configuration (MUST fill this out):

What did you do? (required. The issue will be closed when not provided.)

What did you do? (required. The issue will be closed when not provided.)

What did you expect to happen?

What happened instead?

Steps to reproduce the behaviour

Configuration (MUST fill this out):

Recommend Projects

Recommend Topics

Recommend Org