pypi_update_check uses python 3.7 and above option capture_output in subprocess.run which makes in incompatible with python 3.6. I am running 3.6.10

File "/usr/local/python3/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 792, in check latest_version = str(subprocess.run([sys.executable, '-m', 'pip', 'install', '{}==random'.format('cvdupdate')], capture_output=True, text=True)) File "/usr/local/python3/lib/python3.6/subprocess.py", line 423, in run with Popen(*popenargs, **kwargs) as process: TypeError: __init__() got an unexpected keyword argument 'capture_output'

This seems like a project that is very usable in larger installations, like ours. I guess we're not the only place that would like to setup our own mirror, but can't run pip on the server we would like to run it on.

Now I need to find out how to install this on our relevant server.

Can I somehow add additional databases to serve?
For example https://database.clamav.net/linux.cvd

Hi,

I occasionally see the following message in my logs and noticed, that _query_cvd_version_http uses the retries whereas _query_cvd_version_dns does not, if I read the source correctly.

Would it be possible to add retries for the DNS query as well?

Thanks!
Tim

2022-08-26 09:24:14 cvdupdate-1.1.1 DEBUG Checking for a newer version of cvdupdate.
2022-08-26 09:24:16 cvdupdate-1.1.1 DEBUG cvdupdate is up-to-date: 1.1.1.
2022-08-26 09:24:16 cvdupdate-1.1.1 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2022-08-26 09:24:16 cvdupdate-1.1.1 INFO Using system configured nameservers
2022-08-26 09:24:21 cvdupdate-1.1.1 DEBUG EXCEPTION OCCURRED: The resolution lifetime expired after 5.007 seconds: Server 172.24.8.8 UDP port 53 answered The DNS operation timed out.
2022-08-26 09:24:21 cvdupdate-1.1.1 WARNING Failed to determine available version via DNS TXT query!
2022-08-26 09:24:21 cvdupdate-1.1.1 ERROR Failed to update: DNS query failed.

Hi,

We want to support the an environment in which clients download signatures from one server. This server will be in offline mode – meaning that the signatures will be put there by an administrator manually.
I have some questions regarding it:

1. We thought to bring only the daily.cvd every day (after the main.cvd is there, of course). Is this ok?
2. In order to know what the current version of the signatures on the server side, we wanted to use ‘sigtool --info daily.cvd’ and take the version attribute from there.
   a. Is this is the correct way to achieve it?
   b. Is there another type of signatures version beside the version of the main.cvd and the daily.cvd?
3. For regular environment on which we use mirror (using cvd update), we would like to have a way to check if the if our signatures on the mirror are most updated. For that, we would like to have a way to get the signatures version on the remote location (the one the mirror downloads from). Is there a way to achieve it?

Thanks a lot for your help.

I was attempting to add cvdupdate to a config management tool. We want to keep logs and db in a non-default location, so I need to edit those. config.json has some config options, but also appears to be tracking application state?

Usually I try to just create the config file I want, then compare it to what's on the system (if any), and replace it with the desired one as necessary. But if the config file changes periodically, that doesn't work. Would you be willing to split the dbs portion out to something like ~/.cvdupdate/dbs.json? I might be able to put together a PR with that.

If the main.cvd or dailt.cvd is missing, the update continues, but does not re-pull the daily or main. This has been addresses in the pull request at #35

CVD deletes ALL files in the target directory on a "clean" request. This wipes 3rd party db's NOT managed by CVD as well. This has been addressed by pull request #34

I'm using ClamAV on ECS Fargate containers in different AWS accounts.
I think startup time would be improved and bandwidth reduced if I could have a mirror of the databases.

Are CVD files patched when downloading a CDIFF file?
When are CDIFFs needed? Are they removed at some point?
Are CVD files redownloaded at any time?

I would like to make use of an S3 static website (that I don't need to monitor) to share the databases among all the accounts.
I was thinking of using a lambda function to run on a cron to keep the mirror up to date.

It would be great if cvdupdate could upload the files as they are downloaded but, if files don't need to be patched, I could seed the bucket with an offline run and then update with CDIFFs after a run.

Hi team,

We just saw a change in behaviour which we'd like to validate with you.

Until version 1.0.2, cdiff files were being downloaded on updates:

$ cvd config set --dbdir $PWD
$ cvd update

2022-05-28 23:19:47 cvdupdate-1.0.2 INFO Downloaded main-62.cdiff
2022-05-28 23:19:57 cvdupdate-1.0.2 INFO Downloaded main.cvd. Version: 62
2022-05-28 23:19:58 cvdupdate-1.0.2 INFO Downloaded daily-26555.cdiff
2022-05-28 23:20:01 cvdupdate-1.0.2 INFO Downloaded daily.cvd. Version: 26555
2022-05-28 23:20:02 cvdupdate-1.0.2 INFO Downloaded bytecode-333.cdiff
2022-05-28 23:20:03 cvdupdate-1.0.2 INFO Downloaded bytecode.cvd. Version: 333

Since version 1.1.0 the cdiff files are no longer being downloaded:

$ cvd config set --dbdir $PWD
$ cvd update

2022-05-29 01:20:48 cvdupdate-1.1.0 INFO Using system configured nameservers
2022-05-29 01:20:50 cvdupdate-1.1.0 INFO Downloaded main.cvd
2022-05-29 01:20:50 cvdupdate-1.1.0 INFO Downloaded daily.cvd
2022-05-29 01:20:50 cvdupdate-1.1.0 INFO Downloaded bytecode.cvd

Reading the changelog it's not clear to me whether this is expected or it's a side effect of the config.json + state.json split or something else.

I'd appreciate any help in clarifying this :)

The string:
You are running cvdupate version: {current_version}.
Should be:
You are running cvdupdate version: {current_version}.

Thanks a lot for your script. It works fine.

It would be nice to keep the file dns.txt on local mirror up to date. This file contains version numbers of main, daily and bytecode. So after downloading the files successfully your script could store current version information in this file. Or just copy it from clamav.net (if every download worked fine).

Some common check scripts (on clients) use dns.txt to check if clamav is up to date. Especially clients that are only allowed to access local clamav mirror need this information.

Short version: the replaced script did it ;-)

Thank you in advance
stekaa

It appears that cloudflare doesn't distinguish between full downloads (HTTP 200) and partial downloads (HTTP 206) in terms of billing. While the partial download to check the HTTP version is pretty efficient, checking the database via DNS when using database.clamav.net will be have to be required.

Bit of an odd one pertaining to https://github.com/Cisco-Talos/cvdupdate/blob/main/cvdupdate/cvdupdate.py#L401 ....

System Type: VM
OS: CentOS 7.9.2009
Python Version: 3.6.8 (default available via yum install python3)
pip3 Version: 9.0.3 (default available via yum install python3)
Dependency versions:

• click - 7.1.2
• coloredlogs - 15.0
• colorama - 0.4.4
• requests - 2.25.1
• dnspython - 2.1.0
• rangehttpserver - 1.2.0

Issue Encountered
After installing cvdupdate, trying to run and produce updates results in DNS Timeout occurring, even though dig/nslookup from host system returns TXT record without issue. Here is the exact error encountered when running python3 -m cvdupdate update -V -D:

2021-04-15 10:11:13 cvdupdate-0.3.0 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2021-04-15 10:11:13 cvdupdate-0.3.0 INFO Using nameserver specified in the config: 10.9.64.116
2021-04-15 10:11:18 cvdupdate-0.3.0 DEBUG EXCEPTION OCCURRED: The DNS operation timed out after 5.109134912490845 seconds
2021-04-15 10:11:18 cvdupdate-0.3.0 WARNING Failed to determine available version via DNS TXT query!

Steps to Reproduce

• Setup CentOS 7.9.2009 System
• Run yum install -y python3
• Run pip3 install cvdupdate
• Run python3 -m cvdupdate update

Mitigation
On the above linked line (https://github.com/Cisco-Talos/cvdupdate/blob/main/cvdupdate/cvdupdate.py#L401), this function is where the error is coming from. I specified a direct value for timeout for the resolver object by inserting a line after 409, as follows: our_resolver.timeout = 10. The actual lookup still happens in sub 1 second time, but for some reason couldn't return anything and would timeout without this timeout being specified.

Other Notes
I realize I could have maybe made a pull request for this (and can still if we determine that's what we want to do), but this seemed a rather specific and peculiar issue to run into. If I install cvdupdate on my Mac natively, it also installs dnspython 2.1.0, and doesn't encounter this issue - it works right away. If nothing else, it may be a good idea to just declare a default timeout value to avoid whatever bug this is. I have tested with other values, such as 5 (which appeared to be the default given the error produced), and it still works fine. Perhaps this could also be another configuration option too?

After install cvdupdate with:

python3 -m pip install --user cvdupdate

on Debian 10 Buster server, theres not cvd command working.

Installation seems to be fine. Any solution?
root@clamav-mirror:~# python3 -m pip install --user cvdupdate
Requirement already satisfied: cvdupdate in ./.local/lib/python3.7/site-packages (1.1.0)
Requirement already satisfied: click>=7.0 in ./.local/lib/python3.7/site-packages (from cvdupdate) (8.1.3)
Requirement already satisfied: rangehttpserver in ./.local/lib/python3.7/site-packages (from cvdupdate) (1.2.0)
Requirement already satisfied: colorama in ./.local/lib/python3.7/site-packages (from cvdupdate) (0.4.4)
Requirement already satisfied: coloredlogs>=10.0 in ./.local/lib/python3.7/site-packages (from cvdupdate) (15.0.1)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from cvdupdate) (2.21.0)
Requirement already satisfied: dnspython>=2.1.0 in ./.local/lib/python3.7/site-packages (from cvdupdate) (2.2.1)
Requirement already satisfied: importlib-metadata; python_version < "3.8" in ./.local/lib/python3.7/site-packages (from click>=7.0->cvdupdate) (4.11.4)
Requirement already satisfied: humanfriendly>=9.1 in ./.local/lib/python3.7/site-packages (from coloredlogs>=10.0->cvdupdate) (10.0)
Requirement already satisfied: typing-extensions>=3.6.4; python_version < "3.8" in ./.local/lib/python3.7/site-packages (from importlib-metadata; python_version < "3.8"->click>=7.0->cvdupdate) (4.2.0)
Requirement already satisfied: zipp>=0.5 in ./.local/lib/python3.7/site-packages (from importlib-metadata; python_version < "3.8"->click>=7.0->cvdupdate) (3.8.0)

• the version in pypi is 1.0.2 -- link
• the current version, with the patch for DNX TXT query issue, is 1.1.0

Please push the latest stable build to pypi so users of cvdupdate can use pypi to get the patched version.

ty 🍻 😃

As title, I've setup a clamav mirror server behind a proxy server(adopted Squid).

However, there still existing issues that cannot be resolved...

First I've followed the instruction to setup nameserver by
cvd config set --nameserver my_proxy_ip

And config was shown in following...

#cvd config show
Config file: /root/.cvdupdate/config.json

Config:
{
    "nameserver": "my_proxy_ip",
    "max retry": 3,
    "log directory": "/root/.cvdupdate/logs",
    "rotate logs": true,
    "# logs to keep": 30,
    "db directory": "/var/www/html/clamav/",
    "rotate cdiffs": true,
    "# cdiffs to keep": 30,
    "state file": "/root/.cvdupdate/state.json"
}

Then issue the command and the log was shown as following:

#http_proxy=http://my_proxy_ip:my_proxy_port https_proxy=http://my_proxy_ip:my_proxy_port cvd update -V
2023-08-15 06:32:27 cvdupdate-1.1.1 DEBUG Checking for a newer version of cvdupdate.
2023-08-15 06:32:29 cvdupdate-1.1.1 DEBUG cvdupdate is up-to-date: 1.1.1.
2023-08-15 06:32:29 cvdupdate-1.1.1 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2023-08-15 06:32:29 cvdupdate-1.1.1 INFO Found configuration provided nameservers: my_proxy_ip
2023-08-15 06:32:29 cvdupdate-1.1.1 INFO Using nameservers: ['my_proxy_ip']
2023-08-15 06:32:34 cvdupdate-1.1.1 DEBUG EXCEPTION OCCURRED: The resolution lifetime expired after 5.105 seconds: Server Do53:my_proxy_ip@53 answered The DNS operation timed out.
2023-08-15 06:32:34 cvdupdate-1.1.1 WARNING Failed to determine available version via DNS TXT query!
2023-08-15 06:32:34 cvdupdate-1.1.1 ERROR Failed to update: DNS query failed.

Does there anything I needs to setup on no matter proxy server or this virus definition mirror server?

Hello,
due to the latest changes I am trying to download signatures using cvdupdate tool. I am performing it from inside of company’s network and I get warning: “Failed to determine available version via DNS TXT query!”.
However when I am looking for address names resolution using our internal proxy I am able to get an answer:

# nslookup database.clamav.net <proxy_ip>
Server:         <proxy_ip>
Address:        <proxy_ip>#53

Non-authoritative answer:
database.clamav.net     canonical name = database.clamav.net.cdn.cloudflare.net.
Name:   database.clamav.net.cdn.cloudflare.net
Address: 104.16.219.84
Name:   database.clamav.net.cdn.cloudflare.net
Address: 104.16.218.84
Name:   database.clamav.net.cdn.cloudflare.net
Address: 2606:4700::6810:da54
Name:   database.clamav.net.cdn.cloudflare.net
Address: 2606:4700::6810:db54

Proxy server can forward DNS requests but cannot act as name server. When updating “nameserver” in /root/.cvdupdate/config.json with proxy address I got timeout like below:

# cvd update -V
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Checking main.cvd for update from https://database.clamav.net/main.cvd
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2021-03-16 07:40:28 cvdupdate-0.2.0 INFO Using nameserver specified in the config: 10.111.0.162
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Checking main.cvd version via DNS TXT advertisement.
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG main.cvd version advertised by DNS: 59
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Downloading CDIFFs first...
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Checking for main-59.cdiff
2021-03-16 07:40:28 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): database.clamav.net:443
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 170, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib64/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 353, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 182, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f7ccf06e6a0>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='database.clamav.net', port=443): Max retries exceeded with url: /main-59.cdiff (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7ccf06e6a0>: Failed to establish a new connection: [Errno -2] Name or service not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/cvd", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/cvdupdate/__main__.py", line 257, in update_alias
    ctx.forward(db_update)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 628, in forward
    return self.invoke(cmd, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/cvdupdate/__main__.py", line 100, in db_update
    errors = m.db_update(db)
  File "/usr/local/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 781, in db_update
    if update(db) == False:
  File "/usr/local/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 767, in update
    return self._download_cvd(db, advertised_version)
  File "/usr/local/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 612, in _download_cvd
    'User-Agent': f'ClamAV/{self.dns_version_tokens[0]} (cvdupdate-{self.version})',
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='database.clamav.net', port=443): Max retries exceeded with url: /main-59.cdiff (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7ccf06e6a0>: Failed to establish a new connection: [Errno -2] Name or service not known',))

I am wondering if it’s and option to forward request via proxy correctly, i.e. putting proxy ip in /root/.cvdupdate/config.json as a feature.

vcdupdate isn't working for us as the DNS entry current.cvd.clamav.net doesn't resolve

$ ~/.cvdupdate/logs$ cat 2021-03-18.log 
2021-03-18 03:17:20 PM - WARNING:  Failed to determine available version via DNS TXT query!
2021-03-18 03:17:46 PM - WARNING:  Failed to determine available version via DNS TXT query!
2021-03-18 03:18:57 PM - DEBUG:  Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2021-03-18 03:18:57 PM - DEBUG:  EXCEPTION OCCURRED: 'Resolver' object has no attribute 'resolve'
2021-03-18 03:18:57 PM - WARNING:  Failed to determine available version via DNS TXT query!

dig doesn't resolve DNS

byronc@byronc-lt001:~$ dig current.cvd.clamav.net

; <<>> DiG 9.16.1-Ubuntu <<>> current.cvd.clamav.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63876
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;current.cvd.clamav.net.		IN	A

;; Query time: 8 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Thu Mar 18 15:26:15 NZDT 2021
;; MSG SIZE  rcvd: 51

byronc@byronc-lt001:~$ 

when running vcd update

$:~/.cvdupdate$ cvd update -V
2021-03-18 15:18:57 cvdupdate-0.3.0 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2021-03-18 15:18:57 cvdupdate-0.3.0 DEBUG EXCEPTION OCCURRED: 'Resolver' object has no attribute 'resolve'
2021-03-18 15:18:57 cvdupdate-0.3.0 WARNING Failed to determine available version via DNS TXT query!
Traceback (most recent call last):
  File "/home/byronc/.local/bin/cvd", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/byronc/.local/lib/python3.8/site-packages/cvdupdate/__main__.py", line 259, in update_alias
    ctx.forward(db_update)
  File "/usr/lib/python3/dist-packages/click/core.py", line 573, in forward
    return self.invoke(cmd, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/byronc/.local/lib/python3.8/site-packages/cvdupdate/__main__.py", line 101, in db_update
    errors = m.db_update(db, debug_mode)
  File "/home/byronc/.local/lib/python3.8/site-packages/cvdupdate/cvdupdate.py", line 795, in db_update
    self.logger.error(f"Failed to update {db}. Missing or invalid URL: {self.config['dbs'][db]['url']}")
KeyError: ''
$:~/.cvdupdate$ cd los

Current setup automatically goes to database.clamav.net. It would be nice if there is a way to configure the tool so that it can point to a different URL.

Add some sort of cvdupdate version check to notify users if there is a newer versions.
Ideally cvdupdate would update itself as well, but I don't know if there's a safe way to do that with pip packages, as it could be run globally or with --user or in a virtual env, etc.

Hello,

i have following problem.
I habe a proxy in the company.
I can download per pid over the proxy.
For download the virus pattern i need to config a proxy.
In the config.json file i have no point with proxy settings.
How can i set the proxy for download the virus patterns.

Thanks.

I found that if the DNS query for the version check times out, it prints a stack trace:

❯ cvd update -V
2021-07-15 09:15:50 cvdupdate-1.0.2 DEBUG Checking for a newer version of cvdupdate.
2021-07-15 09:15:59 cvdupdate-1.0.2 DEBUG cvdupdate is up-to-date: 1.0.2.
2021-07-15 09:15:59 cvdupdate-1.0.2 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2021-07-15 09:16:04 cvdupdate-1.0.2 DEBUG EXCEPTION OCCURRED: The DNS operation timed out after 5.005509376525879 seconds
2021-07-15 09:16:04 cvdupdate-1.0.2 WARNING Failed to determine available version via DNS TXT query!
Traceback (most recent call last):
  File "/home/micasnyd/.local/bin/cvd", line 11, in <module>
    load_entry_point('cvdupdate', 'console_scripts', 'cvd')()
  File "/home/micasnyd/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/micasnyd/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/micasnyd/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/micasnyd/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/micasnyd/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/micasnyd/.local/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/micasnyd/.local/lib/python3.8/site-packages/cvdupdate/__main__.py", line 259, in update_alias
    ctx.forward(db_update)
  File "/home/micasnyd/.local/lib/python3.8/site-packages/click/core.py", line 628, in forward
    return self.invoke(cmd, **kwargs)
  File "/home/micasnyd/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/micasnyd/.local/lib/python3.8/site-packages/cvdupdate/__main__.py", line 101, in db_update
    errors = m.db_update(db, debug_mode)
  File "/home/micasnyd/.local/lib/python3.8/site-packages/cvdupdate/cvdupdate.py", line 837, in db_update
    self.logger.error(f"Failed to update {db}. Missing or invalid URL: {self.config['dbs'][db]['url']}")
KeyError: ''

Side note: I have no idea why it failed. Manual testing with dig and using freshclam both worked fine:

❯ dig TXT current.cvd.clamav.net  +short
"0.103.3:61:26233:1626362940:1:90:49192:333"

My environment uses a proxy for access to the internet, and internet DNS lookups are not permitted except from the proxy itself.
(When an HTTP/HTTPS proxy is used, the proxy performs the necessary DNS lookups, and the client does not perform any DNS lookups. So, for most use cases, clients that are behind a proxy don't technically need to be able to perform DNS lookups themselves.)

Looking through the code and history, I see that cvdupdate intentionally doesn't work without DNS, since DNS is being used to reduce the CloudFlare request count: #19
Unfortunately, that makes it incompatible with my environment.

Could an exception be made for environments that do not support DNS?
Could dns.txt be posted on database.clamav.net so that cvdupdate could pull down the version numbers with a single HTTP request (as an alternative way to help reduce the CloudFlare request count when DNS is unavailable)?

I have installed cvdupdate using python pip installer. When I ran cvd update, it is downloading daily, main, bytecode cvd files but not safebrowsing.cvd files. Please help me if there a way to get it downloaded.
Thanks
Venkat

I must download updates via a proxy and was sometimes getting truncated cvd files. I had to manually edit the json to force a redownload of the corrupted file.
Can file size or some other validation of the file download be added before adding it to the json as complete?

Hello,
The script is helpful, just that how to automate this like to run it on an OS level cron? since it has to run on user privilege

Does [ crontab -e ] command like this will work?

20 10 * * * {username} cvd update >> ~/var/log/clamav/definitions-update/cvd-update.log 2>&1

I'm using the tool to setup a local mirror in our internal network. The server for the local mirror has no direct internet connection and would need to use the corporate proxy to access external resources. It seems that the cvdupdate tool currently is unable to use a proxy server.

I tried to set environment variables http_proxy and https_proxy but it did not work.

The documentation indicates that requests.request() needs an additional proxies dictionary to configure a proxy server.

It would be nice to implement this to support local mirrors behind a corporate proxy.

#16 1.949 2023-11-15 12:39:21 cvdupdate-1.1.1 WARNING Failed to download main.cvd. #16 1.949 2023-11-15 12:39:21 cvdupdate-1.1.1 WARNING Download request rejected because we've downloaded the same file too frequently. #16 1.949 2023-11-15 12:39:21 cvdupdate-1.1.1 WARNING We won't try main.cvd again for 13:20:14 hours. #16 1.952 2023-11-15 12:39:21 cvdupdate-1.1.1 DEBUG Checking daily.cvd for update from https://database.clamav.net/daily.cvd

I have been using the CVD update for the past six months, but haven't come across these kind of failures, these is happening very frequently.
How to avoid this from happening and is there any other way to get latest definition other than cvdupdate or freshclam

Current info messages appear to go to stderr:

$ /usr/local/cvdupdate/bin/cvd update >/dev/null
2022-05-24 11:27:00 cvdupdate-1.1.0 INFO Using system configured nameservers
2022-05-24 11:27:00 cvdupdate-1.1.0 INFO main.cvd is up-to-date. Version: 62
2022-05-24 11:27:00 cvdupdate-1.1.0 INFO daily.cvd is up-to-date. Version: 26551
2022-05-24 11:27:00 cvdupdate-1.1.0 INFO bytecode.cvd is up-to-date. Version: 333

This makes it hard to capture errors from cron jobs.

Hi

I try to install the cvdupdate using pip install on my linux (based on centos 7) machine.

I get the following issue:

./cvd --help

Traceback (most recent call last):
File "./cvd", line 6, in
from cvdupdate.main import cli
File "/Python/lib/python3.7/site-packages/cvdupdate/main.py", line 38, in
import coloredlogs
File "/Python/lib/python3.7/site-packages/coloredlogs/init.py", line 211, in
from humanfriendly import coerce_boolean
File "/Python/lib/python3.7/site-packages/humanfriendly/init.py", line 20, in
from humanfriendly.compat import is_string, monotonic
File "/Python/lib/python3.7/site-packages/humanfriendly/compat.py", line 68, in
import unittest
ModuleNotFoundError: No module named 'unittest'

What is the issue here?

Hi,

During running 'cvd update' on my machine, I get an error of:

"requests.exceptions.SSLError: HTTPSConnectionPool(host='database.clamav.net', port=443): Max retries exceeded with url: /main-59.cdiff (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1076)')))"

Is there a way to disable the certificate check here?
Or do you have any other suggestion for solution. I have tried several of different solutions, but couldn't solve it.

Thanks s lot.

Ubuntu 18.04 only has 6.7-3
Users stuck with distro packages for python packages would appreciate it if we can lower the version requirement.