cisco-talos / cvdupdate Goto Github PK
View Code? Open in Web Editor NEWClamAV Private Database Mirror Updater Tool
License: Apache License 2.0
ClamAV Private Database Mirror Updater Tool
License: Apache License 2.0
Hi team,
We just saw a change in behaviour which we'd like to validate with you.
Until version 1.0.2
, cdiff files were being downloaded on updates:
$ cvd config set --dbdir $PWD
$ cvd update
2022-05-28 23:19:47 cvdupdate-1.0.2 INFO Downloaded main-62.cdiff
2022-05-28 23:19:57 cvdupdate-1.0.2 INFO Downloaded main.cvd. Version: 62
2022-05-28 23:19:58 cvdupdate-1.0.2 INFO Downloaded daily-26555.cdiff
2022-05-28 23:20:01 cvdupdate-1.0.2 INFO Downloaded daily.cvd. Version: 26555
2022-05-28 23:20:02 cvdupdate-1.0.2 INFO Downloaded bytecode-333.cdiff
2022-05-28 23:20:03 cvdupdate-1.0.2 INFO Downloaded bytecode.cvd. Version: 333
Since version 1.1.0
the cdiff files are no longer being downloaded:
$ cvd config set --dbdir $PWD
$ cvd update
2022-05-29 01:20:48 cvdupdate-1.1.0 INFO Using system configured nameservers
2022-05-29 01:20:50 cvdupdate-1.1.0 INFO Downloaded main.cvd
2022-05-29 01:20:50 cvdupdate-1.1.0 INFO Downloaded daily.cvd
2022-05-29 01:20:50 cvdupdate-1.1.0 INFO Downloaded bytecode.cvd
Reading the changelog it's not clear to me whether this is expected or it's a side effect of the config.json
+ state.json
split or something else.
I'd appreciate any help in clarifying this :)
I found that if the DNS query for the version check times out, it prints a stack trace:
❯ cvd update -V
2021-07-15 09:15:50 cvdupdate-1.0.2 DEBUG Checking for a newer version of cvdupdate.
2021-07-15 09:15:59 cvdupdate-1.0.2 DEBUG cvdupdate is up-to-date: 1.0.2.
2021-07-15 09:15:59 cvdupdate-1.0.2 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2021-07-15 09:16:04 cvdupdate-1.0.2 DEBUG EXCEPTION OCCURRED: The DNS operation timed out after 5.005509376525879 seconds
2021-07-15 09:16:04 cvdupdate-1.0.2 WARNING Failed to determine available version via DNS TXT query!
Traceback (most recent call last):
File "/home/micasnyd/.local/bin/cvd", line 11, in <module>
load_entry_point('cvdupdate', 'console_scripts', 'cvd')()
File "/home/micasnyd/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/micasnyd/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/micasnyd/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/micasnyd/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/micasnyd/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/micasnyd/.local/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/micasnyd/.local/lib/python3.8/site-packages/cvdupdate/__main__.py", line 259, in update_alias
ctx.forward(db_update)
File "/home/micasnyd/.local/lib/python3.8/site-packages/click/core.py", line 628, in forward
return self.invoke(cmd, **kwargs)
File "/home/micasnyd/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/micasnyd/.local/lib/python3.8/site-packages/cvdupdate/__main__.py", line 101, in db_update
errors = m.db_update(db, debug_mode)
File "/home/micasnyd/.local/lib/python3.8/site-packages/cvdupdate/cvdupdate.py", line 837, in db_update
self.logger.error(f"Failed to update {db}. Missing or invalid URL: {self.config['dbs'][db]['url']}")
KeyError: ''
Side note: I have no idea why it failed. Manual testing with dig
and using freshclam
both worked fine:
❯ dig TXT current.cvd.clamav.net +short
"0.103.3:61:26233:1626362940:1:90:49192:333"
Please push the latest stable build to pypi so users of cvdupdate
can use pypi to get the patched version.
ty 🍻 😃
Current setup automatically goes to database.clamav.net. It would be nice if there is a way to configure the tool so that it can point to a different URL.
I'm using ClamAV on ECS Fargate containers in different AWS accounts.
I think startup time would be improved and bandwidth reduced if I could have a mirror of the databases.
Are CVD files patched when downloading a CDIFF file?
When are CDIFFs needed? Are they removed at some point?
Are CVD files redownloaded at any time?
I would like to make use of an S3 static website (that I don't need to monitor) to share the databases among all the accounts.
I was thinking of using a lambda function to run on a cron to keep the mirror up to date.
It would be great if cvdupdate could upload the files as they are downloaded but, if files don't need to be patched, I could seed the bucket with an offline run and then update with CDIFFs after a run.
My environment uses a proxy for access to the internet, and internet DNS lookups are not permitted except from the proxy itself.
(When an HTTP/HTTPS proxy is used, the proxy performs the necessary DNS lookups, and the client does not perform any DNS lookups. So, for most use cases, clients that are behind a proxy don't technically need to be able to perform DNS lookups themselves.)
Looking through the code and history, I see that cvdupdate
intentionally doesn't work without DNS, since DNS is being used to reduce the CloudFlare request count: #19
Unfortunately, that makes it incompatible with my environment.
Could an exception be made for environments that do not support DNS?
Could dns.txt be posted on database.clamav.net so that cvdupdate could pull down the version numbers with a single HTTP request (as an alternative way to help reduce the CloudFlare request count when DNS is unavailable)?
As title, I've setup a clamav mirror server behind a proxy server(adopted Squid).
However, there still existing issues that cannot be resolved...
First I've followed the instruction to setup nameserver by
cvd config set --nameserver my_proxy_ip
And config was shown in following...
#cvd config show
Config file: /root/.cvdupdate/config.json
Config:
{
"nameserver": "my_proxy_ip",
"max retry": 3,
"log directory": "/root/.cvdupdate/logs",
"rotate logs": true,
"# logs to keep": 30,
"db directory": "/var/www/html/clamav/",
"rotate cdiffs": true,
"# cdiffs to keep": 30,
"state file": "/root/.cvdupdate/state.json"
}
Then issue the command and the log was shown as following:
#http_proxy=http://my_proxy_ip:my_proxy_port https_proxy=http://my_proxy_ip:my_proxy_port cvd update -V
2023-08-15 06:32:27 cvdupdate-1.1.1 DEBUG Checking for a newer version of cvdupdate.
2023-08-15 06:32:29 cvdupdate-1.1.1 DEBUG cvdupdate is up-to-date: 1.1.1.
2023-08-15 06:32:29 cvdupdate-1.1.1 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2023-08-15 06:32:29 cvdupdate-1.1.1 INFO Found configuration provided nameservers: my_proxy_ip
2023-08-15 06:32:29 cvdupdate-1.1.1 INFO Using nameservers: ['my_proxy_ip']
2023-08-15 06:32:34 cvdupdate-1.1.1 DEBUG EXCEPTION OCCURRED: The resolution lifetime expired after 5.105 seconds: Server Do53:my_proxy_ip@53 answered The DNS operation timed out.
2023-08-15 06:32:34 cvdupdate-1.1.1 WARNING Failed to determine available version via DNS TXT query!
2023-08-15 06:32:34 cvdupdate-1.1.1 ERROR Failed to update: DNS query failed.
Does there anything I needs to setup on no matter proxy server or this virus definition mirror server?
Hello,
The script is helpful, just that how to automate this like to run it on an OS level cron? since it has to run on user privilege
Does [ crontab -e ] command like this will work?
20 10 * * * {username} cvd update >> ~/var/log/clamav/definitions-update/cvd-update.log 2>&1
Add some sort of cvdupdate version check to notify users if there is a newer versions.
Ideally cvdupdate would update itself as well, but I don't know if there's a safe way to do that with pip packages, as it could be run globally or with --user
or in a virtual env, etc.
Hi,
During running 'cvd update' on my machine, I get an error of:
"requests.exceptions.SSLError: HTTPSConnectionPool(host='database.clamav.net', port=443): Max retries exceeded with url: /main-59.cdiff (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1076)')))"
Is there a way to disable the certificate check here?
Or do you have any other suggestion for solution. I have tried several of different solutions, but couldn't solve it.
Thanks s lot.
Hello,
i have following problem.
I habe a proxy in the company.
I can download per pid over the proxy.
For download the virus pattern i need to config a proxy.
In the config.json file i have no point with proxy settings.
How can i set the proxy for download the virus patterns.
Thanks.
Hi
I try to install the cvdupdate using pip install on my linux (based on centos 7) machine.
I get the following issue:
Traceback (most recent call last):
File "./cvd", line 6, in
from cvdupdate.main import cli
File "/Python/lib/python3.7/site-packages/cvdupdate/main.py", line 38, in
import coloredlogs
File "/Python/lib/python3.7/site-packages/coloredlogs/init.py", line 211, in
from humanfriendly import coerce_boolean
File "/Python/lib/python3.7/site-packages/humanfriendly/init.py", line 20, in
from humanfriendly.compat import is_string, monotonic
File "/Python/lib/python3.7/site-packages/humanfriendly/compat.py", line 68, in
import unittest
ModuleNotFoundError: No module named 'unittest'
What is the issue here?
After install cvdupdate with:
python3 -m pip install --user cvdupdate
on Debian 10 Buster server, theres not cvd command working.
Installation seems to be fine. Any solution?
root@clamav-mirror:~# python3 -m pip install --user cvdupdate
Requirement already satisfied: cvdupdate in ./.local/lib/python3.7/site-packages (1.1.0)
Requirement already satisfied: click>=7.0 in ./.local/lib/python3.7/site-packages (from cvdupdate) (8.1.3)
Requirement already satisfied: rangehttpserver in ./.local/lib/python3.7/site-packages (from cvdupdate) (1.2.0)
Requirement already satisfied: colorama in ./.local/lib/python3.7/site-packages (from cvdupdate) (0.4.4)
Requirement already satisfied: coloredlogs>=10.0 in ./.local/lib/python3.7/site-packages (from cvdupdate) (15.0.1)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from cvdupdate) (2.21.0)
Requirement already satisfied: dnspython>=2.1.0 in ./.local/lib/python3.7/site-packages (from cvdupdate) (2.2.1)
Requirement already satisfied: importlib-metadata; python_version < "3.8" in ./.local/lib/python3.7/site-packages (from click>=7.0->cvdupdate) (4.11.4)
Requirement already satisfied: humanfriendly>=9.1 in ./.local/lib/python3.7/site-packages (from coloredlogs>=10.0->cvdupdate) (10.0)
Requirement already satisfied: typing-extensions>=3.6.4; python_version < "3.8" in ./.local/lib/python3.7/site-packages (from importlib-metadata; python_version < "3.8"->click>=7.0->cvdupdate) (4.2.0)
Requirement already satisfied: zipp>=0.5 in ./.local/lib/python3.7/site-packages (from importlib-metadata; python_version < "3.8"->click>=7.0->cvdupdate) (3.8.0)
Hi,
I occasionally see the following message in my logs and noticed, that _query_cvd_version_http
uses the retries whereas _query_cvd_version_dns
does not, if I read the source correctly.
Would it be possible to add retries for the DNS query as well?
Thanks!
Tim
2022-08-26 09:24:14 cvdupdate-1.1.1 DEBUG Checking for a newer version of cvdupdate.
2022-08-26 09:24:16 cvdupdate-1.1.1 DEBUG cvdupdate is up-to-date: 1.1.1.
2022-08-26 09:24:16 cvdupdate-1.1.1 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2022-08-26 09:24:16 cvdupdate-1.1.1 INFO Using system configured nameservers
2022-08-26 09:24:21 cvdupdate-1.1.1 DEBUG EXCEPTION OCCURRED: The resolution lifetime expired after 5.007 seconds: Server 172.24.8.8 UDP port 53 answered The DNS operation timed out.
2022-08-26 09:24:21 cvdupdate-1.1.1 WARNING Failed to determine available version via DNS TXT query!
2022-08-26 09:24:21 cvdupdate-1.1.1 ERROR Failed to update: DNS query failed.
The string:
You are running cvdupate version: {current_version}.
Should be:
You are running cvdupdate version: {current_version}.
If the main.cvd or dailt.cvd is missing, the update continues, but does not re-pull the daily or main. This has been addresses in the pull request at #35
I'm using the tool to setup a local mirror in our internal network. The server for the local mirror has no direct internet connection and would need to use the corporate proxy to access external resources. It seems that the cvdupdate tool currently is unable to use a proxy server.
I tried to set environment variables http_proxy
and https_proxy
but it did not work.
The documentation indicates that requests.request()
needs an additional proxies
dictionary to configure a proxy server.
It would be nice to implement this to support local mirrors behind a corporate proxy.
pypi_update_check uses python 3.7 and above option capture_output in subprocess.run which makes in incompatible with python 3.6. I am running 3.6.10
File "/usr/local/python3/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 792, in check latest_version = str(subprocess.run([sys.executable, '-m', 'pip', 'install', '{}==random'.format('cvdupdate')], capture_output=True, text=True)) File "/usr/local/python3/lib/python3.6/subprocess.py", line 423, in run with Popen(*popenargs, **kwargs) as process: TypeError: __init__() got an unexpected keyword argument 'capture_output'
Can I somehow add additional databases to serve?
For example https://database.clamav.net/linux.cvd
This seems like a project that is very usable in larger installations, like ours. I guess we're not the only place that would like to setup our own mirror, but can't run pip
on the server we would like to run it on.
Now I need to find out how to install this on our relevant server.
I was attempting to add cvdupdate to a config management tool. We want to keep logs and db in a non-default location, so I need to edit those. config.json
has some config options, but also appears to be tracking application state?
Usually I try to just create the config file I want, then compare it to what's on the system (if any), and replace it with the desired one as necessary. But if the config file changes periodically, that doesn't work. Would you be willing to split the dbs
portion out to something like ~/.cvdupdate/dbs.json
? I might be able to put together a PR with that.
I have installed cvdupdate using python pip installer. When I ran cvd update, it is downloading daily, main, bytecode cvd files but not safebrowsing.cvd files. Please help me if there a way to get it downloaded.
Thanks
Venkat
Bit of an odd one pertaining to https://github.com/Cisco-Talos/cvdupdate/blob/main/cvdupdate/cvdupdate.py#L401 ....
System Type: VM
OS: CentOS 7.9.2009
Python Version: 3.6.8 (default available via yum install python3
)
pip3 Version: 9.0.3 (default available via yum install python3
)
Dependency versions:
Issue Encountered
After installing cvdupdate, trying to run and produce updates results in DNS Timeout occurring, even though dig/nslookup from host system returns TXT record without issue. Here is the exact error encountered when running python3 -m cvdupdate update -V -D
:
2021-04-15 10:11:13 cvdupdate-0.3.0 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2021-04-15 10:11:13 cvdupdate-0.3.0 INFO Using nameserver specified in the config: 10.9.64.116
2021-04-15 10:11:18 cvdupdate-0.3.0 DEBUG EXCEPTION OCCURRED: The DNS operation timed out after 5.109134912490845 seconds
2021-04-15 10:11:18 cvdupdate-0.3.0 WARNING Failed to determine available version via DNS TXT query!
Steps to Reproduce
yum install -y python3
pip3 install cvdupdate
python3 -m cvdupdate update
Mitigation
On the above linked line (https://github.com/Cisco-Talos/cvdupdate/blob/main/cvdupdate/cvdupdate.py#L401), this function is where the error is coming from. I specified a direct value for timeout for the resolver object by inserting a line after 409, as follows: our_resolver.timeout = 10
. The actual lookup still happens in sub 1 second time, but for some reason couldn't return anything and would timeout without this timeout being specified.
Other Notes
I realize I could have maybe made a pull request for this (and can still if we determine that's what we want to do), but this seemed a rather specific and peculiar issue to run into. If I install cvdupdate on my Mac natively, it also installs dnspython 2.1.0, and doesn't encounter this issue - it works right away. If nothing else, it may be a good idea to just declare a default timeout value to avoid whatever bug this is. I have tested with other values, such as 5 (which appeared to be the default given the error produced), and it still works fine. Perhaps this could also be another configuration option too?
Hello,
due to the latest changes I am trying to download signatures using cvdupdate tool. I am performing it from inside of company’s network and I get warning: “Failed to determine available version via DNS TXT query!”.
However when I am looking for address names resolution using our internal proxy I am able to get an answer:
# nslookup database.clamav.net <proxy_ip>
Server: <proxy_ip>
Address: <proxy_ip>#53
Non-authoritative answer:
database.clamav.net canonical name = database.clamav.net.cdn.cloudflare.net.
Name: database.clamav.net.cdn.cloudflare.net
Address: 104.16.219.84
Name: database.clamav.net.cdn.cloudflare.net
Address: 104.16.218.84
Name: database.clamav.net.cdn.cloudflare.net
Address: 2606:4700::6810:da54
Name: database.clamav.net.cdn.cloudflare.net
Address: 2606:4700::6810:db54
Proxy server can forward DNS requests but cannot act as name server. When updating “nameserver” in /root/.cvdupdate/config.json with proxy address I got timeout like below:
# cvd update -V
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Checking main.cvd for update from https://database.clamav.net/main.cvd
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2021-03-16 07:40:28 cvdupdate-0.2.0 INFO Using nameserver specified in the config: 10.111.0.162
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Checking main.cvd version via DNS TXT advertisement.
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG main.cvd version advertised by DNS: 59
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Downloading CDIFFs first...
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Checking for main-59.cdiff
2021-03-16 07:40:28 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): database.clamav.net:443
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 170, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/lib64/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 353, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 182, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f7ccf06e6a0>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 573, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='database.clamav.net', port=443): Max retries exceeded with url: /main-59.cdiff (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7ccf06e6a0>: Failed to establish a new connection: [Errno -2] Name or service not known',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/cvd", line 11, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/cvdupdate/__main__.py", line 257, in update_alias
ctx.forward(db_update)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 628, in forward
return self.invoke(cmd, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/cvdupdate/__main__.py", line 100, in db_update
errors = m.db_update(db)
File "/usr/local/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 781, in db_update
if update(db) == False:
File "/usr/local/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 767, in update
return self._download_cvd(db, advertised_version)
File "/usr/local/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 612, in _download_cvd
'User-Agent': f'ClamAV/{self.dns_version_tokens[0]} (cvdupdate-{self.version})',
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='database.clamav.net', port=443): Max retries exceeded with url: /main-59.cdiff (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7ccf06e6a0>: Failed to establish a new connection: [Errno -2] Name or service not known',))
I am wondering if it’s and option to forward request via proxy correctly, i.e. putting proxy ip in /root/.cvdupdate/config.json as a feature.
It appears that cloudflare doesn't distinguish between full downloads (HTTP 200) and partial downloads (HTTP 206) in terms of billing. While the partial download to check the HTTP version is pretty efficient, checking the database via DNS when using database.clamav.net will be have to be required.
Hi,
We want to support the an environment in which clients download signatures from one server. This server will be in offline mode – meaning that the signatures will be put there by an administrator manually.
I have some questions regarding it:
Thanks a lot for your help.
Thanks a lot for your script. It works fine.
It would be nice to keep the file dns.txt on local mirror up to date. This file contains version numbers of main, daily and bytecode. So after downloading the files successfully your script could store current version information in this file. Or just copy it from clamav.net (if every download worked fine).
Some common check scripts (on clients) use dns.txt to check if clamav is up to date. Especially clients that are only allowed to access local clamav mirror need this information.
Short version: the replaced script did it ;-)
Thank you in advance
stekaa
I must download updates via a proxy and was sometimes getting truncated cvd files. I had to manually edit the json to force a redownload of the corrupted file.
Can file size or some other validation of the file download be added before adding it to the json as complete?
Ubuntu 18.04 only has 6.7-3
Users stuck with distro packages for python packages would appreciate it if we can lower the version requirement.
vcdupdate isn't working for us as the DNS entry current.cvd.clamav.net doesn't resolve
$ ~/.cvdupdate/logs$ cat 2021-03-18.log
2021-03-18 03:17:20 PM - WARNING: Failed to determine available version via DNS TXT query!
2021-03-18 03:17:46 PM - WARNING: Failed to determine available version via DNS TXT query!
2021-03-18 03:18:57 PM - DEBUG: Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2021-03-18 03:18:57 PM - DEBUG: EXCEPTION OCCURRED: 'Resolver' object has no attribute 'resolve'
2021-03-18 03:18:57 PM - WARNING: Failed to determine available version via DNS TXT query!
dig doesn't resolve DNS
byronc@byronc-lt001:~$ dig current.cvd.clamav.net
; <<>> DiG 9.16.1-Ubuntu <<>> current.cvd.clamav.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63876
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;current.cvd.clamav.net. IN A
;; Query time: 8 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Thu Mar 18 15:26:15 NZDT 2021
;; MSG SIZE rcvd: 51
byronc@byronc-lt001:~$
when running vcd update
$:~/.cvdupdate$ cvd update -V
2021-03-18 15:18:57 cvdupdate-0.3.0 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2021-03-18 15:18:57 cvdupdate-0.3.0 DEBUG EXCEPTION OCCURRED: 'Resolver' object has no attribute 'resolve'
2021-03-18 15:18:57 cvdupdate-0.3.0 WARNING Failed to determine available version via DNS TXT query!
Traceback (most recent call last):
File "/home/byronc/.local/bin/cvd", line 8, in <module>
sys.exit(cli())
File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/byronc/.local/lib/python3.8/site-packages/cvdupdate/__main__.py", line 259, in update_alias
ctx.forward(db_update)
File "/usr/lib/python3/dist-packages/click/core.py", line 573, in forward
return self.invoke(cmd, **kwargs)
File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/byronc/.local/lib/python3.8/site-packages/cvdupdate/__main__.py", line 101, in db_update
errors = m.db_update(db, debug_mode)
File "/home/byronc/.local/lib/python3.8/site-packages/cvdupdate/cvdupdate.py", line 795, in db_update
self.logger.error(f"Failed to update {db}. Missing or invalid URL: {self.config['dbs'][db]['url']}")
KeyError: ''
$:~/.cvdupdate$ cd los
Current info messages appear to go to stderr:
$ /usr/local/cvdupdate/bin/cvd update >/dev/null
2022-05-24 11:27:00 cvdupdate-1.1.0 INFO Using system configured nameservers
2022-05-24 11:27:00 cvdupdate-1.1.0 INFO main.cvd is up-to-date. Version: 62
2022-05-24 11:27:00 cvdupdate-1.1.0 INFO daily.cvd is up-to-date. Version: 26551
2022-05-24 11:27:00 cvdupdate-1.1.0 INFO bytecode.cvd is up-to-date. Version: 333
This makes it hard to capture errors from cron jobs.
#16 1.949 2023-11-15 12:39:21 cvdupdate-1.1.1 WARNING Failed to download main.cvd. #16 1.949 2023-11-15 12:39:21 cvdupdate-1.1.1 WARNING Download request rejected because we've downloaded the same file too frequently. #16 1.949 2023-11-15 12:39:21 cvdupdate-1.1.1 WARNING We won't try main.cvd again for 13:20:14 hours. #16 1.952 2023-11-15 12:39:21 cvdupdate-1.1.1 DEBUG Checking daily.cvd for update from https://database.clamav.net/daily.cvd
I have been using the CVD update for the past six months, but haven't come across these kind of failures, these is happening very frequently.
How to avoid this from happening and is there any other way to get latest definition other than cvdupdate or freshclam
CVD deletes ALL files in the target directory on a "clean" request. This wipes 3rd party db's NOT managed by CVD as well. This has been addressed by pull request #34
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.