Comments (37)
Thank you for your issue @srikr
Here is a sample solution to give you an idea where to look for.
Clone this branch (private_key)
https://github.com/marcinguy/betterscan-ce/tree/private_key
go to docker folder and run ./start.sh
It will read your .ssh/id_rsa (private key) and use it on the platform
You can clone then private repos with the key from .ssh/id_rsa on the computer you started it (you can change key location in docker-compose.yml)
WARNING! The whole platform will use this key now. So make sure you can only access it, otherwise other platform users can also clone using this key.
Please clone like this:
ssh://[email protected]/avinetworks/avi-dev.git
Hope that helps.
Feel free to make a PR to change it in the Frontend and Backend so it can be done via User Interface (On the TODO list and with paid/PRO version)
from betterscan-ce.
Hi @marcinguy,
Thanks for the response. I did following based on your above comments
- checked out private branch name private_key as shown below:
# git branch
master
* private_key
- cd to docker directory and confirmed docker-compose.yaml file, which had private key reference on worker pod and not server(port 5000) pod
└─# cat docker-compose.yml
version: '3'
services:
server:
build:
context: .
dockerfile: docker/backend/Dockerfile
restart: always
depends_on:
- postgres
working_dir: /srv/scanmycode/
entrypoint: sh
command: -c "if [ -f /srv/scanmycode/setup_state/setup_done ]; then python /srv/scanmycode/manage.py runserver; else python /srv/scanmycode/manage.py setup && mkdir -p /srv/scanmycode/setup_state && touch /srv/scanmycode/setup_state/setup_done; fi"
ports:
- 5000:5000
volumes:
- ./data1:/srv/scanmycode/quantifiedcode/data/
- ./data2:/srv/scanmycode/quantifiedcode/backend/data/
- ./setup_state:/srv/scanmycode/setup_state
links:
- "postgres"
postgres:
image: postgres:13.2
restart: unless-stopped
environment:
POSTGRES_DB: qc
POSTGRES_USER: qc
POSTGRES_PASSWORD: qc
PGDATA: /var/lib/postgresql/data/pgdata
ports:
- "5432:5432"
volumes:
- db-data:/var/lib/postgresql/data
worker_1:
build:
context: .
dockerfile: docker/worker/Dockerfile
hostname: worker_1
restart: on-failure
depends_on:
- rabbitmq3
- postgres
- server
working_dir: /srv/scanmycode/
entrypoint: python
command: /srv/scanmycode/manage.py runworker
volumes:
- ./data1:/srv/scanmycode/quantifiedcode/data/
- ./data2:/srv/scanmycode/quantifiedcode/backend/data/
- ../../.ssh/:/root/.ssh
environment:
- PRIVATE_KEY=/root/.ssh/id_rsa
links:
- "rabbitmq3"
- "server"
- "postgres"
rabbitmq3:
container_name: "rabbitmq"
image: rabbitmq:3.8-management-alpine
environment:
- RABBITMQ_DEFAULT_USER=qc
- RABBITMQ_DEFAULT_PASS=qc
ports:
- 5672:5672
- 15672:15672
healthcheck:
test: [ "CMD", "nc", "-z", "localhost", "5672" ]
interval: 5s
timeout: 15s
retries: 1
volumes:
db-data:
driver: local
- Started the Server using ./start.sh
- Using Web Browser did http://localhost:5000 and logged using newly created user account which was already existing
- Added the Project with URL: ssh://[email protected]/avinetworks/avi-dev.git . please refer below screenshot
Note: Tried without ssh:// as well - In following All the above steps resulting in same SSH Key Issue.
Let me know if I am missing something here?
from betterscan-ce.
@srikr Better solution is to make it in backend and frontend.
The above setup assumes you have cloned the repo to user directory (i.e /home/user/betterscan-ce or so) and private key is under /home/user/.ssh/id_rsa (which public key id_rsa.pub is added on the Git server). Is this the case by you?
Feel free to post full "Log" tab output. It can tell more.
P.S We have also a paid version with GitHub support integration in Web Interface. It is easier.
from betterscan-ce.
@marcinguy I am a root user and I have keys placed in /root/.ssh/ . Is it mandatory I have to be regular user?
Below are the Logs I see after adding the project for scan:
[INFO / 2022-08-17 06:20:40] Running pre-analysis hooks for project AVI (7a70e5a7b25e4f33aa1e7d71f42cc8cd).
[INFO / 2022-08-17 06:20:40] Fetching data for project AVI (7a70e5a7b25e4f33aa1e7d71f42cc8cd).
[ERROR / 2022-08-17 06:20:42] Fetching data for project AVI (7a70e5a7b25e4f33aa1e7d71f42cc8cd) failed!
[ERROR / 2022-08-17 06:20:42] Analysis of project AVI (7a70e5a7b25e4f33aa1e7d71f42cc8cd) failed!
[ERROR / 2022-08-17 06:20:42] Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 42, in hook_step
settings.hooks.call(hook, project)
File "/srv/scanmycode/quantifiedcode/helpers/hooks.py", line 42, in call
hook(*args, **kwargs)
File "/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/fetch.py", line 99, in fetch_remote
raise IOError("Cannot fetch git repository!")
OSError: Cannot fetch git repository!
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 76, in analyze_project
_analyze_project(project)
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 92, in _analyze_project
hook_step(project, "project.analyze.fetch", "Fetching data")
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 45, in hook_step
logger.error("Exception {} {}.".format(e.__class__.__name__, e.message))
AttributeError: 'OSError' object has no attribute 'message'
[INFO / 2022-08-17 06:20:40] Running pre-analysis hooks for project AVI (7a70e5a7b25e4f33aa1e7d71f42cc8cd).
[INFO / 2022-08-17 06:20:40] Fetching data for project AVI (7a70e5a7b25e4f33aa1e7d71f42cc8cd).
[ERROR / 2022-08-17 06:20:42] Fetching data for project AVI (7a70e5a7b25e4f33aa1e7d71f42cc8cd) failed!
[ERROR / 2022-08-17 06:20:42] Analysis of project AVI (7a70e5a7b25e4f33aa1e7d71f42cc8cd) failed!
[ERROR / 2022-08-17 06:20:42] Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 42, in hook_step
settings.hooks.call(hook, project)
File "/srv/scanmycode/quantifiedcode/helpers/hooks.py", line 42, in call
hook(*args, **kwargs)
File "/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/fetch.py", line 99, in fetch_remote
raise IOError("Cannot fetch git repository!")
OSError: Cannot fetch git repository!
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 76, in analyze_project
_analyze_project(project)
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 92, in _analyze_project
hook_step(project, "project.analyze.fetch", "Fetching data")
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 45, in hook_step
logger.error("Exception {} {}.".format(e.__class__.__name__, e.message))
AttributeError: 'OSError' object has no attribute 'message'
[INFO / 2022-08-17 06:20:40] Running pre-analysis hooks for project AVI (7a70e5a7b25e4f33aa1e7d71f42cc8cd).
[INFO / 2022-08-17 06:20:40] Fetching data for project AVI (7a70e5a7b25e4f33aa1e7d71f42cc8cd).
[ERROR / 2022-08-17 06:20:42] Fetching data for project AVI (7a70e5a7b25e4f33aa1e7d71f42cc8cd) failed!
[ERROR / 2022-08-17 06:20:42] Analysis of project AVI (7a70e5a7b25e4f33aa1e7d71f42cc8cd) failed!
[ERROR / 2022-08-17 06:20:42] Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 42, in hook_step
settings.hooks.call(hook, project)
File "/srv/scanmycode/quantifiedcode/helpers/hooks.py", line 42, in call
hook(*args, **kwargs)
File "/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/fetch.py", line 99, in fetch_remote
raise IOError("Cannot fetch git repository!")
OSError: Cannot fetch git repository!
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 76, in analyze_project
_analyze_project(project)
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 92, in _analyze_project
hook_step(project, "project.analyze.fetch", "Fetching data")
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 45, in hook_step
logger.error("Exception {} {}.".format(e.__class__.__name__, e.message))
AttributeError: 'OSError' object has no attribute 'message'
from betterscan-ce.
@srikr It should also run as root.
Hmmm
Do you have cloned code in /root/betterscan-ce?
Try to clone using git command in /root to make sure key (priv/pub) is ok
from betterscan-ce.
I have cloned it in /root/srikr/Tools/betterscan-ce . Ok so you want me to move this directory betterscan-ce from /root/srikr/Tools/ to /root do you think that will work?
from betterscan-ce.
@srikr Yes. See docker-compose.yml mappings.
It has to be in user directory. Otherwise you need to adjust mappings.
Place betterscan in /root/betterscan-ce
from betterscan-ce.
@marcinguy I tried moving betterscan-ce directory to /root still I see same issue mentioned in above logs. Adjust mappings in docker-compose.yml meaning should I change the below line considering my code is /root/srikr/Tools/betterscan-ce/:
from
- ../../.ssh/:/root/.ssh
to
- ../../../../.ssh/:/root/.ssh
Above change in mapping will work?
from betterscan-ce.
@srikr Yes, you can change the mapping, but with /root/betterscan-ce and current mapping it should work
Can you try to run
- git -c core.sshCommand="ssh -i /root/.ssh/id_rsa" clone [email protected]:avinetworks/avi-dev.git
Does it work?
- Maybe remove all betterscan-ce images and rebuild it
from betterscan-ce.
ok @marcinguy thanks.
I tried step 1) it works:
git -c core.sshCommand="ssh -i /root/.ssh/id_rsa" clone [email protected]:avinetworks/avi-dev.git
Cloning into 'avi-dev'...
Enter passphrase for key '/root/.ssh/id_rsa':
remote: Enumerating objects: 2780016, done.
remote: Counting objects: 100% (968/968), done.
remote: Compressing objects: 100% (566/566), done.
I will try step 2 later during my day and update thanks.
from betterscan-ce.
@srikr I think I see the problem
Your key is passphrase protected. See:
Enter passphrase for key '/root/.ssh/id_rsa':
So it will not work like this with Betterscan. Key needs to be without this protection, because Betterscan process waits for your passphrase, which is never provided.
from betterscan-ce.
@srikr FYI It works on my end, with key without passphrase protection
from betterscan-ce.
@srikr If you need to have a key with passphrase you can try this:
ssh-add /root/.ssh/id_rsa
Enter your passphrase
Start betterscan-ce under the same user
Hopefully betterscan can later see it, this I am not sure.
from betterscan-ce.
@marcinguy I did following based on your above comments:
- Copied betterscan-ce directory to /root or user directory
- Removed the passphrase of the private key
- Deleted all the docker downloaded images
- Ran start.sh script to reinstall all the packages
- Accessed the betterscan server(http://localhost:5000), this time around it tried running pre-analysis hook for the project multiple times and failed with following error:
[INFO / 2022-08-17 14:00:28] Running pre-analysis hooks for project AVI (afef9d9b66fb4cd4a8dcff2563298669).
[INFO / 2022-08-17 14:00:28] Fetching data for project AVI (afef9d9b66fb4cd4a8dcff2563298669).
[ERROR / 2022-08-17 14:02:29] Fetching data for project AVI (afef9d9b66fb4cd4a8dcff2563298669) failed!
[ERROR / 2022-08-17 14:02:29] Analysis of project AVI (afef9d9b66fb4cd4a8dcff2563298669) failed!
[ERROR / 2022-08-17 14:02:29] Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 42, in hook_step
settings.hooks.call(hook, project)
File "/srv/scanmycode/quantifiedcode/helpers/hooks.py", line 42, in call
hook(*args, **kwargs)
File "/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/fetch.py", line 90, in fetch_remote
git_config=git_config, git_credentials=git_credentials)
File "/usr/local/lib/python3.7/site-packages/checkmate3-0.2.0-py3.7.egg/checkmate/contrib/plugins/git/lib/repository.py", line 264, in fetch
["git", "fetch", remote]+extra_args, env=env, timeout=120)
File "/usr/local/lib/python3.7/site-packages/checkmate3-0.2.0-py3.7.egg/checkmate/contrib/plugins/git/lib/repository.py", line 161, in call
return self._call(args, kwargs, timeout=timeout)
File "/usr/local/lib/python3.7/site-packages/checkmate3-0.2.0-py3.7.egg/checkmate/contrib/plugins/git/lib/repository.py", line 143, in _call
timeout)
TypeError: can't concat str to bytes
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 76, in analyze_project
_analyze_project(project)
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 92, in _analyze_project
hook_step(project, "project.analyze.fetch", "Fetching data")
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 45, in hook_step
logger.error("Exception {} {}.".format(e.__class__.__name__, e.message))
AttributeError: 'TypeError' object has no attribute 'message'
[INFO / 2022-08-17 14:00:28] Running pre-analysis hooks for project AVI (afef9d9b66fb4cd4a8dcff2563298669).
[INFO / 2022-08-17 14:00:28] Fetching data for project AVI (afef9d9b66fb4cd4a8dcff2563298669).
[ERROR / 2022-08-17 14:02:29] Fetching data for project AVI (afef9d9b66fb4cd4a8dcff2563298669) failed!
[ERROR / 2022-08-17 14:02:29] Analysis of project AVI (afef9d9b66fb4cd4a8dcff2563298669) failed!
[ERROR / 2022-08-17 14:02:29] Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 42, in hook_step
settings.hooks.call(hook, project)
File "/srv/scanmycode/quantifiedcode/helpers/hooks.py", line 42, in call
hook(*args, **kwargs)
File "/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/fetch.py", line 90, in fetch_remote
git_config=git_config, git_credentials=git_credentials)
File "/usr/local/lib/python3.7/site-packages/checkmate3-0.2.0-py3.7.egg/checkmate/contrib/plugins/git/lib/repository.py", line 264, in fetch
["git", "fetch", remote]+extra_args, env=env, timeout=120)
File "/usr/local/lib/python3.7/site-packages/checkmate3-0.2.0-py3.7.egg/checkmate/contrib/plugins/git/lib/repository.py", line 161, in call
return self._call(args, kwargs, timeout=timeout)
File "/usr/local/lib/python3.7/site-packages/checkmate3-0.2.0-py3.7.egg/checkmate/contrib/plugins/git/lib/repository.py", line 143, in _call
timeout)
TypeError: can't concat str to bytes
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 76, in analyze_project
_analyze_project(project)
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 92, in _analyze_project
hook_step(project, "project.analyze.fetch", "Fetching data")
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 45, in hook_step
logger.error("Exception {} {}.".format(e.__class__.__name__, e.message))
AttributeError: 'TypeError' object has no attribute 'message'
[INFO / 2022-08-17 14:00:28] Running pre-analysis hooks for project AVI (afef9d9b66fb4cd4a8dcff2563298669).
[INFO / 2022-08-17 14:00:28] Fetching data for project AVI (afef9d9b66fb4cd4a8dcff2563298669).
[ERROR / 2022-08-17 14:02:29] Fetching data for project AVI (afef9d9b66fb4cd4a8dcff2563298669) failed!
[ERROR / 2022-08-17 14:02:29] Analysis of project AVI (afef9d9b66fb4cd4a8dcff2563298669) failed!
[ERROR / 2022-08-17 14:02:29] Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 42, in hook_step
settings.hooks.call(hook, project)
File "/srv/scanmycode/quantifiedcode/helpers/hooks.py", line 42, in call
hook(*args, **kwargs)
File "/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/fetch.py", line 90, in fetch_remote
git_config=git_config, git_credentials=git_credentials)
File "/usr/local/lib/python3.7/site-packages/checkmate3-0.2.0-py3.7.egg/checkmate/contrib/plugins/git/lib/repository.py", line 264, in fetch
["git", "fetch", remote]+extra_args, env=env, timeout=120)
File "/usr/local/lib/python3.7/site-packages/checkmate3-0.2.0-py3.7.egg/checkmate/contrib/plugins/git/lib/repository.py", line 161, in call
return self._call(args, kwargs, timeout=timeout)
File "/usr/local/lib/python3.7/site-packages/checkmate3-0.2.0-py3.7.egg/checkmate/contrib/plugins/git/lib/repository.py", line 143, in _call
timeout)
TypeError: can't concat str to bytes
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 76, in analyze_project
_analyze_project(project)
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 92, in _analyze_project
hook_step(project, "project.analyze.fetch", "Fetching data")
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 45, in hook_step
logger.error("Exception {} {}.".format(e.__class__.__name__, e.message))
AttributeError: 'TypeError' object has no attribute 'message'
It looks like some issue with private repo as I did not see this issue earlier.
from betterscan-ce.
Good and bad news.
You have to debug your issue. It seems like a different issue. Try also different repositories.
I never encountered it like you.
For me the SSH PRIVATE KEY via environmental variable works.
Added also option to change it from UI (Web interface)
You can try the second option. BUT most likely you will get similar error as above.
Clone the "private_key_web" branch
After adding the project you can change the SSH Private key per project, per user. ("cat" it out and copy and paste)
Also make sure you add your project as:
ssh://[email protected]/avinetworks/avi-dev.git
In your case.
from betterscan-ce.
Sure @marcinguy I can give it a try with the new branch with web fix you have sent out but before that I have following questions:
- if I want to fix the above syntax errors, I can fix the errors directly in the file under my user directory (eg: /root/betterscan-ce/quantifiedcode/backend/tasks/project/analyze.py) and go inside docker directory and run start.sh this should have code refelected. Am I correct here?
- or I have to get into container that is running port 5000 using docker exec -it command and change the code inside(eg: /srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py) and then access the web link again: http://localhost:5000. Will this work?
- or what is the procedure to fix the code for running locally
Thanks
Srikanth
from betterscan-ce.
@srikr Very good questions.
You need to do the second approach.
Exec into "docker_worker_1"
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8537eceea5a8 docker_worker_1 "python /srv/scanmyc…" 6 seconds ago Up 4 seconds docker_worker_1_1
31cdbb32113f docker_server "sh -c 'if [ -f /srv…" 6 seconds ago Up 5 seconds 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp docker_server_1
3095aa797513 rabbitmq:3.8-management-alpine "docker-entrypoint.s…" 12 hours ago Up 16 seconds (healthy) 4369/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp, :::5672->5672/tcp, 15671/tcp, 15691-15692/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp, :::15672->15672/tcp rabbitmq
410cd210b823 postgres:13.2 "docker-entrypoint.s…" 12 hours ago Up 16 seconds 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp docker_postgres_1
i.e
docker exec -it 8537eceea5a8 /bin/bash
Edit this file in container:
/usr/local/lib/python3.7/site-packages/checkmate3-0.2.0-py3.7.egg/checkmate/contrib/plugins/git/lib/repository.py
and remove
if timeout_occured:
self.stderr += "\n[process timed out after %d seconds]" % int(
timeout)
or change to:
if timeout_occured:
self.stderr = self.stderr + ("\n[process timed out after %d seconds]" % int(
timeout)).decode('utf-8')
Reset or add new project.
I think you have some timeout
from betterscan-ce.
@srikr after you modify file in container do "killall -HUP python" it will restart the workers with new changes (do it every time you modify)
from betterscan-ce.
super @marcinguy will do that and update here before my End of the day Today. Thanks thats good piece of info.
from betterscan-ce.
I fixed above issues based on your inputs but I again hit following error, seems like it not taking the private key properly
[INFO / 2022-08-18 15:44:51] Running pre-analysis hooks for project AVI (9afad0a3eee14584b88fa27f76b2be36).
[INFO / 2022-08-18 15:44:51] Fetching data for project AVI (9afad0a3eee14584b88fa27f76b2be36).
[ERROR / 2022-08-18 15:46:52] Fetching data for project AVI (9afad0a3eee14584b88fa27f76b2be36) failed!
[ERROR / 2022-08-18 15:46:52] Analysis of project AVI (9afad0a3eee14584b88fa27f76b2be36) failed!
[ERROR / 2022-08-18 15:46:52] Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 42, in hook_step
settings.hooks.call(hook, project)
File "/srv/scanmycode/quantifiedcode/helpers/hooks.py", line 42, in call
hook(*args, **kwargs)
File "/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/fetch.py", line 106, in fetch_remote
raise IOError("Cannot fetch git repository!")
OSError: Cannot fetch git repository!
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 76, in analyze_project
_analyze_project(project)
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 92, in _analyze_project
hook_step(project, "project.analyze.fetch", "Fetching data")
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 45, in hook_step
logger.error("Exception {} {}.".format(e.__class__.__name__, e.message))
AttributeError: 'OSError' object has no attribute 'message'
[INFO / 2022-08-18 15:44:51] Running pre-analysis hooks for project AVI (9afad0a3eee14584b88fa27f76b2be36).
[INFO / 2022-08-18 15:44:51] Fetching data for project AVI (9afad0a3eee14584b88fa27f76b2be36).
[ERROR / 2022-08-18 15:46:52] Fetching data for project AVI (9afad0a3eee14584b88fa27f76b2be36) failed!
[ERROR / 2022-08-18 15:46:52] Analysis of project AVI (9afad0a3eee14584b88fa27f76b2be36) failed!
[ERROR / 2022-08-18 15:46:52] Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 42, in hook_step
settings.hooks.call(hook, project)
File "/srv/scanmycode/quantifiedcode/helpers/hooks.py", line 42, in call
hook(*args, **kwargs)
File "/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/fetch.py", line 106, in fetch_remote
raise IOError("Cannot fetch git repository!")
OSError: Cannot fetch git repository!
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 76, in analyze_project
_analyze_project(project)
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 92, in _analyze_project
hook_step(project, "project.analyze.fetch", "Fetching data")
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 45, in hook_step
logger.error("Exception {} {}.".format(e.__class__.__name__, e.message))
AttributeError: 'OSError' object has no attribute 'message'
Will use your new private branch and update here...
from betterscan-ce.
I used the new private branch I was able to see the private key UI option but when I copy my private key by removing existing key listed and update I don't see it is using the new private key to fetch the git code. This the screenshot I see
Error remains the same:
[INFO / 2022-08-18 17:36:27] Running pre-analysis hooks for project AVI-New (12c026f0e75f4b62b96062e6c93b11db).
[INFO / 2022-08-18 17:36:27] Fetching data for project AVI-New (12c026f0e75f4b62b96062e6c93b11db).
[ERROR / 2022-08-18 17:36:29] Fetching data for project AVI-New (12c026f0e75f4b62b96062e6c93b11db) failed!
[ERROR / 2022-08-18 17:36:29] Analysis of project AVI-New (12c026f0e75f4b62b96062e6c93b11db) failed!
[ERROR / 2022-08-18 17:36:29] Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 42, in hook_step
settings.hooks.call(hook, project)
File "/srv/scanmycode/quantifiedcode/helpers/hooks.py", line 42, in call
hook(*args, **kwargs)
File "/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/fetch.py", line 100, in fetch_remote
raise IOError("Cannot fetch git repository!")
OSError: Cannot fetch git repository!
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 76, in analyze_project
_analyze_project(project)
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 92, in _analyze_project
hook_step(project, "project.analyze.fetch", "Fetching data")
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 45, in hook_step
logger.error("Exception {} {}.".format(e.__class__.__name__, e.message))
AttributeError: 'OSError' object has no attribute 'message'
[INFO / 2022-08-18 17:36:27] Running pre-analysis hooks for project AVI-New (12c026f0e75f4b62b96062e6c93b11db).
[INFO / 2022-08-18 17:36:27] Fetching data for project AVI-New (12c026f0e75f4b62b96062e6c93b11db).
[ERROR / 2022-08-18 17:36:29] Fetching data for project AVI-New (12c026f0e75f4b62b96062e6c93b11db) failed!
[ERROR / 2022-08-18 17:36:29] Analysis of project AVI-New (12c026f0e75f4b62b96062e6c93b11db) failed!
[ERROR / 2022-08-18 17:36:29] Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 42, in hook_step
settings.hooks.call(hook, project)
File "/srv/scanmycode/quantifiedcode/helpers/hooks.py", line 42, in call
hook(*args, **kwargs)
File "/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/fetch.py", line 100, in fetch_remote
raise IOError("Cannot fetch git repository!")
OSError: Cannot fetch git repository!
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 76, in analyze_project
_analyze_project(project)
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 92, in _analyze_project
hook_step(project, "project.analyze.fetch", "Fetching data")
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 45, in hook_step
logger.error("Exception {} {}.".format(e.__class__.__name__, e.message))
AttributeError: 'OSError' object has no attribute 'message'
will check it out if there is any other way I can directly copy this private key into the code for a variable and use from that inside a container.
Thanks
Srikanth
from betterscan-ce.
@srikr try to use the "cat" command if you are under Linux and cat the .ssh/id_rsa (private key) and copy and paste it in UI. When I used editor (vim) it added some line breaks etc
For your reference both methods in private_key branch and in UI worked by me.
from betterscan-ce.
I tried both "cat" command and using file to copy the Keys both resulted same git fetch issue. I am not sure is my private key have some issue after conversion(passphrase removal)? Note: I am able to git clone directly with old/new private keys but issue happens only with betterscan:
Before Passphrase Removal, Private Keys was having headers:
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,C6272F7C9B3DC7B1A753DB621CC87638
blablabla
-----END RSA PRIVATE KEY-----
After Passphrase Removal, Private Keys is having headers
-----BEGIN OPENSSH PRIVATE KEY-----
blablabla
-----END OPENSSH PRIVATE KEY-----
Anyways thanks @marcinguy for all the details. Appreciate it. I can and will try to figure out some other ways to achieve the same.
from betterscan-ce.
@srikr no problem. BTW my key starts with:
-----BEGIN RSA PRIVATE KEY-----
So maybe convert OPENSSH key to RSA KEY:
https://stackoverflow.com/questions/54994641/openssh-private-key-to-rsa-private-key
Anyway, after you update the key, you need to go to Settings->Danger zone->Reset project, I hope you did this.
Good luck!
from betterscan-ce.
@marcinguy I did following but same Git Fetch Issue:
a) Changed Key Header from -----BEGIN OPENSSH PRIVATE KEY----- to -----BEGIN RSA PRIVATE KEY----- no luck here
b) I did reset, reset is successful then did update resulting in git fetch issue
that forced me to look into the code and I landed into this function fetch which does the git fetch and returns non zero return code
def fetch(self, remote="origin", branch=None, ssh_identity_file=None, git_config=None, git_credentials=None):
if not re.match(r"^[\w\d]+$", remote):
raise ValueError("Invalid remote: %s" % remote)
try:
directory = tempfile.mkdtemp()
env = {'HOME': directory}
if ssh_identity_file:
# To Do: Security audit
logger.debug("Fetching with SSH key")
env.update({'CONFIG_FILE': directory+"/ssh_config",
'GIT_SSH': self._get_ssh_wrapper()})
with open(directory+"/ssh_config", "w") as ssh_config_file:
ssh_config_file.write(
self._get_ssh_config(ssh_identity_file))
if git_config:
env.update({'GIT_CONFIG_NOSYSTEM': '1'})
with open(directory+"/.gitconfig", "w") as git_config_file:
git_config_file.write(git_config)
if git_credentials:
with open(directory+"/.git-credentials", "w") as git_credentials_file:
git_credentials_file.write(git_credentials)
extra_args = []
if branch is not None:
extra_args.append(branch)
return_code, stdout = self.call(
["git", "fetch", remote]+extra_args, env=env, timeout=120)
finally:
shutil.rmtree(directory)
return return_code
I am presuming based on the log error we are not passing values to any of this parameters (ssh_identity_file, git_config, git_credentials( which makes me to think it would have executed following line of code:
return_code, stdout = self.call(
["git", "fetch", remote]+extra_args, env=env, timeout=120)
Now are we expecting clone should happen in 120 seconds timeout else we complain git fetch error.. I know the code what we are cloning will take more than 120 seconds for sure to download..
from betterscan-ce.
@srikr ok
If your projects are that big initial scanning will also take long :)
Next rescans, commits scan should be fast (seconds to minutes)
See this:
https://github.com/marcinguy/betterscan-ce/wiki/Big(er)-Projects-Scanning
from betterscan-ce.
@marcinguy seems like everytime the git fetch is failing after 120 seconds so I went inside the container(docker_worker_1) and change the timeout as below and did "killall -HUP python"
return_code, stdout = self.call(
["git", "fetch", remote]+extra_args, env=env, timeout=14400)
Still I see git fetch is failing after 120 seconds as shown in below screenshot. How can I incorporate the above modified code into the container.
from betterscan-ce.
@srikr Try now. Remove all container images and build or get from dockerhub. I added the 14400 timeout to master.
Also you can now clone with standard SSH format:
i.e
[email protected]:marcinguy/test.git
etc
from betterscan-ce.
thanks @marcinguy finally I was able to scan but resulted in the below error:
[INFO / 2022-08-21 11:22:30] Annotating and saving file revisions...
[INFO / 2022-08-21 11:22:31] Analyzing and saving: 450 - 460 (9550 remaining)
[INFO / 2022-08-21 11:22:31] Analyzing file revision test/avitest/functional/l7/Old/suite_N_S/configs/L7SrvrSideSSLHdrChk.json
[INFO / 2022-08-21 11:22:53] Analyzing file revision test/Stress/Tests/pool1/certs/ca/intermediate/csr/www.systest3427.com.csr.pem
[INFO / 2022-08-21 11:23:18] Analyzing file revision test/Stress/Tests/pool1/certs/ca/intermediate/private/www.systest3668.com.key.pem
[ERROR / 2022-08-21 11:23:37] Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 76, in analyze_project
_analyze_project(project)
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 96, in _analyze_project
hook_step(project, "project.analyze.analyze", "Running analysis")
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 42, in hook_step
settings.hooks.call(hook, project)
File "/srv/scanmycode/quantifiedcode/helpers/hooks.py", line 42, in call
hook(*args, **kwargs)
File "/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/analyze.py", line 74, in analyze
analyze_branch(project, branch)
File "/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/analyze.py", line 112, in analyze_branch
analyze_command.run()
File "/usr/local/lib/python3.7/site-packages/checkmate3-0.2.0-py3.7.egg/checkmate/contrib/plugins/git/commands/analyze.py", line 339, in run
snapshots_to_analyze, diffs_to_generate)
File "/usr/local/lib/python3.7/site-packages/checkmate3-0.2.0-py3.7.egg/checkmate/contrib/plugins/git/commands/analyze.py", line 149, in analyze_and_generate_diffs
file_revisions, save_if_empty=True, snapshot=snapshot)
File "/usr/local/lib/python3.7/site-packages/checkmate3-0.2.0-py3.7.egg/checkmate/lib/code/environment.py", line 690, in analyze
file_revisions_slice)
File "/usr/local/lib/python3.7/site-packages/checkmate3-0.2.0-py3.7.egg/checkmate/lib/code/environment.py", line 542, in analyze_file_revisions
for analyzer_name, analyzer_params in list(self.analyzers.items())
File "/usr/local/lib/python3.7/site-packages/checkmate3-0.2.0-py3.7.egg/checkmate/lib/code/environment.py", line 560, in analyze_file_revision
analyzer_results = analyzer.analyze(file_revision)
File "/usr/local/lib/python3.7/site-packages/checkmate3-0.2.0-py3.7.egg/checkmate/contrib/plugins/all/semgrepccpp/analyzer.py", line 54, in analyze
f.name])
File "/usr/local/lib/python3.7/subprocess.py", line 411, in check_output
**kwargs).stdout
File "/usr/local/lib/python3.7/subprocess.py", line 490, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "/usr/local/lib/python3.7/subprocess.py", line 951, in communicate
stdout = self.stdout.read()
File "/usr/local/lib/python3.7/site-packages/billiard/common.py", line 119, in _shutdown_cleanup
sys.exit(-(256 - signum))
File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 283, in exit
return _exit(status)
SystemExit: -241
from betterscan-ce.
@srikr Was it after 14400 seconds? If yes, you have to click "Check for new commits". It will pick up where it stopped.
It scanned 460 items in ca. 4 hours
Analyzing and saving: 450 - 460 (9550 remaining)
21 x times more items to scan still (9550), ca. maybe 3 days, in your case now
After you scan the whole thing, next scans will be only differences (seconds, minutes)
You can queue up "Check for new commits", write a script that will do it every 14000 seconds. Many ideas are possible.
from betterscan-ce.
yes @marcinguy I started 2 days ago after your response came. From the UI it seems like Git Fetch downloaded all the code at Sun, 21 Aug 2022 08:17:59 GMT.:
After that it started analysing and failed.
Yes it would be definitely more than 4 hours. I have also clicked "Check for new commits" at the top. Will look into the progress and get back to you thanks.
from betterscan-ce.
@marcinguy I am little confused I see following screens:
a) when clicked on Issues Its saying analysis is in progress
b) when clicked on Logs I don't see any Logs which shows in progress
c) I also click "Check for new commits" and got queued.. when will it run? Is there some time after which it will run?
Also one more question I have, is there a way I can run static analysis on all python files in code which is already downloaded to local directories or it is expected atleast one time we need to clone via betterscan for to get the code.
Thanks
Srikanth
from betterscan-ce.
@srikr check if all 4 Betterscan containers are running. I think the Worker container could be off (docker ps)
Please paste Log tab items that you see.
You can use Betterscanin in folder and generate Terminal or HTML output, if you use Webinterface, you need to stay within interface. Can be changed in the future.
How many files your project have?
from betterscan-ce.
@marcinguy yes you are correct, docker_worker_1 was down, I restarted the processes again.
This was the Error Logs during that time
[INFO / 2022-08-23 15:06:59] Running pre-analysis hooks for project AVI (4adc50aa720948f685fc373398463a5c).
[INFO / 2022-08-23 15:06:59] Fetching data for project AVI (4adc50aa720948f685fc373398463a5c).
[ERROR / 2022-08-23 15:08:06] Fetching data for project AVI (4adc50aa720948f685fc373398463a5c) failed!
[ERROR / 2022-08-23 15:08:06] Analysis of project AVI (4adc50aa720948f685fc373398463a5c) failed!
[ERROR / 2022-08-23 15:08:06] Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 42, in hook_step
settings.hooks.call(hook, project)
File "/srv/scanmycode/quantifiedcode/helpers/hooks.py", line 42, in call
hook(*args, **kwargs)
File "/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/fetch.py", line 71, in fetch_remote
shutil.copytree(repository.path, tmp_repo_path)
File "/usr/local/lib/python3.7/shutil.py", line 368, in copytree
raise Error(errors)
shutil.Error: [('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/objects/pack/pack-d5567bc915d8033c81fdbc8c702aa59b7a65f4ab.pack', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/objects/pack/pack-d5567bc915d8033c81fdbc8c702aa59b7a65f4ab.pack', '[Errno 28] No space left on device'), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/objects/info', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/objects/info', "[Errno 28] No space left on device: '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/objects/info'"), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/description', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/description', '[Errno 28] No space left on device'), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/HEAD', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/HEAD', '[Errno 28] No space left on device'), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/branches', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/branches', "[Errno 28] No space left on device: '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/branches'"), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/hooks', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/hooks', "[Errno 28] No space left on device: '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/hooks'"), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/config', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/config', '[Errno 28] No space left on device'), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/logs', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/logs', "[Errno 28] No space left on device: '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/logs'"), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/info', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/info', "[Errno 28] No space left on device: '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/info'")]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 76, in analyze_project
_analyze_project(project)
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 92, in _analyze_project
hook_step(project, "project.analyze.fetch", "Fetching data")
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 45, in hook_step
logger.error("Exception {} {}.".format(e.__class__.__name__, e.message))
AttributeError: 'Error' object has no attribute 'message'
[INFO / 2022-08-23 15:06:59] Running pre-analysis hooks for project AVI (4adc50aa720948f685fc373398463a5c).
[INFO / 2022-08-23 15:06:59] Fetching data for project AVI (4adc50aa720948f685fc373398463a5c).
[ERROR / 2022-08-23 15:08:06] Fetching data for project AVI (4adc50aa720948f685fc373398463a5c) failed!
[ERROR / 2022-08-23 15:08:06] Analysis of project AVI (4adc50aa720948f685fc373398463a5c) failed!
[ERROR / 2022-08-23 15:08:06] Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 42, in hook_step
settings.hooks.call(hook, project)
File "/srv/scanmycode/quantifiedcode/helpers/hooks.py", line 42, in call
hook(*args, **kwargs)
File "/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/fetch.py", line 71, in fetch_remote
shutil.copytree(repository.path, tmp_repo_path)
File "/usr/local/lib/python3.7/shutil.py", line 368, in copytree
raise Error(errors)
shutil.Error: [('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/objects/pack/pack-d5567bc915d8033c81fdbc8c702aa59b7a65f4ab.pack', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/objects/pack/pack-d5567bc915d8033c81fdbc8c702aa59b7a65f4ab.pack', '[Errno 28] No space left on device'), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/objects/info', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/objects/info', "[Errno 28] No space left on device: '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/objects/info'"), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/description', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/description', '[Errno 28] No space left on device'), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/HEAD', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/HEAD', '[Errno 28] No space left on device'), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/branches', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/branches', "[Errno 28] No space left on device: '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/branches'"), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/hooks', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/hooks', "[Errno 28] No space left on device: '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/hooks'"), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/config', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/config', '[Errno 28] No space left on device'), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/logs', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/logs', "[Errno 28] No space left on device: '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/logs'"), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/info', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/info', "[Errno 28] No space left on device: '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/info'")]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 76, in analyze_project
_analyze_project(project)
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 92, in _analyze_project
hook_step(project, "project.analyze.fetch", "Fetching data")
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 45, in hook_step
logger.error("Exception {} {}.".format(e.__class__.__name__, e.message))
AttributeError: 'Error' object has no attribute 'message'
[INFO / 2022-08-23 15:06:59] Running pre-analysis hooks for project AVI (4adc50aa720948f685fc373398463a5c).
[INFO / 2022-08-23 15:06:59] Fetching data for project AVI (4adc50aa720948f685fc373398463a5c).
[ERROR / 2022-08-23 15:08:06] Fetching data for project AVI (4adc50aa720948f685fc373398463a5c) failed!
[ERROR / 2022-08-23 15:08:06] Analysis of project AVI (4adc50aa720948f685fc373398463a5c) failed!
[ERROR / 2022-08-23 15:08:06] Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 42, in hook_step
settings.hooks.call(hook, project)
File "/srv/scanmycode/quantifiedcode/helpers/hooks.py", line 42, in call
hook(*args, **kwargs)
File "/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/fetch.py", line 71, in fetch_remote
shutil.copytree(repository.path, tmp_repo_path)
File "/usr/local/lib/python3.7/shutil.py", line 368, in copytree
raise Error(errors)
shutil.Error: [('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/objects/pack/pack-d5567bc915d8033c81fdbc8c702aa59b7a65f4ab.pack', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/objects/pack/pack-d5567bc915d8033c81fdbc8c702aa59b7a65f4ab.pack', '[Errno 28] No space left on device'), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/objects/info', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/objects/info', "[Errno 28] No space left on device: '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/objects/info'"), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/description', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/description', '[Errno 28] No space left on device'), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/HEAD', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/HEAD', '[Errno 28] No space left on device'), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/branches', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/branches', "[Errno 28] No space left on device: '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/branches'"), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/hooks', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/hooks', "[Errno 28] No space left on device: '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/hooks'"), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/config', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/config', '[Errno 28] No space left on device'), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/logs', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/logs', "[Errno 28] No space left on device: '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/logs'"), ('/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d/.git/info', '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/info', "[Errno 28] No space left on device: '/srv/scanmycode/quantifiedcode/data/repositories/a833c9a439ea4badadf39d72d70e156d_temp/.git/info'")]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 76, in analyze_project
_analyze_project(project)
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 92, in _analyze_project
hook_step(project, "project.analyze.fetch", "Fetching data")
File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 45, in hook_step
logger.error("Exception {} {}.".format(e.__class__.__name__, e.message))
AttributeError: 'Error' object has no attribute 'message'
How to use Betterscanin in folder and generate HTML output?
I am not sure about files but its whole a lot of files(mix of C, C++, Python, Java etc)
from betterscan-ce.
@srikr You have too little disk space (see "[Errno 28] No space left on device'")
I modified Dockerfiles and limits a little bit to make it better (restarts and bigger limits)
Actually, everything is in the README.md
To scan a cloned repo (cloned with git clone), just cd/change to that dir and run this:
sh <(curl https://betterscan.io/cli-html.sh)
For HTML report.
It can last in your case few days (as I said), but should not timeout in this approach.
from betterscan-ce.
thanks @marcinguy will try CLI approach as well. I have total of 105GB in that VM. Before I start the test there were some 40GB disk was free, Now I see it is like:
# df -kH
Filesystem Size Used Avail Use% Mounted on
udev 4.2G 0 4.2G 0% /dev
tmpfs 835M 1.6M 833M 1% /run
/dev/sda1 105G 90G 9.5G 91% /
tmpfs 4.2G 0 4.2G 0% /dev/shm
tmpfs 5.3M 0 5.3M 0% /run/lock
overlay 105G 90G 9.5G 91% /var/lib/docker/overlay2/21e5ef9d49aea0d88fc7e247acc82f330a176e0d40ceac259f705687953bbd17/merged
tmpfs 835M 70k 835M 1% /run/user/0
overlay 105G 90G 9.5G 91% /var/lib/docker/overlay2/205ffe7fefaf535ea829dace2f53c0ee5f76f2605637700ad9a14c240abc164d/merged
overlay 105G 90G 9.5G 91% /var/lib/docker/overlay2/a173e65ec34c3ebeee89a8f56d1cdd5be3b12a05ddb0f538e2aaa3ed2b3b1a52/merged
overlay 105G 90G 9.5G 91% /var/lib/docker/overlay2/5d7a9e902aa742720d81af7528ecf7f11c1e3f746381beb4acdfdc5856bf3824/merged
tmpfs 835M 66k 835M 1% /run/user/129
overlay 105G 90G 9.5G 91% /var/lib/docker/overlay2/0dc3793b27e9ef8ef996cb2f01a4ace55fc113ff23f56cd790b500fbd116313e/merged
so basically how much disk space I need to have before running the analysis?
from betterscan-ce.
Closing due to inactivity
from betterscan-ce.
Related Issues (20)
- Private github repo fails to fetch despite adding public ssh key to github account? HOT 1
- only enable specified plugins HOT 2
- CE edition default run requires PRO to view files? HOT 2
- Purchased Pro License in September haven't recieved it yet HOT 2
- No space left on device during build HOT 8
- betterscan/betterscan-ce image cannot be found HOT 3
- Error execution HOT 2
- Issue building docker images HOT 1
- Errors with server (backend) and worker images HOT 3
- Please consider adopting OpenSSF Scorecard HOT 1
- Dependency files should always be scanned HOT 7
- Update osv-scan to v1.1.0 to include support for NuGet and PIP HOT 1
- How can SBOM in cyclonedx format be integrated? HOT 2
- Docker is failing during building HOT 2
- Error during analysis HOT 4
- setup fails to create tasks directory and blocks worker from working on tasks HOT 2
- Getting error in checkmate issue HOT 12
- Is analyzers\yara\WShell_THOR_Webshells.yar an active exploit? HOT 3
- find_unicode_control2.py incorrectly flags UTF-8 BOM as a Trojan Source exploit HOT 1
- Checkmate error(?) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from betterscan-ce.