Code Monkey home page Code Monkey logo

codeanalysis's Introduction


TCA, Tencent Cloud Code Analysis
code-named: CodeDog

license docs

English | 简体中文

TCA Official Website

https://cloud.tencent.com/product/tcap

TCA Github Pages

https://tencent.github.io/CodeAnalysis/

https://github.com/TCATools

Repo Mirror

https://git.code.tencent.com/Tencent_Open_Source/CodeAnalysis.git

What is TCA

Tencent Cloud Code Analysis (TCA for short, code-named CodeDog inside the company early) is a comprehensive platform for code analysis and issue tracking. TCA consist of three components, server, web and client. It integrates of a number of self-developed tools, and also supports dynamic integration of code analysis tools in various programming languages.

Code analysis is a technology, using lexical analysis, syntax analysis, control-flow analysis, data-flow analysis to make a comprehensive analysis of the code, so as to verify whether the code meets the requirements of normative, security, reliability, maintainability and other indicators.

Using TCA can help team find normative, structural, security vulnerabilities and other issues in the code, continuously monitor the quality of the project code and issue alerts. At the same time, TCA opens up APIs to support connection with upstream and downstream systems, so as to integrate code analysis capabilities, ensure code quality, and be more conducive to inheriting an excellent team code culture.

组件图

流程图

Experience

Apply Link

Key Features

  1. Language support: It supports Java/C++/Objective-C/C#/JavaScript/Python/Go/PHP and more, covering common programming languages.
  2. Code inspection: Discover code quality defects, code specification problems, code security vulnerabilities, invalid codes, etc. At present, many self-developed and well-known open source analysis tools have been integrated. The layered architecture can support rapid self-service management tools for teams.
  3. Code measurement: Support comprehensive measurement of code from the three dimensions of code cyclomatic complexity, code repetition rate and code statistics.
  4. DevOps integration: The client can be started via the command line. Standard APIs support connection to upstream and downstream systems, and various DevOps systems.

Getting Started

Community

  • WeChat Official Account: 「腾讯云静态分析」. Follow and send "Join the group" to join the official open source communication WeChat group.

  • WeChat Group:

  • QQ Group: 361791391

  • Discussion

  • Wiki

  • White Paper

Changelogs

Contributing

License

TCA is MIT licensed

TCA Team Member

腾讯云代码分析团队成员

codeanalysis's People

Contributors

aaaa-j avatar akitata avatar anyuan95 avatar auula avatar baozidai avatar bensonhome avatar cloudusers avatar cocorunning avatar cyw3 avatar d0n9 avatar dependabot[bot] avatar didebughu avatar faberiii avatar freyalo avatar g3g4x5x6 avatar lingghh avatar m0renet avatar mapyjjj avatar nickcdon avatar owlmk avatar russgan avatar sanshix avatar xiezheng-xd avatar yalechen-cyw3 avatar yql70 avatar yyyangw avatar zhang9w0v5 avatar zhuay-a avatar zthzhang111 avatar zzzfwww avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

codeanalysis's Issues

ModuleNotFoundError: No module named 'task.scmmgr'

执行:python3 codepuppy.py localscan报如下错误!

[root@idss-centos-7 client]# python3 -V
Python 3.8.11
[root@idss-centos-7 client]#
[root@centos-7 client]# python3 codepuppy.py localscan
Traceback (most recent call last):
File "codepuppy.py", line 22, in
from tool.util.pythontool import PythonTool
File "/opt/Tencent/CodeAnalysis/client/tool/util/pythontool.py", line 15, in
from task.scmmgr import SCMMgr
ModuleNotFoundError: No module named 'task.scmmgr'
[root@centos-7 client]#
[root@centos-7 client]#

427-ERROR-node.localtask.taskprocessmgr: 扫描失败! 工具(code count)扫描进程为空,请联系管理员配置工具进程!

在win7和ubuntu里使用client都出现了这个问题

2022-01-12 14:38:53,601-INFO-util.logutil: Initialize tools done.(use time: 0.03s)
2022-01-12 14:38:53,616-DEBUG-util.envset: 设置生效清单: []
2022-01-12 14:38:53,632-DEBUG-util.envset: 设置生效清单: []
2022-01-12 14:38:54,427-DEBUG-util.envset: 设置生效清单: []
2022-01-12 14:38:54,427-ERROR-node.localtask.taskprocessmgr: 扫描失败! 工具(codecount)扫描进程为空,请联系管理员配置工具进程!

牛批

竞品分析结果:100%覆盖当前开源工具的所有规则

code analysis stuck in init tools

`[Tools init]: 10%|█████ | 1/10 [00:04<00:36, 4.10s/it]2022-01-10 18:35:55,168-WARNING-cmdgit: no .git/config file in work_tree
2022-01-10 18:52:33,028-INFO-util.logutil:


代码分析结果:
NodeError: Error[203]: error: external filter 'git-lfs filter-process' failed
fatal: bin/model/lexeme/model.tflite: smudge filter lfs failed


2022-01-10 18:52:33,028-INFO-util.logutil: json report: XXX\CodeAnalysis/client/scan_status.json
2022-01-10 18:52:33,028-ERROR-util.logutil: NodeError: Error[203]: error: external filter 'git-lfs filter-process' failed
fatal: bin/model/lexeme/model.tflite: smudge filter lfs failed
Traceback (most recent call last):
File "XXX\CodeAnalysis\client\node\gitload.py", line 153, in __retry_checkout
self._scm_client.checkout()
File "src_cmdgit.py", line 1585, in src._cmdgit.CmdGit.checkout
File "src_cmdgit.py", line 1426, in src._cmdgit.CmdGit._clone
File "src_cmdgit.py", line 224, in src._cmdgit.CmdGit._cmd_error_check
util._scmbase.ScmCommandError: error: external filter 'git-lfs filter-process' failed
fatal: bin/model/lexeme/model.tflite: smudge filter lfs failed`

scan_status.json:

{
"status": "error",
"error_code": 203,
"url": "http://XXXX/code-analysis/repos/1/projects/1/scan-history",
"text": "扫描异常",
"description": "NodeError: Error[203]: error: external filter 'git-lfs filter-process' failed\nfatal: bin/model/lexeme/model.tflite: smudge filter lfs failed",
"scan_report": {}
}

Docker部署文档完善

使用Docker部署过程中
请增加
wait dbEnter password:
.Enter password:
两个字段的说明
image

通过 docker-compose 启动时提示输入密码是多少?

sh ./compose_init.sh

[+] Running 19/19
⠿ redis Pulled 19.7s
⠿ b8f262c62ec6 Pull complete 15.7s
⠿ 93789b5343a5 Pull complete 15.8s
⠿ 49cdbb315637 Pull complete 15.9s
⠿ 2c1ff453e5c9 Pull complete 16.2s
⠿ 9341ee0a5d4a Pull complete 16.2s
⠿ 770829e1df34 Pull complete 16.3s
⠿ mysql Pulled 60.5s
⠿ 177e7ef0df69 Pull complete 5.4s
⠿ cac25352c4c8 Pull complete 5.5s
⠿ 8585afabb40a Pull complete 5.7s
⠿ 1e4af4996053 Pull complete 5.8s
⠿ c326522894da Pull complete 5.9s
⠿ 9020d6b6b171 Pull complete 9.4s
⠿ 55eb37ec6e5f Pull complete 9.5s
⠿ 1a9d2f77e0e7 Pull complete 9.6s
⠿ d7e648ad64aa Pull complete 56.9s
⠿ 4120d828ea6b Pull complete 57.1s
⠿ 3b39dc5451af Pull complete 57.2s
[+] Running 3/3
⠿ Network codeanalysis_default Created 0.0s
⠿ Container codeanalysis-mysql-1 Started
⠿ Container codeanalysis-redis-1 Started 0.7s
wait dbEnter password:
.Enter password:

upload_file windows环境报错

========================================================================
2022-01-11 15:33:50,366-INFO-util.logutil: Task_89 (Flake8) starts ...
2022-01-11 15:34:00,491-INFO-util.logutil: Task_89 is done.
2022-01-11 15:34:00,998-ERROR-util.wrapper: method(upload_file) call fails on error: Expecting value: line 1 column 1 (char 0)
2022-01-11 15:34:00,999-INFO-util.wrapper: retrying method(upload_file) after 5 seconds
2022-01-11 15:34:06,009-ERROR-util.wrapper: method(upload_file) call fails on error: Expecting value: line 1 column 1 (char 0)
2022-01-11 15:34:06,009-INFO-util.wrapper: retrying method(upload_file) after 5 seconds
2022-01-11 15:34:11,021-ERROR-util.wrapper: method(upload_file) call fails on error: Expecting value: line 1 column 1 (char 0)
2022-01-11 15:34:11,022-WARNING-util.wrapper: method(upload_file) call still fails after 2 retries

2022-01-11 15:34:11,025-ERROR-util.logutil: JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "H:\CodeAnalysis-main\client\node\localrunner.py", line 392, in run
run_task_mgr.scan_project(cur_execute_request_list, proj_conf)
File "H:\CodeAnalysis-main\client\node\localtask\runlocaltask.py", line 96, in scan_project
self._proj_id).run()
File "H:\CodeAnalysis-main\client\node\localtask\runtask.py", line 170, in run
ResultCheck.upload_task_result(task, self._proj_id)
File "H:\CodeAnalysis-main\client\node\localtask\runtask.py", line 97, in upload_task_result
data_url, log_url = TaskRunner.upload_result_detail(proj_id, task.task_id, task_dir, data, task.task_log)
File "node\taskrunner.py", line 59, in node.taskrunner.TaskRunner.upload_result_detail
File "util\wrapper.py", line 51, in util.wrapper._RetryMethod.call
File "util\wrapper.py", line 107, in util.wrapper.Retry.__retry_call
File "util\wrapper.py", line 92, in util.wrapper.Retry.__retry_call
File "H:\CodeAnalysis-main\client\util\api\fileserver.py", line 103, in upload_file
return self.__upload_data(fp, rel_url, headers)
File "H:\CodeAnalysis-main\client\util\api\fileserver.py", line 87, in __upload_data
HttpClient(self._server_url, rel_url, headers=headers, data=data, proxies=self.proxies).put()
File "H:\CodeAnalysis-main\client\util\api\httpclient.py", line 82, in put
result = HttpRequest.request(url=self.url, headers=self.headers, body=self.data, method="PUT")
File "H:\CodeAnalysis-main\client\util\api\httpclient.py", line 45, in request
dict_result = json.loads(str_result)
File "C:\Python37\lib\json_init
.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\Python37\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Python37\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

下载的main分支的代码,最后结果上传的时候报错了

codepuppy localscan raise Error: No module named 'util.subprocc'

root@iZj6c2l4nh4l1ykl9h4r99Z:~/CodeAnalysis/client# python3 codepuppy.py localscan
Traceback (most recent call last):
  File "codepuppy.py", line 18, in <module>
    from util.gitconfig import GitConfig
  File "/root/CodeAnalysis/client/util/gitconfig.py", line 10, in <module>
    from util.subprocc import SubProcController
ModuleNotFoundError: No module named 'util.subprocc'
root@iZj6c2l4nh4l1ykl9h4r99Z:~/CodeAnalysis/client# python3 --version
Python 3.8.10

报错

➜ client git:(main) ✗ python3 codepuppy.py localscan Traceback (most recent call last): File "/Users/xxxxx/tools/CodeAnalysis/client/codepuppy.py", line 18, in <module> from util.gitconfig import GitConfig File "/Users/xxxxx/tools/CodeAnalysis/client/util/gitconfig.py", line 10, in <module> from util.subprocc import SubProcController ModuleNotFoundError: No module named 'util.subprocc' ➜ client git:(main) ✗ pwd /Users/xxxxx/tools/CodeAnalysis/client

按照教程安装了pip3 install -r client/requirements/app_reqs.pip
也执行了
进入到client/requirements目录
在命令行中执行安装脚本install.sh(linux/mac环境)或install.bat(windows环境)

运行报错

没注意需要先运行./compose_init.sh,在运行docker-compose up -d 中途通过ctrl+z中断后,删除docker容器重启运行./compose_init.sh有如下提示:./compose_init.sh: 7: Syntax error: "(" unexpected

Killed循环运行

Killed
Sending build context to Docker daemon 1.613MB
Step 1/6 : FROM python:3.7.12-slim
---> d3c9ad326043
Step 2/6 : ARG EXTRA_TOOLS="gnupg curl wget jq vim-tiny net-tools procps python3-dev default-libmysqlclient-dev locales inotify-tools gcc subversion git telnet iputils-ping vim openssh-client"
---> Using cache
---> 4027bf0e6245
Step 3/6 : RUN set -ex && cd / && apt-get update && apt-get install -y --no-install-recommends $EXTRA_TOOLS && apt-get update && apt-get install -y mariadb-client && apt-get clean && echo "LC_ALL=zh_CN.UTF-8" >> /etc/environment && echo "zh_CN.UTF-8 UTF-8" > /etc/locale.gen && echo "LANG=zh_CN.UTF-8" > /etc/locale.conf && locale-gen && ln -sf /usr/share/zoneinfo/Asia/Hong_Kong /etc/localtime && rm -rf /var/cache/apt/* /root/.cache
---> Using cache
---> 8e3f43254e55
Step 4/6 : WORKDIR /var/www/django/codedog
---> Using cache
---> c64d8d3ca0b0
Step 5/6 : COPY . .
---> 22854c958f51
Step 6/6 : RUN mkdir -p log/ && pip install -U setuptools pip && pip install -r requirements.txt
---> Running in e2ac50ac740e
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (57.5.0)
Collecting setuptools
Downloading setuptools-60.5.0-py3-none-any.whl (958 kB)
Requirement already satisfied: pip in /usr/local/lib/python3.7/site-packages (21.2.4)
Collecting pip
Downloading pip-21.3.1-py3-none-any.whl (1.7 MB)
Installing collected packages: setuptools, pip
Attempting uninstall: setuptools
Found existing installation: setuptools 57.5.0
Uninstalling setuptools-57.5.0:
Successfully uninstalled setuptools-57.5.0
Killed
Sending build context to Docker daemon 86.27kB
Step 1/6 : FROM python:3.7.12-slim
---> d3c9ad326043
Step 2/6 : ARG EXTRA_TOOLS="gnupg curl wget jq vim-tiny net-tools procps python3-dev default-libmysqlclient-dev locales inotify-tools gcc subversion git telnet iputils-ping vim openssh-client"
---> Using cache
---> 4027bf0e6245
Step 3/6 : RUN set -ex && cd / && apt-get update && apt-get install -y --no-install-recommends $EXTRA_TOOLS && apt-get update && apt-get install -y mariadb-client && apt-get clean && echo "LC_ALL=zh_CN.UTF-8" >> /etc/environment && echo "zh_CN.UTF-8 UTF-8" > /etc/locale.gen && echo "LANG=zh_CN.UTF-8" > /etc/locale.conf && locale-gen && ln -sf /usr/share/zoneinfo/Asia/Hong_Kong /etc/localtime && rm -rf /var/cache/apt/* /root/.cache
---> Using cache
---> 8e3f43254e55
Step 4/6 : WORKDIR /var/www/django/codedog
---> Using cache
---> c64d8d3ca0b0
Step 5/6 : COPY . .
---> 563e879a8cd5
Step 6/6 : RUN mkdir -p log/ && pip install -U setuptools pip && pip install -r requirements.txt
---> Running in 55874b73280d
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (57.5.0)
Collecting setuptools
Downloading setuptools-60.5.0-py3-none-any.whl (958 kB)
Requirement already satisfied: pip in /usr/local/lib/python3.7/site-packages (21.2.4)
Collecting pip
Downloading pip-21.3.1-py3-none-any.whl (1.7 MB)
Installing collected packages: setuptools, pip
Attempting uninstall: setuptools
Found existing installation: setuptools 57.5.0
Uninstalling setuptools-57.5.0:
Successfully uninstalled setuptools-57.5.0

报错

今天替换了python3.7,一直报这个错, 不知道怎么解决。
image

NodeError: Error[204]: 当前环境不支持cobra工具步骤:['analyze']

由于是windows机器,因此使用和server一样的php镜像做了个客户端的镜像,如下,将一些github下载慢的包等手动安装了【不手动安装也一样有问题】:

FROM python:3.7.12-slim

#肯能用不了这么多的软件包,为了快速构建使用server的构建缓存,保留一致
ARG EXTRA_TOOLS="gnupg curl wget jq vim-tiny net-tools procps python3-dev default-libmysqlclient-dev locales inotify-tools gcc subversion git telnet iputils-ping vim openssh-client"
COPY ./sources.list /etc/apt/sources.list

RUN set -ex && cd / \
    && apt-get update \
    && apt-get install -y --no-install-recommends $EXTRA_TOOLS \
    && apt-get update \
    && apt-get install -y mariadb-client \
    && apt-get clean \
    && echo "LC_ALL=zh_CN.UTF-8" >> /etc/environment \
    && echo "zh_CN.UTF-8 UTF-8" > /etc/locale.gen \
    && echo "LANG=zh_CN.UTF-8" > /etc/locale.conf \
    && locale-gen \
    && ln -sf /usr/share/zoneinfo/Asia/Hong_Kong /etc/localtime \
    && rm -rf /var/cache/apt/* /root/.cache

COPY ./client /opt/client

RUN apt-get install -y flex bison #phantomjs
# https://phantomjs.org/download.html 下载 phantomjs
# docker cp .\pkg\phantomjs tca-client:/usr/bin/
COPY ./pkg/phantomjs /usr/bin/phantomjs
RUN apt-get install -y bzip2 fontconfig && \
    chmod +x /usr/bin/phantomjs && \
    echo "export OPENSSL_CONF=/etc/ssl/" >> /etc/profile

RUN pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -r /opt/client/requirements/app_reqs.pip
#cobra
RUN pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -r /opt/client/requirements/tool/cobra_requirements.txt

#从https://github.com/TCATools/下载的个工具打包后执行展开放入
ADD ./pkg/datatool.tar.gz /opt/client/

# https://hub.fastgit.org/Tencent/CodeAnalysis/issues/42
#RUN /opt/client/requirements/install.sh && \
RUN pip install pylint semgrep && \
    mkdir -p /opt/code
RUN ln -s /usr/lib/x86_64-linux-gnu/libffi.so.7 /usr/lib/x86_64-linux-gnu/libffi.so

然后,运行这个客户端 docker run -it --name tca-client --network host tca-client:2022 sh
执行 cd /opt/client/; python3 codepuppy.py localscan 出现如下的错误

[Tools init]: 100%|█████████████████████████████████████████████████| 13/13 [11:13<00:00, 51.81s/it]
2022-01-12 15:08:55,985-INFO-util.logutil: Initialize tools done.(use time: 673.59s)
2022-01-12 15:08:57,514-ERROR-util.logutil: cobra不可用,建议客户使用data/tools/cobra-v2.0.0-alpha.5/install.sh脚本进行
安装!
2022-01-12 15:08:57,515-INFO-util.logutil:
****************************************************************************************************
代码分析结果:
NodeError: Error[204]: 当前环境不支持cobra工具步骤:['analyze'].
****************************************************************************************************
2022-01-12 15:08:57,515-INFO-util.logutil: json report: /opt/client/scan_status.json
2022-01-12 15:08:57,515-ERROR-util.logutil: NodeError: Error[204]: 当前环境不支持cobra工具步骤:['analyze'].
Traceback (most recent call last):
  File "/opt/client/node/localrunner.py", line 380, in run
    request_generator.generate_request(proj_conf)
  File "/opt/client/node/localtask/requestgenerator.py", line 153, in generate_request
    execute_request_list = self._get_execute_request_list(job_context, task_list)
  File "/opt/client/node/localtask/requestgenerator.py", line 183, in _get_execute_request_list
    raise NodeError(code=errcode.E_NODE_TASK_CONFIG, msg=f"当前环境不支持{task_name}工具步骤:{remote_procs}.")
util.exceptions.NodeError: Error[204]: 当前环境不支持cobra工具步骤:['analyze'].

按照 #42 问题的描述,这个是完全正常的
# /opt/client/data/tools/common-cobra-v2.0.0-alpha.5/cobra.py --help
usage: cobra [-h] [-t <target>] [-f <format>] [-o <output>] [-r <rule_id>]
             [-d] [-sid SID] [-H <host>] [-P <port>]

运行报错

Traceback (most recent call last):
File "codepuppy.py", line 18, in
from util.gitconfig import GitConfig
File "/home/test/CodeAnalysis/client/util/gitconfig.py", line 10, in
from util.subprocc import SubProcController
ModuleNotFoundError: No module named 'util.subprocc'

Missing the Code of Conduct

Hello,
This project is currently missing the Code of Conduct. Do you have any plan on adding that? If you want me to do the task using a PR, then I would also need an address (email address) so that I can also enlist that in the Code of Conduct.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.