Code Monkey home page Code Monkey logo

sirius's Introduction

Sirius是什么

Sirius是由百分点科技开发的一套云中心服务管理dashboard。

QQ讨论群:378093490 Sirius

概览

目前支持已经管理的服务统计功能:

云服务

目前包含HDFS云和CODIS云服务的管理

HDFS云

目前支持HDFS文件系统的增加, 删除, 恢复, 分享, 上传, 下载, 统计等功能:

CODIS云

目前支持codis信息, 主机信息:

codis信息

主机信息

云容器

k8s相关监控

离线计算任务

k8s使用说明

用户管理

支持对用户的增加, 删除功能:

如何部署

普通环境部署

(1) 代码拷贝

git clone https://github.com/baifendian/Sirius.git

(2) 创建Sirius隔离环境: virtualenv

[root@bjlg-24p100-40-xtjc06 opt]# pip install virtualenv   #安装 virtualenv 环境
[root@bjlg-24p100-40-xtjc06 opt]# virtualenv  sirius-pyenv --no-site-packages  #创建属于sirius的独立python环境
[root@bjlg-24p100-40-xtjc06 sirius-pyenv]# source bin/activate # 激活环境

(3) 安装Sirius python依赖包

pip install -r requirements.txt

(4) 配置Sirius

修改 Aries.yaml文件

SHARE_PROXY_BASE_URI: "http://172.24.3.64:10012"
REST_BASE_URI: "172.24.3.64:10012"

DATABASES:
  default:
    ENGINE: django.db.backends.mysql
    NAME: aries
    HOST: 172.24.3.64
    PORT: 3306
    USER: root
    PASSWORD: baifendian
  kd_agent_bdms:
    ENGINE: django.db.backends.mysql # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
    NAME: bdms_web10010        # Or path to database file if using sqlite3.
    USER: bdms                       # Not used with sqlite3.
    PASSWORD: bdms                  # Not used with sqlite3.
    HOST: 172.24.100.40             # Set to empty string for localhost. Not used with sqlite3.
    PORT: 3306                       # Set to empty string for default. Not used with sqlite3.

AUTH_LDAP:
  SERVER_URI: 'ldap://192.168.49.241:389'
  USER_DN_TEMPLATE: 'uid=%(user)s,ou=mrbs,dc=baifendian,dc=com'
  BIND_AS_AUTHENTICATING_USER: True
  CACHE_GROUPS: True
  GROUP_CACHE_TIMEOUT: 3600
  USER_ATTR_MAP:
    username: "givenName"
    password: "password"

AMBARI:
  AMBARI_URL: "http://172.24.3.64:8080/api/v1/clusters/hlg_ambari/"
  HDFS_URL: "http://172.24.3.156:50070/webhdfs/v1/"
  HADOOP_CLIENT: "hlg3p64-lupan"
  AMBARI_USER: "admin"
  AMBARI_PASSWORD: "admin"


K8S:
  K8S_IP: '172.24.3.150'
  K8S_PORT: 8080
  INFLUXDB_IP: 'k8sinfluxapi.bfdcloud.com'
  INFLUXDB_PORT: 80
  INFLUXDB_DATABASE: 'k8s'

BDMS:
  IP: '172.24.100.40'
  PORT: '10001'
  USERNAME: 'aiping.liang'
  PASSWORD: 'aiping.liang'


WEBHDFS:
  HOSTS:
    - 172.24.3.155:50070
    - 172.24.3.156:50070
  PORT: 50070
  PATH: "/webhdfs/v1"
  USER: "hadoop"
  TIMEOUT: 10
  MAX_TRIES: 2
  RETRY_DELAY: 3

CODIS:
  INDEX_LINE: 1
  ZK_ADDR: '172.24.3.64:2181'
  HOST_INFO:
    - 0
    - 172.24.3.64
    - root
    -
    - 0
    - a
    - 0
  MEMORY_MAX: 10
  PENTSDB_URL: "http://172.24.4.33:4242"
  SSH_PKEY: '/root/.ssh/id_rsa'
  SSH_KNOWN_HOSTS: '/root/.ssh/known_hosts'

OPENSTACK:
  IP_KEYSTONE: "192.168.190.11"
  PORT_KEYSTONE: 5000
  IP_NOVA: "192.168.190.11"
  PORT_NOVA: 8774
  IP_CINDER:  "192.168.190.11"
  PORT_CINDER: 8776

(5) 启动Sirius

  sh sbin/Aries.sh start

(6) 停止Sirius

  sh sbin/Aries.sh stop

(7) 登录

127.0.0.1:10086

用户名, 密码为ldap的用户名和密码

k8s集群的部署

提示: 由于目前sirius的docker image放在 docker.baifendian.com 私服中.

(1) 代码拷贝并修改配置文件

git clone https://github.com/baifendian/Sirius.git

(2) 下载kubectl客户端

(3) 配置kubectl客户端

$ kubectl config set-cluster default-cluster --server=https://${MASTER_HOST} --certificate-authority=${CA_CERT}
$ kubectl config set-credentials default-admin --certificate-authority=${CA_CERT} --client-key=${ADMIN_KEY} --client-certificate=${ADMIN_CERT}
$ kubectl config set-context default-system --cluster=default-cluster --user=default-admin --namespace=${NAMESPACE}
$ kubectl config use-context default-system

(4) 部署sirius在k8s集群中

$ cd $SIRIUS_HOME/docker-k8s/k8s_config  #进入k8s配置文件目录
$ kubectl create -f sirius-svc.yaml
$ kubectl create -f sirius-rc.yaml
$ kubectl create -f sirius-ingress.yaml

(5) 登录

sirius.bfdcloud.com

用户名, 密码为ldap的用户名和密码

sirius's People

Contributors

abigbigbird avatar haiiii76 avatar hyndaniel avatar linuxwind avatar lsglsf avatar lupan2015 avatar lupan92 avatar qqzxingchen avatar sjx323 avatar zd1990 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sirius's Issues

制定完整的云中心导航栏

现在的导航栏分为四个一级目录:

  • 概览
  • 存储管理
  • 计算管理
  • 用户管理

后续如果要整合更多的服务,这样的导航会有些难以找到对应功能。
初步设想:

  • 首页
    • 开始使用
    • 文档
  • 计算管理
    • k8s相关
    • openstack计算节点相关
    • 离线任务调度相关
    • storm调度相关
  • 存储管理
    • HDFS
    • Ceph块存储(for docker and for openstack)
    • docker镜像
    • openstack镜像
    • 快照
  • Codis云
  • MySQL云
  • ES云
  • ......
  • 用户管理

组织结构讨论下制定一个合理的版本

计算管理 - 我的任务 的表格组件

当刚打开这个页面并切换到表格组件子页面的时候,显示正常
但是如果此时点击表格的表头部分,如点击 “准备时间” ,则表头的布局会完全错位

【讨论】云中心的Space如何对应到Kubernetes,HDFS等系统上?

RT,刚和攀攀讨论的两种方案:

  • 创建Space时自动关联所有系统创建对应Space
    • 优点:实现相对简单
    • 缺点:没有审批流程,默认会访问所有服务,会带来权限的问题。
  • 创建Space时可选择的关联系统
    • 优点:关联时也可以选择配额等信息,功能更全面
    • 缺点:需要单独开发页面以及对应的审批流程

k8s 页面数据展示

Pod、Service、RC的详情信息需要改变显示样式

  • 涉及的页面包括:
    • /CalcManage/PodInfo
    • /CalcManage/ServiceInfo
    • /CalcManage/ReplicationControllerInfo

[全局] Aries.xml中的pythonpath的路径改成相对目录。

现在如下:

<uwsgi>
<pythonpath>/opt/pan.lu/gitsource/Sirius-patch/Sirius/Aries</pythonpath>
  <module>Aries.wsgi</module>
  <master/>
  <harakiri>100</harakiri>
  <socket-timeout>100</socket-timeout>
  <processes>2</processes>
  <daemonize>/tmp/sirius_log/uwsgi.log</daemonize>
  <http>0.0.0.0:10086</http>
  <enable-threads></enable-threads>
</uwsgi>

k8s 页面样式

动态计算SplitPanel组件的高度

  • 涉及的页面包括:
    • /CalcManage/PodInfo
    • /CalcManage/ServiceInfo
    • /CalcManage/ReplicationControllerInfo

存储管理中HDFS文件上传报错

报错信息如下:

[ERROR 2016-08-15 02:32:47,741 @ 6217] (/opt/pan.lu/gitsource/SiriusTest/Sirius/Aries/hdfs/rests.py/post:51) - Traceback (most recent call last):
  File "/opt/pan.lu/gitsource/SiriusTest/Sirius/Aries/hdfs/rests.py", line 49, in post
    result = OP_DICT.get("POST").get(op)(request,path)
  File "/opt/pan.lu/gitsource/SiriusTest/Sirius/Aries/hdfs/service.py", line 369, in upload
    return hdfs.upload(path, request)
  File "/opt/pan.lu/gitsource/SiriusTest/Sirius/Aries/hdfs/function.py", line 377, in upload
    return self.upload_file_by_http(path, request)
  File "/opt/pan.lu/gitsource/SiriusTest/Sirius/Aries/hdfs/function.py", line 109, in upload_file_by_http
    f = request.FILES["files"]
  File "/opt/Python-2.7.8/lib/python2.7/site-packages/django/utils/datastructures.py", line 322, in __getitem__
    raise MultiValueDictKeyError(repr(key))
MultiValueDictKeyError: "'files'"

k8s 页面样式

Datatable组件的选中行高亮功能需要自己实现

  • UI开发人员通过讨论,建议选中行高亮功能由页面开发人员自己实现
  • 涉及的页面包括:
    • /CalcManage/PodInfo
    • /CalcManage/ServiceInfo
    • /CalcManage/ReplicationControllerInfo

[codis] codis模块需要的依赖包添加到requirements.txt 文件

如下:

ImportError at /
No module named redis
Request Method: GET
Request URL:    http://{ip}:{port}/
Django Version: 1.8.6
Exception Type: ImportError
Exception Value:    
No module named redis
Exception Location: /opt/Sirius/Aries/codis/rests.py in <module>, line 17
ImportError at /
No module named paramiko
Request Method: GET
Request URL:    http://172.24.100.40:8000/
Django Version: 1.8.6
Exception Type: ImportError
Exception Value:    
No module named paramiko

去掉源码中的二进制以及压缩包

一般来说源码中不要包含压缩包和二进制文件,package目录下的压缩包和二进制文件需要去掉,这些都应该是上传至release中的。

DAMN ! worker 10 (pid: 18950) died, killed by signal 9 :( trying respawn ...

Sirius启动后访问不了. uwsgi log如下:

Wed Jul 27 07:25:07 2016 - *** HARAKIRI ON WORKER 9 (pid: 18949, try: 1) ***
Wed Jul 27 07:25:07 2016 - HARAKIRI !!! worker 9 status !!!
Wed Jul 27 07:25:07 2016 - HARAKIRI [core 0] 10.12.4.126 - GET / since 1469604246
Wed Jul 27 07:25:07 2016 - HARAKIRI !!! end of worker 9 status !!!
DAMN ! worker 9 (pid: 18949) died, killed by signal 9 :( trying respawn ...
Respawned uWSGI worker 9 (new pid: 19013)
Wed Jul 27 07:26:07 2016 - *** HARAKIRI ON WORKER 10 (pid: 18950, try: 1) ***
Wed Jul 27 07:26:07 2016 - HARAKIRI !!! worker 10 status !!!
Wed Jul 27 07:26:07 2016 - HARAKIRI [core 0] 10.12.4.126 - GET / since 1469604306
Wed Jul 27 07:26:07 2016 - HARAKIRI !!! end of worker 10 status !!!
DAMN ! worker 10 (pid: 18950) died, killed by signal 9 :( trying respawn ...
Respawned uWSGI worker 10 (new pid: 19051)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.