Code Monkey home page Code Monkey logo

9n-mpc's Introduction

九数联邦学习整体解决方案

概述

数据是人工智能的基石,打破数据孤岛,实现数据共享是加速人工智能高速发展的必要条件。联邦学习,作为数据安全计算的业界前沿技术方案,在保护数据隐私的同时深度连接各个合作方,达到技术赋能、共创共赢的目标。

p1

p2

1 整体架构

  • 整个系统分为四个大模块
    • 整体调度与转发模块
    • 资源管理与调度模块
    • 数据求交模块
    • 训练器模块

architecture.png

2 整体调度与转发模块

  • 整体控制数据求交与训练的调度
  • 训练器的配对工作
  • 高效的流量转发

3 资源调度与管理模块

  • 使用k8s屏蔽底层资源差异
  • 使用k8s进行资源的动态调度

4 数据求交

  • 异步分布式框架提升拼接效率

LICNESE

9nFL使用apache2.0许可

9n-mpc's People

Contributors

982945902 avatar chkplusplus avatar jd-9n avatar loneystar1983 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

9n-mpc's Issues

加密操作问题

你好,我在github上看到贵公司联邦学习框架的项目,但是我在项目中没有找到你们在数据对齐时用到的加密协议或方法。请问下,贵公司在数据对齐中使用的方法是什么,在联邦学习时使用了什么同态加密操作?

start_server.sh运行成功,结果进程并未启动

在运行单机版时,运行start_server.sh join之后,显示success,但是后台进程并未成功.建议如下,添加2行,而不是只是检测nohup的返回值是否为0来判断.感谢!

可将

nohup python $CURRENT_DIR/data_join/data_join_server.py $REMOTE_IP $INDEX \    
    $PARTITION_ID $DATA_SOURCE_NAME $DATA_BLOCK_DIR $RAW_DATA_DIR $ROLE -m=$MODE \    
    -p=$PORT0 --raw_data_iter=$RAW_DATA_ITER --compressed_type=$COMPRESSED_TYPE \    
    --example_joiner=$EXAMPLE_JOINER $EAGER_MODE >>"${data_join_log_dir}/console_${ROLE}.log" \    
    2>>"${data_join_log_dir}/error_${ROLE}.log" &    
    ans=`ps -ef|grep data_join_server|grep ${ROLE}`  # 可简单通过查询进程是否启动    
    if [[ -n "${ans}" ]]; then # nohup启动正常,且原代码直接将错误输出流输入到文件,看该文件可解决问题,但是用户体检上,还是应该报错的.    
        echo "data join service start successfully"    
        echo "log dir: ${data_join_log_dir}"    
        echo "input dir: ${RAW_DATA_DIR}"    
        echo "output dir: ${DATA_BLOCK_DIR}"    
    else    
        echo "data join service start failed"    
    fi    

联邦学习的性能情况

想了解下支持多少个联邦学习的参与方?
亿级别的隐私数据求交需要多大的存储量、计算耗时

单机版联邦模型测试报错

训练完成mnist_demo后,进行模型测试。
在两个文件mnist_demo/mnist_leader.py, mnist_demo/mnist_follower.py中分别配置:
run_config["eval"] = 1
run_config["model_dir"] = "models/follower_model/"
运行run_mnist_dist_leader.sh。

报错内容:
mnist_demo/mnist_leader.py line 136
with ops.constrol_denpendences([xl])
NameError: name 'xl' is not defines.

关于是否有第三方参与

您好,阅读完 README 文档之后有下面几个问题想请教下:

  1. 数据求交过程中,涉及第三方参与吗?还是只在两方之间进行?
  2. 训练过程中,是否需要一个可信的第三方参与?还是训练的协议是一个两方协议。

另外,还想请问,目前实现了纵向联邦学习的算法吗?如果有,请问有哪几个算法呢?

感谢花时间阅读本 issue,期待回复~

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.