diff --git a/README.md b/README.md index 59ad3a0c..6c7040a9 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,9 @@ [DOC](https://paddlefl.readthedocs.io/en/latest/) | [Quick Start](https://paddlefl.readthedocs.io/en/latest/compile_and_intall.html) | [中文](./README_cn.md) +[![Release](https://img.shields.io/github/release/PaddlePaddle/PaddleFL.svg)](https://github.com/PaddlePaddle/PaddleFL/releases) +[![License](https://img.shields.io/badge/license-Apache%202-blue.svg)](LICENSE) + PaddleFL is an open source federated learning framework based on PaddlePaddle. Researchers can easily replicate and compare different federated learning algorithms with PaddleFL. Developers can also benefit from PaddleFL in that it is easy to deploy a federated learning system in large scale distributed clusters. In PaddleFL, several federated learning strategies will be provided with application in computer vision, natural language processing, recommendation and so on. Application of traditional machine learning training strategies such as Multi-task learning, Transfer Learning in Federated Learning settings will be provided. Based on PaddlePaddle's large scale distributed training and elastic scheduling of training job on Kubernetes, PaddleFL can be easily deployed based on full-stack open sourced software. @@ -82,8 +85,25 @@ Upon completion of the secure training (or inference) job, the models (or predic For more instructions, please refer to [mpc examples](./python/paddle_fl/mpc/examples) + ## Installation +### Environment preparation + +* CentOS 7 (64 bit) +* Python 3.5/3.6/3.7 ( 64 bit) or above +* pip3 9.0.1+ (64 bit) +* PaddlePaddle 1.8.5 +* Redis 5.0.8 (64 bit) +* GCC or G++ 8.3.1 +* cmake 3.15+ + +### Installation steps + +We provide three ways to install PaddleFL： + +1. Use PaddleFL in docker + We **highly recommend** to run PaddleFL in Docker ```sh @@ -91,12 +111,36 @@ We **highly recommend** to run PaddleFL in Docker docker pull paddlepaddle/paddlefl:1.1.2 docker run --name --net=host -it -v $PWD:/paddle /bin/bash -#Install paddle_fl +``` + +2. Install PaddleFL via installation package + +We provide compiled PaddlePaddle and PaddleFL installation packages, you can download and install them directly. + +First, install PaddlePaddle +``` +#Install PaddlePaddle +wget https://paddlefl.bj.bcebos.com/paddlepaddle-1.8.5-cp**-cp**-linux_x86_64.whl +pip3 install paddlepaddle-1.8.5-cp**-cp**-linux_x86_64.whl +``` +Please replace ** with the python version in the installation environment. E.g., if you are using python3.8, the commands are as below: +``` +wget https://paddlefl.bj.bcebos.com/paddlepaddle-1.8.5-cp38-cp38-linux_x86_64.whl +pip3 install paddlepaddle-1.8.5-cp38-cp38-linux_x86_64.whl +``` +Then, install PaddleFL +``` +#Install PaddleFL pip3 install paddle_fl ``` +The above command will automatically install PaddleFL corresponding to python3.8. For other python3 environments, you can download the corresponding installation package from https://pypi.org/project/paddle-fl/1.1.2/#files and install it manually. + + +3. Install PaddleFL from source code If you want to compile and install from source code, please click [here](./docs/source/md/compile_and_install.md) to get instructions. + If you are using the gloo communication model, you will need Redis. We also provide a stable Redis installation package for download. ```sh diff --git a/README_cn.md b/README_cn.md index 276cdbf5..53c7b3e8 100644 --- a/README_cn.md +++ b/README_cn.md @@ -1,5 +1,8 @@ [English](./README.md) | 简体中文 +[![Release](https://img.shields.io/github/release/PaddlePaddle/PaddleFL.svg)](https://github.com/PaddlePaddle/PaddleFL/releases) +[![License](https://img.shields.io/badge/license-Apache%202-blue.svg)](LICENSE) + # PaddleFL PaddleFL是一个基于PaddlePaddle的开源联邦学习框架。研究人员可以很轻松地用PaddleFL复制和比较不同的联邦学习算法，开发人员也比较容易在大规模分布式集群中部署PaddleFL联邦学习系统。PaddleFL提供很多种联邦学习策略（横向联邦学习、纵向联邦学习）及其在计算机视觉、自然语言处理、推荐算法等领域的应用。此外，PaddleFL还将提供传统机器学习训练策略的应用，例如多任务学习、联邦学习环境下的迁移学习。依靠着PaddlePaddle的大规模分布式训练和Kubernetes对训练任务的弹性调度能力，PaddleFL可以基于全栈开源软件轻松地部署。 @@ -78,22 +81,61 @@ PFM 拥有与PaddlePaddle相同的运行模式。在训练前，用户需要定 请参考[MPC的例子](./python/paddle_fl/mpc/examples)，以获取更多的信息。 - ## 安装 +### 环境依赖 + +* CentOS 7 (64 bit) +* Python 3.5/3.6/3.7/3.8 ( 64 bit) +* pip3 9.0.1+ (64 bit) +* PaddlePaddle 1.8.5 +* Redis 5.0.8 (64 bit) +* GCC or G++ 8.3.1 +* cmake 3.15+ + +### 安装部署 + +我们提供三种方式安装PaddleFL，您可以根据自己的实际情况进行选择： + +1.在Docker中使用PaddleFL + 我们**强烈建议** 您在docker中使用PaddleFL。 ```sh #Pull and run the docker docker pull paddlepaddle/paddlefl:1.1.2 docker run --name --net=host -it -v $PWD:/paddle /bin/bash +``` +Docker中环境配置以及paddlepaddle和paddlefl已经安装完成，您可以直接运行示例代码，开始使用PaddleFL。 + +2.安装包安装 -#Install paddle_fl +我们提供了编译好的PaddlePaddle及PaddleFL安装包，您可以直接进行下载安装。 + +首先安装PaddlePaddle +``` +#Install PaddlePaddle +wget https://paddlefl.bj.bcebos.com/paddlepaddle-1.8.5-cp**-cp**-linux_x86_64.whl +pip3 install paddlepaddle-1.8.5-cp**-cp**-linux_x86_64.whl +``` + +安装时，请将**替换成安装环境中的python版本。例如，如果您使用的python版本为python3.8，那么使用下面的命令： +``` +wget https://paddlefl.bj.bcebos.com/paddlepaddle-1.8.5-cp38-cp38-linux_x86_64.whl +pip3 install paddlepaddle-1.8.5-cp38-cp38-linux_x86_64.whl +``` + +然后安装PaddleFL +``` +#Install PaddleFL pip3 install paddle_fl ``` +上述命令会自动安装python3.8对应的PaddleFL。对于其他python3环境，您可以从 https://pypi.org/project/paddle-fl/1.1.2/#files 下载对应安装包手动安装。 +3.源码安装 若您希望从源码编译安装，请点击[这里](./docs/source/md/compile_and_install_cn.md)。 + 如果使用gloo通信模型，需要用到redis，我们也提供了稳定的redis安装包, 可供下载。 ```sh diff --git a/docs/source/md/compile_and_install.md b/docs/source/md/compile_and_install.md index fd0642da..daf153f5 100644 --- a/docs/source/md/compile_and_install.md +++ b/docs/source/md/compile_and_install.md @@ -1,62 +1,41 @@ -# Compile and Install - -## Installation - -We **highly recommend** to run PaddleFL in Docker - -```sh -#Pull and run the docker -docker pull hub.baidubce.com/paddlefl/paddle_fl:latest -docker run --name --net=host -it -v $PWD:/root /bin/bash - -#Install paddle_fl -pip install paddle_fl -``` - -We also prepare a stable redis package for you to download and install, which will be used in tasks with MPC. - -```sh -wget --no-check-certificate https://paddlefl.bj.bcebos.com/redis-stable.tar -tar -xf redis-stable.tar -cd redis-stable && make -``` - ## Compile From Source Code #### A. Environment preparation -* CentOS 6 or CentOS 7 (64 bit) -* Python 2.7.15+/3.5.1+/3.6/3.7 ( 64 bit) or above -* pip or pip3 9.0.1+ (64 bit) -* PaddlePaddle release 1.8 (if not build paddle from source) +* CentOS 7 (64 bit) +* Python 3.5/3.6/3.7 ( 64 bit) or above +* pip3 9.0.1+ (64 bit) +* PaddlePaddle 1.8.5 * Redis 5.0.8 (64 bit) -* GCC or G++ 6+ +* GCC or G++ 8.3.1 * cmake 3.15+ #### B. Clone the source code, compile and install -Fetch the source code and checkout stable release +Fetch the source code and prepare for compilation + ```sh git clone https://github.com/PaddlePaddle/PaddleFL cd /path/to/PaddleFL - -# Checkout stable release mkdir build && cd build ``` -Execute compile commands, where `PYTHON_EXECUTABLE` is path to the python binary where the PaddlePaddle is installed, `CMAKE_C_COMPILER` is the path of gcc and `CMAKE_CXX_COMPILER` is the path of g++. +Execute compile commands, where `CMAKE_C_COMPILER` is the path of gcc and `CMAKE_CXX_COMPILER` is the path of g++, `PYTHON_EXECUTABLE` is path to the python binary where the PaddlePaddle is installed, `PYTHON_INCLUDE_DIRS` is path to the file `Python.h`. Then you can put the directory in the following command and make: ```sh -cmake ../ -DPYTHON_EXECUTABLE=${PYTHON_EXECUTABLE} -DCMAKE_C_COMPILER=${gcc_path} -DCMAKE_CXX_COMPILER=${g++_path} +cmake .. -DCMAKE_C_COMPILER=${gcc_path} -DCMAKE_CXX_COMPILER=${g++_path} -DPYTHON_EXECUTABLE=${PYTHON_EXECUTABLE} -DPYTHON_INCLUDE_DIRS=${PYTHON_INCLUDE_DIRS} -DBUILD_PADDLE_FROM_SOURCE=ON -DWITH_GRPC=ON make -j$(nproc) ``` - -Note that paddle must be built from source after paddle-fl version 1.1.2 by turning on cmake option `BUILD_PADDLE_FROM_SOURCE` (paddlepaddle1.8.5). +For example, +``` +cmake .. -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DPYTHON_EXECUTABLE=/usr/local/python/bin/python3.8 -DPYTHON_INCLUDE_DIRS=/usr/local/python/include/python3.8/ -DBUILD_PADDLE_FROM_SOURCE=ON -DWITH_GRPC=ON +``` +If you have installed PaddlePaddle in advance and only want to compile PaddleFL, then change "-DBUILD_PADDLE_FROM_SOURCE=ON" in the above command to "-DBUILD_PADDLE_FROM_SOURCE=OFF". Install paddle if BUILD_PADDLE_FROM_SOURCE=on: ```sh -pip or pip3 install ./third_party/paddle/src/extern_paddle-build/python/dist/paddlepaddle-1.8.5-cp38-cp38-linux_x86_64.whl -U +pip3 install ./third_party/paddle/src/extern_paddle-build/python/dist/paddlepaddle-1.8.5-cp38-cp38-linux_x86_64.whl -U ``` Install the package: @@ -65,5 +44,5 @@ Install the package: make install cd /path/to/PaddleFL/python ${PYTHON_EXECUTABLE} setup.py sdist bdist_wheel -pip or pip3 install dist/***.whl -U +pip3 install dist/***.whl -U ``` diff --git a/docs/source/md/compile_and_install_cn.md b/docs/source/md/compile_and_install_cn.md index f28f48fa..4e9e6786 100644 --- a/docs/source/md/compile_and_install_cn.md +++ b/docs/source/md/compile_and_install_cn.md @@ -2,44 +2,46 @@ #### A. 环境准备 -* CentOS 6 or CentOS 7 (64 bit) -* Python 2.7.15+/3.5.1+/3.6/3.7 ( 64 bit) or above -* pip or pip3 9.0.1+ (64 bit) -* PaddlePaddle release 1.8 (如果选择不从源代码构建paddle) +* CentOS 7 (64 bit) +* Python 3.5/3.6/3.7 ( 64 bit) or above +* pip3 9.0.1+ (64 bit) +* PaddlePaddle 1.8.5 * Redis 5.0.8 (64 bit) -* GCC or G++ 4.8.3+ +* GCC or G++ 8.3.1 * cmake 3.15+ #### B. 克隆源代码并编译安装 -获取源代码 +1.获取源代码 ```sh git clone https://github.com/PaddlePaddle/PaddleFL cd /path/to/PaddleFL - -# Checkout stable release mkdir build && cd build ``` -执行编译指令, `PYTHON_EXECUTABLE` 为安装了PaddlePaddle的可执行python路径, `CMAKE_C_COMPILER` 为指定的gcc路径, `CMAKE_CXX_COMPILER` 为指定的g++路径。 +2.执行编译指令, `CMAKE_C_COMPILER` 为指定的gcc路径, `CMAKE_CXX_COMPILER` 为指定的g++路径,`PYTHON_EXECUTABLE` 为安装了PaddlePaddle的可执行python路径,`DPYTHON_INCLUDE_DIRS`为指定Python.h文件所在路径。 之后就可以执行编译和安装的指令 ```sh -cmake ../ -DPYTHON_EXECUTABLE=${PYTHON_EXECUTABLE} -DCMAKE_C_COMPILER=${gcc_path} -DCMAKE_CXX_COMPILER=${g++_path} +cmake .. -DCMAKE_C_COMPILER=${gcc_path} -DCMAKE_CXX_COMPILER=${g++_path} -DPYTHON_EXECUTABLE=${PYTHON_EXECUTABLE} -DPYTHON_INCLUDE_DIRS=${PYTHON_INCLUDE_DIRS} -DBUILD_PADDLE_FROM_SOURCE=ON -DWITH_GRPC=ON make -j$(nproc) ``` +例如： +``` +cmake .. -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DPYTHON_EXECUTABLE=/usr/local/python/bin/python3.8 -DPYTHON_INCLUDE_DIRS=/usr/local/python/include/python3.8/ -DBUILD_PADDLE_FROM_SOURCE=ON -DWITH_GRPC=ON +``` +如果您事先安装好了PaddlePaddle，希望只编译PaddleFL，那么将上面命令中 "-DBUILD_PADDLE_FROM_SOURCE=ON" 改为 "-DBUILD_PADDLE_FROM_SOURCE=OFF" 即可。 -注意, paddle-fl 1.1.2 版(或更高)必须从源码构建paddle, 即打开cmake选项`BUILD_PADDLE_FROM_SOURCE`. -安装paddle (如果选择从源代码构建paddle): +3.安装paddle (如果选择从源代码构建paddle): ```sh -pip or pip3 install ./third_party/paddle/src/extern_paddle-build/python/dist/paddlepaddle-1.8.0-cp36-cp36m-linux_x86_64.whl -U +pip3 install ./third_party/paddle/src/extern_paddle-build/python/dist/paddlepaddle-1.8.5-cp38-cp38-linux_x86_64.whl -U ``` -安装对应的安装包 +4.安装PaddleFL对应的安装包 ```sh make install cd /path/to/PaddleFL/python ${PYTHON_EXECUTABLE} setup.py sdist bdist_wheel -pip or pip3 install dist/***.whl -U +pip3 install dist/***.whl -U ``` diff --git a/docs/source/md/introduction.md b/docs/source/md/introduction.md index e74e7672..155fa863 100644 --- a/docs/source/md/introduction.md +++ b/docs/source/md/introduction.md @@ -82,5 +82,5 @@ For more instructions, please refer to [mpc examples](./python/paddle_fl/mpc/exa - Add K8S deployment scheme for PFM. - FL mobile simulator will be open sourced in following versions. -PaddleFL + diff --git a/python/paddle_fl/mpc/examples/linear_reg_with_uci/decrypt_save.py b/python/paddle_fl/mpc/examples/linear_reg_with_uci/decrypt_save.py index af8970e7..0b514067 100644 --- a/python/paddle_fl/mpc/examples/linear_reg_with_uci/decrypt_save.py +++ b/python/paddle_fl/mpc/examples/linear_reg_with_uci/decrypt_save.py @@ -22,7 +22,7 @@ decrypt_loss_file=sys.argv[1] decrypt_prediction_file=sys.argv[2] BATCH_SIZE=10 -process_data.load_decrypt_data("/tmp/uci_loss", (1, ), decrypt_loss_file) +process_data.load_decrypt_data("./mpc_infer_data/uci_loss", (1, ), decrypt_loss_file) print("uci_loss done") -process_data.load_decrypt_data("/tmp/uci_prediction", (BATCH_SIZE, ), decrypt_prediction_file) +process_data.load_decrypt_data("./mpc_infer_data/uci_prediction", (BATCH_SIZE, ), decrypt_prediction_file) print("prediction done") diff --git a/python/paddle_fl/mpc/examples/model_decryption/decrypt_and_inference.py b/python/paddle_fl/mpc/examples/model_decryption/decrypt_and_inference.py index 0607f8d1..8a984e49 100644 --- a/python/paddle_fl/mpc/examples/model_decryption/decrypt_and_inference.py +++ b/python/paddle_fl/mpc/examples/model_decryption/decrypt_and_inference.py @@ -18,7 +18,7 @@ import paddle import paddle.fluid as fluid -import paddle_fl.mpc.data_utils.data_utils import get_datautils +import paddle_fl.mpc.data_utils.data_utils mpc_du = get_datautils('aby3')