Code Monkey home page Code Monkey logo

libprio's Introduction

libprio - A Prio library in C using NSS

NOTE - this repository is being used for Firefox as part of a pilot experiment, this repository will be archived in favor of the new Rust implementation, libprio-rs. Please direct contributions that are not related to the Firefox pilot to this new project.

Warning: We do our best to write bug-free code, but I have no doubt that there are scary bugs, side-channel attacks, and memory leaks lurking herein.

Security bugs: If you find a security-critical bug in libprio, please report it to Mozilla using the contact information on this page.

Overview

This is an implementation of the core cryptographic routines for the Prio system for the private computation of aggregate statistics:

"Prio: Private, Robust, and Scalable Computation of Aggregate Statistics"
by Henry Corrigan-Gibbs and Dan Boneh
USENIX Symposium on Networked Systems Design and Implementation
March 2017

Available online at: https://crypto.stanford.edu/prio/

Usage scenario. The library implements the cryptographic routines necessary for the following application scenario: Each client holds a vector of boolean values. Each client uses the library to encode her private vector into two encoded packets—one for server A and one for server B.

After receiving shares from a client, the servers can use the routines implemented here to check whether the client-provided packets are well formed. (Without this check, a single malicious client can corrupt the output of the computation.)

After collecting data packets from many clients, the servers can combine their state to learn how many clients had the ith bit of their data vector set to true and how many clients had the ith bit of their data vector set to false. As long as at least one of the two servers is honest (i.e., runs the correct code), the servers learn nothing else about the clients' data, under standard cryptographic assumptions.

For example, the ith bit of the data vector could indicate whether the client ever visited the ith-ranked website in the Alexa Top 500. The servers would learn how many clients visited each of the Top 500 websites without learning which clients visited which websites.

Efficiency considerations. The code makes no use of public-key crypto, so it should be relatively fast. When each a data packet is of length N, all arithmetic is modulo a prime p (we use an 87-bit prime by default), and "elements" are integers modulo p, the dominant costs of the system are:

  • Client compute: O(N log N) multiplications
  • Client-to-server communication: 2N + O(1) elements
  • Server compute: O(N log N) multiplications to check each packet
    (NOTE: Using an optimization we haven't yet implemented, we can drop this cost to O(N) multiplications per packet.)
  • Server-to-server communication: O(1) elements
  • Server storage: O(N) elements

Running the code

You must first install NSS/NSPR with NSPR version 4.13.1 (or newer?) and NSS version 3.35 (or newer?), SCons version 3.0.1 (or newer?), and msgpack-c version 2.1.5 (or newer?).

On Ubuntu Bionic (18.04LTS), you can install NSS and scons with:

$ sudo apt install scons libnspr4-dev libnss3-dev 

and you will have to download msgpack-c 2.1.5 or newer here, since the Ubuntu packages for msgpack are far out of date.

For macOS using Homebrew:

$ brew install nss nspr scons msgpack

To compile the code, run:

$ scons

To run the test suite, execute:

$ build/ptest/ptest -v

To print debug messages while compiling:

$ scons VERBOSE=1

To compile with debug symbols, run:

$ scons DEBUG=1

To clean up the object and binary files, run:

$ scons -c

The library can be built and tested with docker.

$ docker-compose build
$ docker-compose run libprio

The files in this directory are:

/build      - Binaries, object files, etc.
/include    - Exported header files
              (Note: The public header is <mprio.h> since
              NSPR already has a file called <prio.h>.)
/mpi        - NSS MPI bignum library 
/pclient    - Example code that uses the Prio library
/prio       - Prio library core code
/ptest      - Tests and test runner

Optimizations and features not yet implemented

  • Server compute. By using a fast polynomial interpolation-and-evaluation routine, we can reduce the cost of checking a single client request from O(N log N) multiplications down to O(N) multiplications, for a data packet of N items.
  • Differential privacy. It would be very straightforward to add some small amount of noise to the final statistics to provide differential privacy. If this would be useful, I can add it.
  • Misc. There are TODO notes scattered throughout code indicating places for potential performance optimizations.

libprio's People

Contributors

acmiyaguchi avatar choller avatar degregat avatar franziskuskiefer avatar froydnj avatar gabrielganne avatar henrycg avatar jbeich avatar mozilla-github-standards avatar rhelmer avatar tititiou36 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

libprio's Issues

linux build failing

libprio is failing to build for Linux, although it works for macOS (which is a very similar environment which makes the failure kinda confusing to me)

build NSS from source and remove mpi/ dir

Currently we have a partial copy of non-exported NSS code in the mpi/ subdir, and require the Travis script to work in lock-step with the NSS version. This is done to try to allow consumers of libprio to build against binary versions of NSS, since this can be really painful especially on Windows.

This doesn't seem worth the headache however, and it makes it look like libprio is quite a bit more code than it is (including a bunch of scary-looking ASM!)

Instead I think we should just download NSS releases from source and build it: https://github.com/nss-dev/nss

Add option for differential privacy

Optionally, Prio servers should add Laplace-distributed noise to their shares of the final statistic before publishing them. In this way, the final output of the system can be differentially private w.r.t. to the data of any particular client.

use types from <stdint.h> for integer types instead of implementation-defined

@hsivonen mentioned this in a recent review: https://phabricator.services.mozilla.com/D16267#412614

Seems unfortunate that libprio uses integer types whose size is implementation-defined instead of using the types from <stdint.h> (in this case uint64_t).

@henrycg any reason not to make this change? I think this basically means giving up on C90 support and requiring C99. The only C90 compiler in wide use that I am aware of is MSVC (issue #17) and I am less convinced that this is important given that clang works well on that platform now.

automated code coverage, static analysis, fuzzing, style checking, etc.

Now that Travis CI is running it'd be pretty easy to add things like test coverage, static analysis tools (clang-tidy and scan-build are OK, there are more), fuzzing, style checking etc.

I know that there is a particular code style already in-place for libprio but I'd very much suggest that some out-of-the-box style using clang-format be used (it comes with a Mozilla style even!) and enforced when PRs are opened. More for the sake of future contributors than as a personal preference (but that too ;)

Add option for fixed-point encoding with additive shift

To use prio as a backend for e.g. federated learning, support for fixed-point encoding would be nice. Since we only have summation of bits, preserving sign bits is not an option so an additive shift should be applied to the encoding.

I started work on this and will do a (draft) PR when it's presentable.

windows build failing

I noticed this doing tryserver (pre-checkin CI) builds:

z:/build/build/src/third_party/prio/prio/client.c(10,10): fatal error: 'strings.h' file not found

  • It looks like <string.h> is present on Windows, AFAICT <strings.h> is an BSD fork that didn't make it's way to Windows. Not 100% sure of the lineage here or what's available to C++ vs. C, it looks like <string.h> is available on macOS and Windows for C code at least.
  • std::isxdigit doesn't seem to work on Windows but isxdigit does

add basic CI for libprio

Since we're building libprio in Firefox there is OK coverage for basic stuff, but I think it'd make everyone's lives easier if there was some basic CI like Travis or CircleCI etc. that compiled, linted, ran tests, etc. when folks opened PRs and after merges, so it doesn't require someone to remember to do this locally before merging.

We'd catch this on import to Firefox but that could be quite a long lag time :)

I'll have a bit of free time to do this and have done it before so can set it up if you like.

rewrite browser-test.c / encode-once.js for latest API

I was just taking a look at browser-test.c and encode-once.js, I took a quick stab at rewriting the latter:

/*
 * Copyright (c) 2018, Henry Corrigan-Gibbs
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, You can obtain one at http://mozilla.org/MPL/2.0/.
 */

ChromeUtils.import('resource://gre/modules/Services.jsm');

let [publicKeyA, publicKeyB, batchID, param1, param2, param3] = arguments;

Services.prefs.setStringPref('prio.publicKeyA', publicKeyA);
Services.prefs.setStringPref('prio.publicKeyB', publicKeyB);

async function test() {
  let params =  {
    'browserIsUserDefault': Boolean(param1),
    'safeModeUsage': Boolean(param2),
    'startupCrashDetected': Boolean(param3)
  };

  let result = await PrioEncoder.encode(batchID, params);
}

test().then(console.log("finished"));

This still needs to return the correct output, but I was doing a quick test run with is and seeing:

Hit MOZ_CRASH(NSS_Shutdown failed) at /Users/rhelmer/src/mozilla-central/xpcom/build/XPCOMInit.cpp:1044

I suspect that this has to do with how NSS gets init'd in the xpcshell environment, as I am seeing no problems when running equivalent code in the browser.

libprio does not build on non-64bit and noopt (-O0) builds

I haven't tried directly with Scons but I suspect it'll fail in the same way:

https://treeherder.mozilla.org/logviewer.html#?job_id=195528704&repo=try&lineNumber=32847

14:45:50 INFO - /builds/worker/workspace/build/src/third_party/prio/prio/client.c:48: error: undefined reference to 'next_power_of_two'
32848 14:45:50 INFO - /builds/worker/workspace/build/src/third_party/prio/prio/config.c:130: error: undefined reference to 'next_power_of_two'
32849 14:45:50 INFO - /builds/worker/workspace/build/src/third_party/prio/prio/server.c:255: error: undefined reference to 'next_power_of_two'
32850 14:45:50 INFO - /builds/worker/workspace/build/src/third_party/prio/prio/server.c:282: error: undefined reference to 'next_power_of_two'

I'll spin up an Android dev environment at some point, this platform is not really a top priority for me at the moment but will be good to have it eventually.

Travis should automatically fetch the latest version of NSS

Right now, the .travis.yml file points to a non-permanent version of NSS. Whenever Mozilla releases a new version of NSS, the file name changes and our Travis CI builds all fail. It would be nice to have the tests automatically use the latest version of NSS.

Many tests fail on FreeBSD

$ pkg install -qy bash scons msgpack nss
$ scons
$ ./build/ptest/ptest -v
build/ptest/client_test.c:99: mu_check((rv = (PrioClient_encode(cfg, data_items, &for_a, &aLen, &for_b, &bLen))) == SECSuccess) failed, resuming test case
build/ptest/client_test.c:126: mu_check(rv == SECSuccess) failed, resuming test case
build/ptest/client_test.c:99: mu_check((rv = (PrioClient_encode(cfg, data_items, &for_a, &aLen, &for_b, &bLen))) == SECSuccess) failed, resuming test case
build/ptest/client_test.c:126: mu_check(rv == SECSuccess) failed, resuming test case
build/ptest/client_test.c:99: mu_check((rv = (PrioClient_encode(cfg, data_items, &for_a, &aLen, &for_b, &bLen))) == SECSuccess) failed, resuming test case
build/ptest/client_test.c:126: mu_check(rv == SECSuccess) failed, resuming test case
build/ptest/client_test.c:99: mu_check((rv = (PrioClient_encode(cfg, data_items, &for_a, &aLen, &for_b, &bLen))) == SECSuccess) failed, resuming test case
build/ptest/client_test.c:126: mu_check(rv == SECSuccess) failed, resuming test case
build/ptest/encrypt_test.c:67: mu_check((rv = (PublicKey_encrypt(pubkey, bytes_enc, &encryptedBytes, enclen, bytes_in, inlen))) == SECSuccess) failed, resuming test case
build/ptest/encrypt_test.c:67: mu_check((rv = (PublicKey_encrypt(pubkey, bytes_enc, &encryptedBytes, enclen, bytes_in, inlen))) == SECSuccess) failed, resuming test case
build/ptest/encrypt_test.c:67: mu_check((rv = (PublicKey_encrypt(pubkey, bytes_enc, &encryptedBytes, enclen, bytes_in, inlen))) == SECSuccess) failed, resuming test case
build/ptest/encrypt_test.c:67: mu_check((rv = (PublicKey_encrypt(pubkey, bytes_enc, &encryptedBytes, enclen, bytes_in, inlen))) == SECSuccess) failed, resuming test case
build/ptest/encrypt_test.c:91: mu_check(bad ? (rv == SECFailure) : (rv == SECSuccess)) failed, resuming test case
build/ptest/encrypt_test.c:67: mu_check((rv = (PublicKey_encrypt(pubkey, bytes_enc, &encryptedBytes, enclen, bytes_in, inlen))) == SECSuccess) failed, resuming test case
build/ptest/encrypt_test.c:91: mu_check(bad ? (rv == SECFailure) : (rv == SECSuccess)) failed, resuming test case
build/ptest/encrypt_test.c:67: mu_check((rv = (PublicKey_encrypt(pubkey, bytes_enc, &encryptedBytes, enclen, bytes_in, inlen))) == SECSuccess) failed, resuming test case
build/ptest/encrypt_test.c:301: mu_check((rv = (PublicKey_encrypt(pubkey, output, &outputLen, sizeof(output), input, sizeof(input)))) == SECSuccess) failed, resuming test case
build/ptest/encrypt_test.c:313: mu_check(rv == SECSuccess) failed, resuming test case
build/ptest/encrypt_test.c:232: mu_check((rv = (PublicKey_encrypt(pubkey, output, &outputLen, sizeof(output), input, sizeof(input)))) == SECSuccess) failed, resuming test case
build/ptest/encrypt_test.c:244: mu_check(rv == SECSuccess) failed, resuming test case
build/ptest/server_test.c:191: mu_check((rv = (PrioClient_encode(cfg, data_items, &for_a, &aLen, &for_b, &bLen))) == SECSuccess) failed, resuming test case
build/ptest/server_test.c:191: mu_check((rv = (PrioClient_encode(cfg, data_items, &for_a, &aLen, &for_b, &bLen))) == SECSuccess) failed, resuming test case
build/ptest/server_test.c:191: mu_check((rv = (PrioClient_encode(cfg, data_items, &for_a, &aLen, &for_b, &bLen))) == SECSuccess) failed, resuming test case
build/ptest/server_test.c:191: mu_check((rv = (PrioClient_encode(cfg, data_items, &for_a, &aLen, &for_b, &bLen))) == SECSuccess) failed, resuming test case
build/ptest/server_test.c:191: mu_check((rv = (PrioClient_encode(cfg, data_items, &for_a, &aLen, &for_b, &bLen))) == SECSuccess) failed, resuming test case
build/ptest/server_test.c:191: mu_check((rv = (PrioClient_encode(cfg, data_items, &for_a, &aLen, &for_b, &bLen))) == SECSuccess) failed, resuming test case
build/ptest/server_test.c:236: mu_check(rv == SECSuccess) failed, resuming test case
build/ptest/server_test.c:93: mu_check((rv = (PrioClient_encode(cfg, data_items, &for_a, &aLen, &for_b, &bLen))) == SECSuccess) failed, resuming test case
build/ptest/server_test.c:121: mu_check(rv == SECSuccess) failed, resuming test case

Tests done:
        8 test suite(s) passed, 3 failed, 0 skipped.
        56 test case(s) passed, 19 failed.
        1437180 check(s) passed, 29 failed.

libprio does not compile with MSVC

Most Firefox builders use Clang/LLVM nowadays but one of our builders still uses MSVC so this came up:

https://treeherder.mozilla.org/logviewer.html#?job_id=195697225&repo=mozilla-central&lineNumber=6090

10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(80): error C2057: expected constant expression
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(80): error C2466: cannot allocate an array of constant size 0
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(80): error C2133: 'key_bytes': unknown size
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(84): error C2057: expected constant expression
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(84): error C2466: cannot allocate an array of constant size 0
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(84): error C2133: 'spki_data': unknown size
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(281): error C2057: expected constant expression
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(281): error C2466: cannot allocate an array of constant size 0
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(281): error C2133: 'aadBuf': unknown size
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(319): error C2057: expected constant expression
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(319): error C2466: cannot allocate an array of constant size 0
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(319): error C2133: 'aad_buf': unknown size

I believe this is because MSVC doesn't support VLAs (variable length arrays), it wants the array size to be specified at compile time:

https://stackoverflow.com/questions/5246900/enabling-vlasvariable-length-arrays-in-ms-visual-c

Make libprio easier to build on Windows

The README has working instructions for Debian-flavored Linux (Travis-CI uses Ubuntu Trusty), and I've been keeping the macOS instructions up-to-date (we could have Travis-CI check this too FWIW but it hasn't been a priority for me at least :) )

However building on Windows is super painful:

  1. there aren't convenient NSS packages AFAIK, and that is a pain to build (it's being worked on)... https://bugzilla.mozilla.org/show_bug.cgi?id=1434943 helps a lot for getting it working w/ MSVC, but I don't think there's a great way to build with clang-cl outside of mozilla-central

Even in the presence of convenient Windows NSS packages, it is sometimes desirable to do debug builds of NSS or otherwise modify the NSS source, and getting this going took the most time.

  1. msgpack requires CMake which can be pretty hairy to get going and working with MSVC (I haven't even tried w/ clang yet)

  2. libprio requires the latest SCons and Python 2.7 or so, which is doable but can be pretty tricky to get right

We have all of the above building in mozilla-central using our moz.build build system only and Clang for everything, which is nice but really hard to reuse anywhere else.

However, I think we could take a few notes from it:

I am wondering if we could reduce the problem just to getting Python 2.7 + SCons + Clang installed on Windows, and drive the rest from SCons... those are probably the least painful dependencies involved here :)

I think the process could look like this:

  1. install git, python 2.7, clang on Windows (there are installers for these)
  2. from Windows command line, pip install scons gyp then scons build

Then SCons would:

  • check out NSS/NSPR and build them using gyp, passing along the DEBUG flag
  • check out msgpack and just include sources from it similar to how m-c works

libprio could stop having a vendored copy of the NSS mpi files too since it would know where the NSS sources were.

Travis-CI unfortunately does not support Windows and I don't know of any solid plans to do so - I think that https://github.com/marketplace/appveyor is an option however.

PrioClient_encode fails for boolean arrays with length greater than 2047

While trying to plot the relationship between the client input and payload size, I ran into a few issues.

Here are some simplified examples:

from prio import prio

# number of input bits
k = 2047

_, pk = prio.create_keypair()
config = prio.Config(k, pk, pk, b"test")
client = prio.Client(config)
data = bytes([1]*k)
client.encode(data)

This returns two bytestrings of size 49251 and 155 respectively.

When k = 2048:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-79-44942f1cbd1c> in <module>
      7 client = prio.Client(config)
      8 data = bytes([1]*k)
----> 9 client.encode(data)

~/Work/python-libprio/prio/prio.py in encode(self, data)
    117 
    118     def encode(self, data):
--> 119         return prio.PrioClient_encode(self.config.instance, data)
    120 
    121 

RuntimeError: PrioClient_encode was not succesful.

A different error occurs at k = 4096:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-82-29aeb9b7d227> in <module>
      4 
      5 _, pk = prio.create_keypair()
----> 6 config = prio.Config(k, pk, pk, b"test")
      7 client = prio.Client(config)
      8 data = bytes([1]*k)

~/Work/python-libprio/prio/prio.py in __init__(self, n_fields, server_a, server_b, batch_id)
     33             server_a.instance,
     34             server_b.instance,
---> 35             batch_id)
     36 
     37     def num_data_fields(self):

ValueError: PyCapsule_New called with null pointer

I added basic tracing statements to debug the Python wrapper; both of the errors return this error:

build/prio/client.c:72
build/prio/client.c:111
build/prio/client.c:219
build/prio/client.c:301

It looks like something along these lines may be causing issues:

libprio/prio/client.c

Lines 71 to 72 in 2c06b4b

// Evaluate at the 2N-th roots of unity
P_CHECKC(poly_fft(evals_out, poly_f, cfg, false));

Is there a documented input size for the encode function?

pip install prio not working on Windows

Running pip install prio results in the following error on Windows: Cannot open include file: 'blapit.h'

Environment: Windows 10; Python 3.10.0; pip 21.2.3; (tried it on multiple windows-pc's). It works on Linux,

The full error:

Collecting prio
  Downloading prio-0.2.tar.gz (207 kB)
     |████████████████████████████████| 207 kB 3.3 MB/s
Using legacy 'setup.py install' for prio, since package 'wheel' is not installed.
Installing collected packages: prio
    Running setup.py install for prio ... error
    ERROR: Command errored out with exit status 1:
     command: 'C:\Users\xxxxx\test\venv\Scripts\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\xxxxx\\AppData\\Local\\Temp\\pip-install-xl6_jnan\\prio_4b0d94ba5e1c42df93abeac942cbd6b6\\setup.py'"'"'; __file__='"'"'C:\\Users\\xxxxx\\AppData\\Local\\Temp\\pip-install-xl6_jnan\\prio_4b0d94ba5e1c42df93abeac942cbd6b6\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\xxxxx\AppData\Local\Temp\pip-record-onq_uk0h\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\Users\xxxxx\test\venv\include\site\python3.10\prio'
         cwd: C:\Users\xxxxx\AppData\Local\Temp\pip-install-xl6_jnan\prio_4b0d94ba5e1c42df93abeac942cbd6b6\
    Complete output (17 lines):
    running install
    running build
    running build_py
    creating build
    creating build\lib.win-amd64-3.10
    creating build\lib.win-amd64-3.10\prio
    copying prio\libprio.py -> build\lib.win-amd64-3.10\prio
    copying prio\prio.py -> build\lib.win-amd64-3.10\prio
    copying prio\__init__.py -> build\lib.win-amd64-3.10\prio
    running build_ext
    building '_libprio' extension
    creating build\temp.win-amd64-3.10
    creating build\temp.win-amd64-3.10\Release
    C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I/usr/include/nspr -I/usr/include/nss -IC:\Users\xxxxx\test\venv\include -IC:\Python310\include -IC:\Python310\Include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt /Tclibprio_wrap.c /Fobuild\temp.win-amd64-3.10\Release\libprio_wrap.obj
    libprio_wrap.c
    C:\Users\xxxxxr\AppData\Local\Temp\pip-install-xl6_jnan\prio_4b0d94ba5e1c42df93abeac942cbd6b6\libprio/include/mprio.h(16): fatal error C1083: Cannot open include file: 'blapit.h': No such file or directory
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'C:\Users\xxxxx\test\venv\Scripts\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\xxxxx\\AppData\\Local\\Temp\\pip-install-xl6_jnan\\prio_4b0d94ba5e1c42df93abeac942cbd6b6\\setup.py'"'"'; __file__='"'"'C:\\Users\\xxxxx\\AppData\\Local\\Temp\\pip-install-xl6_jnan\\prio_4b0d94ba5e1c42df93abeac942cbd6b6\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\xxxxx\AppData\Local\Temp\pip-record-onq_uk0h\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\Users\xxxxx\test\venv\include\site\python3.10\prio' Check the logs for full command output.

Switch from Travis CI to Taskcluster

I've spent some time this week debugging issues with Travis CI (some of the github admins at Mozilla needed to do some work), it seems to still not be working... I see builds happening and get emailed about them (since Travis is set up via my account), but it's not working in PRs.

I was talking to a co-worker earlier about the state of GitHub integration for Mozilla's Taskcluster CI, and it sounds like there are quite a few projects using it successfully. I think it'd be pretty easy to switch over.

This would also allow us to use a modern Linux distro, as Travis is stuck on an older Ubuntu we've had to do extra work to make sure there are newish versions of our build dependencies.

libprio does not build on macOS (with scons)

Not super urgent at the moment and I hacked around it, but before I forget:

The macOS instructions in the README no longer work, because the latest SConstruct doesn't seem to honor the LDFLAGS to find the shared libs and CPPFLAGS to find the headers.

Should be pretty trivial to fix, I'll get it when I have a free moment (as I mentioned I just hacked the file to hardcode the paths I get from the homebrew installs)

Implement server-side optimizations

In the appendix to the full Prio paper, we describe an optimization that reduces the server-side computation to O(n), for a data vector of n items, from O(n log n). It would be nice to implement this optimization.

With a bit more work, it may also be possible to get down to O(n) work on the client side. Ask me if you're interested in working on this.

Should not use locale-dependent C standard-library functions

libprio uses locale-dependent character classification and manipulation functions like, isdigit, isxdigit, isupper, islower and toupper.

This is problematic.

First, libprio passes char to these, but these are defined to take int whose value is the value of an unsigned char or EOF. On platforms where char is signed, passing it directly to these functions without casting to unsigned char first results in sign-extension and Undefined Behavior when the high bit of the char is set.

Second, these are locale-dependent, which makes them inappropriate for processing protocol text. Even the ones that aren't locale-dependent by standard are made so by Microsoft.

See:
https://stackoverflow.com/questions/2898228/can-isdigit-legitimately-be-locale-dependent-in-c
https://daniel.haxx.se/blog/2018/01/30/isalnum-is-not-my-friend/
https://daniel.haxx.se/blog/2008/10/15/strcasecmp-in-turkish/

I suggest doing the intended comparisons with char literals. That is, ('0' <= c && c <= '9') in place of isdigit(c).

Cannot import private keys that start with zero bytes

First seen on Travis, I can reproduce locally by running ptest -v in a loop:

build/ptest/encrypt_test.c:306: mu_check((rv = (PrivateKey_decrypt(pvtkey_imp, decrypt, &plainLen, sizeof(decrypt), output, outputLen))) == SECSuccess) failed, resuming test case
build/ptest/encrypt_test.c:313: mu_check(rv == SECSuccess) failed, resuming test ca

Tests done:
	10 test suite(s) passed, 1 failed, 0 skipped.
	73 test case(s) passed, 1 failed.
	1441815 check(s) passed, 2 failed.

I've reproduced it on master (a898b55) after ~100 tries.

replace msgpack

We've found at least one drawback to msgpack so far:
https://bugzilla.mozilla.org/show_bug.cgi?id=1486478

I think we could go with something simpler/smaller since libprio only uses it internally and not on the wire.

One of my colleagues suggested we could potentially use Rust for this, since doing this sort of thing is super easy and can be done in safe rust code, you just derive(Serialize, Deserialize) on your structs for example: https://github.com/mozilla/sccache/blob/master/src/protocol.rs

This could be done as a small library which exposes a C API, however I do worry a bit about whether that might expose us to timing or other side-channel since the codegen for Rust is different than C and (presumably) not as well understood by the crypto community at large.

In any case, we could probably go with something simpler/smaller than msgpack, and maybe not even have to write it :)

Add PrivateKey export/import

The API makes it possible to generate a key pair and to serialize the public key. The private key should also have accompanying functions for serialization to aid with server development.

This seems to be the intended usage of the API:

  1. Start server A, export public key
  2. Start server B, export public key
  3. Create a client instance with public keys from server A & B

The servers currently don't have a way of persisting their private keys across sessions (reboots, etc).

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

  1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
  2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please see Mozilla-GitHub-Standards or email [email protected].

(Message COC001)

Travis CI free usage ends Dec 3; mozilla repos should switch to other CI platforms

We're opening this issue because your project has used Travis CI within the last 6 months. If you have already migrated off it, you can close and ignore this issue.

Travis CI is ending free builds on public repositories. travis-ci.com stopped providingthem in early November, and travis-ci.org will stop after December 31, 2020. To avoid disruptions to your workflows, you must migrate to another CI service.

For production use cases, we recommend switching to CircleCI. This service is already widely used within Mozilla. There is a guide to migrating from Travis CI to CircleCI available here.

For non production use cases, we recommend either CircleCI or Github Actions. There is a guide to migrating from Travis CI to Github Actions available here. Github Actions usage within Mozilla is new, and you will have to work with our github administrators to enable specific actions following this process.

If you have any questions, reach out in #github-admin:mozilla.org on matrix.

Wiki changes

FYI: The following changes were made to this repository's wiki:

These were made as the result of a recent automated defacement of publically writeable wikis.

Remove precomputed table of roots of unity

The file prio/params.h contains a large precomputed table of roots of unity, which we use to get a modest speedup when computing FFTs. The speedup is not large enough to justify carrying around this huge table and shipping it with the binary—we should just compute this table on the fly as needed.

Add a SWIG Python binding

I wrote a functional Python binding, python-libprio. It uses SWIG for interface definition and has support for a large number of languages. I've found the binding to be useful -- it's been straightforward to use in data analysis tools like pandas.

It looks like SCons has native support for SWIG, so it would fit well in this project. The python binding references this library as a submodule, which can be inverted to be a modular piece of this codebase.

It would be nice to deploy on this on an alpine-python docker image (~60 mb) for testing. Alpine alone is 5mb, but python provides a richer tool-set for testing the entire system. As a security oriented distribution, alpine has up-to-date packages for nss and nspr. The one thing that might need to be tackled as part of a PR is to optionally build the library using musl libc instead of glibc.

Upgrade to clang-format 9.0

Currently we use clang-format 3.9 because in ded8b10 this was the latest version on Ubuntu Trusty, which was what Travis CI stuck us with at the time.

Since 52643ce we've been on Ubuntu Bionic, so we can upgrade to a much newer version (9.0), which is what Homebrew is at right now as well.

We'll need to re-format the code though since one of the reasons we've been stuck in 3.9 is that there were breaking changes after that version.

Add a comparison function for PrioConfig objects

Re: #100 (comment).

When the state of two servers are merged, the config objects are compared for consistency.

libprio/prio/server.c

Lines 103 to 115 in aebda0e

static int
server_cmp(PrioServer s, const_PrioServer s_i)
{
return public_key_cmp(s->cfg->server_a_pub, s_i->cfg->server_a_pub) ||
public_key_cmp(s->cfg->server_b_pub, s_i->cfg->server_b_pub) ||
s->idx != s_i->idx ||
s->cfg->num_data_fields != s_i->cfg->num_data_fields ||
mp_cmp(&s->cfg->modulus, &s_i->cfg->modulus) ||
s->cfg->batch_id_len != s_i->cfg->batch_id_len ||
strncmp((const char*)s->cfg->batch_id,
(const char*)s_i->cfg->batch_id,
s->cfg->batch_id_len);
}

This could be refactored to use a PrioConfig_cmp function that compares the config of the two servers, as well as the other server bits.

Compile ptest with clang sanitizers

Since we're already using clang by default, it would be nice to compile the ptest utility using MemorySanitizer, AddressSanitizer, and UBSanitizer. This way, Travis CI can catch these types of bad behavior during testing.

Some details on how to compile with the sanitizers are here.

Implement easy optimizations for `PrioConfig_new`

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.