ncdns reproducible build harnesses for RBM.
First, initialize the stuff that's derived from upstream Tor:
./tools/setup-submodule-symlinks.sh
./tools/patch-tor-to-namecoin.sh
Then follow the Tor documentation to build via the Makefile.
RBM-based reproducible build harness for ncdns
Home Page: https://www.namecoin.org/
License: Other
https://github.com/namecoin/ncp11 should be added as a project.
It's a bit difficult to audit this code, because it duplicates several components of upstream tor-browser-build. Would it be a good idea to add upstream tor-browser-build as a Git submodule, and make the common
, container-image
, debootstrap-image
, go
, and goxnet
projects symlinks to that submodule's corresponding projects?
tor-browser-build
Git commit hash 177afba4be247dd88d089d7f5e769625003a181c.ncdns-repro
Git commit hash cafb2b1 as a Git submodule of tor-browser-build
.goxnetip
for GNU/Linux x86_64.goxnetip
should build without errors.
cannot find package "golang.org/x/sys/unix"
Adding goxsysunix
as a dependency of goxnetip
should fix the issue.
https://github.com/namecoin/qlib should be added as a project.
Empirically it appears that "./tools/clean-old" downloads the Git and HTTPS dependencies of its own accord, so we don't need "showconf filename" for that purpose. This should save circa 1 minute.
OpenTimestamps improves verifiability if signing keys are later revoked.
The github.com,alecthomas,template
project needs to be updated from upstream. Our commit hash is a0175ee3bccc567396460bf5acd36800cb10c49c, latest upstream is fb15b899a75114aa79cc930e33c46b577cc664b1.
https://github.com/namecoin/dns-prop279 should be added as a project.
https://github.com/namecoin/crosssign/tree/master/cross_sign_tool should be added as a project.
https://github.com/namecoin/ncprop279 should be added as a project.
I see this in the Cirrus logs when building gcc
:
[19:58:06.213] + echo 'Cleaning cache...'
[19:58:06.213] Cleaning cache...
[19:58:06.213] + rm -rfv out/container-image
[19:58:06.214] removed directory 'out/container-image'
[19:58:06.214] + [[ 0 -eq 0 ]]
[19:58:06.214] + ./tools/clean-old --dry-run
[19:58:06.512] Getting input files for release release ncdns-linux-i686
[19:59:19.608] Removing file /tmp/cirrus-ci-build/out/plain-binaries/plain-binaries-10.5a10-linux-i686-220112.tar.gz
So the ./tools/clean-old
run is taking more than a minute. (For reference, that entire task's build script took 1:29
, so over 2/3 of the build script was consumed by this step.) AFAICT we shouldn't need to run ./tools/clean-old
on every task; it should be sufficient to run it only on the download
task. This would speed up every build task by more than a minute.
The github.com,golang,groupcache
project needs to be updated from upstream. Our commit hash is 5b532d6fd5efaf7fa130d4e859a2fde0fc3a9e1b, latest upstream is 869f871628b6baa9cfbc11732cdf6546b17c1298.
https://github.com/namecoin/pkcs11mod should be added as a project.
Is there any reason that the various subpackages of ncdns are in the same project? With the exception of meek
, upstream tor-browser-build
seems to build one Go package per rbm project, and I suspect that the only reason meek
is an exception is that the extra packages are required in order to use meek, which doesn't seem to be the case for ncdns's subpackages.
https://github.com/namecoin/tlsrestrictnss/tree/master/tlsrestrict_nss_tool should be added as a project.
This will require:
The FLAGS variable for upstream Tor's linux-i686 target is embedded inside var/configure_opt_i686
and therefore isn't properly accessible from pkcs11mod's build scripts. We should refactor it into its own variable and send the result to upstream Tor.
CFLAGS
values can be found in rbm.conf
by grepping for CFLAGS
(in the case of linux-i686 and windows) or FLAGS
(in the case of osx). Also (for osx) in the snowflake build script by grepping for CGO_CFLAGS
.
make submodule-update
produces spurious warnings due to rm
or rmdir
failing. This bug was introduced in #75 . We should hide those warnings unless the overall operation fails.
We should consider relicensing this repo under the tor-browser-build license.
@domob1812 Can you enable the ncdns-repro
repository for Travis builds? You should be able to do so at https://travis-ci.org/namecoin/ncdns-repro . I apparently don't have permissions to do it myself, since I'm not an admin of the Namecoin GitHub org.
We should log timestamps for each line of Cirrus output, so that we can more easily notice performance bottlenecks.
The ncdns
project uses go install
in its build script, whereas everything in upstream tor-browser-build is using go build
. Is there a reason for this? If so, it should probably be documented.
The github.com,alecthomas,units
project needs to be updated from upstream. Our commit hash is 2efee857e7cfd4f3d0138cc3cbb1b4966962b93a, latest upstream is c3de453c63f4bdb4dadffab9805ec00426c505f7.
The locale
folder for Electrum-NMC should be built inside rbm.
Based on past experience, I don't think anyone at Tor is auditing their intermediate projects for reproducibility. Making Cirrus test these projects' reproducibility would be highly useful.
Electrum-NMC should be added as a project. Initially, using the Electrum-NMC Python tarball as an input, and outputting only a Python tarball, is acceptable to close this issue.
We should add a Cirrus check that re-generates the Cirrus YML and confirms that it is unchanged. This would help us catch any YML determinism issues caused by merge conflicts.
https://github.com/namecoin/tlsrestrictnss should be added as a project.
https://github.com/namecoin/qlib/tree/master/qexe should be added as a project.
https://github.com/namecoin/crosssignnameconstraint should be added as a project.
The github.com,coreos,go-systemd
project needs to be updated from upstream. Our commit hash is 95778dfbb74eb7e4dbaf43bf7d71809650ef8076 (which is from master
branch somewhere between v18
and v19
), latest upstream is tag v20
.
https://github.com/namecoin/crosssign should be added as a project.
The ncdns
project needs to be updated from upstream. Our commit hash is 4ea036742bd5b10a7b56765355dcc88724632f32, latest upstream is 006f537e01cf20db01b18e399ed51f57211db91c.
Current Electrum-NMC uses PyQt tools to compile Qt Designer forms into PyQt code; we should do this step inside RBM.
The github.com,mattn,go-isatty
project needs to be updated from upstream. Our commit hash is 7fcbc72f853b92b5720db4a6b8482be612daef24 (prior to tag v0.0.1
), latest upstream is tag v0.0.8
.
https://github.com/namecoin/safetlsa should be added as a project.
The packages
folder for Electrum-NMC should be built inside rbm.
https://github.com/namecoin/tlsrestrictnss/tree/master/tlsrestrictnsssync should be added as a project.
When uploading the Git cache, I see this a lot:
[11:05:10.404] SHA for cache folders (/tmp/cirrus-ci-build/git_clones) is 'f7a7f6a796f55a086657d4067c0a71da80e4ab34a4e91b366aed00027406769d'
[11:05:10.404] Cache git_release_osx_x86_64 has changed!
[11:05:10.404] List of changes for cache folders (/tmp/cirrus-ci-build/git_clones):
[11:05:10.457] modified: ncp11/.git/index
[11:05:10.457] modified: goeasyconfig/.git/index
[11:05:10.457] modified: gowebsocket/.git/index
[11:05:10.457] modified: cmake/.git/index
[11:05:10.457] modified: gobtcd/.git/index
[11:05:10.457] modified: cctools/.git/index
[11:05:10.457] modified: godns/.git/index
[11:05:10.457] modified: goservice/.git/index
[11:05:10.458] modified: ncprop279/.git/index
[11:05:10.458] modified: goisatty/.git/index
[11:05:10.458] modified: gobtcd2/.git/index
[11:05:10.458] modified: goncrpcclient/.git/index
[11:05:10.458] modified: goansicolor/.git/index
[11:05:10.458] modified: gotext/.git/index
[11:05:10.458] modified: gogroupcache/.git/index
[11:05:10.458] modified: gosocks/.git/index
[11:05:10.458] modified: gounits/.git/index
[11:05:10.458] modified: ninja/.git/index
[11:05:10.458] modified: gobtcutil/.git/index
[11:05:10.458] modified: gosvcutils/.git/index
[11:05:10.458] modified: goconfigurable/.git/index
[11:05:10.458] modified: goxcryptoripemd160/.git/index
[11:05:10.458] modified: gokingpin/.git/index
[11:05:10.458] modified: goxcryptoed25519/.git/index
[11:05:10.458] modified: gopkcs11/.git/index
[11:05:10.458] modified: gosplicesign/.git/index
[11:05:10.458] modified: gopflag/.git/index
[11:05:10.458] modified: gotemplate/.git/index
[11:05:10.458] modified: goxnet/.git/index
[11:05:10.458] modified: gobtclog/.git/index
[11:05:10.458] modified: gotlsrestrictnss/.git/index
[11:05:10.458] modified: libtapi/.git/index
[11:05:10.458] modified: goncbtcjson/.git/index
[11:05:10.458] modified: ncdns/.git/index
[11:05:10.458] modified: gocrosssignnameconstraint/.git/index
[11:05:10.458] modified: goxnetip/.git/index
[11:05:10.458] modified: gomadns/.git/index
[11:05:10.458] modified: godegoutils/.git/index
[11:05:10.458] modified: goxlog/.git/index
[11:05:10.458] modified: goxsys/.git/index
[11:05:10.458] modified: gosystemd/.git/index
[11:05:10.458] modified: gobuildinfo/.git/index
[11:05:10.458] modified: gopkcs11mod/.git/index
[11:05:10.458] modified: godexlogconfig/.git/index
[11:05:10.458] modified: goxsysunix/.git/index
[11:05:10.458] modified: gopretty/.git/index
[11:05:10.458] modified: gotoml/.git/index
[11:05:15.398] git_release_osx_x86_64 cache size is 418Mb.
[11:05:15.398] Uploading cache git_release_osx_x86_64...
The resulting upload took 17 seconds, all because Git mutates its index
file in all repos that we touched. Is there a way around this? Will Git get unhappy if we, say, just delete all the index
files prior to uploading the cache?
The github.com,kr,pretty
project needs to be updated from upstream. Our commit hash is a0c635c0aede2cb0c2f8bb0f0d70485fbdd11bd4, latest upstream is 71e7e49937503c662b9b636fd6b2c14b1aa818a5.
make release-linux-x86_64
sudo ./ncdns
ncdns should drop privileges
Error in service: Daemon must not run as root or with capabilities; run as non-root user or use -uid
My understanding is that this behavior shows up when cgo is disabled. Looking at the source code for the ncdns
rbm project, I don't see any export CGO_ENABLED=1
like shows up in the go-webrtc
project from Tor Browser; nor do I see '[% c("var/compiler") %]'
as an input file. I suspect that cgo may need to be enabled for both ncdns and the gopkg.in,hlandau,service.v2
project, and maybe some of the latter's dependencies as well. I'm pretty sure that we only need to enable cgo on linux targets, since the only usage of cgo in ncdns that I'm aware of is libcap, which is solely to work around a Linux bug.
./rbm/rbm build ncp11 --target release --target ncdns-linux-x86_64
Files should be identical.
--- a/ncp11-0.0.1-linux-x86_64-7197ac.tar.gz
+++ b/ncp11-0.0.1-linux-x86_64-7197ac.tar.gz
├── ncp11-0.0.1-linux-x86_64-7197ac.tar
│ ├── ./libncp11.so
│ │ ├── readelf --wide --notes {}
│ │ │ @@ -1,8 +1,8 @@
│ │ │
│ │ │ Displaying notes found in: .note.go.buildid
│ │ │ Owner Data size Description
│ │ │ - Go 0x00000053 Unknown note type: (0x00000004) description data: 78 72 67 71 62 7a 62 2d 4d 45 6d 2d 31 37 69 4b 50 57 66 75 2f 6f 76 46 63 4e 48 6a 32 46 6d 4d 55 71 44 66 52 4a 57 55 4f 2f 4f 32 39 39 57 30 76 4f 4c 47 2d 66 53 61 77 55 62 6d 36 39 2f 68 53 77 6a 4a 6f 70 72 51 4f 30 34 43 69 41 48 4b 78 4c 47
│ │ │ + Go 0x00000053 Unknown note type: (0x00000004) description data: 78 72 67 71 62 7a 62 2d 4d 45 6d 2d 31 37 69 4b 50 57 66 75 2f 6f 76 46 63 4e 48 6a 32 46 6d 4d 55 71 44 66 52 4a 57 55 4f 2f 4f 32 39 39 57 30 76 4f 4c 47 2d 66 53 61 77 55 62 6d 36 39 2f 30 37 5a 55 44 54 4b 44 75 6d 55 6b 31 54 6e 36 38 64 4c 6a
│ │ │
│ │ │ Displaying notes found in: .note.gnu.gold-version
│ │ │ Owner Data size Description
│ │ │ GNU 0x00000009 NT_GNU_GOLD_VERSION (gold version) Version: gold 1.16
│ │ ├── readelf --wide --version-info {}
│ │ │ @@ -84,15 +84,15 @@
│ │ │ 140: 1 (*global*) 1 (*global*) 1 (*global*) 1 (*global*)
│ │ │ 144: 1 (*global*) 1 (*global*) 1 (*global*) 1 (*global*)
│ │ │ 148: 1 (*global*) 1 (*global*) 1 (*global*) 1 (*global*)
│ │ │ 14c: 1 (*global*) 1 (*global*)
│ │ │
│ │ │ Version definition section '.gnu.version_d' contains 1 entry:
│ │ │ Addr: 0x0000000000005330 Offset: 0x005330 Link: 3 (.dynstr)
│ │ │ - 000000: Rev: 1 Flags: BASE Index: 1 Cnt: 1 Name: /tmp/go-build832923583/b001/exe/a.out
│ │ │ + 000000: Rev: 1 Flags: BASE Index: 1 Cnt: 1 Name: /tmp/go-build272286888/b001/exe/a.out
│ │ │
│ │ │ Version needs section '.gnu.version_r' contains 3 entries:
│ │ │ Addr: 0x000000000000534c Offset: 0x00534c Link: 3 (.dynstr)
│ │ │ 000000: Version: 1 File: libc.so.6 Cnt: 3
│ │ │ 0x0010: Name: GLIBC_2.2.5 Flags: none Version: 2
│ │ │ 0x0020: Name: GLIBC_2.3.4 Flags: none Version: 3
│ │ │ 0x0030: Name: GLIBC_2.4 Flags: none Version: 4
│ │ ├── readelf --wide --decompress --hex-dump=.dynstr {}
│ │ │ @@ -1,15 +1,15 @@
│ │ │
│ │ │ Hex dump of section '.dynstr':
│ │ │ 0x000021f0 0072756e 74696d65 2e746c73 67005f5f .runtime.tlsg.__
│ │ │ 0x00002200 676d6f6e 5f737461 72745f5f 00736967 gmon_start__.sig
│ │ │ 0x00002210 69736d65 6d626572 00474c49 42435f32 ismember.GLIBC_2
│ │ │ 0x00002220 2e322e35 006c6962 632e736f 2e36002f .2.5.libc.so.6./
│ │ │ - 0x00002230 746d702f 676f2d62 75696c64 38333239 tmp/go-build8329
│ │ │ - 0x00002240 32333538 332f6230 30312f65 78652f61 23583/b001/exe/a
│ │ │ + 0x00002230 746d702f 676f2d62 75696c64 32373232 tmp/go-build2722
│ │ │ + 0x00002240 38363838 382f6230 30312f65 78652f61 86888/b001/exe/a
│ │ │ 0x00002250 2e6f7574 005f696e 69740073 69676164 .out._init.sigad
│ │ │ 0x00002260 64736574 005f6669 6e690073 69676163 dset._fini.sigac
│ │ │ 0x00002270 74696f6e 006c6962 70746872 6561642e tion.libpthread.
│ │ │ 0x00002280 736f2e30 00756e73 6574656e 76005f49 so.0.unsetenv._I
│ │ │ 0x00002290 544d5f64 65726567 69737465 72544d43 TM_deregisterTMC
│ │ │ 0x000022a0 6c6f6e65 5461626c 65007369 67656d70 loneTable.sigemp
│ │ │ 0x000022b0 74797365 74005f49 544d5f72 65676973 tyset._ITM_regis
This seems to be caused by golang/go#33772 . The recommendation in that issue is to pass -ldflags=-buildid=
to go build
.
Signify is much better regarded by crypto experts than OpenPGP.
The github.com,miekg,dns
project needs to be updated from upstream. Our commit hash is 8a56deec68a78bfea6e73db7b086175eb8beec8b (between tags v1.1.13
and v1.1.14
), latest upstream is tag v1.1.15
.
https://github.com/namecoin/crosssignnameconstraint/tree/master/cross_sign_name_constraint_tool should be added as a project.
We should rotate the Cirrus RBM signing key before we use it to sign any Nightlies that actual people try to download.
I'm seeing this in the plain-binaries
task:
[13:17:09.603] SHA for cache folders (/tmp/cirrus-ci-build/out) is '1d984823efc7aa958bbde08b075d67ee82d9a3a723cce6bf92081d81893bd862'
[13:17:09.603] Cache out_release_linux_x86_64 has changed!
[13:17:09.603] List of changes for cache folders (/tmp/cirrus-ci-build/out):
[13:17:09.603] created: plain-binaries/ncdns-c947efb679dd-linux-x86_64-2241e6.tar.xz
[13:17:09.603] created: q/q-0.0.4-linux-x86_64-126875.tar.gz
[13:17:14.556] out_release_linux_x86_64 cache size is 975Mb.
[13:17:14.556] Uploading cache out_release_linux_x86_64...
This cache upload took 27 seconds. Seems like it'd be faster if we only had to upload the output folders for the projects that actually got a change.
We should be able to do this by making the YML generator iterate over each of the subfolders of projects
. The YML will need to be regenerated when new projects are added, but now that we verify determinism of the YML, Cirrus will remind us to do so.
Right now we're pulling in Namecoin projects by hash; we should add Namecoin release keys to the keyring and then pull them in by version.
https://github.com/namecoin/certdehydrate-dane-rest-api should be added as a project.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.