Code Monkey home page Code Monkey logo

native-toolchain's Introduction

Environment setup

We currently support building on the following platforms:

  • x86-64 & AArch64 Ubuntu 16.04, 18.04, 20.04
  • x86-64 CentOS 7.x, 8.x
  • x86-86 SLES 12, 15

Other platforms might also work but are not well tested. To set up the build environment from scratch, refer to the specific docker files, e.g. docker/ubuntu1804.df.

Building Dependencies

Running ./buildall.sh in the top level directory produces the binaries for the current versions of all packages. This will likely take several hours the first time you run it. If you rerun the script, it checks the build and check directories for preexisting artifacts and only regenerates them if not present.

Sources

By default, the sources for the different packages are downloaded from an S3 bucket provided by Cloudera. If desired, it's possible to download the exact version of the package and simply move it to the source directory.

For example, if you want to download the gcc source manually, find the gcc-4.9.2.tar.gz archive and copy it to source/gcc. If the file is present it will not be downloaded again.

Building a Specific Package

To build a specific package run:

./build.sh package package.version

for example:

./build.sh python 2.7.15

Its possible as well to build several packages at once.

./build.sh python 2.7.15 llvm 3.3-p5

Here, the arguments are package version pairs.

Mac OS X

To build on Mac we cannot use a custom GCC, so we have to use the system compiler:

SYSTEM_GCC=1 DEBUG=1 ./buildall.sh

How do I contribute code?

You need to first sign and return an ICLA and CCLA before we can accept and redistribute your contribution. Once these are submitted you are free to start contributing to native-toolchain. Submit these to [email protected].

Find

We use Github issues to track bugs for this project. Find an issue that you would like to work on (or file one if you have discovered a new issue!). If no-one is working on it, assign it to yourself only if you intend to work on it shortly.

It’s a good idea to discuss your intended approach on the issue. You are much more likely to have your patch reviewed and committed if you’ve already got buy-in from the native-toolchain community before you start.

Fix

Now start coding! As you are writing your patch, please keep the following things in mind:

First, please include tests with your patch. If your patch adds a feature or fixes a bug and does not include tests, it will generally not be accepted. If you are unsure how to write tests for a particular component, please ask on the issue for guidance.

Second, please keep your patch narrowly targeted to the problem described by the issue. It’s better for everyone if we maintain discipline about the scope of each patch. In general, if you find a bug while working on a specific feature, file a issue for the bug, check if you can assign it to yourself and fix it independently of the feature. This helps us to differentiate between bug fixes and features and allows us to build stable maintenance releases.

Finally, please write a good, clear commit message, with a short, descriptive title and a message that is exactly long enough to explain what the problem was, and how it was fixed.

Please post your patch to the native-toolchain project at https://gerrit.cloudera.org for review. See Impala's guide on using gerrit to submit and review patches for instructions on how to send patches to http://gerrit.cloudera.org, except make sure to send your patch to the native-toolchain project instead of Impala-ASF.

native-toolchain's People

Contributors

alexeyserbin avatar bartash avatar bhavikdbavishi avatar boroknagyz avatar danburkert avatar dtsirogiannis avatar fredyw avatar gaallas avatar glglwty avatar granthenke avatar grundprinzip avatar gshimansky avatar hacosta avatar huangtianhua avatar joemcdonnell avatar lekv avatar mikaelsmith avatar mikesbrown avatar rizaon avatar sahiltakiar avatar smukil avatar stiga-huang avatar timarmstrong avatar tmater avatar toddlipcon avatar twmarshall avatar wesm avatar wzhou-code avatar yjwater avatar zhaorenhai avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

native-toolchain's Issues

native-toolchain/build/binutils-2.26.1/bin/ld.gold: internal error in global, at aarch64.cc:6213 collected2: error: ld returned 1 exit status

I was trying to build native-toolchain with kudu modified for arm64 and based on 5c610bf40, then there is an error raise:

native-toolchain/build/binutils-2.26.1/bin/ld.gold: internal error in global, at aarch64.cc:6213

collected2: error: ld returned 1 exit status

Then I tried to change build.sh of binutiils and configure binutils without --enable-gold, and then kudu built completely and the native-toolchain built completely, maybe it's relate with the commits of kudu: https://github.com/liusheng/kudu/commits/aarch64-support/cmake_modules/KuduLinker.cmake

Can someone confirm this? and I will propose a patch if needed, thanks.

选项附件

automake fails to build in Ubuntu 18.04

Using SYSTEM_GCC=1 GCC_VERSION=7.3.0-15ubuntu2 ./buildall.sh to dodge #24, I get the following error trying to build automake:

  GEN      doc/aclocal-1.14.1
  GEN      doc/automake-1.14.1
help2man: can't get `--help' info from automake-1.14
Try `--no-discard-stderr' if option outputs to stderr
Makefile:3707: recipe for target 'doc/automake-1.14.1' failed
make: *** [doc/automake-1.14.1] Error 255
make: *** Waiting for unfinished jobs....

(That's the tail of check/automake-1.14.1.log.)

gcc make exits with no warnings

Hello,

i'm running buildall.sh
and it appears on wrap make -j... when building gcc 4.9.2 that everything seems to max out for several minutes then the process will exit with status 2.
-j is getting 4 as the BUILD_THREADS.
Again, there is no error message it simply stops running. i've tried adding other flags to make but i get no output whatsoever.

it should be said that I'm running inside of a docker container on debian 9.

EDIT
i'm not sure why i didn't try this before, but i ran make from the build directory without the aid of the build scripts:

In file included from ./tm.h:27:0,
                 from ../../gcc-4.9.2/gcc/cp/except.c:27:
../../gcc-4.9.2/gcc/config/elfos.h:102:21: warning: invalid suffix on literal; C++11 requires a space between literal and string macro [-Wliteral-suffix]
    fprintf ((FILE), "%s"HOST_WIDE_INT_PRINT_UNSIGNED"\n",\
                     ^
../../gcc-4.9.2/gcc/config/elfos.h:170:24: warning: invalid suffix on literal; C++11 requires a space between literal and string macro [-Wliteral-suffix]
       fprintf ((FILE), ","HOST_WIDE_INT_PRINT_UNSIGNED",%u\n",  \
                        ^
In file included from ./tm.h:45:0,
                 from ../../gcc-4.9.2/gcc/cp/except.c:27:
../../gcc-4.9.2/gcc/defaults.h:126:24: warning: invalid suffix on literal; C++11 requires a space between literal and string macro [-Wliteral-suffix]
       fprintf ((FILE), ","HOST_WIDE_INT_PRINT_UNSIGNED",%u\n",  \
                        ^
In file included from ../../gcc-4.9.2/gcc/cp/except.c:1013:0:
cfns.gperf: In function ‘const char* libc_name_p(const char*, unsigned int)’:
cfns.gperf:101:1: error: ‘const char* libc_name_p(const char*, unsigned int)’ redeclared inline with ‘gnu_inline’ attribute
cfns.gperf:26:14: note: ‘const char* libc_name_p(const char*, unsigned int)’ previously declared here
cfns.gperf: At global scope:
cfns.gperf:26:14: warning: inline function ‘const char* libc_name_p(const char*, unsigned int)’ used but never defined
Makefile:1058: recipe for target 'cp/except.o' failed
make[3]: *** [cp/except.o] Error 1
make[3]: Leaving directory '/impala/native-toolchain/source/gcc/build-4.9.2/gcc'
Makefile:4233: recipe for target 'all-stage1-gcc' failed
make[2]: *** [all-stage1-gcc] Error 2
make[2]: Leaving directory '/impala/native-toolchain/source/gcc/build-4.9.2'
Makefile:22967: recipe for target 'stage1-bubble' failed
make[1]: *** [stage1-bubble] Error 2
make[1]: Leaving directory '/impala/native-toolchain/source/gcc/build-4.9.2'
Makefile:900: recipe for target 'all' failed
make: *** [all] Error 2

$RELEASE_NAME Matching error

buildall.sh:222:  if [[ ! "$RELEASE_NAME" =~ CentOS.*5\.[[:digit:]] ]]; then
buildall.sh:293:if [[ ! "$RELEASE_NAME" =~ CentOS.*5\.[[:digit:]] ]]; then
source/llvm/build-source-tarball.sh:55:  if [[ ! "$RELEASE_NAME" =~ CentOS.*5\.[[:digit:]] ]]; then

CentOS.*5\.[[:digit:]] matches my CenOS 7.5

# lsb_release -r -i
Distributor ID:	CentOS
Release:	7.5.1804

Thrift build fails in toolchain build attempt on fresh Ubuntu 14.04 LTS

Snippet from build:

checking for gcc... /home/wesm/code/native-toolchain/build/gcc-4.9.2/bin/gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether /home/wesm/code/native-toolchain/build/gcc-4.9.2/bin/gcc accepts -g... yes
checking for /home/wesm/code/native-toolchain/build/gcc-4.9.2/bin/gcc option to accept ISO C89... none needed
checking whether /home/wesm/code/native-toolchain/build/gcc-4.9.2/bin/gcc understands -c and -o together... yes
checking dependency style of /home/wesm/code/native-toolchain/build/gcc-4.9.2/bin/gcc... gcc3
checking whether we are using the GNU C++ compiler... yes
checking whether /home/wesm/code/native-toolchain/build/gcc-4.9.2/bin/g++ accepts -g... yes
checking dependency style of /home/wesm/code/native-toolchain/build/gcc-4.9.2/bin/g++... gcc3
checking for ranlib... ranlib
checking for bash... /bin/bash
checking for perl... /usr/bin/perl
checking for python... /usr/bin/python
checking for ar... /usr/bin/ar
checking for ant... no
checking Checking EXTERNAL_PATH set to... /home/wesm/code/native-toolchain/source/thrift/thrift-0.9.0-p2/contrib/fb303
checking whether to enable optimized build... yes
checking whether to enable static mode... yes
checking Checking thrift_home set to... /usr/local
checking for boostlib >= 1.33.1... configure: error: We could not detect the boost libraries (version 1.33 or higher). If you have a staged boost library (still not installed) please specify $BOOST_ROOT in your environment and do not give a PATH to --with-boost option.  If you are sure you have boost installed, then check your version number looking in <boost/version.hpp>. See http://randspringer.de/boost for more documentation.

Setting export BOOST_ROOT=$TOOLCHAIN_CLONE/build/boost-1.57.0 beforehand doesn't fix the problem.

Installed libboost-dev with apt and the build succeeds, but this defeats the goal of the toolchain.

ARCH_FLAGS: unbound variable

Hello,

Getting the below error when running on my mac with:
SYSTEM_GCC=1 DEBUG=1 ./buildall.sh

./init-compiler.sh: line 41: ARCH_FLAGS: unbound variable

mac os:
10.12.5 (16F73)

Any help would be appreciated.

Thanks

Build failed on ubuntu:16.04, getting error as "ccache.lock: No such file or directory"

Build failed on ubuntu:16.04, getting error as "native-toolchain/ccache/ccache.lock: No such file or directory: No such file or directory"

root@d6c77eb423d5:/home/impdev/native-toolchain# ./buildall.sh
++ : generic
++ : generic
++ export BUILD_LABEL
++ set +x
Build ID is 3ddbd51e-c164-448c-8924-e5118576c34a-53d8a71d77
/home/impdev/native-toolchain/ccache
/home/impdev/native-toolchain/functions.sh: line 517: /home/impdev/native-toolchain/ccache/ccache.lock: No such file or directory
root@d6c77eb423d5:/home/impdev/native-toolchain# which ccache
/usr/bin/ccache
root@d6c77eb423d5:/home/impdev/native-toolchain#

Failed to download native-toolchain code with git clone

The error message is as follows:

$ git clone https://github.com/cloudera/native-toolchain.git
Cloning into 'native-toolchain'...
remote: Enumerating objects: 145, done.
remote: Counting objects: 100% (145/145), done.
remote: Compressing objects: 100% (111/111), done.
remote: Total 2702 (delta 67), reused 104 (delta 29), pack-reused 2557
Receiving objects: 100% (2702/2702), 1.27 MiB | 206.00 KiB/s, done.
Resolving deltas: 100% (1541/1541), done.
error: invalid path 'source/thrift/thrift-0.9.0-patches/0006-Revert-Thrift-p5:IMPALA-3159-Add-SAN-Wildcard-SSL-cert-client-support-in-python.patch'
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

rpath being set incorrectly

When setting LDFLAGS in init-compiler, we do the escaping of "$ORIGIN" in rpath incorrectly - if you look at the output of a build you'll see rpath being set to something of the form 'RIGIN/../lib', when it should be '$ORIGIN/../lib'

I played around with it for awhile and unfortunately was not able to come up with an escaping that seems to actually work.

One possible solution would be to use a tool like chrpath or patchelf to update the rpath after everything is done, as I've done here: https://github.com/twmarshall/native-toolchain/tree/rpath, but this adds a dependency on a non-standard package, which isn't ideal.

sys/ustat.h not found for gcc-4.9.2 build

There is a problem with building libsanitizer in gcc-4.9.2 on Ubuntu 19.10. There is no sys/ustat.h file in the system. Suggested patch looks like this. Idea is taken from this link https://github.com/vmware/photon/blob/master/SPECS/gcc/libsanitizer-avoidustat.h-glibc-2.28.patch

diff --git a/source/gcc/build.sh b/source/gcc/build.sh
index c87e0bd..632f4e6 100755
--- a/source/gcc/build.sh
+++ b/source/gcc/build.sh
@@ -76,7 +76,7 @@ if [ ! -f $SOURCE_DIR/check/$PACKAGE_STRING ]; then
     # We apply the patches manually here (instead of bumping the patch level) because
     # some components (boost) fail to compile with a modified gcc version.
     PATCH_DIR=${THIS_DIR}/gcc-${PACKAGE_VERSION}-patches
-    apply_patches 4 $PATCH_DIR
+    apply_patches 5 $PATCH_DIR
   fi

   download_gcc_prerequisites
diff --git a/source/gcc/gcc-4.9.2-patches/945-ustat.patch b/source/gcc/gcc-4.9.2-patches/945-ustat.patch
new file mode 100644
index 0000000..5aeb09f
--- /dev/null
+++ b/source/gcc/gcc-4.9.2-patches/945-ustat.patch
@@ -0,0 +1,31 @@
+--- gcc-4.9.2.orig/libsanitizer/sanitizer_common/sanitizer_platform_limits_posix.cc    2019-12-17 17:56:37.654678258 -0600
++++ gcc-4.9.2/libsanitizer/sanitizer_common/sanitizer_platform_limits_posix.cc 2019-12-17 17:58:23.089867685 -0600
+@@ -81,7 +81,6 @@
+ #include <sys/statvfs.h>
+ #include <sys/timex.h>
+ #include <sys/user.h>
+-#include <sys/ustat.h>
+ #include <linux/cyclades.h>
+ #include <linux/if_eql.h>
+ #include <linux/if_plip.h>
+@@ -163,7 +162,19 @@ namespace __sanitizer {
+   unsigned struct_old_utsname_sz = sizeof(struct old_utsname);
+   unsigned struct_oldold_utsname_sz = sizeof(struct oldold_utsname);
+   unsigned struct_itimerspec_sz = sizeof(struct itimerspec);
+-  unsigned struct_ustat_sz = sizeof(struct ustat);
++// Use pre-computed size of struct ustat to avoid <sys/ustat.h> which
++// has been removed from glibc 2.28.
++#if defined(__aarch64__) || defined(__s390x__) || defined (__mips64) \
++  || defined(__powerpc64__) || defined(__arch64__) || defined(__sparcv9) \
++  || defined(__x86_64__)
++#define SIZEOF_STRUCT_USTAT 32
++#elif defined(__arm__) || defined(__i386__) || defined(__mips__) \
++  || defined(__powerpc__) || defined(__s390__)
++#define SIZEOF_STRUCT_USTAT 20
++#else
++#error Unknown size of struct ustat
++#endif
++  unsigned struct_ustat_sz = SIZEOF_STRUCT_USTAT;
+ #endif // SANITIZER_LINUX
+
+ #if SANITIZER_LINUX && !SANITIZER_ANDROID

Ubuntu:16.04 PowerPC platform, libunwind shared library .so file not generated

Followed the below steps :
git clone https://github.com/cloudera/native-toolchain.git ~/native-toolchain
cd ~/native-toolchain
mkdir ccache
./buildall.sh

libunwind build is successful, But it does not generate the 'libunwind.so' file.

impdev@ebfd4dac5024:~/native-toolchain/build/libunwind-1.3-rc1-p3/lib64$ ls
libunwind-generic.a libunwind-ppc64.la libunwind-ptrace.la libunwind-setjmp.la libunwind.la
libunwind-ppc64.a libunwind-ptrace.a libunwind-setjmp.a libunwind.a pkgconfig

When using the native-toolchain libraries to build impala , it gives following error
$IMPALA_HOME/buildall.sh

CMake Error at CMakeLists.txt:154 (message):
Library 'libunwind' has neither shared nor static library files
Call Stack (most recent call first):
CMakeLists.txt:373 (IMPALA_ADD_THIRDPARTY_LIB)

impdev@ebfd4dac5024:~/$ uname -a
Linux ebfd4dac5024 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.