chen3feng / blade-build Goto Github PK

There are four output dirs now, (for different -m -p),
I suggest name output target dirs configurable, and make them subdirs of the same output ROOT dir, such as
BLADE_OUT/32opt
BLADE_OUT/64opt
BLADE_OUT/32dbg
BLADE_OUT/64dbg

we can make BLADE_OUT hidden to avoid dirtying svn

Run test with coverage enabled.
Display coverage test result, maybe by less -R or vim

Blade(error): Traceback (most recent call last):
File "/root/typhoon-blade-develop/blade.zip/blade_main.py", line 253, in main
exit_code = _main(blade_path)
File "/root/typhoon-blade-develop/blade.zip/blade_main.py", line 162, in _main
blade_command=command)
File "/root/typhoon-blade-develop/blade.zip/blade.py", line 181, in init
self.build_environment = BuildEnvironment(self.current_source_path)
File "/root/typhoon-blade-develop/blade.zip/build_environment.py", line 28, in init
self.ccache_installed = self._check_ccache_install()
File "/root/typhoon-blade-develop/blade.zip/build_environment.py", line 69, in _check_ccache_install
universal_newlines=True)
File "/usr/local/python/lib/python2.7/subprocess.py", line 679, in init
errread, errwrite)
File "/usr/local/python/lib/python2.7/subprocess.py", line 1228, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

考虑的feature:

• 根据#include中的头文件名自动得到deps。
• 能否根据 cc/cpp 自动生成 cc_library/cc_binary/cc_test，比如语法：

···bash
genbuild cc_library xxx *.cpp
···

自动产生 BUILD 规则

cc_library(
     name = 'xxx',
     srcs = ['xxx.cpp', 'yyy.cpp'],
     deps = ...
)

并自动再追加到当前目录下的 BUILD文件里（需要先检查是否已存在同名 target）

@talebook

blade checkout
blade update
...

三丰兄：
就我自己的使用情况来看，目前的blade只支持高版本gcc，如gcc4.7，我用ubuntu10.04自带的gcc是无法成功变异的，我升级到ubuntu12.04后，就顺利编译成功了，ubuntu12.04默认是gcc4.7或者接近的版本（具体我记不太清了），如果能在readme中说明一下支持的gcc版本，就容易理解多了，否则大家使用的时候总是碰到各种各样的编译错误。

When i build scribe.thrift from the project scribe, blade said something wrong and exited.

My BUILD file is like this:

thrift_library(
  name = 'scribe_thrift',
  srcs = 'if/scribe.thrift',
  deps = '//thirdparty/thrift-0.9.1:fb303_thrift',
)

and BUILD of library 'fb303_thrift' from thrift is like this:

thrift_library(
  name = 'fb303_thrift',
  srcs = 'contrib/fb303/if/fb303.thrift',
)

BTW, i can build 'fb303_thrift' successfully.

When i build 'scribe_thrift', there is always some error message like this:

[root@skuang-001 scribe]# blade build -j1 :scribe_thrift
... ...
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: building associated VariantDir targets: build64_release
Compiling scribe/if/scribe.thrift to cc source
[INFO] Output directory is: build64_release/scribe/if
Output directory build64_release/scribe/if is unusable: No such file or directory
scons: *** [build64_release/scribe/scribe_constants.cpp] Error 255
scons: building terminated because of errors.
Blade(error): building failure

there are only key messages above. the line beginning with '[INFO]' is printed by me.

Now, i have no idea why scribe.thrift is not output into 'gen-cpp' directory?
is this a bug?

比如加入-Wno-unused-local-typedefs等

which is similar to cc_library, but it's prebuilt.
we can known from target type that it has no source files.

貌似是按照各个cc_test倚赖的源文件的最后修改时间顺序来执行的。

已经尝试：
简单尝试了一下：blade clean ,然后添加-t=1指定单任务并且给每个cc_test添加exclusive = True，blade test --full-test。

应用场景：
因为编写一个多进程的服务，如果能按照固定顺序运行test，库内各个测试就可以有前后倚赖，模拟出服务实际运行环境下的一些操作。

当前情况：
现在有些是在单一测试文件中使用多个test case，有些是通过fork,dead test来进行测试，但还是剩下一些逻辑这样写起来太繁琐，而让多个test有时序倚赖可以非常简单的完成测试，尤其是整个模块一起测试。

cc_config(
optimize = 'O2'
)

Can be overwriten by actural cc target.

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.6.1/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.6.1-9ubuntu3' --with-bugurl=file:///usr/share/doc/gcc-4.6/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++,go --prefix=/usr --program-suffix=-4.6 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.6 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-objc-gc --disable-werror --with-arch-32=i686 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3)

version.cpp文件如下：

/* This file was generated by blade /
extern "C" {
namespace binary_version {
extern const int kSvnInfoCount = 0;
extern const char const kSvnInfo[0] = {};
extern const char kBuildType[] = "release";
extern const char kBuildTime[] = "Tue Jun 18 01:21:02 2013";
extern const char kBuilderName[] = "shunpingye";
extern const char kHostName[] = "shunpingye";
extern const char kCompiler[] = "GCC 4.6.1-9ubuntu3)";
}}

这里
kCompiler[] = "GCC 4.6.1-9ubuntu3)";
有点瑕疵，要不就写全了"gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) "
要么就把上面的右括号去掉了吧。

cc_library(
...
deps = '.../xxx.so',
deps = '.../yyy.a',
)

yeshunping:
Query 功能很赞，是我最想要的，不够是否能更彻底点，将query depended,build结合起来，编译所有依赖它的模块，这个功能对改动基础库最有价值。有这个功能，对Hudson之类的持续集成平台的依赖就低很多，可以有效减少代码提交造成现有模块编译失败的情况出现。

比如说，我们一直使用的都是third_party...

比如现在有这样的目录结构：

root/
   BLADE_ROOT
   a/
      BUILD
   b/
      BUILD

我希望能够在root/下面直接用blade build来产生 a 和 b

这个需要怎么做的？

是否在 root 下弄个 BUILD，然后里面利用自定义的 rule来依赖a 和 b ？

Blade(info): Auto upgrade success: 跳过“/home/users/wenxiang/workspace/baidu/blade”
Blade: Entering directory `/home/users/wenxiang/workspace/baidu'
Blade(warning): blade_path=/home/users/wenxiang/workspace/baidu/blade/src/blade
Blade(warning): working_dir=/home/users/wenxiang/workspace/baidu/cloudatlas/papi
Blade(warning): current_building_path=build64_release
Blade(warning): blade_root_dir=/home/users/wenxiang/workspace/baidu
Blade(info): ccache found
Blade(info): loading BUILDs...
Blade(error): Parse error in cloudatlas/papi/src/main/BUILD, exit...
Traceback (most recent call last):
File "/home/users/wenxiang/workspace/baidu/blade/src/blade/load_build_files.py", line 138, in _load_build_file
execfile(build_file, build_rules.get_all(), None)
File "cloudatlas/papi/src/main/BUILD", line 8, in
'com/baidu/cloudatlas/mapred/ReadInfo.java'
File "/home/users/wenxiang/workspace/baidu/blade/src/blade/java_targets.py", line 125, in java_library
kwargs)
TypeError: init() takes exactly 4 arguments (7 given)

We can consider using scons under svn , or make SCONS_BIN_DIR as configurable,
So users need NOT to install scons using sudo.
If we write code on shared online servers, we will benefit from this changelist.(What is more, blade doesn't need to detect scons version)

For all I know, google has a gyp tool which use to build chrome's project file. Chrome is a so complicated C++ project, but gyp can handle it.
And the ninja build system will speed up the Edit-Compile cycles. So I want to know why we need another build system like typhoon-blade?

cu_library...

下面是我在本地尝试做的修改，暂时可以工作了。还没有来得及跑测试脚本，第一次接触Python和开源社区，楼主请多指教。

1. 更改以后的配置（BLADE_ROOT/blade.conf:）
   proto_library_config(
   protoc='//thirdparty/protobuf-2.4.1/src:protoc.bin',
   protobuf_libs=['//thirdparty/protobuf:protobuf'],
   protobuf_path='thirdparty',
   protobuf_incs = 'thirdparty',
   protobuf_php_path='thirdparty/Protobuf-PHP/library',
   protoc_php_plugin='thirdparty/Protobuf-PHP/protoc-gen-php.php'
   )
2. 在proto类型的scons脚本中，加上protoc_bin为它的依赖(cc_targets.py)
   1451 #liwei12
   1452 proto_config = configparse.blade_config.get_config('proto_library_config')
   1453 protoc_bin = proto_config['protoc']
   1454 #Now protoc_bin is //thirdparty/protobuf/src:protoc.bin
   1455 #build_dir is build64_release
   1456 protoc_bin = self.blade.get_build_path() + protoc_bin.replace(":","/");
   1457 self._write_rule("%s.Proto(['%s', '%s'], ['%s','%s'])" % (
   1458 env_name,
   1459 proto_src,
   1460 proto_hdr,
   1461 os.path.join(self.data['path'], src),
   1462 protoc_bin));
3. 在生成scons脚本的header部分的时候，更改proto可执行文件的路径(rules_generator.py)
   315 proto_config = configparse.blade_config.get_config('proto_library_config')
   316 protoc_bin = proto_config['protoc']
   317
   318 #liwei12
   319 #Now protoc_bin is //thirdparty/protobuf/src:protoc.bin
   320 #build_dir is build64_release
   321 protoc_bin = self.build_dir + protoc_bin.replace(":","/");
   322 protobuf_path = proto_config['protobuf_path']
4. 在proto_library中，第一次运行的时候，添加一个target到target_base!!!(cc_target.py)
   1298 self._add_hardcode_library(protobuf_lib)
   1299 #liwei12
   1300 proto_config = configparse.blade_config.get_config('proto_library_config')
   1301 protoc_bin = proto_config['protoc']
   1302 self._add_hardcode_library(var_to_list(protoc_bin))

你好，

这算不得是issue，就是我有这样的场景，我产生了一个binary之后，需要调试运行，这个binary的运行需要一个放在当前目录下的.ini文件，像这种情况，我想知道best practices是怎么样的呢？

我现在是手动拷贝一个.ini文件到debug目录下，然后运行该程序。但总感觉不是很方便，因为一来默认的.ini文件是属于代码仓库的一部分，而我需要每次修改之后复制到debug目录下。

We should support feature to print warning message when the same source files belong to more than one targets

for example:
cc_library(name = 'a',
srcs = ['a.cc',
'c.cc'],
)

cc_library(name = 'b',
srcs = ['a.cc',
'b.cc'],
)

a.cc belongs to target a and b, we should print warning message to avoid this ossasion.
In most cases , a.cc should seperated into another target.

使用gen_rule写了一个简单的规则, 但是好像运行后并没有在/tmp下创建文件

gen_rule(
  name = "install",
  cmd = "touch /tmp/tt.tt",
  deps = [":smsgd"],
  outs = ["/tmp/tt.tt"]
)

我想要的功能就是简单的把编译生成的程序拷到某个地方

To run unittest , we should using pprof script, but If I don't install it, it fails to work.
To fix it, we should make pprof path configurable,
for example, I put pprof under /devel/pprof
PPROF_PATH="$BLADE_ROOT/devel/pprof"

Passing the testdata by cmd-line using --testargs=your/testdata is ok, but some test cases which use testdata will fail when just run "blade test" without no testargs. Acturally, i have configured the testdata option in cc_test()

我们的场景是openssl有平台相关的源文件，i386和x86_64分别需要编译不同的文件
希望有一种机制能够感觉target arch分别选择不同的文件
或者定义target arch相关的library target

经分析，cc_library需要加上对proto_library的顺序依赖，因为pb.h需要先生成，但是不要引入对dep的cc_library的顺序依赖，因为不同的静态cc_library之间本来就是可并行的。动态库则存在顺序依赖，但是也不需要显式地写出来，因为链接命令里已经隐含。

When I run some unittest, some cases failed, since it were run in ROOT dir, but some test files
are under BLADE_ROOT, we should try to run unittest using BLADE_ROOT as current dir.

可以考虑实现为如果含空格，就split然后打个警告。
@talebook

貌似在32位环境下也给加了-m32选项, 我的gcc版本是4.6, 64位下识别-m32和-m64, 32位下不识别-m32选项, 我改了下下面的内容:

diff --git a/src/blade/blade_platform.py b/src/blade/blade_platform.py
index e1de000..6df06f9 100644
--- a/src/blade/blade_platform.py
+++ b/src/blade/blade_platform.py
@@ -152,8 +152,8 @@ class CcFlagsManager(object):

     def get_flags_except_warning(self):
         """Get the flags that are not warning flags. """
-        flags_except_warning = ['-m%s' % self.options.m, '-mcx16', '-pipe']
-        linkflags = ['-m%s' % self.options.m]
+        flags_except_warning = ['-mcx16', '-pipe']
+        linkflags = []
         if self.options.profile == 'debug':
             flags_except_warning += ['-ggdb3', '-fstack-protector']
         elif self.options.profile == 'release':
diff --git a/src/blade/rules_generator.py b/src/blade/rules_generator.py
index 3430dab..4d7f7c8 100644
--- a/src/blade/rules_generator.py
+++ b/src/blade/rules_generator.py
@@ -54,7 +54,6 @@ class SconsFileHeaderGenerator(object):
         self.version_cpp_compile_template = string.Template("""
 env_version = Environment(ENV = os.environ)
 env_version.Append(SHCXXCOMSTR = '%s$updateinfo%s' % (colors('cyan'), colors('end')))
-env_version.Append(CPPFLAGS = '-m$m')
 version_obj = env_version.SharedObject('$filename')
 """)
         self.blade_config = configparse.blade_config

是我安装配置的问题 还是默认至少会带一个-m32或者-m64的选项?

Current support only svn

Now, the path pattern is hard-coded as lib64/lib.a, this design make it configuable:
cc_library(
prebuilt=True,
pattern = 'lib$bits/lib$name.$suffix'
)
here, $bits means bits, $name means name, $suffix means suffix. other vars are:
$arch
$os
$cc
$profile

Such as

$ blade help --rule cc_library
Define a c/c++ library target...

$ blade help --config cc_config

Implementation:
Add a help.py

help.register(type, name, doc)

About config doc:
by contract:

'cc_config' : {
},
'cc_config__doc__' : '...'

hi, 3feng:
使用中有个困惑，求答疑解惑：
某第三方库(oracle)只有.so库(但是想采用静态编译方式)。所以想用#lib的方式调用oracle库，但是oracle库的rpm包安装不是系统路径，编译的时候就会找不到。这个冲突有解决的办法吗？blade有加入libpath的参数设置吗？
期待您的回复。谢谢。

Almost all unittest are expected to be done in short time,
It is better to set time limit for test case running time.
For some large test,we can offer test_time option for cc_test target type.

--generate-dynamic has bug for deps missing detection,
I has a library, which using snappy, but I don't deps on snappy, But
blade build util/... --generate-dynamic
No errors were reported.

We know a solution for it, for example
cc_library(name = "bar",
desp = [":foo1",
"foo2",
],
)

we can fake a cc_binary using
g++ -o bar_dep_check_dummy -Wl,--whole-archive -lbar -Wl,--no-whole-archive xxx/libmain.a libfoo1.a libfoo2.a

NOTE:
xxx/libmain.a is a library, it has a empty main function, but do NOTHING at all.
We whole-archive target library, so if it miss any library, linking error will be reported.

I try to implement this feature, but it's a little hard for me now.

_check_ccache_install use ccache --version to detect version of ccache , but for some lower version of ccache, it won't work.
We should use "ccache -V " to detect version of ccache

Generate result according to java package into same root dir.
Add more colors.
Each proto_library generate a jar.
Support generate all-in-one jar.

It is sucks using eclipse to edit the source code, and blade / run test in the console.
Please develop a eclipse plugin to enhance it.

$ bb toft/base/:benchmark
Blade(info): Auto upgrade success: Skipped '/usr/local/code/github/devel/typhoon-blade'
Blade(warning): unknown source client type, NOT svn OR git