Comments (9)
我发现文档中有一点没有说清楚。
这儿并发度的控制是client和本地daemon协作完成的。 client会首先向daemon请求quota,然后才会运行gcc做预处理。
所以并发度在daemon配置,而命令由client执行。
请求quota的逻辑位于https://github.com/Tencent/yadcc/blob/master/yadcc/client/rewrite_file.cc#L137
这么做的原因也是多用户环境。多用户环境下,总的并行度受制于机器(尤其是内存,oom容易死机),所以需要有一个全局共享的daemon来控制这些。
from yadcc.
通过gcc -E做的:https://github.com/Tencent/yadcc/blob/master/yadcc/client/rewrite_file.cc#L157
应该会有执行gcc的日志:https://github.com/Tencent/yadcc/blob/master/yadcc/client/command.cc#L249
from yadcc.
通过gcc -E做的:https://github.com/Tencent/yadcc/blob/master/yadcc/client/rewrite_file.cc#L157
应该会有执行gcc的日志:https://github.com/Tencent/yadcc/blob/master/yadcc/client/command.cc#L249
我其实是想了解,预处理是客户端直接发起,还是提交给守护进程执行?我看守护进程的文档中有描述
--lightweight_local_task_overprovisioning_ratio
:针对预处理等轻量任务,我们会额外允许一些任务并发度,这个“额外”的部分为max_local_tasks * lightweight_local_task_overprovisioning_ratio。
现象是在加大并发的情况下,预处理的耗时似乎也跟着提升了,感觉并没有被限制住。
from yadcc.
客户端。多用户环境下守护进程去访问代码可能会有权限问题。
可以top看看实际运行的gcc数量,不管是排队还是一起跑,如果cpu核数受限,60个一起预处理应该耗时都会涨。
另外可以参考https://github.com/Tencent/yadcc/blob/master/yadcc/doc/debugging.md#守护进程 来查询守护进程里面的状态,可能可以看出一些信息。
from yadcc.
可以着重观察字段lightweight_waiters,估计是在排队了
from yadcc.
在make的过程中执行 curl http://localhost:8334/inspect/vars/yadcc
。
守护进程直接挂掉了,还是必现的,只有在没有本地任务的时候才能打印信息。
terminate called after throwing an instance of 'Json::LogicError'
what(): in Json::Value::operator[](ArrayIndex): requires arrayValue
*** Aborted at 1666318308 (unix time) try "date -d @1666318308" if you are using GNU date ***
PC: @ 0x7fcf68e162e7 __GI_raise
*** SIGABRT (@0x5e900195653) received by PID 1660499 (TID 0x7fcf61dd2700) from PID 1660499; stack trace: ***
@ 0xad5209 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7fcf69ac2140 (unknown)
@ 0x7fcf68e162e7 __GI_raise
@ 0x7fcf68e176c8 __GI_abort
@ 0x498633 __gnu_cxx::__verbose_terminate_handler()
@ 0xafc1a6 __cxxabiv1::__terminate()
@ 0xafc1e1 std::terminate()
@ 0xafc313 __cxa_throw
@ 0x48f9de Json::throwLogicError()
@ 0xa97a56 Json::Value::operator[]()
@ 0xa97d3b Json::Value::append()
@ 0xa10016 yadcc::daemon::local::LocalTaskMonitor::DumpInternals()
@ 0xa100b4 _ZZN5flare8FunctionIFN4Json5ValueEvEE15ErasedCopySmallIZN5yadcc6daemon5local16LocalTaskMonitorC4EvEUlvE_EEPKNS4_7TypeOpsEPvOT_ENUlSE_E_4_FUNESE_
@ 0x62087e _ZZN5flare8FunctionIFN4Json5ValueEvEE15ErasedCopySmallIZNS_17ExposedVarDynamicIS2_EC4ESt17basic_string_viewIcSt11char_traitsIcEES4_PNS_15ExposedVarGroupEEUlvE_EEPKNS4_7TypeOpsEPvOT_ENUlSI_E_4_FUNESI_
@ 0xa85301 _ZZN5flare8FunctionIFSt8optionalIN4Json5ValueEESt17basic_string_viewIcSt11char_traitsIcEEEE15ErasedCopyLargeIZNS_15ExposedVarGroup3AddES8_NS0_IFS3_vEEEEUlS8_E2_EEPKNSA_7TypeOpsEPvOT_ENUlSJ_OS8_E_4_FUNESJ_SM_
@ 0xa8560b flare::ExposedVarGroup::Dump()
@ 0xa86185 flare::ExposedVarGroup::TryGet()
@ 0x4d160b flare::rpc::builtin::ExposedVarsHandler::OnGet()
@ 0x4edf8c flare::http::Service::FastCall()
@ 0x662735 flare::rpc::detail::NormalConnectionHandler::ServiceFastCall()
@ 0x662ace _ZZN5flare8FunctionIFvvEE15ErasedCopyLargeIZNS_3rpc6detail23NormalConnectionHandler16ProcessOnePacketEPNS_19NoncontiguousBufferEmEUlvE1_EEPKNS2_7TypeOpsEPvOT_ENUlSD_E_4_FUNESD_
@ 0xa1b6d7 flare::fiber::detail::FiberProc()
from yadcc.
看了下代码,确实有bug。应该是内部用的jsoncpp版本太老然后没做类似的类型检查所以没报。
https://github.com/Tencent/yadcc/blob/master/yadcc/daemon/local/local_task_monitor.cc#L217 。L217和L223用了相同的字段名,分别作为scalar和array。你可以先改个字段名,晚点我更新下代码。
from yadcc.
这儿并发度的控制是client和本地daemon协作完成的。 client会首先向daemon请求quota,然后才会运行gcc做预处理。
了解了,感谢答复 。
from yadcc.
rapidjson不也是腾讯开源的么?你们咋不用?
from yadcc.
Related Issues (20)
- Memory Usage for Large File Operations HOT 2
- build failed, undefined reference to `idn2_xxx' HOT 5
- Importing changes from branch `next` HOT 1
- Can i Preprocess with PCH and compile the job remotely? HOT 5
- How to disable assertions similar to FLARE_PCHECK in the release version? HOT 1
- Fail to build yadcc HOT 1
- 对于跨操作系统的分布式编译应该怎么做 HOT 2
- 安装报错 HOT 1
- ./blade build yadcc/... 报错 HOT 3
- Any develop roadmap on yadcc about more features ? HOT 2
- 单机测试环境下如何生成启动调度器scheduler所需要的token? HOT 4
- 如果使用的编译器不是gcc而是llvm,yadcc是否支持呢?应该如何整合构建系统? HOT 4
- 请发布一个 Release 或者打一个 tag HOT 2
- 安装报错 HOT 7
- [yadcc arm] 资源包打包了源码目录的 libfakeroot.so, 期望打包构建出来的库 HOT 7
- scheduler Unrecognized compilation environment HOT 1
- 请问yadcc 是否支持c++ modules HOT 1
- 技术原理请教 HOT 2
- Cannot contact delegate daemon. Daemon died? HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from yadcc.