Code Monkey home page Code Monkey logo

Comments (9)

0x804d8000 avatar 0x804d8000 commented on July 28, 2024 1

我发现文档中有一点没有说清楚。

这儿并发度的控制是client和本地daemon协作完成的。 client会首先向daemon请求quota,然后才会运行gcc做预处理。

所以并发度在daemon配置,而命令由client执行。

请求quota的逻辑位于https://github.com/Tencent/yadcc/blob/master/yadcc/client/rewrite_file.cc#L137

这么做的原因也是多用户环境。多用户环境下,总的并行度受制于机器(尤其是内存,oom容易死机),所以需要有一个全局共享的daemon来控制这些。

from yadcc.

0x804d8000 avatar 0x804d8000 commented on July 28, 2024

通过gcc -E做的:https://github.com/Tencent/yadcc/blob/master/yadcc/client/rewrite_file.cc#L157

应该会有执行gcc的日志:https://github.com/Tencent/yadcc/blob/master/yadcc/client/command.cc#L249

from yadcc.

zcfh avatar zcfh commented on July 28, 2024

通过gcc -E做的:https://github.com/Tencent/yadcc/blob/master/yadcc/client/rewrite_file.cc#L157

应该会有执行gcc的日志:https://github.com/Tencent/yadcc/blob/master/yadcc/client/command.cc#L249

我其实是想了解,预处理是客户端直接发起,还是提交给守护进程执行?我看守护进程的文档中有描述

--lightweight_local_task_overprovisioning_ratio:针对预处理等轻量任务,我们会额外允许一些任务并发度,这个“额外”的部分为max_local_tasks * lightweight_local_task_overprovisioning_ratio。

现象是在加大并发的情况下,预处理的耗时似乎也跟着提升了,感觉并没有被限制住。

from yadcc.

0x804d8000 avatar 0x804d8000 commented on July 28, 2024

客户端。多用户环境下守护进程去访问代码可能会有权限问题。

可以top看看实际运行的gcc数量,不管是排队还是一起跑,如果cpu核数受限,60个一起预处理应该耗时都会涨。

另外可以参考https://github.com/Tencent/yadcc/blob/master/yadcc/doc/debugging.md#守护进程 来查询守护进程里面的状态,可能可以看出一些信息。

from yadcc.

0x804d8000 avatar 0x804d8000 commented on July 28, 2024

可以着重观察字段lightweight_waiters,估计是在排队了

from yadcc.

zcfh avatar zcfh commented on July 28, 2024

在make的过程中执行 curl http://localhost:8334/inspect/vars/yadcc
守护进程直接挂掉了,还是必现的,只有在没有本地任务的时候才能打印信息。

terminate called after throwing an instance of 'Json::LogicError'
  what():  in Json::Value::operator[](ArrayIndex): requires arrayValue
*** Aborted at 1666318308 (unix time) try "date -d @1666318308" if you are using GNU date ***
PC: @     0x7fcf68e162e7 __GI_raise
*** SIGABRT (@0x5e900195653) received by PID 1660499 (TID 0x7fcf61dd2700) from PID 1660499; stack trace: ***
    @           0xad5209 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fcf69ac2140 (unknown)
    @     0x7fcf68e162e7 __GI_raise
    @     0x7fcf68e176c8 __GI_abort
    @           0x498633 __gnu_cxx::__verbose_terminate_handler()
    @           0xafc1a6 __cxxabiv1::__terminate()
    @           0xafc1e1 std::terminate()
    @           0xafc313 __cxa_throw
    @           0x48f9de Json::throwLogicError()
    @           0xa97a56 Json::Value::operator[]()
    @           0xa97d3b Json::Value::append()
    @           0xa10016 yadcc::daemon::local::LocalTaskMonitor::DumpInternals()
    @           0xa100b4 _ZZN5flare8FunctionIFN4Json5ValueEvEE15ErasedCopySmallIZN5yadcc6daemon5local16LocalTaskMonitorC4EvEUlvE_EEPKNS4_7TypeOpsEPvOT_ENUlSE_E_4_FUNESE_
    @           0x62087e _ZZN5flare8FunctionIFN4Json5ValueEvEE15ErasedCopySmallIZNS_17ExposedVarDynamicIS2_EC4ESt17basic_string_viewIcSt11char_traitsIcEES4_PNS_15ExposedVarGroupEEUlvE_EEPKNS4_7TypeOpsEPvOT_ENUlSI_E_4_FUNESI_
    @           0xa85301 _ZZN5flare8FunctionIFSt8optionalIN4Json5ValueEESt17basic_string_viewIcSt11char_traitsIcEEEE15ErasedCopyLargeIZNS_15ExposedVarGroup3AddES8_NS0_IFS3_vEEEEUlS8_E2_EEPKNSA_7TypeOpsEPvOT_ENUlSJ_OS8_E_4_FUNESJ_SM_
    @           0xa8560b flare::ExposedVarGroup::Dump()
    @           0xa86185 flare::ExposedVarGroup::TryGet()
    @           0x4d160b flare::rpc::builtin::ExposedVarsHandler::OnGet()
    @           0x4edf8c flare::http::Service::FastCall()
    @           0x662735 flare::rpc::detail::NormalConnectionHandler::ServiceFastCall()
    @           0x662ace _ZZN5flare8FunctionIFvvEE15ErasedCopyLargeIZNS_3rpc6detail23NormalConnectionHandler16ProcessOnePacketEPNS_19NoncontiguousBufferEmEUlvE1_EEPKNS2_7TypeOpsEPvOT_ENUlSD_E_4_FUNESD_
    @           0xa1b6d7 flare::fiber::detail::FiberProc()

from yadcc.

0x804d8000 avatar 0x804d8000 commented on July 28, 2024

看了下代码,确实有bug。应该是内部用的jsoncpp版本太老然后没做类似的类型检查所以没报。

https://github.com/Tencent/yadcc/blob/master/yadcc/daemon/local/local_task_monitor.cc#L217 。L217和L223用了相同的字段名,分别作为scalar和array。你可以先改个字段名,晚点我更新下代码。

from yadcc.

zcfh avatar zcfh commented on July 28, 2024

这儿并发度的控制是client和本地daemon协作完成的。 client会首先向daemon请求quota,然后才会运行gcc做预处理。

了解了,感谢答复 。

from yadcc.

aguludunu avatar aguludunu commented on July 28, 2024

rapidjson不也是腾讯开源的么?你们咋不用?

from yadcc.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.