Code Monkey home page Code Monkey logo

Comments (9)

T-ze-yu avatar T-ze-yu commented on July 17, 2024

机器资源状况:
image
Kuscia log:
Uploading p207.log…
Uploading p208.log…

from kuscia.

T-ze-yu avatar T-ze-yu commented on July 17, 2024

p207.log
p208.log

from kuscia.

zimu-yuxi avatar zimu-yuxi commented on July 17, 2024

kuscia API的调用参数和测试数据可以发一下吗?

from kuscia.

T-ze-yu avatar T-ze-yu commented on July 17, 2024

通过命令docker update --memory 32g --memory-swap 40g 增加了容器的可用内存,同时使用了heu设备,重新运行该任务在data目录有结果数据产生,但在日志还是报错:status = StatusCode.RESOURCE_EXHAUSTED
2024-06-26T11:42:04.915790123+08:00 stderr F details = "grpc: received message larger than max (4598659 vs. 4194304)"
2024-06-26T11:42:04.915792996+08:00 stderr F debug_error_string = "UNKNOWN:Error received from peer ipv4:172.18.0.2:8071 {grpc_message:"grpc: received message larger than max (4598659 vs. 4194304)", grpc_status:8, created_time:"2024-06-26T03:42:04.914109278+00:00"}";看上去是grpc的消息大小限制了传输,下面是完整的日志文件:
p208.log
p207.log

from kuscia.

T-ze-yu avatar T-ze-yu commented on July 17, 2024

kuscia API的调用参数和测试数据可以发一下吗?

{'job_id': 'bpa911oghtjhm7cg', 'initiator': 'p208', 'max_parallelism': 1, 'tasks': [{'app_image': 'secretflow-image', 'parties': [{'domain_id': 'p208'}, {'domain_id': 'p207'}], 'alias': 'VertWoeBinning', 'task_id': 'o1q3q14fp6vrwpl8', 'task_input_config': '{"sf_datasource_config": {"p208": {"id": "default-data-source"}, "p207": {"id": "default-data-source"}}, "sf_cluster_desc": {"parties": ["p208", "p207"], "devices": [{"name": "spu", "type": "spu", "parties": ["p208", "p207"], "config": "{\"runtime_config\":{\"protocol\":\"REF2K\",\"field\":\"FM64\"},\"link_desc\":{\"connect_retry_times\":60,\"connect_retry_interval_ms\":1000,\"brpc_channel_protocol\":\"http\",\"brpc_channel_connection_type\":\"pooled\",\"recv_timeout_ms\":1200000,\"http_timeout_ms\":1200000}}"}, {"name": "heu", "type": "heu", "parties": ["p208", "p207"], "config": "{\"mode\": \"PHEU\", \"schema\": \"paillier\", \"key_size\": 2048}"}], "ray_fed_config": {"cross_silo_comm_backend": "brpc_link"}}, "sf_node_eval_param": {"domain": "feature", "name": "vert_woe_binning", "version": "0.0.2", "attr_paths": ["input/input_data/feature_selects", "input/input_data/label", "secure_device_type", "binning_method", "bin_num", "positive_label", "chimerge_init_bins", "chimerge_target_bins", "chimerge_target_pvalue", "report_rules"], "attrs": [{"ss": ["ze6555_1", "ze6555_2", "ze6555_3", "ze6555_4", "ze6555_5", "ze6555_6", "ze6555_7", "ze6555_8", "ze6555_9", "ze6555_10", "ze6555_11", "ze6555_12", "ze6555_13", "ze6555_14", "ze6555_15", "ze6555_16", "ze6555_17", "ze6555_18", "ze6555_19", "ze6555_20", "ze6555_21", "ze6555_22", "ze6555_23", "ze6555_24", "ze6555_25", "ze6555_26", "ze6555_27", "ze6555_28", "ze6555_29", "ze6555_30", "ze6555_31", "ze6555_32", "ze6555_33", "ze6555_34", "ze6555_35", "ze6555_36", "ze6555_37", "ze6555_38", "ze6555_39", "ze6555_40", "ze6555_41", "ze6555_42", "ze6555_43", "ze6555_44", "ze6555_45", "ze6555_46", "ze6555_47", "ze6555_48", "ze6555_49", "ze6555_50", "ze6555_51", "ze6555_52", "ze6555_53", "ze6555_54", "ze6555_55", "ze6555_56", "ze6555_57", "ze6555_58", "ze6555_59", "ze6555_60", "ze6555_61", "ze6555_62", "ze6555_63", "ze6555_64", "ze6555_65", "ze6555_66", "ze6555_67", "ze6555_68", "ze6555_69", "ze6555_70", "ze6555_71", "ze6555_72", "ze6555_73", "ze6555_74", "ze6555_75", "ze6555_76", "ze6555_77", "ze6555_78", "ze6555_79", "ze6555_80", "ze6555_81", "ze6555_82", "ze6555_83", "ze6555_84", "ze6555_85", "ze6555_86", "ze6555_87", "ze6555_88", "ze6555_89", "ze6555_90", "ze6555_91", "ze6555_92", "ze6555_93", "ze6555_94", "ze6555_95", "ze6555_96", "ze6555_97", "ze6555_98", "ze6555_99", "ze6555_100", "ze6555_101", "ze6555_102", "ze6555_103", "ze6555_104", "ze6555_105", "ze6555_106", "ze6555_107", "ze6555_108", "ze6555_109", "ze6555_110", "ze6555_111", "ze6555_112", "ze6555_113", "ze6555_114", "ze6555_115", "ze6555_116", "ze6555_117", "ze6555_118", "ze6555_119", "ze6555_120", "ze6555_121", "ze6555_122", "ze6555_123", "ze6555_124", "ze6555_125", "ze6555_126", "ze6555_127", "ze6555_128", "ze6555_129", "ze6555_130", "ze6555_131", "ze6555_132", "ze6555_133", "ze6555_134", "ze6555_135", "ze6555_136", "ze6555_137", "ze6555_138", "ze6555_139", "ze6555_140", "ze6555_141", "ze6555_142", "ze6555_143", "ze6555_144", "ze6555_145", "ze6555_146", "ze6555_147", "ze6555_148", "ze6555_149", "ze6555_150", "ze6555_151", "ze6555_152", "ze6555_153", "ze6555_154", "ze6555_155", "ze6555_156", "ze6555_157", "ze6555_158", "ze6555_159", "ze6555_160", "ze6555_161", "ze6555_162", "ze6555_163", "ze6555_164", "ze6555_165", "ze6555_166", "ze6555_167", "ze6555_168", "ze6555_169", "ze6555_170", "ze6555_171", "ze6555_172", "ze6555_173", "ze6555_174", "ze6555_175", "ze6555_176", "ze6555_177", "ze6555_178", "ze6555_179", "ze6555_180", "ze6555_181", "ze6555_182", "ze6555_183", "ze6555_184", "ze6555_185", "ze6555_186", "ze6555_187", "ze6555_188", "ze6555_189", "ze6555_190", "ze6555_191", "ze6555_192", "ze6555_193", "ze6555_194", "ze6555_195", "ze6555_196", "ze6555_197", "ze6555_198", "ze6555_199", "ze6555_200", "ze6555_201", "ze6555_202", "ze6555_203", "ze6555_204", "ze6555_205", "ze6555_206", "ze6555_207", "ze6555_208", "ze6555_209", "ze6555_210", "ze6555_211", "ze6555_212", "ze6555_213", "ze6555_214", "ze6555_215", "ze6555_216", "ze6555_217", "ze6555_218", "ze6555_219", "ze6555_220", "ze6555_221", "ze6555_222", "ze6555_223", "ze6555_224", "ze6555_225", "ze6555_226", "ze6555_227", "ze6555_228", "ze6555_229", "ze6555_230", "ze6555_231", "ze6555_232", "ze6555_233", "ze6555_234", "ze6555_235", "ze6555_236", "ze6555_237", "ze6555_238", "ze6555_239", "ze6555_240", "ze6555_241", "ze6555_242", "ze6555_243", "ze6555_244", "ze6555_245", "ze6555_246", "ze6555_247", "ze6555_248", "ze6555_249", "ze6555_250", "ze6555_251", "ze6555_252", "ze6555_253", "ze6555_254", "ze6555_255", "ze6555_256", "ze6555_257", "ze6555_258", "ze6555_259", "ze6555_260", "ze6555_261", "ze6555_262", "ze6555_263", "ze6555_264", "ze6555_265", "ze6555_266", "ze6555_267", "ze6555_268", "ze6555_269", "ze6555_270", "ze6555_271", "ze6555_272", "ze6555_273", "ze6555_274", "ze6555_275", "ze6555_276", "ze6555_277", "ze6555_278", "ze6555_279", "ze6555_280", "ze6555_281", "ze6555_282", "ze6555_283", "ze6555_284", "ze6555_285", "ze6555_286", "ze6555_287", "ze6555_288", "ze6555_289", "ze6555_290", "ze6555_291", "ze6555_292", "ze6555_293", "ze6555_294", "ze6555_295", "ze6555_296", "ze6555_297", "ze6555_298", "ze6555_299", "ze6555_300", "ze6555_301", "ze6555_302", "ze6555_303", "ze6555_304", "ze6555_305", "ze6555_306", "ze6555_307", "ze6555_308", "ze6555_309", "ze6555_310", "ze6555_311", "ze6555_312", "ze6555_313", "ze6555_314", "ze6555_315", "ze6555_316", "ze6555_317", "ze6555_318", "ze6555_319", "ze6555_320", "ze6555_321", "ze6555_322", "ze6555_323", "ze6555_324", "ze6555_325", "ze6555_326", "ze6555_327", "ze6555_328", "ze6555_329", "ze6555_330", "ze6555_331", "ze6555_332", "ze6555_333", "ze6555_334", "ze6555_335", "ze6555_336", "ze6555_337", "ze6555_338", "ze6555_339", "ze6555_340", "ze6555_341", "ze6555_342", "ze6555_343", "ze6555_344", "ze6555_345", "ze6555_346", "ze6555_347", "ze6555_348", "ze6555_349", "ze6555_350", "ze6555_351", "ze6555_352", "ze6555_353", "ze6555_354", "ze6555_355", "ze6555_356", "ze6555_357", "ze6555_358", "ze6555_359", "ze6555_360", "ze6555_361", "ze6555_362", "ze6555_363", "ze6555_364", "ze6555_365", "ze6555_366", "ze6555_367", "ze6555_368", "ze6555_369", "ze6555_370", "ze6555_371", "ze6555_372", "ze6555_373", "ze6555_374", "ze6555_375", "ze6555_376", "ze6555_377", "ze6555_378", "ze6555_379", "ze6555_380", "ze6555_381", "ze6555_382", "ze6555_383", "ze6555_384", "ze6555_385", "ze6555_386", "ze6555_387", "ze6555_388", "ze6555_389", "ze6555_390", "ze6555_391", "ze6555_392", "ze6555_393", "ze6555_394", "ze6555_395", "ze6555_396", "ze6555_397", "ze6555_398", "ze6555_399", "ze6555_400", "ze6555_401", "ze6555_402", "ze6555_403", "ze6555_404", "ze6555_405", "ze6555_406", "ze6555_407", "ze6555_408", "ze6555_409", "ze6555_410", "ze6555_411", "ze6555_412", "ze6555_413", "ze6555_414", "ze6555_415", "ze6555_416", "ze6555_417", "ze6555_418", "ze6555_419", "ze6555_420", "ze6555_421", "ze6555_422", "ze6555_423", "ze6555_424", "ze6555_425", "ze6555_426", "ze6555_427", "ze6555_428", "ze6555_429", "ze6555_430", "ze6555_431", "ze6555_432", "ze6555_433", "ze6555_434", "ze6555_435", "ze6555_436", "ze6555_437", "ze6555_438", "ze6555_439", "ze6555_440", "ze6555_441", "ze6555_442", "ze6555_443", "ze6555_444", "ze6555_445", "ze6555_446", "ze6555_447", "ze6555_448", "ze6555_449", "ze6555_450", "ze6555_451", "ze6555_452", "ze6555_453", "ze6555_454", "ze6555_455", "ze6555_456", "ze6555_457", "ze6555_458", "ze6555_459", "ze6555_460", "ze6555_461", "ze6555_462", "ze6555_463", "ze6555_464", "ze6555_465", "ze6555_466", "ze6555_467", "ze6555_468", "ze6555_469", "ze6555_470", "ze6555_471", "ze6555_472", "ze6555_473", "ze6555_474", "ze6555_475", "ze6555_476", "ze6555_477", "ze6555_478", "ze6555_479", "ze6555_480", "ze6555_481", "ze6555_482", "ze6555_483", "ze6555_484", "ze6555_485", "ze6555_486", "ze6555_487", "ze6555_488", "ze6555_489", "ze6555_490", "ze6555_491", "ze6555_492", "ze6555_493", "ze6555_494", "ze6555_495", "ze6555_496", "ze6555_497", "ze6555_498", "ze6555_499", "ze6555_500", "ze6555_501", "ze6555_502", "ze6555_503", "ze6555_504", "ze6555_505", "ze6555_506", "ze6555_507", "ze6555_508", "ze6555_509", "ze6555_510", "ze6555_511", "ze6555_512", "ze6555_513", "ze6555_514", "ze6555_515", "ze6555_516", "ze6555_517", "ze6555_518", "ze6555_519", "ze6555_520", "ze6555_521", "ze6555_522", "ze6555_523", "ze6555_524", "ze6555_525", "ze6555_526", "ze6555_527", "ze6555_528", "ze6555_529", "ze6555_530", "ze6555_531", "ze6555_532", "ze6555_533", "ze6555_534", "ze6555_535", "ze6555_536", "ze6555_537", "ze6555_538", "ze6555_539", "ze6555_540", "ze6555_541", "ze6555_542", "ze6555_543", "ze6555_544", "ze6555_545", "ze6555_546", "ze6555_547", "ze6555_548", "ze6555_549", "ze6555_550", "ze6555_551", "ze6555_552", "ze6555_553", "ze6555_554", "ze6555_555", "ze6555_556", "ze6555_557", "ze6555_558", "ze6555_559", "ze6555_560", "ze6555_561", "ze6555_562", "ze6555_563", "ze6555_564", "ze6555_565", "ze6555_566", "ze6555_567", "ze6555_568", "ze6555_569", "ze6555_570", "ze6555_571", "ze6555_572", "ze6555_573", "ze6555_574", "ze6555_575", "ze6555_576", "ze6555_577", "ze6555_578", "ze6555_579", "ze6555_580", "ze6555_581", "ze6555_582", "ze6555_583", "ze6555_584", "ze6555_585", "ze6555_586", "ze6555_587", "ze6555_588", "ze6555_589", "ze6555_590", "ze6555_591", "ze6555_592", "ze6555_593", "ze6555_594", "ze6555_595", "ze6555_596", "ze6555_597", "ze6555_598", "ze6555_599", "ze6555_600", "ze6555_601", "ze6555_602", "ze6555_603", "ze6555_604", "ze6555_605", "ze6555_606", "ze6555_607", "ze6555_608", "ze6555_609", "ze6555_610", "ze6555_611", "ze6555_612", "ze6555_613", "ze6555_614", "ze6555_615", "ze6555_616", "ze6555_617", "ze6555_618", "ze6555_619", "ze6555_620", "ze6555_621", "ze6555_622", "ze6555_623", "ze6555_624", "ze6555_625", "ze6555_626", "ze6555_627", "ze6555_628", "ze6555_629", "ze6555_630", "ze6555_631", "ze6555_632", "ze6555_633", "ze6555_634", "ze6555_635", "ze6555_636", "ze6555_637", "ze6555_638", "ze6555_639", "ze6555_640", "ze6555_641", "ze6555_642", "ze6555_643", "ze6555_644", "ze6555_645", "ze6555_646", "ze6555_647", "ze6555_648", "ze6555_649", "ze6555_650", "ze6555_651", "ze6555_652", "ze6555_653", "ze6555_654", "ze6555_655", "ze6555_656", "ze6555_657", "ze6555_658", "ze6555_659", "ze6555_660", "ze6555_661", "ze6555_662", "ze6555_663", "ze6555_664", "ze6555_665", "ze6555_666", "ze6555_667", "ze6555_668", "ze6555_669", "ze6555_670", "ze6555_671", "ze6555_672", "ze6555_673", "ze6555_674", "ze6555_675", "ze6555_676", "ze6555_677", "ze6555_678", "ze6555_679", "ze6555_680", "ze6555_681", "ze6555_682", "ze6555_683", "ze6555_684", "ze6555_685", "ze6555_686", "ze6555_687", "ze6555_688", "ze6555_689", "ze6555_690", "ze6555_691", "ze6555_692", "ze6555_693", "ze6555_694", "ze6555_695", "ze6555_696", "ze6555_697", "ze6555_698", "ze6555_699", "ze6555_700", "ze6555_701", "ze6555_702", "ze6555_703", "ze6555_704", "ze6555_705", "ze6555_706", "ze6555_707", "ze6555_708", "ze6555_709", "ze6555_710", "ze6555_711", "ze6555_712", "ze6555_713", "ze6555_714", "ze6555_715", "ze6555_716", "ze6555_717", "ze6555_718", "ze6555_719", "ze6555_720", "ze6555_721", "ze6555_722", "ze6555_723", "ze6555_724", "ze6555_725", "ze6555_726", "ze6555_727", "ze6555_728", "ze6555_729", "ze6555_730", "ze6555_731", "ze6555_732", "ze6555_733", "ze6555_734", "ze6555_735", "ze6555_736", "ze6555_737", "ze6555_738", "ze6555_739", "ze6555_740", "ze6555_741", "ze6555_742", "ze6555_743", "ze6555_744", "ze6555_745", "ze6555_746", "ze6555_747", "ze6555_748", "ze6555_749", "ze6555_750", "ze6555_751", "ze6555_752", "ze6555_753", "ze6555_754", "ze6555_755", "ze6555_756", "ze6555_757", "ze6555_758", "ze6555_759", "ze6555_760", "ze6555_761", "ze6555_762", "ze6555_763", "ze6555_764", "ze6555_765", "ze6555_766", "ze6555_767", "ze6555_768", "ze6555_769", "ze6555_770", "ze6555_771", "ze6555_772", "ze6555_773", "ze6555_774", "ze6555_775", "ze6555_776", "ze6555_777", "ze6555_778", "ze6555_779", "ze6555_780", "ze6555_781", "ze6555_782", "ze6555_783", "ze6555_784", "ze6555_785", "ze6555_786", "ze6555_787", "ze6555_788", "ze6555_789", "ze6555_790", "ze6555_791", "ze6555_792", "ze6555_793", "ze6555_794", "ze6555_795", "ze6555_796", "ze6555_797", "ze6555_798", "ze6555_799", "ze6555_800", "ze6555_801", "ze6555_802", "ze6555_803", "ze6555_804", "ze6555_805", "ze6555_806", "ze6555_807", "ze6555_808", "ze6555_809", "ze6555_810", "ze6555_811", "ze6555_812", "ze6555_813", "ze6555_814", "ze6555_815", "ze6555_816", "ze6555_817", "ze6555_818", "ze6555_819", "ze6555_820", "ze6555_821", "ze6555_822", "ze6555_823", "ze6555_824", "ze6555_825", "ze6555_826", "ze6555_827", "ze6555_828", "ze6555_829", "ze6555_830", "ze6555_831", "ze6555_832", "ze6555_833", "ze6555_834", "ze6555_835", "ze6555_836", "ze6555_837", "ze6555_838", "ze6555_839", "ze6555_840", "ze6555_841", "ze6555_842", "ze6555_843", "ze6555_844", "ze6555_845", "ze6555_846", "ze6555_847", "ze6555_848", "ze6555_849", "ze6555_850", "ze6555_851", "ze6555_852", "ze6555_853", "ze6555_854", "ze6555_855", "ze6555_856", "ze6555_857", "ze6555_858", "ze6555_859", "ze6555_860", "ze6555_861", "ze6555_862", "ze6555_863", "ze6555_864", "ze6555_865", "ze6555_866", "ze6555_867", "ze6555_868", "ze6555_869", "ze6555_870", "ze6555_871", "ze6555_872", "ze6555_873", "ze6555_874", "ze6555_875", "ze6555_876", "ze6555_877", "ze6555_878", "ze6555_879", "ze6555_880", "ze6555_881", "ze6555_882", "ze6555_883", "ze6555_884", "ze6555_885", "ze6555_886", "ze6555_887", "ze6555_888", "ze6555_889", "ze6555_890", "ze6555_891", "ze6555_892", "ze6555_893", "ze6555_894", "ze6555_895", "ze6555_896", "ze6555_897", "ze6555_898", "ze6555_899", "ze6555_900"]}, {"ss": ["y"]}, {"s": "heu"}, {"s": "quantile"}, {"i64": 10}, {"s": "1"}, {"i64": 100}, {"i64": 10}, {"f": 0.1}, {"b": true}]}, "sf_input_ids": ["6be6d1bc-psi-dataset"], "sf_output_ids": ["c611496c-bin-rule", "c611496c-report"], "sf_output_uris": ["jobs/202406241242475313460/c611496c-bin-rule", "jobs/202406241242475313460/c611496c-report"]}', 'priority': '100'}, {'app_image': 'secretflow-image', 'parties': [{'domain_id': 'p208'}, {'domain_id': 'p207'}], 'alias': 'VertBinSubstitution', 'task_id': 'p9ufb5yzmkstq60e', 'task_input_config': '{"sf_datasource_config": {"p208": {"id": "default-data-source"}, "p207": {"id": "default-data-source"}}, "sf_cluster_desc": {"parties": ["p208", "p207"], "devices": [{"name": "spu", "type": "spu", "parties": ["p208", "p207"], "config": "{\"runtime_config\":{\"protocol\":\"REF2K\",\"field\":\"FM64\"},\"link_desc\":{\"connect_retry_times\":60,\"connect_retry_interval_ms\":1000,\"brpc_channel_protocol\":\"http\",\"brpc_channel_connection_type\":\"pooled\",\"recv_timeout_ms\":1200000,\"http_timeout_ms\":1200000}}"}, {"name": "heu", "type": "heu", "parties": ["p208", "p207"], "config": "{\"mode\": \"PHEU\", \"schema\": \"paillier\", \"key_size\": 2048}"}], "ray_fed_config": {"cross_silo_comm_backend": "brpc_link"}}, "sf_node_eval_param": {"domain": "preprocessing", "name": "vert_bin_substitution", "version": "0.0.1", "attr_paths": [], "attrs": []}, "sf_input_ids": ["6be6d1bc-psi-dataset", "c611496c-bin-rule"], "sf_output_ids": ["c611496c-vertbinsubstitution-dataset"], "sf_output_uris": ["jobs/202406241242475313460/c611496c-vertbinsubstitution-dataset"]}', 'priority': '100'}]}

from kuscia.

zimu-yuxi avatar zimu-yuxi commented on July 17, 2024

通过命令docker update --memory 32g --memory-swap 40g 增加了容器的可用内存,同时使用了heu设备,重新运行该任务在data目录有结果数据产生,但在日志还是报错:status = StatusCode.RESOURCE_EXHAUSTED 2024-06-26T11:42:04.915790123+08:00 stderr F details = "grpc: received message larger than max (4598659 vs. 4194304)" 2024-06-26T11:42:04.915792996+08:00 stderr F debug_error_string = "UNKNOWN:Error received from peer ipv4:172.18.0.2:8071 {grpc_message:"grpc: received message larger than max (4598659 vs. 4194304)", grpc_status:8, created_time:"2024-06-26T03:42:04.914109278+00:00"}";看上去是grpc的消息大小限制了传输,下面是完整的日志文件: p208.log p207.log

目前正常能够拿到结果了吗?

from kuscia.

T-ze-yu avatar T-ze-yu commented on July 17, 2024

目录里面是有一个bin-rule的文件,但通过api查询任务状态是失败的,所以也不知道结果是不是正确的

from kuscia.

T-ze-yu avatar T-ze-yu commented on July 17, 2024

通过命令docker update --memory 32g --memory-swap 40g 增加了容器的可用内存,同时使用了heu设备,重新运行该任务在data目录有结果数据产生,但在日志还是报错:status = StatusCode.RESOURCE_EXHAUSTED 2024-06-26T11:42:04.915790123+08:00 stderr F details = "grpc: received message larger than max (4598659 vs. 4194304)" 2024-06-26T11:42:04.915792996+08:00 stderr F debug_error_string = "UNKNOWN:Error received from peer ipv4:172.18.0.2:8071 {grpc_message:"grpc: received message larger than max (4598659 vs. 4194304)", grpc_status:8, created_time:"2024-06-26T03:42:04.914109278+00:00"}";看上去是grpc的消息大小限制了传输,下面是完整的日志文件: p208.log p207.log

能够解除这个grpc通信量的设置吗

from kuscia.

zimu-yuxi avatar zimu-yuxi commented on July 17, 2024

通过命令docker update --memory 32g --memory-swap 40g 增加了容器的可用内存,同时使用了heu设备,重新运行该任务在data目录有结果数据产生,但在日志还是报错:status = StatusCode.RESOURCE_EXHAUSTED 2024-06-26T11:42:04.915790123+08:00 stderr F details = "grpc: received message larger than max (4598659 vs. 4194304)" 2024-06-26T11:42:04.915792996+08:00 stderr F debug_error_string = "UNKNOWN:Error received from peer ipv4:172.18.0.2:8071 {grpc_message:"grpc: received message larger than max (4598659 vs. 4194304)", grpc_status:8, created_time:"2024-06-26T03:42:04.914109278+00:00"}";看上去是grpc的消息大小限制了传输,下面是完整的日志文件: p208.log p207.log

能够解除这个grpc通信量的设置吗

暂时不支持设置的。
问题原因是:woe 分箱产生的report文件太大了,超过了 4M,目前grpc 最大的数据传输量是 4M。
解决方式:
1.可以将数据维数改小一点试下
2.可以通过二开kuscia来修改传输量大小限制,在此处上方增加opts = append(opts, grpc.MaxRecvMsgSize(1010241024)),尝试一下

from kuscia.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.