ray-project / mobius Goto Github PK

Mobius is an AI infrastructure platform for distributed online learning, including online sample processing, training and serving.

Home Page: https://ray-project.github.io/mobius/

License: Apache License 2.0

Starlark 1.61% Shell 1.51% Java 56.29% Python 13.80% Cython 1.44% C++ 24.75% C 0.59%

training realtime fusion machine-learning distributed-computing online-learning

mobius's Introduction

Mobius : Online Machine Learning. ============

Mobius is an AI infra platform including realtime computing and training.

Ray Streaming

Ray Streaming is a data processing framework built on ray.

Key Features

#.: Cross Language. Based on Ray's multi-language actor, Ray Streaming can also run in multiple languages(only Python and Java is supported currently) with high efficiency. You can implement your operator in different languages and run them in one job.
#.: Single Node Failover. We designed a special failover mechanism that only needs to rollback the failed node it's own, in most cases, to recover the job. This will be a huge benefit if your job is sensitive about failure recovery time. In other frameworks like Flink, instead, the entire job should be restarted once a node has failure.
#.: AutoScaling. (Moved from internal in the future). Generate a new graph with different configurations in runtime without stopping job.
#.: Fusion Training. (Moved from internal in the future). Combine TensorFlow/Pytorch and streaming, then building an e2e online machine learning pipeline.

Examples

Python

import ray
from ray.streaming import StreamingContext

ctx = StreamingContext.Builder() \
    .build()
ctx.read_text_file(__file__) \
    .set_parallelism(1) \
    .flat_map(lambda x: x.split()) \
    .map(lambda x: (x, 1)) \
    .key_by(lambda x: x[0]) \
    .reduce(lambda old_value, new_value:
            (old_value[0], old_value[1] + new_value[1])) \
    .filter(lambda x: "ray" not in x) \
    .sink(lambda x: print("result", x))
ctx.submit("word_count")

Java

StreamingContext context = StreamingContext.buildContext();
List<String> text = Collections.singletonList("hello world");
DataStreamSource.fromCollection(context, text)
    .flatMap((FlatMapFunction<String, WordAndCount>) (value, collector) -> {
        String[] records = value.split(" ");
        for (String record : records) {
            collector.collect(new WordAndCount(record, 1));
        }
    })
    .filter(pair -> !pair.word.contains("world"))
    .keyBy(pair -> pair.word)
    .reduce((oldValue, newValue) ->
            new WordAndCount(oldValue.word, oldValue.count + newValue.count))
    .sink(result -> System.out.println("sink result=" + result));
context.execute("testWordCount");

Use Java Operators in Python

import ray
from ray.streaming import StreamingContext

ctx = StreamingContext.Builder().build()
ctx.from_values("a", "b", "c") \
    .as_java_stream() \
    .map("io.ray.streaming.runtime.demo.HybridStreamTest$Mapper1") \
    .filter("io.ray.streaming.runtime.demo.HybridStreamTest$Filter1") \
    .as_python_stream() \
    .sink(lambda x: print("result", x))
ctx.submit("HybridStreamTest")

Use Python Operators in Java

StreamingContext context = StreamingContext.buildContext();
DataStreamSource<String> streamSource =
    DataStreamSource.fromCollection(context, Arrays.asList("a", "b", "c"));
streamSource
    .map(x -> x + x)
    .asPythonStream()
    .map("ray.streaming.tests.test_hybrid_stream", "map_func1")
    .filter("ray.streaming.tests.test_hybrid_stream", "filter_func1")
    .asJavaStream()
    .sink(value -> System.out.println("HybridStream sink=" + value));
context.execute("HybridStreamTestJob");

Training

Training solution is one of the major topics for online machine learning systems, different from the traditional batch training approach, online training needs to learn from infinite streaming data, with high stability and performance for both system and algorithm level.

Key Features

#.: Elastic. Both ps and worker level elastic during long term running, support dynamic networking for new node add and remove without restart job.
#.: Single Node Failover. Based on ray streaming's capability of Single Node Failover, dynamic networking support single-node failover without restarting the entire job.
#.: Large scale sparse embedding. Provide add-ones of tensorflow, support training with large scale and elastic sparse embedding features.
#.: Streaming input support. A general dataset creator to support all data sources as backend, including both streaming and batch data.
#.: Algorithm toolkits for online learning. An algorithm toolkit to help the long-term training models keep converging.
#.: Validation for continuous model delivery. A validation mechanism to help our system keep delivering high-quality models and intercept all the abnormal models.

Build

Build from source code : - Build a docker using docker/Dockerfile-env - Execute scripts/install.sh

Getting Involved

Forum: For discussions about development, questions about usage, and feature requests.
GitHub Issues: For reporting bugs.
Slack: Join our Slack channel.
StackOverflow: For questions about how to use Ray-Mobius.

mobius's People

Contributors

Stargazers

Watchers

Forkers

alipay ashione lpydhbt eric2016fly shism2 socioprophet zjx1230 asbaharoon java-cds-club omarmahamid visioninhope jovany-wang gofullthrottle

mobius's Issues

Bazel can not support riscv64/mips64 instruction jni build with lastest bazel.

bazel 5.0 build on mips64 or riscv64 occurs

ERROR: /github/home/.cache/bazel/_bazel_root/fa5a074cd6f1[25](https://github.com/ray-project/mobius/runs/5273958213?check_suite_focus=true#step:5:25)5c5f5cefe240bb7613/external/bazel_tools/src/conditions/BUILD:61:15: no such target '@platforms//cpu:riscv64': target 'riscv64' not declared in package 'cpu' defined by /github/home/.cache/bazel/_bazel_root/fa5a074cd6f1255c5f5cefe240bb7613/external/platforms/cpu/BUILD and referenced by '@bazel_tools//src/conditions:linux_riscv64'
INFO: Repository remote_coverage_tools instantiated at:
  /DEFAULT.WORKSPACE.SUFFIX:3:13: in <toplevel>
Repository rule http_archive defined at:
  /github/home/.cache/bazel/_bazel_root/fa5a074cd6f1255c5f5cefe240bb7613/external/bazel_tools/tools/build_defs/repo/http.bzl:364:[31](https://github.com/ray-project/mobius/runs/5273958213?check_suite_focus=true#step:5:31): in <toplevel>
INFO: Repository com_google_absl instantiated at:
  /__w/mobius/mobius/streaming/WORKSPACE:16:15: in <toplevel>
  /github/home/.cache/bazel/_bazel_root/fa5a074cd6f1255c5f5cefe240bb7613/external/com_github_ray_project_ray/bazel/ray_deps_setup.bzl:217:22: in ray_deps_setup
  /github/home/.cache/bazel/_bazel_root/fa5a074cd6f1255c5f5cefe240bb7613/external/com_github_ray_project_ray/bazel/ray_deps_setup.bzl:76:24: in auto_http_archive
Repository rule http_archive defined at:
  /github/home/.cache/bazel/_bazel_root/fa5a074cd6f1255c5f5cefe240bb7613/external/bazel_tools/tools/build_defs/repo/http.bzl:364:31: in <toplevel>
ERROR: /github/home/.cache/bazel/_bazel_root/fa5a074cd6f1255c5f5cefe240bb7613/external/bazel_tools/tools/jdk/BUILD:90:11: errors encountered resolving select() keys for @bazel_tools//tools/jdk:jni

Zero-Copy buffer pool between java/python and native layer.

We'd better avoid duplicated copy from JNI/Cython to cpp objects.

Current data transporting pipeline:

User writes a object in any schema, invoking collect and passing a bytes format data to Data Writer
Data Writer copy and move them to transporting, then send bytes data to other process.

There are two copy operations.

Actually, we don't need copy them between transfer and data writer, which is why we proposal zero-copy buffer pool.
How buffer pool works:

Get a fixed data buffer and wrap them in cython or java ByteBuffer
User function can serilize their data to ByteBuffer directly
Data Writer only capture data's pointer and transfer them in pointer
Release them after upstream consumed and recevied a notification from peer target.

Is mobius suitable to be used in streaming data processing and model inference？

As title says, I am wondering if this repo is suitable to be used in streaming data processing and model inference? like data processing and inference in Flink

Independent Vetex/Actor API

Mobius/Ray Streaming might need users create independent actors these out of streaming DAG.
In our prevois design, job master will invoke and manage all lifecyle of independent actors, such as Parameter Server actor that supported for training inside.

Actually, different training infra need user-define independent actors so engine dev should design a more general api for creating and managing independent actors/vertices.

ctx = context.build_context(context_config)
ctx.independent_vertex(module_name = 'xx', class_name = 'xx', number = 10, config = {}, resources = {}, lazy=false)

For exmplae, we define a parameter server group:

ctx.independent_vertex(module_name = "penrose.ps_actor", class_name = "PsActor", number  = 10)

什么时间开源

请问，什么时间开源，谢谢！！

[Docs] compile and setup instructions

can we have document about how to compile and setup ray streaming?

I am trying to do some mobius evaluation. I tried to do this manually, but I am not sure if I am doing it correctly.

We may need to run mobius in the following environments.

container
linux environment.
macosx environment

General question on mobius design

From what I can see, it seems like mobius is doing something similar to flink and spark, saving states in two level storages(local & remote).

Yet, there is a paper Hazelcast https://arxiv.org/abs/2103.10169 In there design, they directly use in-memory storage to handle states distributedly. If mobius can use the fundamental ray core object store to do something similar, will it be much faster?

Add an e2e integration example showing how we do sampling, training, prediction and serving.

Java test fails to run without opencesus symbols exported.

Version
0.0.1

Describe the bug
2022-02-21 10:58:03,621 INFO JobSchedulerImpl [pool-3-thread-1]: Begin creating workers.
2022-02-21 10:58:03,630 INFO WorkerLifecycleController [ForkJoinPool.commonPool-worker-2]: Start to create worker actor for vertex: 4-[ReduceOperator -> SinkOperator]-0 with resource: {}, workeConfig: {streaming.job.id=00000000, streaming.job.name=testUserDefinedSourceWordCount, channel_type=MEMORY_CHANNEL}.
2022-02-21 10:58:03,630 INFO WorkerLifecycleController [ForkJoinPool.commonPool-worker-9]: Start to create worker actor for vertex: 1-[SourceOperatorImpl -> FlatMapOperator -> KeyByOperator]-0 with resource: {}, workeConfig: {streaming.job.id=00000000, streaming.job.name=testUserDefinedSourceWordCount, channel_type=MEMORY_CHANNEL}.
dyld: lazy symbol binding failed: Symbol not found: __ZN10opencensus4tags6TagKey8RegisterENSt3__117basic_string_viewIcNS2_11char_traitsIcEEEE
Referenced from: /private/var/folders/mq/04v30h4d6y94mslbyrys55nr0000gn/T/native_libs338051773672306680/libstreaming_java.dylib
Expected in: flat namespace
dyld: Symbol not found: __ZN10opencensus4tags6TagKey8RegisterENSt3__117basic_string_viewIcNS2_11char_traitsIcEEEE
Referenced from: /private/var/folders/mq/04v30h4d6y94mslbyrys55nr0000gn/T/native_libs338051773672306680/libstreaming_java.dylib
Expected in: flat namespace

To Reproduce

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[Scheduler] Job Master need non-inplacement scheduling.

In current pipeline first schedule strategy, we assign all of resources to each actor/worker instead of ray GCS management.
It's good way to place all of actors once in single job cluster, but for multitenacy there are many job master actors so they can not manage shared resource of others. It would better to fit more elastic strategy and let gcs do more things in multitenacy mode.

[bug] InputStreamTask.run

ray version:1.9.1
java version:Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
env:mac os 12.1
"KeyByFucntion generic type"：HashMap<String,String>

BUG1:on run-mode = SINGLE_PROCESS,when I use dataStream function "keyBy",then transfrom data in other woker node,i found in big data,may hava 2 errors occur。
BUG2:on run-mode = CLUSTER.when I use dataStream function "keyBy",then transfrom data in another woker node,another woker node jvm crashed。
BUG1：may 1st error:

2022-01-19 17:55:30,268 INFO JobMaster [pool-3-thread-1]: Vertex ExecutionVertex{id=1, name=4-[FlatMapOperator -> OpenSinkOperator]-0, resources={}, state=TO_ADD, containerId=ContainerId{id=b6b0fdaa-1b56-4343-b754-2cb53d7747ee}, workerActor=io.ray.runtime.actor.LocalModeActorHandle@67341926} request rollback, exception msg : java.lang.ClassCastException: java.lang.Byte cannot be cast to java.util.List
	at io.ray.streaming.runtime.serialization.CrossLangSerializer.deserialize(CrossLangSerializer.java:38)
	at io.ray.streaming.runtime.serialization.CrossLangSerializer.deserialize(CrossLangSerializer.java:12)
	at io.ray.streaming.runtime.worker.tasks.InputStreamTask.run(InputStreamTask.java:64)
	at io.ray.runtime.AbstractRayRuntime.lambda$wrapRunnable$2(AbstractRayRuntime.java:246)
	at java.lang.Thread.run(Thread.java:748)

BUG1:may 2nd error:

2022-01-19 18:09:03,354 ERROR InputStreamTask [io.ray.streaming.runtime.worker.tasks.OneInputStreamTask-1642586922675]: Last success checkpointId=0, now occur error.
java.lang.IllegalArgumentException: null
	at java.nio.Buffer.limit(Buffer.java:275) ~[?:1.8.0_231]
	at io.ray.streaming.runtime.transfer.DataReader.getDataMessage(DataReader.java:180) ~[streaming-runtime-1.9.2.jar:?]
	at io.ray.streaming.runtime.transfer.DataReader.read(DataReader.java:134) ~[streaming-runtime-1.9.2.jar:?]
	at io.ray.streaming.runtime.worker.tasks.InputStreamTask.run(InputStreamTask.java:47) ~[streaming-runtime-1.9.2.jar:?]
	at io.ray.runtime.AbstractRayRuntime.lambda$wrapRunnable$2(AbstractRayRuntime.java:246) ~[ray-runtime-1.9.2.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]

BUG2:

 #
 # A fatal error has been detected by the Java Runtime Environment:
 #
 #  SIGSEGV (0xb) at pc=0x0000000124c25a03, pid=52940, tid=0x0000000000009f03
 #
 # JRE version: Java(TM) SE Runtime Environment (8.0_231-b11) (build 1.8.0_231-b11)
 # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.231-b11 mixed mode bsd-amd64 compressed oops)
 # Problematic frame:
 # C  [libstreaming_java.dylib+0x37a03]  _ZN3ray9streaming12MockConsumer21NotifyChannelConsumedEy+0x53
 #
 # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
 #
 # If you would like to submit a bug report, please visit:
 #   http://bugreport.java.com/bugreport/crash.jsp
 # The crash happened outside the Java Virtual Machine in native code.
 # See problematic frame for where to report the bug.
 
 Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
 C  [libstreaming_java.dylib+0x37a03]  _ZN3ray9streaming12MockConsumer21NotifyChannelConsumedEy+0x53
 C  [libstreaming_java.dylib+0x10ac9]  _ZN3ray9streaming10DataReader18NotifyConsumedItemERNS0_19ConsumerChannelInfoEy+0x1b9
 C  [libstreaming_java.dylib+0x23278]  _ZN3ray9streaming17ReliabilityHelper17HandleNoValidItemERNS0_19ConsumerChannelInfoE+0x1a8
 C  [libstreaming_java.dylib+0xd9e0]  _ZN3ray9streaming10DataReader21GetMessageFromChannelERNS0_19ConsumerChannelInfoERNSt3__110shared_ptrINS0_10DataBundleEEEjj+0x6f0
 C  [libstreaming_java.dylib+0xc402]  _ZN3ray9streaming10DataReader17InitChannelMergerEj+0x392
 C  [libstreaming_java.dylib+0xfdac]  _ZN3ray9streaming10DataReader9GetBundleEjRNSt3__110shared_ptrINS0_10DataBundleEEE+0x25c
 C  [libstreaming_java.dylib+0x26be]  Java_io_ray_streaming_runtime_transfer_DataReader_getBundleNative+0x2e
 j  io.ray.streaming.runtime.transfer.DataReader.getBundleNative(JJJJ)V+0
 j  io.ray.streaming.runtime.transfer.DataReader.getBundle(J)V+20
 j  io.ray.streaming.runtime.transfer.DataReader.read(J)Lio/ray/streaming/runtime/transfer/message/ChannelMessage;+14
 j  io.ray.streaming.runtime.worker.tasks.InputStreamTask.run()V+25
 j  io.ray.runtime.AbstractRayRuntime.lambda$wrapRunnable$2(Ljava/lang/Object;Ljava/lang/Runnable;)V+11
 j  io.ray.runtime.AbstractRayRuntime$$Lambda$65.run()V+12
 j  java.lang.Thread.run()V+11
 v  ~StubRoutines::call_stub
 V  [libjvm.dylib+0x2f0792]
 V  [libjvm.dylib+0x2f0f36]
 V  [libjvm.dylib+0x2f10e2]
 V  [libjvm.dylib+0x34b9d1]
 V  [libjvm.dylib+0x571817]
 V  [libjvm.dylib+0x572f12]
 V  [libjvm.dylib+0x48f0fa]
 C  [libsystem_pthread.dylib+0x64f4]  _pthread_start+0x7d
 C  [libsystem_pthread.dylib+0x200f]  thread_start+0xf
 C  0x0000000000000000
 
 Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
 j  io.ray.streaming.runtime.transfer.DataReader.getBundleNative(JJJJ)V+0
 j  io.ray.streaming.runtime.transfer.DataReader.getBundle(J)V+20
 j  io.ray.streaming.runtime.transfer.DataReader.read(J)Lio/ray/streaming/runtime/transfer/message/ChannelMessage;+14
 j  io.ray.streaming.runtime.worker.tasks.InputStreamTask.run()V+25
 j  io.ray.runtime.AbstractRayRuntime.lambda$wrapRunnable$2(Ljava/lang/Object;Ljava/lang/Runnable;)V+11
 j  io.ray.runtime.AbstractRayRuntime$$Lambda$65.run()V+12
 j  java.lang.Thread.run()V+11
 v  ~StubRoutines::call_stub

Bump first alpha 0.0.1 version

Streaming part of mobius will publish 0.0.1 version in branch 0.1-version-bump

Python HybirdStream Test Exception

When I was running python test

python -m pytest python/raystreaming/tests/test_hybrid_stream.py --capture=no

outputs are


(raylet) java.util.concurrent.CompletionException: java.lang.ExceptionInInitializerError
(raylet)        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
(raylet)        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
(raylet)        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
(raylet)        at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596)
(raylet)        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
(raylet)        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
(raylet)        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) 
(raylet)        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
(raylet) Caused by: java.lang.ExceptionInInitializerError
(raylet)        at java.lang.Class.forName0(Native Method)
(raylet)        at java.lang.Class.forName(Class.java:348)
(raylet)        at io.ray.runtime.util.MethodUtils.getReturnTypeFromSignature(MethodUtils.java:51)
(raylet)        at io.ray.runtime.util.ConcurrencyGroupUtils.extractConcurrencyGroupsByAnnotations(ConcurrencyGroupUtils.java:25)
(raylet)        at io.ray.runtime.AbstractRayRuntime.extractConcurrencyGroups(AbstractRayRuntime.java:288)
(raylet)        at io.ray.api.call.ActorCreator.<init>(ActorCreator.java:25)
(raylet)        at io.ray.api.RayCall.actor(RayCall.java:1314)
(raylet)        at io.ray.streaming.runtime.master.scheduler.controller.WorkerLifecycleController.createWorker(WorkerLifecycleController.java:54)
(raylet)        at io.ray.streaming.runtime.master.scheduler.controller.WorkerLifecycleController.lambda$asyncBatchExecute$3(WorkerLifecycleController.java:200)
(raylet)        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
(raylet)        ... 5 more
(raylet) Caused by: java.lang.NullPointerException: {} doesn't exist. [native/linux/libstreaming_java.so]
(raylet)        at io.ray.shaded.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:1007)
(raylet)        at io.ray.runtime.util.BinaryFileUtil.getNativeFile(BinaryFileUtil.java:64)
(raylet)        at io.ray.runtime.util.JniUtils.loadLibrary(JniUtils.java:67)
(raylet)        at io.ray.runtime.util.JniUtils.loadLibrary(JniUtils.java:49)
(raylet)        at io.ray.runtime.util.JniUtils.loadLibrary(JniUtils.java:25)
(raylet)        at io.ray.streaming.runtime.util.EnvUtil.loadNativeLibraries(EnvUtil.java:33)
(raylet)        at io.ray.streaming.runtime.worker.JobWorker.<clinit>(JobWorker.java:48)
(raylet)        ... 15 more
(pid=174960) 2022-04-02 07:00:53,095 ERROR JobClientImpl [main]: Failed to submit job: HybridStreamTest.
(pid=174960) java.lang.RuntimeException: submitting job failed
(pid=174960)    at io.ray.streaming.runtime.client.JobClientImpl.submit(JobClientImpl.java:48) [raystreaming_dist.jar:?]

which means ray binary file util fails to extract libstreaming java so file

Contributor Guideline & Design Doc Collections[Current Status Quo & Future Plan]

Main Modules

Reference & Design Doc

Modules of Ray Streaming:

CrossLang API
Runtime
Coordinator/Scheduler
Reliability/Fault Tolerance
Fusion Training
Advanced Functions

✅ : Finished
⚠️ : Move From Internal/Developing
🏃🏻‍♀️: Welcome to contribute(If you want to contribute features please let us know)

CrossLang API

X-Lang:

Java <-> Python ✅
Arrow ⚠️

API:

Source ✅
Union ✅
Sink ✅
Map ✅
FlatMap ✅
Reduce ✅
Join ⚠️
Window ⚠️

Popular Connectors:

Kafka 🏃🏻‍♀️
Mysql 🏃🏻‍♀️
etc.

Runtime

Transfer:

Low Latency RingBuffer Auto Flush ✅
RandomShuffle ✅
HashShuffle ✅
DynamicRebalance ⚠️
Colocated SharedMemory ⚠️
FlowControl ⚠️ (Without Empty Message): 04 Feb Empty Message

State Backend:

State backend common library ⚠️
Memory Backend ✅
Rocksdb Backend ⚠️
S3 Backend 🏃🏻‍♀️

Buffer Optimization:

BufferPool ⚠️ Part1 28 Mar
ElasticBuffer ⚠️

Coordinator/Scheduler

Coordinator:

Rewrite JobMaster in Python 🏃🏻‍♀️

Scheduler & AutoScale:

PlacementGroup-Pipeline First ⚠️
Pod-Wise-Random-Scheduler ✅
Random Scheduler ⚠️
United Distributed Controller ⚠️🏃🏻‍♀️
Rescale Adaption ⚠️🏃🏻‍♀️

Reliability/Fault Tolerance

Reliability:

Checkpoint Sync ✅
Checkpoint Async ⚠️🏃🏻‍♀️
At Least Once ✅
Exactly Once ⚠️
Exactly Same ⚠️

Fusion Training

Training:

CircularBuffer shared with tensorflow/pytorch reader ⚠️
Parameter Server Scheduler ⚠️
Evaluator Scheduler ⚠️
Parameter Server AutoPartition ⚠️

Advanced Functions

Distributed RPC over DAG ⚠️
Metrics & Profiling ⚠️

format shell script does not work.

Version
Describe the version you used.
master version
Describe the bug
A clear and concise description of what the bug is.
When I execute sh script/format.sh
Output is

'buildifier' is not installed. Use '-sb' to skip formatting bazel files or install the buildifier from
      'https://github.com/bazelbuild/buildtools/tree/master/buildifier'.

then I run this script with parameter -sb, it still shows

cripts git:(e210963) ✗ ./format.sh -sb
'buildifier' is not installed. Use '-sb' to skip formatting bazel files or install the buildifier from
      'https://github.com/bazelbuild/buildtools/tree/master/buildifier'.

My laptop is MacOs
Additional context
Add any other context about the problem here.

[Exception] create actor exception

Version
Describe the version you used.
lasteset with ray version > 2.2
Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

run simple wordcount

Expected behavior
A clear and concise description of what you expected to happen.
No crash and exception raised.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.
Just found pyarrow load library crash in m3 mac book pro

[META] Tracking the code migration from ray repo

Description

Source code compilation and installation

CI/CD

streaming cpp
streaming java
streaming python : #43 enable python ut excludes hybird stream tests
streaming java package & release
streaming python package & many linux release

Docs

doc for streaming installation and basic using
doc for mobius installation and basic using
doc for advanced using

Streaming Detail

Mobius Detail

API
Plugin
Runtime

Build tools access from object storage service (Aliyun)

We provide a static object storage service for cache build tools in binary or source codes to make build easily.

MacOS(11.5)

buildifier: https://ray-mobius-us.oss-us-west-1.aliyuncs.com/ci/macos/buildifier

Linux(Ubuntu)

buildifier: https://ray-mobius-us.oss-us-west-1.aliyuncs.com/ci/linux/buildifier

ray-project / mobius Goto Github PK

mobius's Introduction

Ray Streaming

Key Features

Examples

Python

Java

Use Java Operators in Python

Use Python Operators in Java

Training

Key Features

Build

Getting Involved

mobius's People

Contributors

Stargazers

Watchers

Forkers

mobius's Issues

Main Modules

Reference & Design Doc

Modules of Ray Streaming:

CrossLang API

Runtime

Coordinator/Scheduler

Reliability/Fault Tolerance

Fusion Training

Advanced Functions

Description

Source code compilation and installation

CI/CD

Docs

Streaming Detail

Mobius Detail

MacOS(11.5)

Linux(Ubuntu)

Recommend Projects

Recommend Topics

Recommend Org