Thread-per-core Architecture about photonlibos HOT 18 OPEN

Kindly pinging :)

from photonlibos.

Yes, Photon coroutines can satisfy your requirements. Every resource is located in a single thread and shared among coroutines. You can read the documents for more details.

from photonlibos.

Linux thread is referred to as vCPU in Photon. Each of them has a dedicated scheduler for coroutines (Photon's threads), and a dedicated instance of event engine (e.g. epoll or io_uring). Their execution is basically independent of each other, unless you conduct inter-vCPU task coordination or migration.

from photonlibos.

You can realize interrupt affinity in the same way you do to other applications, e.g. pinning interrupt handler and corresponding photon vCPU (Linux thread) to the same physical CPU core.

from photonlibos.

What would you recommend to separate CPU and IO-bound jobs in the programming models? As far as I know, coroutines are for IO-bound jobs.

from photonlibos.

You can use the migrate API to move your CPU bound tasks to specific vCPU. It’s lightweight

from photonlibos.

What would you recommend for Interprocessor communication if shared memory is absolutely no?

from photonlibos.

Is it a Photon related issue?

from photonlibos.

Already appreciate your answers so I apologize if it sounds unrelated. I am evaluating and comparing Photon with Seastar. Trying to map approaches in Seastar to Photon.

For example, In Seastar, it is mostly done by passing a lambda to a neighbor VCPU. I was wondering what do you think is a best approach to take as communication between VCPUs.

from photonlibos.

A Photon thread (coroutine) is essentially a function. Lambda is also the same.

The underlay implementation of thread migrate is eventfd notification and task queue.

Besides Photon also has a MPMC queue to transmit functions, encapsulated as the so-called WorkPool

from photonlibos.

I've searched the code, how about use sched_setaffinity(linux)/thread_policy_set(macos) to bind vCPU to a single CPU core? @beef9999

from photonlibos.

@loongs-zhang Are you suggesting that we bind vCPU by default?

from photonlibos.

What would you recommend for Interprocessor communication if shared memory is absolutely no?

Multi-process without sharing memory? How about UNIX domain socket?

from photonlibos.

@loongs-zhang Are you suggesting that we bind vCPU by default?

yes

from photonlibos.

What would you recommend for Interprocessor communication if shared memory is absolutely no?
Multi-process without sharing memory? How about UNIX domain socket?

How about deep cloning and sharing?

from photonlibos.

@loongs-zhang Are you suggesting that we bind vCPU by default?

yes

As different apps require different binding configuration, it's difficult for us to do it by default.
For example, a typical scenario is file/storage server. We may need to consider IRQ handlers of the NICs and SSDs, and our service threads (vCPUs). The best binding configuration should minimize CPU switching along the execution.

from photonlibos.

How about deep cloning and sharing?

I not sure whether cloning is feasible, as it may imply sharing in the first place, and @kosav7r said it was "absolutely no".

from photonlibos.

What would you recommend to separate CPU and IO-bound jobs in the programming models? As far as I know, coroutines are for IO-bound jobs.

Photon has a built-in WorkPool to deal with various kinds of background jobs. For IO-bound ones, you can initialize the worker vCPUs to enable coroutines and event engines. For CPU-bound ones, you can simple use kernel threads without initializing photon.

BTW, the jobs are efficiently passed to workers with lock-free shared memory ring queue.

from photonlibos.