Serialization Issue about ray HOT 12 CLOSED

Please refer to the entire code here: https://github.com/Timothy102/soniox/blob/main/main.py.

I have no idea what an EnhancedModule is. I'm running the code above and nothing else.

from ray.

Can you reformat the above description? It seemed that the repro steps are mis-formatted.

Can you also paste the error message?

from ray.

Here's the error I receieve.

TypeError: Could not serialize the function main.store_all_videos_and_captions_from_urls:
================================================================================
Checking Serializability of <function store_all_videos_and_captions_from_urls at 0x7f7e44fb8b80>
================================================================================
!!! FAIL serialization: cannot pickle 'EnhancedModule' object
Detected 3 global variables. Checking serializability...
    Serializing 'Channels' <class '__main__.Channels'>...
    Serializing 'Channel' <class '__main__.Channel'>...
    Serializing 'preprocessing_video' <ray.remote_function.RemoteFunction object at 0x7f7e44faab80>...
    !!! FAIL serialization: cannot pickle 'EnhancedModule' object
        Serializing '_function' <function preprocessing_video at 0x7f7e49233dc0>...
        !!! FAIL serialization: cannot pickle 'EnhancedModule' object
        Detected 1 global variables. Checking serializability...
            Serializing 'Video' <__main__.ActorClass(Video) object at 0x7f7e44faaa90>...
            !!! FAIL serialization: cannot pickle 'EnhancedModule' object
        Serializing '__generator_backpressure_num_objects' None...
================================================================================
Variable: 

        FailTuple(Video [obj=<__main__.ActorClass(Video) object at 0x7f7e44faaa90>, parent=<function preprocessing_video at 0x7f7e49233dc0>])

from ray.

What's the main.py? Can you provide a repro script? Without it, it's very difficult for us to dig down further and do anything.

from ray.

FAIL serialization: cannot pickle 'EnhancedModule' object

What's this EnhancedModule specifically?

from ray.

tag @hongchaodeng

from ray.

Thanks for providing it! Added it to our support backlogs. Will debug it.

from ray.

OK I looked at the code and figured out the root cause here.

The EnhancedModule comes form the yt-dlp library that your code is using:

https://github.com/yt-dlp/yt-dlp/blob/ac817bc83efd939dca3e40c4b527d0ccfc77172b/yt_dlp/compat/compat_utils.py#L30

This one is not serializable. There might be more though.

You have to do customized serialization as documented here: https://github.com/yt-dlp/yt-dlp/blob/ac817bc83efd939dca3e40c4b527d0ccfc77172b/yt_dlp/compat/compat_utils.py#L30

from ray.

Hey, thank you, but you pasted the same link twice and that still doesn't tell me how I can fix this issue.

Your prompt attention would be greatly appreciated.

from ray.

Sorry.
It is this doc: https://docs.ray.io/en/latest/ray-core/objects/serialization.html#customized-serialization

from ray.

So, how exactly should this class of mine look like?

class Channel: """Each Channel is a class of its own. """ def __init__(self, channel_id, url=None, args=None): self.url = url self.channel_id = channel_id self.output_folder = args.output_folder

from ray.

https://github.com/Timothy102/soniox -- please check do.py again. This code is not coming through. I'm dying here.

from ray.