ashleykleynhans / runpod-worker-a1111 Goto Github PK

View Code? Open in Web Editor NEW

85.0 4.0 41.0 578 KB

RunPod Serverless Worker for the Automatic1111 Stable Diffusion API

License: GNU General Public License v3.0

Dockerfile 1.63% Python 85.88% Shell 12.49%

docker generative-ai runpod-worker serverless stable-diffusion

runpod-worker-a1111's Introduction

A1111 Stable Diffusion | RunPod Serverless Worker

This is the source code for a RunPod Serverless worker that uses the Automatic1111 Stable Diffusion API for inference.

Model

The model(s) for inference will be loaded from a RunPod Network Volume.

Extensions

This worker includes the following A1111 extensions:

Testing

Installing, Building and Deploying the Serverless Worker

RunPod API Endpoint

You can send requests to your RunPod API Endpoint using the /run or /runsync endpoints.

Requests sent to the /run endpoint will be handled asynchronously, and are non-blocking operations. Your first response status will always be IN_QUEUE. You need to send subsequent requests to the /status endpoint to get further status updates, and eventually the COMPLETED status will be returned if your request is successful.

Requests sent to the /runsync endpoint will be handled synchronously and are blocking operations. If they are processed by a worker within 90 seconds, the result will be returned in the response, but if the processing time exceeds 90 seconds, you will need to handle the response and request status updates from the /status endpoint until you receive the COMPLETED status which indicates that your request was successful.

RunPod API Examples

A1111 APIs

Helper APIs

Optional Webhook Callbacks

You can optionally Enable a Webhook.

Endpoint Status Codes

Status	Description
IN_QUEUE	Request is in the queue waiting to be picked up by a worker. You can call the `/status` endpoint to check for status updates.
IN_PROGRESS	Request is currently being processed by a worker. You can call the `/status` endpoint to check for status updates.
FAILED	The request failed, most likely due to encountering an error.
CANCELLED	The request was cancelled. This usually happens when you call the `/cancel` endpoint to cancel the request.
TIMED_OUT	The request timed out. This usually happens when your handler throws some kind of exception that does return a valid response.
COMPLETED	The request completed successfully and the output is available in the `output` field of the response.

Serverless Handler

The serverless handler (rp_handler.py) is a Python script that handles the API requests to your Endpoint using the runpod Python library. It defines a function handler(event) that takes an API request (event), runs the inference using the model(s) from your Network Volume with the input, and returns the output in the JSON response.

Acknowledgements

Additional Resources

Community and Contributing

Pull requests and issues on GitHub are welcome. Bug fixes and new features are encouraged.

You can contact me and get help with deploying your Serverless worker to RunPod on the RunPod Discord Server below, my username is ashleyk.

Appreciate my work?

runpod-worker-a1111's People

Contributors

Stargazers

Watchers

runpod-worker-a1111's Issues

Unable to start Web UI when following guide - OSError undefined symbol

Hi
Thank you for your work with this package. I am currently trying to get this running, tried both manual steps and the automatic installation script provided.

I connect my storage and start a new Pod with a lightweight template like Pytorch
However I get the same problem every time, I believe it is related to this warning during installation of xformers.

After running every step, I get this error when trying to start at step 8

Run the Web UI:
deactivate
export HF_HOME="/workspace"
cd /workspace/stable-diffusion-webui
./webui.sh -f

Traceback (most recent call last):
  File "/workspace/stable-diffusion-webui/launch.py", line 48, in <module>
    main()
  File "/workspace/stable-diffusion-webui/launch.py", line 44, in main
    start()
  File "/workspace/stable-diffusion-webui/modules/launch_utils.py", line 432, in start
    import webui
  File "/workspace/stable-diffusion-webui/webui.py", line 13, in <module>
    initialize.imports()
  File "/workspace/stable-diffusion-webui/modules/initialize.py", line 16, in imports
    import pytorch_lightning  # noqa: F401
  File "/workspace/venv/lib/python3.10/site-packages/pytorch_lightning/__init__.py", line 35, in <module>
    from pytorch_lightning.callbacks import Callback  # noqa: E402
  File "/workspace/venv/lib/python3.10/site-packages/pytorch_lightning/callbacks/__init__.py", line 14, in <module>
    from pytorch_lightning.callbacks.batch_size_finder import BatchSizeFinder
  File "/workspace/venv/lib/python3.10/site-packages/pytorch_lightning/callbacks/batch_size_finder.py", line 24, in <module>
    from pytorch_lightning.callbacks.callback import Callback
  File "/workspace/venv/lib/python3.10/site-packages/pytorch_lightning/callbacks/callback.py", line 25, in <module>
    from pytorch_lightning.utilities.types import STEP_OUTPUT
  File "/workspace/venv/lib/python3.10/site-packages/pytorch_lightning/utilities/types.py", line 27, in <module>
    from torchmetrics import Metric
  File "/workspace/venv/lib/python3.10/site-packages/torchmetrics/__init__.py", line 14, in <module>
    from torchmetrics import functional  # noqa: E402
  File "/workspace/venv/lib/python3.10/site-packages/torchmetrics/functional/__init__.py", line 14, in <module>
    from torchmetrics.functional.audio._deprecated import _permutation_invariant_training as permutation_invariant_training
  File "/workspace/venv/lib/python3.10/site-packages/torchmetrics/functional/audio/__init__.py", line 14, in <module>
    from torchmetrics.functional.audio.pit import permutation_invariant_training, pit_permutate
  File "/workspace/venv/lib/python3.10/site-packages/torchmetrics/functional/audio/pit.py", line 22, in <module>
    from torchmetrics.utilities import rank_zero_warn
  File "/workspace/venv/lib/python3.10/site-packages/torchmetrics/utilities/__init__.py", line 14, in <module>
    from torchmetrics.utilities.checks import check_forward_full_state_property
  File "/workspace/venv/lib/python3.10/site-packages/torchmetrics/utilities/checks.py", line 25, in <module>
    from torchmetrics.metric import Metric
  File "/workspace/venv/lib/python3.10/site-packages/torchmetrics/metric.py", line 30, in <module>
    from torchmetrics.utilities.data import (
  File "/workspace/venv/lib/python3.10/site-packages/torchmetrics/utilities/data.py", line 22, in <module>
    from torchmetrics.utilities.imports import _TORCH_GREATER_EQUAL_1_12, _XLA_AVAILABLE
  File "/workspace/venv/lib/python3.10/site-packages/torchmetrics/utilities/imports.py", line 50, in <module>
    _TORCHAUDIO_GREATER_EQUAL_0_10: Optional[bool] = compare_version("torchaudio", operator.ge, "0.10.0")
  File "/workspace/venv/lib/python3.10/site-packages/lightning_utilities/core/imports.py", line 73, in compare_version
    pkg = importlib.import_module(package)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/workspace/venv/lib/python3.10/site-packages/torchaudio/__init__.py", line 1, in <module>
    from . import (  # noqa: F401
  File "/workspace/venv/lib/python3.10/site-packages/torchaudio/_extension/__init__.py", line 45, in <module>
    _load_lib("libtorchaudio")
  File "/workspace/venv/lib/python3.10/site-packages/torchaudio/_extension/utils.py", line 64, in _load_lib
    torch.ops.load_library(path)
  File "/workspace/venv/lib/python3.10/site-packages/torch/_ops.py", line 643, in load_library
    ctypes.CDLL(path)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /workspace/venv/lib/python3.10/site-packages/torchaudio/lib/libtorchaudio.so: undefined symbol: _ZN3c104cuda9SetDeviceEi```

Service not ready yet. Retrying...

Hello, thank you very much.

I used the method you helped me with to create serverless endpoints. When I call the interface, the call is unsuccessful.

The error code keeps prompting as follows:

2023-12-16T07:06:28.281344669Z Worker Initiated

2023-12-16T07:06:28.281408906Z Symlinking files from Network Volume

2023-12-16T07:06:28:284478868Z Starting WebUI API

2023-12-16T07:06:28.291203864Z Starting RunPod Handler

2023-12-12T07:06:33.333222087Z {"requestId": null, "message": "Service not ready yet. Retrying...", "level": "INFO"}

2023-12-16T07:06:36.382023361Z {"requestId": null, "message": "Service not ready yet. Retrying...", "level": "INFO"}

2023-12-16T07:06:39.430904540Z {"requestId": null, "message": "Service not ready yet. Retrying...", "level": "INFO"}

2023-12-16T07:06:42-479959Z {"requestId": null, "message": "Service not ready"

May I ask what the reason is?

Roop support

Hey! First of all thanks for this awesome repo!

Could you please add roop support here?

Can you add a webshell api?

I have an idea,add a webshell api,Pass in the shell statement,Then return the result via Webhook.

This enables online downloading of models

Access the A1111 serverless instance while in_progress

Hello,

I'm currently exploring the feasibility of tracking the progress of image generation via the API. Specifically, I'm interested in real-time updates on the image being generated, not just the operational status of the runpod serverless instance. Previously, we utilized an API endpoint in A1111 WebUI for this purpose. However, since our API requests are now routed through the runpod serverless instance, it's unclear how to maintain this functionality.

Is there a method to access the image generation progress directly from A1111 WebUI in a way that avoids the issue where a request to the get_progress API is either queued until image generation is complete or, in some cases, triggers the launch of a new serverless instance to obtain progress, rendering the information irrelevant for the current task? This feature is critical for us as we aim to provide live generation updates on our frontend.

Any guidance or suggestions you could provide would be greatly appreciated.

Thank you for your support.

Feature request: File download

Adding a new model/Lora/embedding requires spinning up a regular pod to write to the network volume.
A new API endpoint could be created that would accept a file URL and destination path, perform the download, and write the file to the specified path.

img2img inpainting mask example

Hey, could you please add to docs example for img2img with inpainting mask?

Starting runpod handler error

Hello again. Sorry to bother you. Runpod can't start this image with errors:

2023-09-16T10:05:55.025616537Z Starting WebUI API
2023-09-16T10:05:55.026057945Z /start.sh: line 10: /workspace/venv/bin/activate: No such file or directory
2023-09-16T10:05:55.031351566Z /start.sh: line 30: deactivate: command not found
2023-09-16T10:05:55.031601013Z Starting RunPod Handler
2023-09-16T10:05:55.031956529Z /start.sh: line 15: /workspace/logs/webui.log: No such file or directory
2023-09-16T10:06:00.461383810Z INFO   | Service not ready yet. Retrying...

NOOB question... How do i run inference or access web-ui using this setup?

I've run through the instructions in the README, the pod is running which is great. I can connect to Jupyter, but there seems to be no instructions that I can find anywhere on how to actual run inference requests on a pod, or to access the webui.

I can see only port 8888 is open be default, no other ports, so how are they accessed on a 'Pod'? I can see Serverless provides an endpoint url for inference which makes it obvious there (although not clear how webui would run from a serverless pod).

Retrying...

I found that when serverless is set to 0 as the minimum worker, this problem occurs when it runs again after a long period of inactivity. It takes a few minutes to call and the callback is successful

Continuously during: {"requestId": null, "message": "Service not ready yet. Retrying...", "level": "INFO"}.

Is it possible that it is caused by the mechanism of serverless?

How to make the return request of webhook only be requested once

How to make the return request of webhook only be requested once?

Because the logic of my webhook processing is more complex and time-consuming, then I found that the webhook will call back every 12 seconds. How to make webhook only called once?

Unable to apply lora.

Hi! I'm using this repository you created very well.

Finished all the other settings, finally managed to run StableDiffusion with runpod serverless. :)

But while all the other features work well, Lora doesn't work at all. Even though I definitely installed extra network extension and updated it through refresh-loars API

Do you have any idea of this?

S3 image uploads

Hi, first of all, thanks for the repo, it's super helpful

I need to save the images generated from the model to S3 to avoid heavy network usage in my infrastructure. In the runpod documentation there is a helper that handles this feature.

Can this feature be used out of the box with this template? If not, how can I implement it?

Thanks