Code Monkey home page Code Monkey logo

unpaint's Introduction

Introduction

an image showing a windowed application where the left side shows a painter's workshop with a canvas selected, while the right side shows the same image, but with the selected canvas replaced with a variation of Mona Lisa

an image showing a windowed application where the left side shows a depth map of a flying bird, while the right side shows similarly shaped tropical bird

New: Unpaint demonstration on Xbox consoles, see it on YouTube.

This repository contains a fully C++ implementation of Stable Diffusion-based image synthesis tool called Unpaint. It supports basic and ControlNet enhanced implementations of txt2img, img2img, inpainting pipelines and the safety checker. This solution does not depend on Python and runs the entire image generation process in a single process, making deployments significantly simpler and smaller, essentially consisting of a few executable and library files (~30MB), and the model weights (~2.5GB / model). Using the library it is possible to integrate Stable Diffusion into almost any application, which can import C++ or C functions.

For technical details please check the page of the C++ Stable Diffusion library which Unpaint is based on here.

System requirements

The application uses DirectML and it supports current GPUs of NVidia and AMD, but should work on other DirectML capable hardware as well. We encourage you to try it with different hardware and raise issues on this page if you have problems. Make sure to install the latest Windows updates and especially the latest GPU drivers, as Unpaints benefits from Microsoft's recent Build conference announced optimizations greatly.

Please note that ONNX based pipelines are not yet necessarily as optimized as with using pytorch and CUDA directly, however with latest drivers they are pretty close.

How to install the app?

Unpaint is now available in the Microsoft Store, if you do not need the latest dev builds, please install it from there.

You can find the app installers on the releases page.

To install the app the first time follow these steps:

  • Download both the .msix and .crt files
  • Right click the .crt file and select install certificate
  • In the wizard select Local Machine and click next
  • Then select Place all certificates in following Store and click Browse
  • Select Trusted Root Certification Authorities and click OK
  • Click Next and then Finish

This installs Unpaint's self-signed security certificate which is required by the new MSIX installer pipeline for test builds, this is not needed if you install the app from the store.

If you feel unsafe consider using SignTool to resign the Unpaint package with your own certificate (you need to import your own one as well), you can refer to the end of this file on how to do it.

After this just open the .msix file and should install fine and start the app.

Installing models

To use Unpaint you will need to install at least one image generation model. You can install models from your hard drive, or HuggingFace. We have created a handy tutorial on how to convert existing models, you can find it here.

Building the project

Development environment

  • Install Visual Studio 2022
    • Select the following workloads:
      • Desktop development with C++
      • Universal Windows Platform development
      • Game development with C++
    • Also select these individual packages:
      • C++ (v143) Universal Windows Platform tools
  • Install git from here if you do not have it yet
  • I did the project with Windows 11, earlier Windows versions might or might not work

Once you have Visual Studio, for the first build follow these steps:

  • Clone the repo with git clone https://github.com/axodox/unpaint
  • Open Unpaint.sln with Visual Studio
    • If Visual Studio prompts you that you still need to install something, then follow its instructions
    • You could also get a prompt to enable Developer Mode in Windows, which you can do in the Settings app
  • Open the Config folder in the Unpaint project in Solution Explorer, and double click Package.appxmanifest
  • Go to the Packaging tab, and select Choose Certificate
  • Press Create and follow the wizard

After this, you can just start the project and have it be built and run. Enjoy :)

unpaint's People

Contributors

axodox avatar csbarnes avatar milkflavor avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

unpaint's Issues

Filter flagging landscapes as an explicit image.

This is a really cool program you've built, and I prefer it over easy diffusion or others. I also appreciate the built-in NSFW filter. Some things are better not seen. However, I had one of my prompts flagged as an explicit image.
Positive:
Jagged snow covered peaks rise from a snow covered forest, above a lake. A massive glacier descends down the mountain into the lake. realistic, crisp, Alps, Cascades
negative:
blurry, render, fuzzy

model: realistic vision onnx

Settings:
512X512
batching: 9
steps: 75
random seed: 1391526699
guidance strength: 7

Workaround for 'Error downloading Unpaint/Assets'

The Assets won't download, the error Git gives is

Downloading Unpaint/Assets/LockScreenLogo.scale-200.png (4.1 KB)
Error downloading object: Unpaint/Assets/LockScreenLogo.scale-200.png (7fd70ec): Smudge error: Error downloading Unpaint/Assets/LockScreenLogo.scale-200.png (7fd70ec6e4e2854c4e67c15b939b21a04c90e4a4712a20eb530866499792a9b8): batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.

Errors logged to 'C:\Cpp\unpaint\.git\lfs\logs\20230609T123116.492325.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: Unpaint/Assets/LockScreenLogo.scale-200.png: smudge filter lfs failed

They aren't in the source zips either, including the one with the latest release.

The solutionn I found for the time being was to grab the MSIX and extract the Assets from it with 7zip. Maybe this will be helpful to others.

Wrong check of onnx environment

!_onnxEnvironment expression should be evaluated before usage of _onnxEnvironment->DeviceId in the code void StableDiffusionModelExecutor::EnsureEnvironment(std::string_view modelId) { if (_modelId != modelId || _onnxEnvironment->DeviceId != int32_t(_unpaintOptions->AdapterIndex()) || !_onnxEnvironment)

[Bug] Glitchy window header, and UI "blinks"

On Windows 11, app header works glitchy. I'm using the Light mode, and Min/Max/Close buttons are not visible without mouse hovering:
image

Also, the whole app content (UI, image) randomly (not often) blinks.

error in convert model

when i follow the guide of Model-import, then run "python .\stable_diffusion.py --optimize --model_id runwayml/stable-diffusion-v1-5" in "olive-sd\olive\examples\directml\stable_diffusion" directory, report error as follow:

(olive-env) PS D:\E_2_D\temp\axodox\olive-sd\olive\examples\directml\stable_diffusion> python .\stable_diffusion.py --optimize --model_id runwayml/stable-diffusion-v1-5
C:\Users\synov\miniconda3\envs\olive-env\lib\site-packages\pydantic_internal_config.py:269: UserWarning: Valid config keys have changed in V2:

  • 'json_dumps' has been removed
  • 'json_loads' has been removed
    warnings.warn(message, UserWarning)
    C:\Users\synov\miniconda3\envs\olive-env\lib\site-packages\pydantic_internal_config.py:269: UserWarning: Valid config keys have changed in V2:
  • 'json_dumps' has been removed
  • 'json_loads' has been removed
    warnings.warn(message, UserWarning)
    C:\Users\synov\miniconda3\envs\olive-env\lib\site-packages\pydantic_internal_config.py:269: UserWarning: Valid config keys have changed in V2:
  • 'json_dumps' has been removed
  • 'json_loads' has been removed
    warnings.warn(message, UserWarning)
    Traceback (most recent call last):
    File ".\stable_diffusion.py", line 18, in
    from olive.workflows import run as olive_run
    File "C:\Users\synov\miniconda3\envs\olive-env\lib\site-packages\olive\workflows_init_.py", line 5, in
    from olive.workflows.run.run import run
    File "C:\Users\synov\miniconda3\envs\olive-env\lib\site-packages\olive\workflows\run\run.py", line 16, in
    from olive.passes import Pass
    File "C:\Users\synov\miniconda3\envs\olive-env\lib\site-packages\olive\passes_init_.py", line 5, in
    from olive.passes.olive_pass import FullPassConfig, Pass
    File "C:\Users\synov\miniconda3\envs\olive-env\lib\site-packages\olive\passes\olive_pass.py", line 13, in
    from olive.common.config_utils import ConfigBase, validate_config
    File "C:\Users\synov\miniconda3\envs\olive-env\lib\site-packages\olive\common\config_utils.py", line 118, in
    class ConfigListBase(ConfigBase):
    File "C:\Users\synov\miniconda3\envs\olive-env\lib\site-packages\pydantic_internal_model_construction.py", line 98, in new
    private_attributes = inspect_namespace(
    File "C:\Users\synov\miniconda3\envs\olive-env\lib\site-packages\pydantic_internal_model_construction.py", line 291, in inspect_namespace
    raise TypeError("To define root models, use pydantic.RootModel rather than a field called 'root'")
    TypeError: To define root models, use pydantic.RootModel rather than a field called 'root'
    (olive-env) PS D:\E_2_D\temp\axodox\olive-sd\olive\examples\directml\stable_diffusion>

Import onnx model

Hi, I have used optimum from huggingface to convert the model to onnx and this is impossible to open the model in unpaint because it doesn't contain safety_checker folder.

I have tried to change the code to don't use safety checker but I have one error "Saving initialized tensors (failed:open file model.onnx_data)"....
So, impossible to test your software actualy.
If you have a valid solution ?
Thanks.

[Bug] Nvidia device being ignored (but works forcibly)

I have an integrated AMD and discrete Nvidia graphics (laptop). By default, Unpaint uses AMD, and it don't "see" Nvidia at all.
However, I was able to force it in Windows Graphics settings, and after that, it worked.
P.S. It could be useful to show some system info in Settings section or something; like GPU detected, RAM/VRAM used, etc.

Should StableDiffusionTest.cpp generate an image?

It compiles fine, but does not generate anything (not sure if it is meant to).
I also wonder where I have to put the model files exactly for that (libfolder)...
What I would like to do: Run stable diffusion with Open Frameworks (maybe as an addon - there is already one for .onnx files).
For that I do not need Axodox::Graphics, because I would copy the pixels of the resulting tensor to an openGL texture...

[Feedback] Impressive AI UWP app

I would like to thank you for bringing such functionality to UWP application,
it works very well (Using it on my laptop), at least I don't have to run local server to use it.

I do have a little advice if you don't mind regarding the target users:
Microsoft already made big damage to UWP using unnecessary restrictions and wrong environment concept along strict Store policy.
the UWP apps mostly now used in XBOX,
that's why I'm not expecting to see much contributors (I hope I'm wrong about that),
UWP's first impression wasn't very good for developers so they still not very convinced to engage with it,
Windows users still prefer Win32 apps style over UWP because of the limitation,
unless they're looking for simplicity which you already achieved it in your app,
don't depend on UWP security market, users will trade security for features,
so I do suggest to checkout this discord server, and see if there is opportunity to post it in the announcement so it can get more attention from the right community, also users in this server already has experience to deal with dev mode.

I hope to see this app more popular and has more support.

Most things I liked:

  • Results pretty good
  • Safety check is really good
  • Using Modify (option) instead of img2img as text was perfect (specifically for basic users)
  • Mask tool seems easy to use unlike other apps.
  • Super easy to use and modify generated images

What I hope to see:

  • Inverted mask, so I can choose to keep one object and remove every thing else.
  • Choose external location for output images (this maybe for XBOX only, desktop is fine)

btw I cannot use any higher resolution other than 512x512 with my NVIDIA RTX-2070 , Windows 10
I thought it's model issue but for some reason I can generate higher resolutions with other python apps.
If this is limited so I prefer to make 512 default on PC too, so it doesn't fail from the first time.
user will have to switch between models to generate again, otherwise it will stuck at failed state (happens with 768x768).

Sorry if that too long.

Many thanks, I really enjoyed using this app.

Generation failed on Xbox Series S

I installed
Unpaint_1.0.69.0_x64.msix
With dependency
Microsoft.UI.Xaml.2.8_8.230

Using the model
axodoxian/stable_diffusion_onnx

On my Xbox series S and I'm getting "Generation failed"

I tried the default (SraKmd_arden) and the Microsoft basic render driver.

I'm not sure what I'm doing wrong or how to troubleshoot the logs. It would be nice to include a more descriptive error message. Any guidance?

[Feature request] linux support request

Please consider adding support for Linux, as it can be beneficial, especially for deployment on cloud platforms.
Are there any limitations that prevent the code from running on Linux, or is it primarily related to the building process?
It appears that the code utilizes ONNX, which is supported on Linux.

[Question] Safety

any ways to disable the safety check? cant seem to find how (yes im shameless)

[Bug] Windows.UI.Xaml.dll error in the latest version

Hi, I updated the app to the latest version (43), and it stopped to work after updating. After full reinstalling, it worked untill I hit the Install additional tools in Settings.
Getting this error in logs:

AppName Unpaint.exe 
  AppVersion 0.0.0.0 
  AppTimeStamp 64aafa1f 
  ModuleName Windows.UI.Xaml.dll 
  ModuleVersion 10.0.22621.1928 
  ModuleTimeStamp e91f7675 
  ExceptionCode c000027b 
  FaultingOffset 0000000000872360 
  ProcessId 0x2228 
  ProcessCreationTime 0x1d9b3662506736d 
  AppPath C:\Program Files\WindowsApps\Unpaint_1.0.43.0_x64__asy5p9gk2jzay\Unpaint.exe 
  ModulePath C:\Windows\System32\Windows.UI.Xaml.dll 
  IntegratorReportId 4b40cd91-9ed6-444d-8435-5cce5c820e92 
  PackageFullName Unpaint_1.0.43.0_x64__asy5p9gk2jzay 
  PackageRelativeAppId App

Impossible to add stable, diffusion, ONNX schemes on Xbox series S

Tried every model under the sun that was stable diffusion ONNX format but kept saying "The model does not match the Stable Diffusion ONNX schema." Tried to import locally said the same thing seems like a possible platform issue then a software issue because of the difference between windows and Xbox windows. But I'm not a expert on that since MS isn't really opened on that but hopefully this is a simple fix either on my part or yours. Or just might be my mistake.

[Question] A bunch of questions

  1. How to disable NSFW checker and Safe Mode? There are no switchers in UI and Settings:
    image
  2. Got an error a couple of times, but have no idea which one and how to "debug". How to check error log and generating stats (speed, etc.)?
  3. I see some hotkeys are working here (like Del to delete the image). How to see all of them? It would be nice to have some listing visible.

Thank you!

Generation failed on my Lenovo T14S device(Windows 11)

Because it always trigger the exception below, for debugging the issue, I removed the code below:

    catch (...)
    {
      async.update_state(1.f, "Generation failed.");
      return {};
    }

Then I got the below exception call stack. Any idea on how to debug this issue? Thanks in advance!
I downloaded the models from 'https://huggingface.co/axodoxian/stable_diffusion_onnx':

>	Unpaint.exe!winrt::terminate() Line 5206	C++
 	Unpaint.exe!std::coroutine_traits<winrt::fire_and_forget,winrt::Unpaint::implementation::InferenceViewModel &>::promise_type::unhandled_exception() Line 9623	C++
 	Unpaint.exe!`winrt::Unpaint::implementation::InferenceViewModel::GenerateImage$_ResumeCoro$1'::`1'::catch$46() Line 386	C++
 	[External Code]	
 	Unpaint.exe!winrt::Unpaint::implementation::InferenceViewModel::GenerateImage() Line 332	C++
 	[External Code]	
 	Unpaint.exe!winrt::impl::resume_background_callback(void * __formal, void * context) Line 8915	C++
 	[External Code]	

'make_guid': identifier not found

Firstly, congratulations on developing such an impressive project!

I am currently attempting to build this project using Visual Studio 2022, but I am encountering a specific error. Could you possibly provide some insight or guidance on why this might be happening? Any assistance would be greatly appreciated."

image

Failed to build the project

Follow the steps and installed all required package.
Updated all nuget packages to latest.
Windows 11
Visual Studio 2022 17.6.4

errors:

1>pch.cpp
1>C:\workspace\github\unpaint\packages\Axodox.Common.1.0.23\include\Infrastructure\Logger.h(27,48): error C2039: 'format_string': is not a member of 'std'
1>C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\include\numeric(23): message : see declaration of 'std'

App crash on browsing models

Hi, thank you for the great project.
image
After I hit Browse, the app suddenly closes. Is there any dump file I can send you or any static path I can put my models in?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.