resemble-ai / resemble-enhance Goto Github PK

View Code? Open in Web Editor NEW

929.0 14.0 83.0 36 KB

AI powered speech denoising and enhancement

Home Page: https://huggingface.co/spaces/ResembleAI/resemble-enhance

License: MIT License

Python 100.00%

denoise speech-denoising speech-enhancement speech-processing

resemble-enhance's Introduction

Resemble Enhance

EnhanceVideo.mp4

Resemble Enhance is an AI-powered tool that aims to improve the overall quality of speech by performing denoising and enhancement. It consists of two modules: a denoiser, which separates speech from a noisy audio, and an enhancer, which further boosts the perceptual audio quality by restoring audio distortions and extending the audio bandwidth. The two models are trained on high-quality 44.1kHz speech data that guarantees the enhancement of your speech with high quality.

Usage

Installation

Install the stable version:

pip install resemble-enhance --upgrade

Or try the latest pre-release version:

pip install resemble-enhance --upgrade --pre

Enhance

resemble_enhance in_dir out_dir

Denoise only

resemble_enhance in_dir out_dir --denoise_only

Web Demo

We provide a web demo built with Gradio, you can try it out here, or also run it locally:

python app.py

Train your own model

Data Preparation

You need to prepare a foreground speech dataset and a background non-speech dataset. In addition, you need to prepare a RIR dataset (examples).

data
├── fg
│   ├── 00001.wav
│   └── ...
├── bg
│   ├── 00001.wav
│   └── ...
└── rir
    ├── 00001.npy
    └── ...

Training

Denoiser Warmup

Though the denoiser is trained jointly with the enhancer, it is recommended for a warmup training first.

python -m resemble_enhance.denoiser.train --yaml config/denoiser.yaml runs/denoiser

Enhancer

Then, you can train the enhancer in two stages. The first stage is to train the autoencoder and vocoder. And the second stage is to train the latent conditional flow matching (CFM) model.

Stage 1

python -m resemble_enhance.enhancer.train --yaml config/enhancer_stage1.yaml runs/enhancer_stage1

Stage 2

python -m resemble_enhance.enhancer.train --yaml config/enhancer_stage2.yaml runs/enhancer_stage2

Blog

Learn more on our website!

resemble-enhance's People

Contributors

Stargazers

Watchers

Forkers

gevmin94 render-ai sdbds timdesrochers touristshaun kustomzone eos21 liuguoyou daswer123 ishine shaun95 amanaryan007 hopperrr jurisgpt mechanicss videofeedback newoneincntk alexwd26 arthurdamasio st3alth jeffery-work aufr33 bigfly-nb wangqun010101 soi-20 yashlanjewar20 azraelkuan techthiyanes pete1313 ww516617119 andupotorac hussain1406 camenduru lamquangtuong robinysh m3m012y jeffara archkik jags111 jmaigc chuwoo cellinlab ydf feiwei9696 t-bagwell tanyatang suryatmodulus hotliu grahlnn iseeyo f901107 b08240 keyman9848 wyn314 jimjonesbabyfreshout litagin02 navezjt guhfgf alexdemartos princetrunks davidmartinrius timn2008 w4l6 lanlve thelustriva theonejohann haoheliu aixingxy dalilll ganjunhong runngezhang-jx warrensbox user01 zhongshijun kakiac threadabort cvcuiwei dwash96 dmvevents oza75 tsavpyn

resemble-enhance's Issues

在推理完成后发现几乎没有变化，和hugging face的结果差距很大

感谢作者开源了这么伟大的作品！
请问作者是有在web demo上增加额外的操作吗？例如AGC之类的，还是我的参数没有设置对，我都是用的默认的参数；

PosixPath error when running on MacOS

Trying to run on MacOS, but to no avail.
I've placed a "1.wav" file in the current folder and start resemble-enhance like this:
resemble-enhance ./ ./
The same output is given when using an absolute path.

Output:

[2024-01-03 20:17:18,615] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to mps (auto detect)
[2024-01-03 20:17:18,938] torch.distributed.elastic.multiprocessing.redirects: [WARNING] NOTE: Redirects are currently not supported in Windows or MacOs.
CUDA is not available but --device is set to cuda, using CPU instead
Processing 1.wav:   0%|                                   | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/usr/local/bin/resemble-enhance", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/resemble_enhance/enhancer/__main__.py", line 100, in main
    dwav, sr = torchaudio.load(path)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torchaudio/_backend/utils.py", line 204, in load
    return backend.load(uri, frame_offset, num_frames, normalize, channels_first, format, buffer_size)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torchaudio/_backend/sox.py", line 42, in load
    ret = torch.ops.torchaudio.sox_io_load_audio_file(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/_ops.py", line 692, in __call__
    return self._op(*args, **kwargs or {})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: torchaudio::sox_io_load_audio_file() Expected a value of type 'str' for argument '_0' but instead found type 'PosixPath'.
Position: 0
Value: PosixPath('1.wav')
Declaration: torchaudio::sox_io_load_audio_file(str _0, int? _1, int? _2, bool? _3, bool? _4, str? _5) -> (Tensor _0, int _1)
Cast error details: Unable to cast Python instance of type <class 'pathlib.PosixPath'> to C++ type '?' (#define PYBIND11_DETAILED_ERROR_MESSAGES or compile in debug mode for details)

How can i convert the .pt file to core ml model

How can i convert the mp_rank_00_model_states.pt file to Apple core ml model file？Is there anyone successful ?

Pre-trained usage with CPU / without Nvidia GPU

Is there any chance of enabling the usage of the pre-trained model with a CPU / non-Nvidia GPU?
(-> RuntimeError: Found no NVIDIA driver on your system)

Or is there no hope due to cuda?

Thank you for the awesome work!

Question about Dataset

Hi, thanks for releasing this model! Which dataset did you use for background and RIR? Thanks!

Installation error

Hey,

System: MacOS Ventura 13.5.1
After using installation command as described in readme, I'm getting error:

ERROR: Could not find a version that satisfies the requirement resemble-enhance (from versions: none)
ERROR: No matching distribution found for resemble-enhance

I tried to install both versions (stable and pre-release), no luck.

Any ideas what could be wrong?

pre-trained model ?

Query Regarding the Impact of Varied Acoustic Environments on Model Performance

Dear Resemble Enhance Team,

I hope this message finds you well. I am reaching out to inquire about the robustness of the Resemble Enhance AI models, particularly in relation to their performance across diverse acoustic environments.

Having perused your documentation and successfully utilised your tool for speech enhancement and denoising, I've observed impressive results in standard settings. However, I am curious about the model's adaptability when confronted with audio data recorded in atypical acoustic spaces, which may not be well-represented in the training datasets.

Specifically, my questions are as follows:

How does the model cope with audio inputs recorded in highly reverberant spaces, or those with unique echo characteristics that might diverge significantly from the RIR datasets used during training?
Is there a recommended approach to fine-tuning the model on a custom dataset that includes such unique acoustic characteristics, to better tailor the enhancement capabilities to specific environments?
Could you provide insights into the model's limitations when dealing with extreme noise conditions or non-linear distortions that are not commonly found in everyday scenarios?

Understanding these aspects is crucial for my ongoing project, which involves processing archival audio recordings that exhibit a wide range of acoustic anomalies.

I appreciate the cutting-edge work your team has accomplished with Resemble Enhance and look forward to any guidance you can provide on the aforementioned queries.

Best regards,
yihong1120

Windows posixpath error

Setup a test environment (voc_resembleenhance) and started the UI. Weeb page opens. I select a noisy wav and it opens and plays OK. Then when I click Submit the UI shows Error in both output spots. I get this posixpath error on the command line. Linux specific? Any fix for Windows?

Traceback (most recent call last):
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\gradio\queueing.py", line 456, in call_prediction
    output = await route_utils.call_process_api(
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\gradio\blocks.py", line 1522, in process_api
    result = await self.call_function(
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\gradio\blocks.py", line 1144, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\anyio\to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\anyio\_backends\_asyncio.py", line 2134, in run_sync_in_worker_thread
    return await future
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\anyio\_backends\_asyncio.py", line 851, in run
    result = context.run(func, *args)
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\gradio\utils.py", line 674, in wrapper
    response = f(*args, **kwargs)
  File "D:\Tests\Resemble Enhance\app.py", line 24, in _fn
    wav1, new_sr = denoise(dwav, sr, device)
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "D:\Tests\Resemble Enhance\resemble_enhance\enhancer\inference.py", line 29, in denoise
    enhancer = load_enhancer(run_dir, device)
  File "D:\Tests\Resemble Enhance\resemble_enhance\enhancer\inference.py", line 17, in load_enhancer
    hp = HParams.load(run_dir)
  File "D:\Tests\Resemble Enhance\resemble_enhance\hparams.py", line 109, in load
    hps.append(cls.from_yaml(run_dir / "hparams.yaml"))
  File "D:\Tests\Resemble Enhance\resemble_enhance\hparams.py", line 94, in from_yaml
    return cls(**dict(OmegaConf.merge(cls(), OmegaConf.load(path))))
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\omegaconf\omegaconf.py", line 190, in load
    obj = yaml.load(f, Loader=get_yaml_loader())
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\yaml\__init__.py", line 81, in load
    return loader.get_single_data()
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\yaml\constructor.py", line 51, in get_single_data
    return self.construct_document(node)
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\yaml\constructor.py", line 60, in construct_document
    for dummy in generator:
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\yaml\constructor.py", line 413, in construct_yaml_map
    value = self.construct_mapping(node)
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\omegaconf\_utils.py", line 151, in construct_mapping
    return super().construct_mapping(node, deep=deep)
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\yaml\constructor.py", line 218, in construct_mapping
    return super().construct_mapping(node, deep=deep)
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\yaml\constructor.py", line 143, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\yaml\constructor.py", line 100, in construct_object
    data = constructor(self, node)
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\omegaconf\_utils.py", line 183, in <lambda>
    lambda loader, node: pathlib.PosixPath(*loader.construct_sequence(node)),
  File "D:\Python\lib\pathlib.py", line 962, in __new__
    raise NotImplementedError("cannot instantiate %r on your system"
NotImplementedError: cannot instantiate 'PosixPath' on your system
Traceback (most recent call last):
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\gradio\queueing.py", line 456, in call_prediction
    output = await route_utils.call_process_api(
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\gradio\blocks.py", line 1522, in process_api
    result = await self.call_function(
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\gradio\blocks.py", line 1144, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\anyio\to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\anyio\_backends\_asyncio.py", line 2134, in run_sync_in_worker_thread
    return await future
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\anyio\_backends\_asyncio.py", line 851, in run
    result = context.run(func, *args)
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\gradio\utils.py", line 674, in wrapper
    response = f(*args, **kwargs)
  File "D:\Tests\Resemble Enhance\app.py", line 24, in _fn
    wav1, new_sr = denoise(dwav, sr, device)
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "D:\Tests\Resemble Enhance\resemble_enhance\enhancer\inference.py", line 29, in denoise
    enhancer = load_enhancer(run_dir, device)
  File "D:\Tests\Resemble Enhance\resemble_enhance\enhancer\inference.py", line 17, in load_enhancer
    hp = HParams.load(run_dir)
  File "D:\Tests\Resemble Enhance\resemble_enhance\hparams.py", line 109, in load
    hps.append(cls.from_yaml(run_dir / "hparams.yaml"))
  File "D:\Tests\Resemble Enhance\resemble_enhance\hparams.py", line 94, in from_yaml
    return cls(**dict(OmegaConf.merge(cls(), OmegaConf.load(path))))
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\omegaconf\omegaconf.py", line 190, in load
    obj = yaml.load(f, Loader=get_yaml_loader())
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\yaml\__init__.py", line 81, in load
    return loader.get_single_data()
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\yaml\constructor.py", line 51, in get_single_data
    return self.construct_document(node)
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\yaml\constructor.py", line 60, in construct_document
    for dummy in generator:
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\yaml\constructor.py", line 413, in construct_yaml_map
    value = self.construct_mapping(node)
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\omegaconf\_utils.py", line 151, in construct_mapping
    return super().construct_mapping(node, deep=deep)
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\yaml\constructor.py", line 218, in construct_mapping
    return super().construct_mapping(node, deep=deep)
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\yaml\constructor.py", line 143, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\yaml\constructor.py", line 100, in construct_object
    data = constructor(self, node)
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\omegaconf\_utils.py", line 183, in <lambda>
    lambda loader, node: pathlib.PosixPath(*loader.construct_sequence(node)),
  File "D:\Python\lib\pathlib.py", line 962, in __new__
    raise NotImplementedError("cannot instantiate %r on your system"
NotImplementedError: cannot instantiate 'PosixPath' on your system

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\gradio\queueing.py", line 501, in process_events
    response = await self.call_prediction(awake_events, batch)
  File "D:\Tests\Resemble Enhance\voc_resembleenhance\lib\site-packages\gradio\queueing.py", line 465, in call_prediction
    raise Exception(str(error) if show_error else None) from error

reading hparams

I got nothing while reading params

run the project meets errors

I use two methods to try to run the project

I download the project latest source code,and I use pip download all of dependencies,but i meet the error.BTW,the environment is python3.9 and i use Conda virtual environment

(voice_enhance) root@jack-B450M-S2H:~/project/resemble-enhance# python3 app.py 
Traceback (most recent call last):
  File "/root/project/resemble-enhance/app.py", line 5, in <module>
    from resemble_enhance.enhancer.inference import denoise, enhance
  File "/root/project/resemble-enhance/resemble_enhance/enhancer/inference.py", line 7, in <module>
    from ..inference import inference
  File "/root/project/resemble-enhance/resemble_enhance/inference.py", line 11, in <module>
    from .hparams import HParams
  File "/root/project/resemble-enhance/resemble_enhance/hparams.py", line 36, in <module>
    class HParams:
  File "/root/project/resemble-enhance/resemble_enhance/hparams.py", line 105, in HParams
    def load(cls, run_dir, yaml: Path | None = None):
TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
(voice_enhance) root@jack-B450M-S2H:~/project/resemble-enhance# ls
LICENSE  README.md  app.py  config  packages.txt  pyproject.toml  requirements.txt  resemble_enhance  setup.py
(voice_enhance) root@jack-B450M-S2H:~/project/resemble-enhance# resemble-enhance 
Traceback (most recent call last):
  File "/root/anaconda3/envs/voice_enhance/bin/resemble-enhance", line 33, in <module>
    sys.exit(load_entry_point('resemble-enhance==0.0.1', 'console_scripts', 'resemble-enhance')())
  File "/root/anaconda3/envs/voice_enhance/bin/resemble-enhance", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/root/anaconda3/envs/voice_enhance/lib/python3.9/importlib/metadata.py", line 86, in load
    module = import_module(match.group('module'))
  File "/root/anaconda3/envs/voice_enhance/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/root/anaconda3/envs/voice_enhance/lib/python3.9/site-packages/resemble_enhance-0.0.1-py3.9.egg/resemble_enhance/enhancer/__main__.py", line 10, in <module>
    from .inference import denoise, enhance
  File "/root/anaconda3/envs/voice_enhance/lib/python3.9/site-packages/resemble_enhance-0.0.1-py3.9.egg/resemble_enhance/enhancer/inference.py", line 6, in <module>
    from ..inference import inference
  File "/root/anaconda3/envs/voice_enhance/lib/python3.9/site-packages/resemble_enhance-0.0.1-py3.9.egg/resemble_enhance/inference.py", line 11, in <module>
    from .hparams import HParams
  File "/root/anaconda3/envs/voice_enhance/lib/python3.9/site-packages/resemble_enhance-0.0.1-py3.9.egg/resemble_enhance/hparams.py", line 36, in <module>
    class HParams:
  File "/root/anaconda3/envs/voice_enhance/lib/python3.9/site-packages/resemble_enhance-0.0.1-py3.9.egg/resemble_enhance/hparams.py", line 105, in HParams
    def load(cls, run_dir, yaml: Path | None = None):
TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'

I use the pip install the package named resemble-enhance,and when I use it to enhance a wav file,it meets errors.

(base) root@jack-B450M-S2H:/mnt/disk1/test/audio# resemble-enhance input/ output/
[2024-02-08 08:40:35,596] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
CUDA is not available but --device is set to cuda, using CPU instead
Processing output/part2.wav:   0%|                                                                                                                          | 0/1 [04:29<?, ?it/s]
Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.11/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/root/anaconda3/lib/python3.11/http/client.py", line 1286, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/root/anaconda3/lib/python3.11/http/client.py", line 1332, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/root/anaconda3/lib/python3.11/http/client.py", line 1281, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/root/anaconda3/lib/python3.11/http/client.py", line 1041, in _send_output
    self.send(msg)
  File "/root/anaconda3/lib/python3.11/http/client.py", line 979, in send
    self.connect()
  File "/root/anaconda3/lib/python3.11/http/client.py", line 1451, in connect
    super().connect()
  File "/root/anaconda3/lib/python3.11/http/client.py", line 945, in connect
    self.sock = self._create_connection(
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.11/socket.py", line 851, in create_connection
    raise exceptions[0]
  File "/root/anaconda3/lib/python3.11/socket.py", line 836, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/bin/resemble-enhance", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/root/anaconda3/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.11/site-packages/resemble_enhance/enhancer/__main__.py", line 110, in main
    hwav, sr = enhance(
               ^^^^^^^^
  File "/root/anaconda3/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.11/site-packages/resemble_enhance/enhancer/inference.py", line 39, in enhance
    enhancer = load_enhancer(run_dir, device)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.11/site-packages/resemble_enhance/enhancer/inference.py", line 16, in load_enhancer
    run_dir = download()
              ^^^^^^^^^^
  File "/root/anaconda3/lib/python3.11/site-packages/resemble_enhance/enhancer/download.py", line 27, in download
    torch.hub.download_url_to_file(url, str(path))
  File "/root/anaconda3/lib/python3.11/site-packages/torch/hub.py", line 620, in download_url_to_file
    u = urlopen(req)
        ^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.11/urllib/request.py", line 519, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.11/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.11/urllib/request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.11/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 110] Connection timed out

btw, i download the resemble-enhance package in a virtual environment with python3.10 because the python3.9 will return other error

(voice_enhance) root@jack-B450M-S2H:~/project/resemble-enhance# resemble-enhance 
Traceback (most recent call last):
  File "/root/anaconda3/envs/voice_enhance/bin/resemble-enhance", line 33, in <module>
    sys.exit(load_entry_point('resemble-enhance==0.0.1', 'console_scripts', 'resemble-enhance')())
  File "/root/anaconda3/envs/voice_enhance/bin/resemble-enhance", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/root/anaconda3/envs/voice_enhance/lib/python3.9/importlib/metadata.py", line 86, in load
    module = import_module(match.group('module'))
  File "/root/anaconda3/envs/voice_enhance/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/root/anaconda3/envs/voice_enhance/lib/python3.9/site-packages/resemble_enhance-0.0.1-py3.9.egg/resemble_enhance/enhancer/__main__.py", line 10, in <module>
    from .inference import denoise, enhance
  File "/root/anaconda3/envs/voice_enhance/lib/python3.9/site-packages/resemble_enhance-0.0.1-py3.9.egg/resemble_enhance/enhancer/inference.py", line 6, in <module>
    from ..inference import inference
  File "/root/anaconda3/envs/voice_enhance/lib/python3.9/site-packages/resemble_enhance-0.0.1-py3.9.egg/resemble_enhance/inference.py", line 11, in <module>
    from .hparams import HParams
  File "/root/anaconda3/envs/voice_enhance/lib/python3.9/site-packages/resemble_enhance-0.0.1-py3.9.egg/resemble_enhance/hparams.py", line 36, in <module>
    class HParams:
  File "/root/anaconda3/envs/voice_enhance/lib/python3.9/site-packages/resemble_enhance-0.0.1-py3.9.egg/resemble_enhance/hparams.py", line 105, in HParams
    def load(cls, run_dir, yaml: Path | None = None):
TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'

Web Application throws Errors on Submit

Hi,

I am running the Web App (app.yml) and whenever I´ve uploaded a file and click on submit I can see the following output at the webserver:

Traceback (most recent call last):
  File "/opt/resemble-enhance/.venv/lib/python3.11/site-packages/gradio/queueing.py", line 456, in call_prediction
    output = await route_utils.call_process_api(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/resemble-enhance/.venv/lib/python3.11/site-packages/gradio/route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/resemble-enhance/.venv/lib/python3.11/site-packages/gradio/blocks.py", line 1522, in process_api
    result = await self.call_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/resemble-enhance/.venv/lib/python3.11/site-packages/gradio/blocks.py", line 1144, in call_function
    prediction = await anyio.to_thread.run_sync(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/resemble-enhance/.venv/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/resemble-enhance/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2134, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/opt/resemble-enhance/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 851, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/resemble-enhance/.venv/lib/python3.11/site-packages/gradio/utils.py", line 674, in wrapper
    response = f(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^
  File "/opt/resemble-enhance/app.py", line 24, in _fn
    wav1, new_sr = denoise(dwav, sr, device)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/resemble-enhance/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/resemble-enhance/resemble_enhance/enhancer/inference.py", line 29, in denoise
    enhancer = load_enhancer(run_dir, device)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/resemble-enhance/resemble_enhance/enhancer/inference.py", line 16, in load_enhancer
    run_dir = download()
              ^^^^^^^^^^
  File "/opt/resemble-enhance/resemble_enhance/enhancer/download.py", line 40, in download
    run_command(["git", "-C", str(REPO_DIR), "lfs", "pull"], "Failed to pull latest changes, please try again.")
  File "/opt/resemble-enhance/resemble_enhance/enhancer/download.py", line 17, in run_command
    raise RuntimeError(msg) from e
RuntimeError: Failed to pull latest changes, please try again.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/resemble-enhance/.venv/lib/python3.11/site-packages/gradio/queueing.py", line 501, in process_events
    response = await self.call_prediction(awake_events, batch)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/resemble-enhance/.venv/lib/python3.11/site-packages/gradio/queueing.py", line 465, in call_prediction
    raise Exception(str(error) if show_error else None) from error
Exception: None

How can I solve this?
Thanks in advance.

Command line error

I'm using this great app on an Apple M1.

I used a virtual environment to run it in Python 3.10. After it has been installed the Web App works great. But when I try to run the command line version I get a cascade of errors.

This occurs on the dev and main branch, in Python 3.10 and 3.11

(my_env) degner@Davids-MacBook-Pro-2 Desktop % resemble-enhance /Users/degner/Desktop/To\ Clean /Users/degner/Desktop/Cleaned
[2024-03-02 00:32:41,743] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to mps (auto detect)
[2024-03-02 00:32:41,850] torch.distributed.elastic.multiprocessing.redirects: [WARNING] NOTE: Redirects are currently not supported in Windows or MacOs.
CUDA is not available but --device is set to cuda, using CPU instead
Processing /Users/degner/Desktop/Cleaned/Main - synced 2 - Cut down 2 copy.wav:   0%|                                                                                                                                                                                         | 0/2 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "/Users/degner/Desktop/my_env/bin/resemble-enhance", line 8, in <module>
    sys.exit(main())
  File "/Users/degner/Desktop/my_env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/Users/degner/Desktop/my_env/lib/python3.10/site-packages/resemble_enhance/enhancer/__main__.py", line 110, in main
    hwav, sr = enhance(
  File "/Users/degner/Desktop/my_env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/Users/degner/Desktop/my_env/lib/python3.10/site-packages/resemble_enhance/enhancer/inference.py", line 39, in enhance
    enhancer = load_enhancer(run_dir, device)
  File "/Users/degner/Desktop/my_env/lib/python3.10/site-packages/resemble_enhance/enhancer/inference.py", line 20, in load_enhancer
    state_dict = torch.load(path, map_location="cpu")["module"]
  File "/Users/degner/Desktop/my_env/lib/python3.10/site-packages/torch/serialization.py", line 1028, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/Users/degner/Desktop/my_env/lib/python3.10/site-packages/torch/serialization.py", line 1246, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.
(my_env) degner@Davids-MacBook-Pro-2 Desktop %

colab

Thank you for this great tool. I made a colab version here: https://github.com/hopperrr/resemble-enhance

resemble_enhance/enhancer /main.py fixes

Errors running enhancer from the command line. Fixes:

94 dwav, sr = torchaudio.load(path) >> dwav, sr = torchaudio.load(str(path))

115 torchaudio.save(out_path, hwav[None], sr) >> torchaudio.save(str(out_path), hwav[None], sr)

Training data / time

Few questions:

How much data was the demo model trained on?
How many epochs?
How long did training take and what system?

In Ubuntu 22.04, The application stops with an error telling nothing except that cloning the repo has failed

This occurs because you don't have git-lfs package installed on your OS. Install git-lfs on your Distro. For Ubuntu:

sudo apt install git-lfs

This is also the fix to the gradio app telling that an Error Occurred in 2-3 seconds.

Command line instructions

On the github instructions it uses "resemble_enhance" as the command instead of "resemble-enhance"

Great app, Thanks!

Windows: Errors while installing

I tried using the enhancer on a bark audio file on hugging face, and it works wonders!

Trying to pip install it, I get this error:

>>> ERROR
1
  error: subprocess-exited-with-error

  python setup.py egg_info did not run successfully.
  exit code: 1

  [15 lines of output]
  test.c
  LINK : fatal error LNK1181: cannot open input file 'aio.lib'
  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "C:\Users\45239\AppData\Local\Temp\pip-install-phvt4rr_\deepspeed_96322bcee24e46919b71c01d496a21e7\setup.py", line 182, in <module>
      abort(f"Unable to pre-compile {op_name}")
    File "C:\Users\45239\AppData\Local\Temp\pip-install-phvt4rr_\deepspeed_96322bcee24e46919b71c01d496a21e7\setup.py", line 52, in abort
      assert False, msg
  AssertionError: Unable to pre-compile async_io
  DS_BUILD_OPS=1
   [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
   [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
   [WARNING]  One can disable async_io with DS_BUILD_AIO=0
   [ERROR]  Unable to pre-compile async_io
  [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

Encountered error while generating package metadata.

See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

non english speech transformed to weird language

Peace, the non english speech transformed to weird language, i think it only work with english speech right now.

error: subprocess-exited-with-error

D:\AI\resemble-enhance>pip install resemble-enhance --upgrade
Collecting resemble-enhance
Downloading resemble_enhance-0.0.1-py3-none-any.whl.metadata (3.4 kB)
Collecting celluloid==0.2.0 (from resemble-enhance)
Downloading celluloid-0.2.0-py3-none-any.whl (5.4 kB)
Collecting deepspeed==0.12.4 (from resemble-enhance)
Downloading deepspeed-0.12.4.tar.gz (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 2.3 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [20 lines of output]
[WARNING] Unable to import torch, pre-compiling ops will be disabled. Please visit https://pytorch.org/ to see how to properly install torch on your system.
[WARNING] unable to import torch, please install it if you want to pre-compile any deepspeed ops.
DS_BUILD_OPS=1
Traceback (most recent call last):
File "C:\Users\as-aj\AppData\Local\Programs\Python\Python310\lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 353, in
main()
File "C:\Users\as-aj\AppData\Local\Programs\Python\Python310\lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "C:\Users\as-aj\AppData\Local\Programs\Python\Python310\lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
File "C:\Users\as-aj\AppData\Local\Temp\pip-build-env-dqrtwetr\overlay\Lib\site-packages\setuptools\build_meta.py", line 325, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "C:\Users\as-aj\AppData\Local\Temp\pip-build-env-dqrtwetr\overlay\Lib\site-packages\setuptools\build_meta.py", line 295, in _get_build_requires
self.run_setup()
File "C:\Users\as-aj\AppData\Local\Temp\pip-build-env-dqrtwetr\overlay\Lib\site-packages\setuptools\build_meta.py", line 480, in run_setup
super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
File "C:\Users\as-aj\AppData\Local\Temp\pip-build-env-dqrtwetr\overlay\Lib\site-packages\setuptools\build_meta.py", line 311, in run_setup
exec(code, locals())
File "", line 147, in
AssertionError: Unable to pre-compile ops without torch installed. Please install torch before attempting to pre-compile ops.
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Regards,