如何训练自己的声音模型？语音库中可选项太少了

Hi everyone!

My name is David Martin Rius and I have just published this project on GitHub: https://github.com/davidmartinrius/speech-dataset-generator/

Now you can create datasets automatically with any audio or lists of audios.

I hope you find it useful.

Here are the key functionalities of the project:

1. Dataset Generation: Creation of multilingual datasets with Mean Opinion Score (MOS).

2. Silence Removal: It includes a feature to remove silences from audio files, enhancing the overall quality.

3. Sound Quality Improvement: It improves the quality of the audio when needed.

4. Audio Segmentation: It can segment audio files within specified second ranges.

5. Transcription: The project transcribes the segmented audio, providing a textual representation.

6. Gender Identification: It identifies the gender of each speaker in the audio.

7. Pyannote Embeddings: Utilizes pyannote embeddings for speaker detection across multiple audio files.

8. Automatic Speaker Naming: Automatically assigns names to speakers detected in multiple audios.

9. Multiple Speaker Detection: Capable of detecting multiple speakers within each audio file.

10. Store speaker embeddings: The speakers are detected and stored in a Chroma database, so you do not need to assign a speaker name.

11. Syllabic and words-per-minute metrics

Feel free to explore the project at https://github.com/davidmartinrius/speech-dataset-generator

David Martin Rius

~/MeloTTS$ python3 -m unidic download

download url: https://cotonoha-dic.s3-ap-northeast-1.amazonaws.com/unidic-3.1.0.zip
Dictionary version: 3.1.0+2021-08-31
Downloading UniDic v3.1.0+2021-08-31...
unidic-3.1.0.zip: 0.00B [00:00, ?B/s]
Traceback (most recent call last):
File "/usr/lib/python3.10/urllib/request.py", line 1348, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/usr/lib/python3.10/http/client.py", line 1283, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/lib/python3.10/http/client.py", line 1455, in connect
self.sock = self._context.wrap_socket(self.sock,
File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.10/ssl.py", line 1100, in _create
self.do_handshake()
File "/usr/lib/python3.10/ssl.py", line 1371, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

Hi, this is a great project, the production speed is very fast and the quality is high. I want to package it as a websocket connection and need to use streaming audio data. Will this be supported in a future version?

Why are unidic/mecab required if not using Japanese TTS?

1. why need so many epochs for the training, 10000?
2. I only training with 13 sentence and the training cost almost 7 hours, and now it wont stop soon.
3. the log indicate the training from 9500 to 10000 repeatly, why?
   Hope get your answers, Thanks.

Hi,Thank you for your open source project, but the pre-training model downloaded by default does not seem to support Chinese, and the trained model cannot produce Chinese voice.

Is there any voice conversion using ?

RUN apt-get update && apt-get install -y
6

build-essential libsndfile1 \

7

&& rm -rf /var/lib/apt/lists/*

8

9

RUN pip install -e .
10

RUN python -m unidic download
Downloading UniDic v3.1.0+2021-08-31...
download url: https://cotonoha-dic.s3-ap-northeast-1.amazonaws.com/unidic-3.1.0.zip
Dictionary version: 3.1.0+2021-08-31
unidic-3.1.0.zip: 0.00B [00:00, ?B/s]unidic-3.1.0.zip: 0.00B [00:00, ?B/s]
Traceback (most recent call last):
File "/usr/local/lib/python3.9/urllib/request.py", line 1346, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/usr/local/lib/python3.9/http/client.py", line 1285, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.9/http/client.py", line 1331, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.9/http/client.py", line 1280, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.9/http/client.py", line 1040, in _send_output
self.send(msg)
File "/usr/local/lib/python3.9/http/client.py", line 980, in send
self.connect()
File "/usr/local/lib/python3.9/http/client.py", line 1454, in connect
self.sock = self._context.wrap_socket(self.sock,
File "/usr/local/lib/python3.9/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/usr/local/lib/python3.9/ssl.py", line 1040, in _create
self.do_handshake()
File "/usr/local/lib/python3.9/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.9/site-packages/unidic/main.py", line 17, in
plac.call(commands[command], sys.argv[1:])
File "/usr/local/lib/python3.9/site-packages/plac_core.py", line 436, in call
cmd, result = parser.consume(arglist)
File "/usr/local/lib/python3.9/site-packages/plac_core.py", line 287, in consume
return cmd, self.func(*(args + varargs + extraopts), **kwargs)
File "/usr/local/lib/python3.9/site-packages/unidic/download.py", line 104, in download_version
download_and_clean(dictinfo['version'], dictinfo['url'])
File "/usr/local/lib/python3.9/site-packages/unidic/download.py", line 62, in download_and_clean
download_progress(url, fname)
File "/usr/local/lib/python3.9/site-packages/unidic/download.py", line 38, in download_progress
urlretrieve(url, filename=fname, reporthook=t.update_to, data=None)
File "/usr/local/lib/python3.9/urllib/request.py", line 239, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "/usr/local/lib/python3.9/urllib/request.py", line 214, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.9/urllib/request.py", line 517, in open
response = self._open(req, data)
File "/usr/local/lib/python3.9/urllib/request.py", line 534, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/usr/local/lib/python3.9/urllib/request.py", line 494, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.9/urllib/request.py", line 1389, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File "/usr/local/lib/python3.9/urllib/request.py", line 1349, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error EOF occurred in violation of protocol (_ssl.c:1129)>

11

RUN python melo/init_downloads.py
12

13

CMD ["python", "./melo/app.py", "--host", "0.0.0.0", "--port", "8888"]

wsl2:unbuntu 22.04
how can i correct this errors? thank you.

在docker部署后，不能通过python调用，如何进行http接口调用呢？

I tried to read italian with spanish model and it works almost fine. Is possible to support also italian?

Hi, thanks for your great work!

I would like to clarify if https://github.com/myshell-ai/MeloTTS/blob/main/docs/training.md supports training for a new language.

Plus, do you have any tips when training on a new language from your experience with this project?

Thanks again!

Maybe caused by hardcode release cuda cache only?

new address is 'tohoku-nlp/bert-base-japanese-v3'；need to change

Hi!
Could you share the loss images during training to get an idea of how they should look like?
I'm trying to train a new single speaker model but my model cant articulate words at early stages (epoch 500) eventhough the attention matrix looks diagonal.
I attach the config file in case it might help.
Thanks!!

config.json

File ~/work/python/tts/MeloTTS/melo/api.py:81, in TTS.tts_to_file(self, text, speaker_id, output_path, sdp_ratio, noise_scale, noise_scale_w, speed)
79 t = re.sub(r'([a-z])([A-Z])', r'\1 \2', t)
80 device = self.device
---> 81 bert, ja_bert, phones, tones, lang_ids = utils.get_text_for_tts_infer(t, language, self.hps, device, self.symbol_to_id)
82 with torch.no_grad():
83 x_tst = phones.to(device).unsqueeze(0)

File ~/work/python/tts/MeloTTS/melo/utils.py:38, in get_text_for_tts_infer(text, language_str, hps, device, symbol_to_id)
36 ja_bert = torch.zeros(768, len(phone))
37 else:
---> 38 bert = get_bert(norm_text, word2ph, language_str, device)
39 del word2ph
40 assert bert.shape[-1] == len(phone), phone

File ~/work/python/tts/MeloTTS/melo/text/init.py:34, in get_bert(norm_text, word2ph, language, device)
30 from .korean import get_bert_feature as kr_bert
32 lang_bert_func_map = {"ZH": zh_bert, "EN": en_bert, "JP": jp_bert, 'ZH_MIX_EN': zh_mix_en_bert,
33 'FR': fr_bert, 'SP': sp_bert, 'ES': sp_bert, "KR": kr_bert}
---> 34 bert = lang_bert_func_map[language](norm_text, word2ph, device)
35 return bert

File ~/work/python/tts/MeloTTS/melo/text/chinese_mix.py:199, in get_bert_feature(text, word2ph, device)
197 def get_bert_feature(text, word2ph, device):
198 from . import chinese_bert
--> 199 return chinese_bert.get_bert_feature(text, word2ph, model_id='bert-base-multilingual-uncased', device=device)

File ~/work/python/tts/MeloTTS/melo/text/chinese_bert.py:35, in get_bert_feature(text, word2ph, device, model_id)
33 for i in inputs:
34 inputs[i] = inputs[i].to(device)
---> 35 res = model(**inputs, output_hidden_states=True)
36 res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
37 # import pdb; pdb.set_trace()
38 # assert len(word2ph) == len(text) + 2

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
1190 # If we don't have any hooks, we want to skip the rest of the logic in
1191 # this function, and just call forward.
1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1193 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194 return forward_call(*input, **kwargs)
1195 # Do not call functions when jit is used
1196 full_backward_hooks, non_full_backward_hooks = [], []

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py:1358, in BertForMaskedLM.forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, labels, output_attentions, output_hidden_states, return_dict)
1349 r"""
1350 labels (torch.LongTensor of shape (batch_size, sequence_length), optional):
1351 Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., 1352 config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored (masked), the
1353 loss is only computed for the tokens with labels in [0, ..., config.vocab_size]
1354 """
1356 return_dict = return_dict if return_dict is not None else self.config.use_return_dict
-> 1358 outputs = self.bert(
1359 input_ids,
1360 attention_mask=attention_mask,
1361 token_type_ids=token_type_ids,
1362 position_ids=position_ids,
1363 head_mask=head_mask,
1364 inputs_embeds=inputs_embeds,
1365 encoder_hidden_states=encoder_hidden_states,
1366 encoder_attention_mask=encoder_attention_mask,
1367 output_attentions=output_attentions,
1368 output_hidden_states=output_hidden_states,
1369 return_dict=return_dict,
1370 )
1372 sequence_output = outputs[0]
1373 prediction_scores = self.cls(sequence_output)

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
1190 # If we don't have any hooks, we want to skip the rest of the logic in
1191 # this function, and just call forward.
1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1193 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194 return forward_call(*input, **kwargs)
1195 # Do not call functions when jit is used
1196 full_backward_hooks, non_full_backward_hooks = [], []

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py:1013, in BertModel.forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
1006 # Prepare head mask if needed
1007 # 1.0 in head_mask indicate we keep the head
1008 # attention_probs has shape bsz x n_heads x N x N
1009 # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads]
1010 # and head_mask is converted to shape [num_hidden_layers x batch x num_heads x seq_length x seq_length]
1011 head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)
-> 1013 embedding_output = self.embeddings(
1014 input_ids=input_ids,
1015 position_ids=position_ids,
1016 token_type_ids=token_type_ids,
1017 inputs_embeds=inputs_embeds,
1018 past_key_values_length=past_key_values_length,
1019 )
1020 encoder_outputs = self.encoder(
1021 embedding_output,
1022 attention_mask=extended_attention_mask,
(...)
1030 return_dict=return_dict,
1031 )
1032 sequence_output = encoder_outputs[0]

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
1190 # If we don't have any hooks, we want to skip the rest of the logic in
1191 # this function, and just call forward.
1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1193 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194 return forward_call(*input, **kwargs)
1195 # Do not call functions when jit is used
1196 full_backward_hooks, non_full_backward_hooks = [], []

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py:230, in BertEmbeddings.forward(self, input_ids, token_type_ids, position_ids, inputs_embeds, past_key_values_length)
227 token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=self.position_ids.device)
229 if inputs_embeds is None:
--> 230 inputs_embeds = self.word_embeddings(input_ids)
231 token_type_embeddings = self.token_type_embeddings(token_type_ids)
233 embeddings = inputs_embeds + token_type_embeddings

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
1190 # If we don't have any hooks, we want to skip the rest of the logic in
1191 # this function, and just call forward.
1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1193 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194 return forward_call(*input, **kwargs)
1195 # Do not call functions when jit is used
1196 full_backward_hooks, non_full_backward_hooks = [], []

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py:160, in Embedding.forward(self, input)
159 def forward(self, input: Tensor) -> Tensor:
--> 160 return F.embedding(
161 input, self.weight, self.padding_idx, self.max_norm,
162 self.norm_type, self.scale_grad_by_freq, self.sparse)

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torch/nn/functional.py:2210, in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
2204 # Note [embedding_renorm set_grad_enabled]
2205 # XXX: equivalent to
2206 # with torch.no_grad():
2207 # torch.embedding_renorm_
2208 # remove once script supports set_grad_enabled
2209 no_grad_embedding_renorm(weight, input, max_norm, norm_type)
-> 2210 return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

RuntimeError: Placeholder storage has not been allocated on MPS device!

Hi,
This is already pip-packaged, but it is not published on PyPI. This means:

1. It cannot be specified as a requirement on other packages published on PyPI
2. It must be installed via Git

Do you think it might be possible to publish it on PyPI in the future? If so, I'd be happy to make a PR that will use GitHub Actions to automatically publish the package on new updates.

Thanks!

Hi,
Is long-form synthesis possible (perhaps similar to how StyleTTS 2 does it)?
Thanks!

docker build successfully. my macbookpro is M1 chip.
I run the command :
sudo docker run --gpus all -it -p 8888:8888 melotts
then get the error:

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].
ERRO[0000] error waiting for container:

Not supported ?

This is the best tts tool I've ever used, and it's the closest to a human voice. thank you !

However, the voice generated is currently female. How can I change it to a male voice?

Portuguese on roadmap?

目前的音色听起来不太对劲，沙沙的

中文生成的声音忽大忽小是什么原因，特别是长文本的时候

Any tips on how I could go about streaming audio from the docker container?

Explanation

When installing MeloTTS via the PyPI package receiving error indicating that it can't find the requirements.txt file

Environment

• Intel Mac Sonoma 14.0
• Python 3.9.13
• pip 24.0

Comand and Output

$ pip3.9 install melotts
Collecting melotts
  Using cached melotts-0.1.1.tar.gz (4.6 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [17 lines of output]
      Traceback (most recent call last):
        File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/private/var/folders/hc/npzs9snx1m91qqb69ztyym8c0000gq/T/pip-build-env-ismo191n/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/private/var/folders/hc/npzs9snx1m91qqb69ztyym8c0000gq/T/pip-build-env-ismo191n/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/hc/npzs9snx1m91qqb69ztyym8c0000gq/T/pip-build-env-ismo191n/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 487, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/private/var/folders/hc/npzs9snx1m91qqb69ztyym8c0000gq/T/pip-build-env-ismo191n/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 7, in <module>
      FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/hc/npzs9snx1m91qqb69ztyym8c0000gq/T/pip-install-8d_wg6c2/melotts_3fd10f3406e145fbbe4488f43feb9e02/requirements.txt'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Hey, this looks really cool! How is this different from OpenVoice? (And is this the permissively licensed successor?) thanks!

download url: https://cotonoha-dic.s3-ap-northeast-1.amazonaws.com/unidic-3.1.0.zip 国内无法访问，请教下如何解决？

RUN python3 -m unidic download:
2.445 Downloading UniDic v3.1.0+2021-08-31...
2.506 download url: https://cotonoha-dic.s3-ap-northeast-1.amazonaws.com/unidic-3.1.0.zip
2.506 Dictionary version: 3.1.0+2021-08-31
unidic-3.1.0.zip: 0.00B [00:00, ?B/s]
2.866 Traceback (most recent call last):
2.866 File "/usr/local/lib/python3.9/urllib/request.py", line 1346, in do_open
2.900 h.request(req.get_method(), req.selector, req.data, headers,
2.900 File "/usr/local/lib/python3.9/http/client.py", line 1285, in request
2.934 self._send_request(method, url, body, headers, encode_chunked)
2.934 File "/usr/local/lib/python3.9/http/client.py", line 1331, in _send_request
2.934 self.endheaders(body, encode_chunked=encode_chunked)
2.934 File "/usr/local/lib/python3.9/http/client.py", line 1280, in endheaders
2.934 self._send_output(message_body, encode_chunked=encode_chunked)
2.934 File "/usr/local/lib/python3.9/http/client.py", line 1040, in _send_output
2.935 self.send(msg)
2.935 File "/usr/local/lib/python3.9/http/client.py", line 980, in send
2.935 self.connect()
2.935 File "/usr/local/lib/python3.9/http/client.py", line 1454, in connect
2.936 self.sock = self._context.wrap_socket(self.sock,
2.936 File "/usr/local/lib/python3.9/ssl.py", line 500, in wrap_socket
2.936 return self.sslsocket_class._create(
2.936 File "/usr/local/lib/python3.9/ssl.py", line 1040, in _create
2.937 self.do_handshake()
2.937 File "/usr/local/lib/python3.9/ssl.py", line 1309, in do_handshake
2.937 self._sslobj.do_handshake()
2.937 ConnectionResetError: [Errno 104] Connection reset by peer

Hi,
Is voice cloning possible?
Thanks!

推理真的好快，求大佬推理代码，想要训练想要的声音

download_utils.py

torch<2.0 difficult to accept

Hello, based on the information given in training docs, metadata are in this address:
data/example/metadata.list
but no folder named data is available in this repository
and a quick question as well, is it possible to train with cpu and not a gpu?

=> ERROR [7/7] RUN python melo/init_downloads.py

56.33 Traceback (most recent call last):
56.33 File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
56.33 conn = connection.create_connection(
56.33 File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 72, in create_connection
56.33 for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
56.33 File "/usr/local/lib/python3.9/socket.py", line 954, in getaddrinfo
56.33 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
56.33 socket.gaierror: [Errno -3] Temporary failure in name resolution
56.33
56.33 During handling of the above exception, another exception occurred:
56.33
56.33 Traceback (most recent call last):
56.33 File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 715, in urlopen
56.33 httplib_response = self._make_request(
56.33 File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 404, in _make_request
56.33 self._validate_conn(conn)
56.33 File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1058, in _validate_conn
56.33 conn.connect()
56.33 File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 363, in connect
56.33 self.sock = conn = self._new_conn()
56.33 File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 186, in new_conn
56.33 raise NewConnectionError(
56.33 urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7feb1ac74100>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
56.33
56.33 During handling of the above exception, another exception occurred:
56.33
56.33 Traceback (most recent call last):
56.33 File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
56.33 resp = conn.urlopen(
56.33 File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 799, in urlopen
56.33 retries = retries.increment(
56.33 File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
56.33 raise MaxRetryError(pool, url, error or ResponseError(cause))
56.33 urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='cdn-lfs-us-1.huggingface.co', port=443): Max retries exceeded with url: /repos/8a/1e/8a1ef674c4775bc19ceb0e1b85c7136ea7f9b0157416ed9a7087ac84f4160675/acd278040eaf9536908e2b965273df5a731c44d8f0da66cc5fed7972772ed23c?response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27checkpoint.pth%3B+filename%3D%22checkpoint.pth%22%3B&Expires=1709629442&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwOTYyOTQ0Mn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zLzhhLzFlLzhhMWVmNjc0YzQ3NzViYzE5Y2ViMGUxYjg1YzcxMzZlYTdmOWIwMTU3NDE2ZWQ5YTcwODdhYzg0ZjQxNjA2NzUvYWNkMjc4MDQwZWFmOTUzNjkwOGUyYjk2NTI3M2RmNWE3MzFjNDRkOGYwZGE2NmNjNWZlZDc5NzI3NzJlZDIzYz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=eYWAAm3hnYBgV0BaOM7SO4frDGQwsuROYf46iWBwCoBEjZFDZxI1jHqT6y63pfmLWPcg3FmEObNLA6V7iVqTtMUDZj7eNJd-iGBaNW9SBWd6WJu6iCPvIgbz1IIyYKu8FdZGGLmuuvveUWVsH2IawLFYepUkCJeIlo5grAYWFZ94Nr9r6Sp8W-Ivf-AAmx00E1ErGlgLm2K5GmNrSH77DDVJy6k3o01~ysFkDIz2i0ZhsD41RU2Y4DCvbM8wulY9K5SXsyh6uQlCevV1o6YgG0WJwmhKFO7M7naL4UazmexzaTOL-jlakCxhsb-XWYuhy3t9bEYwjqFmJutsTw&Key-Pair-Id=KCD77M1F0VK2B (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7feb1ac74100>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
56.33
56.33 During handling of the above exception, another exception occurred:
56.33
56.33 Traceback (most recent call last):
56.33 File "/app/melo/init_downloads.py", line 8, in
56.33 'EN': TTS(language='EN', device=device),
56.33 File "/app/melo/api.py", line 57, in init
56.33 checkpoint_dict = load_or_download_model(language, device, use_hf=use_hf)
56.33 File "/app/melo/download_utils.py", line 51, in load_or_download_model
56.33 ckpt_path = hf_hub_download(repo_id=LANG_TO_HF_REPO_ID[language], filename="checkpoint.pth")
56.33 File "/usr/local/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
56.33 return fn(*args, **kwargs)
56.33 File "/usr/local/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1457, in hf_hub_download
56.33 http_get(
56.33 File "/usr/local/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 451, in http_get
56.33 r = _request_wrapper(
56.33 File "/usr/local/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 408, in request_wrapper
56.33 response = get_session().request(method=method, url=url, **params)
56.33 File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
56.33 resp = self.send(prep, **send_kwargs)
56.33 File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
56.33 r = adapter.send(request, **kwargs)
56.33 File "/usr/local/lib/python3.9/site-packages/huggingface_hub/utils/http.py", line 67, in send
56.33 return super().send(request, *args, *kwargs)
56.33 File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 519, in send
56.33 raise ConnectionError(e, request=request)
56.33 requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='cdn-lfs-us-1.huggingface.co', port=443): Max retries exceeded with url: /repos/8a/1e/8a1ef674c4775bc19ceb0e1b85c7136ea7f9b0157416ed9a7087ac84f4160675/acd278040eaf9536908e2b965273df5a731c44d8f0da66cc5fed7972772ed23c?response-content-disposition=attachment%3B+filename%3DUTF-8%27%27checkpoint.pth%3B+filename%3D%22checkpoint.pth%22%3B&Expires=1709629442&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwOTYyOTQ0Mn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zLzhhLzFlLzhhMWVmNjc0YzQ3NzViYzE5Y2ViMGUxYjg1YzcxMzZlYTdmOWIwMTU3NDE2ZWQ5YTcwODdhYzg0ZjQxNjA2NzUvYWNkMjc4MDQwZWFmOTUzNjkwOGUyYjk2NTI3M2RmNWE3MzFjNDRkOGYwZGE2NmNjNWZlZDc5NzI3NzJlZDIzYz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=eYWAAm3hnYBgV0BaOM7SO4frDGQwsuROYf46iWBwCoBEjZFDZxI1jHqT6y63pfmLWPcg3FmEObNLA6V7iVqTtMUDZj7eNJd-iGBaNW9SBWd6WJu6iCPvIgbz1IIyYKu8FdZGGLmuuvveUWVsH2IawLFYepUkCJeIlo5grAYWFZ94Nr9r6Sp8W-Ivf-AAmx00E1ErGlgLm2K5GmNrSH77DDVJy6k3o01~ysFkDIz2i0ZhsD41RU2Y4DCvbM8wulY9K5SXsyh6uQlCevV1o6YgG0WJwmhKFO7M7naL4UazmexzaTOL-jlakCxhsb-XWYuhy3t9bEYwjqFmJutsTw&Key-Pair-Id=KCD77M1F0VK2B (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7feb1ac74100>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))"), '(Request ID: ccc80f06-1a04-4eca-a0a4-98ec6e9cba50)')

Dockerfile:11

9 | RUN pip install -e .
10 | RUN python -m unidic download
11 | >>> RUN python melo/init_downloads.py
12 |
13 | CMD ["python", "./melo/app.py", "--host", "0.0.0.0", "--port", "8888"]

Hi,
This looks really cool! Do you think it's possible to mirror the models to the Hugging Face Hub for faster downloads?
Thanks!

Hi, will there be support for the Russian language, if so, can you tell me the approximate time frame?

I've been trying to quantize the Linear and Convolutional layers to try and speed up model inference but I am getting mixed results, this is what I am doing at the moment:

quantized_model = torch.quantization.quantize_dynamic(
    model,
    {torch.nn.Linear, torch.nn.Conv1d},  # Add other layers as needed
    dtype=torch.qint8,
    inplace=True
)

Does anyone have any advice?

I am very interested in this project, when will the scripts for training new models be open? I would like to know how to train a new model.

Do you guys plan to also release a german speaker model ? - would be really nice!

only girl voice available now
https://github.com/myshell-ai/MeloTTS/blob/main/docs/install.md

Running setup.py install for mecab-python3 ... error
error: subprocess-exited-with-error

× Running setup.py install for mecab-python3 did not run successfully.
│ exit code: 1
╰─> [15 lines of output]
/Users/root/miniconda3/envs/py310/lib/python3.10/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
warnings.warn(
WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
running install
/Users/root/miniconda3/envs/py310/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build/lib.macosx-11.1-arm64-cpython-310
creating build/lib.macosx-11.1-arm64-cpython-310/MeCab
copying src/MeCab/init.py -> build/lib.macosx-11.1-arm64-cpython-310/MeCab
copying src/MeCab/cli.py -> build/lib.macosx-11.1-arm64-cpython-310/MeCab
running build_ext
error: [Errno 2] No such file or directory: 'mecab-config'
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> mecab-python3

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Text split to sentences.
The field of text-to-speech has seen rapid development recently.

Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']

• This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  Traceback (most recent call last):
  File "/Users/user/Library/Python/3.9/lib/python/site-packages/gradio/queueing.py", line 495, in call_prediction
  output = await route_utils.call_process_api(
  File "/Users/user/Library/Python/3.9/lib/python/site-packages/gradio/route_utils.py", line 235, in call_process_api
  output = await app.get_blocks().process_api(
  File "/Users/user/Library/Python/3.9/lib/python/site-packages/gradio/blocks.py", line 1627, in process_api
  result = await self.call_function(
  File "/Users/user/Library/Python/3.9/lib/python/site-packages/gradio/blocks.py", line 1173, in call_function
  prediction = await anyio.to_thread.run_sync(
  File "/Users/user/Library/Python/3.9/lib/python/site-packages/anyio/to_thread.py", line 56, in run_sync
  return await get_async_backend().run_sync_in_worker_thread(
  File "/Users/user/Library/Python/3.9/lib/python/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
  return await future
  File "/Users/user/Library/Python/3.9/lib/python/site-packages/anyio/_backends/_asyncio.py", line 851, in run
  result = context.run(func, *args)
  File "/Users/user/Library/Python/3.9/lib/python/site-packages/gradio/utils.py", line 690, in wrapper
  response = f(*args, **kwargs)
  File "/Users/user/Downloads/MeloTTS-main/melo/app.py", line 35, in synthesize
  models[language].tts_to_file(text, models[language].hps.data.spk2id[speaker], bio, speed=speed, pbar=progress.tqdm, format='wav')
  File "/Users/user/Downloads/MeloTTS-main/melo/api.py", line 107, in tts_to_file
  audio = self.model.infer(
  File "/Users/user/Downloads/MeloTTS-main/melo/models.py", line 998, in infer
  x, m_p, logs_p, x_mask = self.enc_p(
  File "/Users/user/Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
  return forward_call(*input, **kwargs)
  File "/Users/user/Downloads/MeloTTS-main/melo/models.py", line 376, in forward
  x = self.encoder(x * x_mask, x_mask, g=g)
  File "/Users/user/Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
  return forward_call(*input, **kwargs)
  File "/Users/user/Downloads/MeloTTS-main/melo/attentions.py", line 107, in forward
  y = self.attn_layers[i](x, x, attn_mask)
  File "/Users/user/Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
  return forward_call(*input, **kwargs)
  File "/Users/user/Downloads/MeloTTS-main/melo/attentions.py", line 263, in forward
  x, self.attn = self.attention(q, k, v, mask=attn_mask)
  File "/Users/user/Downloads/MeloTTS-main/melo/attentions.py", line 280, in attention
  key_relative_embeddings = self._get_relative_embeddings(self.emb_rel_k, t_s)
  File "/Users/user/Downloads/MeloTTS-main/melo/attentions.py", line 344, in _get_relative_embeddings
  padded_relative_embeddings = F.pad(
  IndexError: Dimension out of range (expected to be in range of [-3, 2], but got 3)

installed list:
MyMacBookProM3Max MeloTTS-main % pip list
Package Version

aiofiles 23.2.1
altair 5.2.0
altgraph 0.17.2
annotated-types 0.6.0
anyascii 0.3.2
anyio 4.3.0
attrs 23.2.0
audioread 3.0.1
Babel 2.14.0
boto3 1.34.50
botocore 1.34.50
cached_path 1.6.0
cachetools 5.3.3
certifi 2024.2.2
cffi 1.16.0
charset-normalizer 3.3.2
click 8.1.7
cn2an 0.5.22
colorama 0.4.6
contourpy 1.2.0
cycler 0.12.1
dateparser 1.1.8
decorator 5.1.1
Deprecated 1.2.14
Distance 0.1.3
docopt 0.6.2
easyocr 1.7.0
eng-to-ipa 0.0.2
exceptiongroup 1.2.0
fastapi 0.110.0
ffmpy 0.3.2
filelock 3.13.1
filterpy 1.4.5
fonttools 4.49.0
fsspec 2024.2.0
fugashi 1.3.0
future 0.18.2
g2p-en 2.1.0
g2pkk 0.1.2
google-api-core 2.17.1
google-auth 2.28.1
google-cloud-core 2.4.1
google-cloud-storage 2.14.0
google-crc32c 1.5.0
google-resumable-media 2.7.0
googleapis-common-protos 1.62.0
gradio 4.19.2
gradio_client 0.10.1
gruut 2.2.3
gruut-ipa 0.13.0
gruut-lang-de 2.0.0
gruut-lang-en 2.0.0
gruut-lang-es 2.0.0
gruut-lang-fr 2.0.2
h11 0.14.0
httpcore 1.0.4
httpx 0.27.0
huggingface-hub 0.20.3
idna 3.6
imageio 2.34.0
importlib-resources 6.1.1
inflect 7.0.0
jaconv 0.3.4
jamo 0.4.1
jieba 0.42.1
Jinja2 3.1.3
jmespath 1.0.1
joblib 1.3.2
jsonlines 1.2.0
jsonschema 4.21.1
jsonschema-specifications 2023.12.1
kiwisolver 1.4.5
langid 1.1.6
lazy_loader 0.3
librosa 0.9.1
llvmlite 0.42.0
macholib 1.15.2
markdown-it-py 3.0.0
MarkupSafe 2.1.5
matplotlib 3.8.3
mdurl 0.1.2
mecab-python3 1.0.5
melo 0.1.1
mpmath 1.3.0
networkx 2.8.8
ninja 1.11.1.1
nltk 3.8.1
num2words 0.5.12
numba 0.59.0
numpy 1.24.3
opencv-python 4.7.0.72
opencv-python-headless 4.9.0.80
orjson 3.9.15
packaging 23.2
pandas 2.0.2
pillow 10.2.0
pip 24.0
plac 1.4.3
platformdirs 4.2.0
pooch 1.8.1
proces 0.1.7
protobuf 4.25.3
psutil 5.9.8
pyasn1 0.5.1
pyasn1-modules 0.3.0
pyclipper 1.3.0.post5
pycparser 2.21
pydantic 2.6.2
pydantic_core 2.16.3
pydub 0.25.1
Pygments 2.17.2
pykakasi 2.2.1
pyparsing 3.1.1
pypinyin 0.50.0
python-bidi 0.4.2
python-crfsuite 0.9.10
python-dateutil 2.8.2
python-multipart 0.0.9
pytz 2024.1
PyYAML 6.0.1
referencing 0.33.0
regex 2023.12.25
requests 2.31.0
resampy 0.4.2
rich 13.7.0
rpds-py 0.18.0
rsa 4.9
ruff 0.2.2
s3transfer 0.10.0
scikit-image 0.22.0
scikit-learn 1.4.1.post1
scipy 1.10.1
seaborn 0.13.2
semantic-version 2.10.0
setuptools 69.1.1
shapely 2.0.3
shellingham 1.5.4
six 1.15.0
sniffio 1.3.1
soundfile 0.12.1
starlette 0.36.3
sympy 1.12
threadpoolctl 3.3.0
tifffile 2024.2.12
tokenizers 0.13.3
tomlkit 0.12.0
toolz 0.12.1
torch 1.13.1
torchaudio 0.13.1
torchvision 0.17.0
tqdm 4.66.2
transformers 4.27.4
txtsplit 1.0.0
typer 0.9.0
typing_extensions 4.9.0
tzdata 2024.1
tzlocal 5.2
ultralytics 8.0.114
Unidecode 1.3.7
unidic 1.1.0
unidic-lite 1.0.8
urllib3 1.26.18
uvicorn 0.27.1
wasabi 0.10.1
websockets 11.0.3
wheel 0.37.0
wrapt 1.16.0
zipp 3.17.0

Could not run and just error,how can I fix it?

missing installation step "python -m pip install unidic", otherwise you can get:
"No module named unidic"

so the correct code to install in windows is:

git clone https://github.com/myshell-ai/MeloTTS.git
cd MeloTTS
pip install -e .
python -m pip install unidic
python -m unidic download

Bert-VITS2 use AGPL-V3, but this repo use MIT……？

code:

from melo.api import TTS
import time

start_time=time.time()

# Speed is adjustable
speed = 1.0
device = 'cpu' # or cuda:0

text = "下面请 A 0 0 1 号用户到诊室P 0 0 1就诊"
model = TTS(language='ZH', device=device)
speaker_ids = model.hps.data.spk2id

output_path = 'zh.wav'
model.tts_to_file(text, speaker_ids['ZH'], output_path, speed=speed)

end_time = time.time()  # 获取当前时间
elapsed_time = end_time - start_time  # 计算运行时间
print("程序运行时间：", elapsed_time, "秒")

result:

 > Text split to sentences.
下面请 A 0 0 1 号用户到诊室P 0 0 1就诊
 > ===========================
  0%|                                                                                                                                                                                          | 0/1 [00:00<?, ?it/s]Building prefix dict from the default dictionary ...
Loading model from cache /tmp/jieba.cache
Loading model cost 0.759 seconds.
Prefix dict has been built successfully.
Some weights of the model checkpoint at bert-base-multilingual-uncased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.bias', 'cls.seq_relationship.weight']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
[W NNPACK.cpp:53] Could not initialize NNPACK! Reason: Unsupported hardware.
100%|███████████████████████████████████████████████████████████| 1/1 [02:25<00:00, 145.41s/it]
程序运行时间： 166.48126244544983 秒

你好！你们是否可以支持rust推理？怎样使用rust更有效率的调用你们的接口？

Hi there,

Will you support OR Is it possible to share the training code so we can train on languages other than the ones you already support like Farsi(Persian)?

报错信息如下
_380.7 ERROR: Exception:
380.7 Traceback (most recent call last):
380.7 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/urllib3/response.py", line 438, in _error_catcher
380.7 yield
380.7 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/urllib3/response.py", line 561, in read
380.7 data = self._fp_read(amt) if not fp_closed else b""
380.7 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/urllib3/response.py", line 527, in _fp_read
380.7 return self._fp.read(amt) if amt is not None else self._fp.read()
380.7 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 90, in read
380.7 data = self.__fp.read(amt)
380.7 File "/usr/local/lib/python3.9/http/client.py", line 463, in read
380.7 n = self.readinto(b)
380.7 File "/usr/local/lib/python3.9/http/client.py", line 507, in readinto
380.7 n = self.fp.readinto(b)
380.7 File "/usr/local/lib/python3.9/socket.py", line 704, in readinto
380.7 return self._sock.recv_into(b)
380.7 File "/usr/local/lib/python3.9/ssl.py", line 1275, in recv_into
380.7 return self.read(nbytes, buffer)
380.7 File "/usr/local/lib/python3.9/ssl.py", line 1133, in read
380.7 return self.sslobj.read(len, buffer)
380.7 socket.timeout: The read operation timed out