Hi.

I've noticed that the Unicode character, "⁊", a Tironian note standing for Latin "et", is being changed to the numeral, "7". While they look alike, this doesn't seem like the best way to decode it, as it is never used to represent a numeral.

In Old Irish and Latin texts it was used as short-hand for the full word "et", so perhaps it should be replaced with "et". Alternatively, it is still used as an ampersand in modern Irish, so, maybe "&" would be a better replacement option?

Le meas,
Adrian.

Version 1.3.0 dropped support for Python <3.5, include Python 2. But the setup.cfg didn't change the "universal=1" line, so wheels are still being created with names like this: "Unidecode-1.3.0-py2.py3-none-any.whl", which claim Python 2 support when this is not true.

That line should probably also removed, a new 1.3.1 built, and a PyPI"yank" request done on the current 1.3.0, since as long as it is there pip will incorrectly consider it the most recent Python 2 compatible version, instead of the proper 1.2.0?

I'm using unidecode to remove accent from french words, it work perfect if the word is declared as string like accented_string ="Málaga" but doesn't work as expected if i read the word from text file!

this code working well

from unidecode import unidecode
accented_string = 'Málaga'
unaccented_string = unidecode.unidecode(accented_string)
print(unaccented_string)

the output is "Malaga"
now i want to do the same, but by reading text file

from unidecode import unidecode
import fileinput
fr_txt="fr.txt"
for name in fileinput.input([fr_txt]):
clean = name.replace("\n", "")
unaccented_string = unidecode.unidecode(clean)
print(unaccented_string )

the output is "M',laga" !!! so what's wrong?
i tried also this code

import io
from unidecode import unidecode
fr_txt="txt.txt"
f = io.open(fr_txt, mode="r", encoding="utf-8")
f.read()
for name in f:
clean = name.replace("\n", "")
line = unidecode(clean)
print(line)

the output is "M',laga" !!!
any idea?

import io
from unidecode import unidecode
fr_txt="1.txt"
f = io.open(fr_txt, mode="r", encoding="utf8")
f.read()
for name in f:
clean = name.replace("\n", "")
line = unidecode(clean)
print(line)

There is a problem with the case of accented char

Á →  a, it should be Á →  A

The character u"\xad" (http://www.fileformat.info/info/unicode/char/00ad/index.htm) looks exactly the same as a normal hyphen (http://www.fileformat.info/info/unicode/char/2010/index.htm).

If I try to normalize it to ascii, it ends up as '' however;

 unidecode.unidecode(u"\xad")
 ''

Other nasty characters, like the HORIZONTAL ELLIPSIS (http://www.fileformat.info/info/unicode/char/2026/index.htm) is normalized correctly.

unidecode.unidecode(u"\u2026")
'...'

Can the SOFT HYPHEN also be normalized into an -?

I recently upgraded unidecode, and saw some failing test.

The test in question:

"Pickup 65” TV from Platform 9¾, Kingʹs Cross Station."

The result:

- Pickup 65" TV from Platform 93/4, King's Cross Station.
+ Pickup 65" TV from Platform 9 3/4 , King's Cross Station.

I think that separating the 9 from the 3/4 is a good idea, so as to distinguish it from the possibility of 93 / 4 (which the original is not), however there is also a space placed between the 3/4 and the comma which does not read well.

Not a major issue but probably something that will bug people.

Hi,
Planning to use the library
licensed under GNU Library General Public License v2
we are not doing any modifications in the source code of the library
GPL, or our application can be made proprietary
can you please more insights into this...

Regards
Manju

>>> import unidecode
>>>
>>> unidecode.unidecode("Balabəyov")
'Balab@yov'
>>>

Expected: "Balabeyov", instead of "Balab@yov"

When trying to transliterate

"马云"
I receive

"Ma Yun " (notice the space in the end) instead of

"Ma Yun"

Here's the code you can use to replicate this issue:

import unittest
import unidecode

class TestStrings(unittest.TestCase):
    def test_replace_non_ascii_letters_with_chinese_name(self):
        self.assertEquals(unidecode.unidecode("马云"), "Ma Yun")

The test fails with the following error:

AssertionError: 'Ma Yun ' != 'Ma Yun'
- Ma Yun 
?       -
+ Ma Yun

Run on Python 3.8.5

EDIT:

Google Translate seems to be doing this with no issue, but perhaps Google Translate has the faulty transliteration. Chinese speakers welcome to correct me.

pip install unidecode
Collecting unidecode
Downloading https://files.pythonhosted.org/packages/31/39/53096f9217b057cb049fe872b7fc7ce799a1a89b76cf917d9639e7a558b5/Unidecode-1.0.23-py2.py3-none-any.whl (237kB)

echo de Bernières | unidecode
Unable to decode input: ordinal not in range(128), start: 8, end: 9

Thanks for making the library, it's really helpful in my case for cleaning social media texts.

Here are some cases where the transliteration/conversion was not correct (Version 1.3.2):

>>> from unidecode import unidecode
>>> unidecode("ᕼᗩᑭᑭIᗴᗴ")
'hpokikiIgaga'
>>> unidecode("🇦🇷🇮")
''
>>> unidecode("ωεłł")
'oell'
>>> unidecode("RᗅIPႮ")
'RghoIPP'
>>> unidecode("ғʀᴇᴇ")
"g'REE"

I will update this issue with more examples as I come across. Thanks!

Edit:

It looks like most of the issues is because they are characters of some other scripts, not fancy characters.
So, is there some way to do appearance-based conversion rather than approximate-phonetic conversion?

The tests in 1.1.0 release fail with PyPy:

======================================================================                                                                             
FAIL: test_encoding_error (tests.test_utility.TestUnidecodeUtility)                                                                                
----------------------------------------------------------------------                                                                             
Traceback (most recent call last):                                                                                                                 
  File "/tmp/portage/dev-python/unidecode-1.1.0/work/Unidecode-1.1.0/tests/test_utility.py", line 43, in test_encoding_error
    self.assertEqual(err, expected)                                      
AssertionError: u'Unable to decode input: invalid utf-8, start: 0, end: 1\n' != 'Unable to decode input: invalid start byte, start: 0, end: 1\n'
                                                                                                                                                   
----------------------------------------------------------------------

Hi there,
I find myself in a particuliar situation where I deal with manuscripts transcriptions. Of course, some char are not valid in unicode, but some seems to be from my understanding :

	unidecode.x0f1
	unidecode.x0e6
	unidecode.x0f0
ꝛ	unidecode.x0a7
	unidecode.x0f0
	unidecode.x0f0
	unidecode.x0e8
	unidecode.x0f0
	unidecode.x0e5
	unidecode.x0e4
ꝓ	unidecode.x0a7
	unidecode.x0f1
	unidecode.x0f0
	unidecode.x0f7
	unidecode.x0e6
	unidecode.x0e7
	unidecode.x0e5
	unidecode.x0ee
ꝰ	unidecode.x0a7
ꝙ	unidecode.x0a7
ꝯ	unidecode.x0a7
	unidecode.x0f1
	unidecode.x0f1
	unidecode.x0f1
	unidecode.x0f1
ꝑ	unidecode.x0a7
	unidecode.x0e8
	unidecode.x0f1

Should I pull request for x0a7 and x0f7 ?

The library is used in wagtail cms, when setting up the search, I noticed that the Cyrillic text is converted to transliteration. For the spelling dictionary this was unacceptable.

Hi Tomaž,

can you clarify the license of the data/mapping files?
The original Text-Unidecode was released as Clarified Perl Artistic License.
I am currently working on a transliteration library under MIT license and would love to make use of your excellent updates/refinements to the mapping files.

Would you be willing to release your updated mappings under a dual Clarified Perl Artistic / GPL license? I have also contacted the original Perl author, whether he's willing to release his code under Perl Artistic License 2.0...

Sorry for all the back and forth.

Regards,
Johannes

Current behavior:

unidecode("a"+chr(2000)+"b") == "a[?]b"

Desired behavior:

unidecode("a"+chr(2000)+"b") == "a b"

Rationale

Unicode spaces naturally correspond to the usual ascii spaces.

In reality, 'μmol' is micro mol and mmol is mili mol, so 1 mmol is 1000 μmol

'umol' would also be an acceptable transliteration in this case.

I'd like to see unidecode turn ℉ into something else, although I'm not sure what the best option for replacing the degree symbol is. degF maybe? Just F?

Always transliterating punctuation marks as regular letters could be an issue for some applications. While the paragraph sign ¶ is transliterated to P, I would like to have an option to treat it as unknown.

(I started this issue following 81f938d and regarding the exotic inverted nun ׆‎ that was changed to be transliterated into n as the regular nun נ. but the inverted one is an editorial/punctuation mark)

Hello @avian2. Would it be possible to add a feature that would take as input a set of one or more Unicode codepoints' and/or ranges to leave unchanged? This means that unidecode would effectively ignore such ranges of characters and leave them in place while transliterating everything else.

Thanks,
Diogo

Unidecode gives different values if your string is encoded in NFC (short form) or NFD mode (where a single letter may be represented by the letter, followed by a separate unicode accent mark).

unidecode.unidecode(unicodedata.normalize('NFC', 'Ѐ')) = 'Ie'
unidecode.unidecode(unicodedata.normalize('NFD', 'Ѐ')) = 'E'

Is this expected behavior?

\u30fb and \u30fc are currently both mapped to an empty character. Is there a reason for this?

If not, \u30fc (ー) should probably be mapped to -, and \u30fb (・) to maybe ., although I'm slightly less sure about the latter.

pip install unidecode

in a new py file

from unidecode import unidecode
unidecode('ko\u017eu\u0161\u010dek')

ImportError: cannot import name 'unidecode' from partially initialized module 'unidecode' (most likely due to a circular import)

Example code to test japanese letters in Hiragana, full-width Katakana and half-width Katakana using dakuten and handakuten:

import unidecode as ud

hiragana    = "はひほへほ ばびぶぼべ ぱぴぷぺぽ"
katakana_fw = "ハヒフヘホ バビブベボ パピプペポ"
katakana_hw = "ﾊﾋﾌﾍﾎ ﾊﾞﾋﾞﾌﾞﾍﾞﾎﾞ ﾊﾟﾋﾟﾌﾟﾍﾟﾎﾟ"

print(ud.unidecode(hiragana))
print(ud.unidecode(katakana_fw))
print(ud.unidecode(katakana_hw))

The result for Hiragana and Katakana full width is correct, however the result for half-width Katakana is not:

hahihoheho babibubobe papipupepo
hahihuheho babibubebo papipupepo
hahihuheho ha:hi:hu:he:ho: ha;hi;hu;he;ho;

Instead of the correct transliteration "ba" it returns "ha:" and instead of "pa" it returns "ha;".

Please, support localization for Unicode block and character description translations from https://github.com/unicode-table/unicode-table-data/tree/master/loc If possible, use a gettetxt approach similar to https://pypi.org/project/pycountry/ Implementing this feature will allow users to read Unicode descriptions in their own language, other than English.

I'd like to suggest that the project_urls meta data be added to this project to allow users coming from PyPI easier discovery of that urls available, including submission of Bugs, Location of Documentation, and location of source code.

See Packaging and distributing projects for more details

I have a case where most transliterations only have to deal with ASCII chars, with only a few taking non-ASCII ones (Spanish).

See iki/unidecode#11 (comment)

PR will follow.

Hi, I updated my server today with latest and I now get the following error :

   File "/usr/local/lib/python3.5/dist-packages/unidecode/__init__.py", line 151
     __import__(f'unidecode.{file.stem}', globals(), locals(), ['data']).data,
                                       ^
 SyntaxError: invalid syntax

I am running python 3.5 on debian stretch.
Forcing previous version 1.3.4 solves the issue.
I am not sure if this means that Python 3.5 is not supported anymore on newest version or if there is an error in latest Unidecode version ?

👋 Could you add this repo as the "homepage" for the pypi listing for unidecode? https://pypi.org/project/Unidecode/

Spotted an issue in a Dependabot pull request upgrading unidecode where the dependency url goes to the wrong repo: https://github.com/kmike/text-unidecode

Example pull request: metabrainz/listenbrainz-server#1219

For reference, Dependabot tries to find the "homepage" for the dependency from the pypi json response, first looking at any homepage entries and if none exist scanning the description for any source repo links. In this case the description for Unidecode on pypi includes a link to text-unidecode before the correct github link under the license section Unidecode should use a permissive license such as MIT or the BSD license..

This is a feature request to add a library function that can be called to set custom character replacement mappings. The library function could be called in a similar manner to as follows:

unidecode.setReplacement('™', "(TM)")

after which subsequent calls to unidecode would replace '™' with '(TM)' rather than the default of "(tm).

For my particular use case, this would be useful for standardizing names across data sources. For instance, if one source has the name "Acme™" and the other has the name "Acme", I could make a call to replace all instances of '™' with "", which allows me to easily compare across sources.

However, I could see a large number of other use cases for this as well. For example, in the FAQ, it is mentioned that localization is not supported and that certain german characters translate to english text rather than german phonetics. Having a call like this could allow programmers to make their own "localization" libraries, allowing them to whatever mappings they feel like.

Hi,

in __init__.py:78: if codepoint > 0xeffff:

Is that the correct codepoint. I can see a private use area 0xe000-0xefff.
Is the above a typo, or am I missing something re: utf-16?

I'm trying to understand what's happening here. I have the following text in a file:

"\u003e"

But decoding this has no effect:

dom = open('dom.txt', encoding='utf-8').read()
unidecode(dom)

"\u003e"

But "\u003e" -> ">"

Why is this not decoded as expected?

Traditionally Python Unidecode followed the versioning scheme used by the original Perl module. This doesn't make sense any more because:

• Perl Text::Unidecode itself switched to a different versioning scheme long ago - http://search.cpan.org/~sburke/Text-Unidecode-1.30/
• Python Unidecode has now sufficiently diverged from Perl version so that matched version numbers are no longer useful,
• setuptools (>= 8) began mangling version numbers, yielding confusion because Unidecode 0.04.x now sometimes appears as 0.4.x - pypa/setuptools#302

Next release should be numbered something like 1.0.22.

My code is as follow:

from unidecode import unidecode

s = "中文字café"

output = unidecode(s)
print(output)

It outputs Zhong Wen Zi cafe. Can the function only transliterate European accent characters only, but leave Chinese characters as it is?

Desired output: 中文字cafe

Thanks.

Hi there !
I hope this is really a bug and not me not being sure of what's going on... Anyway.

Context

• unidecode==0.4.21
• python 3.5.2
• Linux Mint Sonya 18.2 (Not that I think it matters here)

Reproduction of the bug

from unidecode import unidecode
print(unidecode("y"))  # prints y
print(unidecode("ў"))  # prints u

Expected

print(unidecode("ў")) should print y as well

I'll try to see if I can understand the reason why it works this way in the codebase but at least the issue is open...

First of all, thanks for this tool.

I'm trying to decode UTF-16 escaped strings to no avail.

For instance, \u00c1 is converted to A instead of Á:

>>> import unidecode
>>> unidecode.unidecode('\u00c1')
'A'
>>> unidecode.unidecode(u'\u00c1')
'A'

(I've used https://www.branah.com/unicode-converter to verify)

Is there an option to specify the UTF-16 encoding ?

Thanks.

Major scripts like Latin, Cyrillic and Arabic are used to write many languages. Right now, the lib is implicitly unidecoding all Cyrillic as if it were Russian specifically.

For example:
unidecode('Халид Бешлић, Цеца, Жељко Јоксимовић') # sr

actual:
'Khalid Beshlitsh, Tsetsa, Zheljko Joksimovitsh'

expected:
'Halid Beslic, Ceca, Zeljko Joksimovic'

That's because there should be an implicit initial conversion to Latin ('Halid Bešlić, Ceca, Željko Joksimović'). Passing that to unidecode works as expected.

Similar is true for Latin. For example:
unidecode('Kadıköy') # tr
unidecode('Schönheimer') # de

actual:
'Kadikoy'
'Schonheimer'

expected:
'Kadikoy'
'Schoenheimer'

The most straightforward fix is an optional parameter lang.

Is there any chance this could be released with a weak copyleft licence (e.g. BSD or MIT)?

The PyPI wheels contain __init__.pyi (checked 1.3.6). Is this intended? This commit removed __init__.pyi two years ago: f877539. It looks like the file is for type annotations, but type annotations have since been added in __init__.py. Thank you!

1. 呆 Ai -> Dai (Pronunciation in Chinese)

2. I found that unidecode will transform all characters to its pronunciation in Chinese even those the text is actually written in Japanese Kanji.

3. While transforming Chinese characters to their pronunciations, a trailing whitespace is always added to the end of the text. 阿傍 -> A Bang (There's is a trailing whitespace at the end)

4. While transforming text composed of Chinese characters and Japanese katakana/hiragana, the whitespace is left out. 阿呆の足下使い -> A Ai noZu Xia Shi i -> A Ai no Zu Xia Shi i (And, these are Japanese characters, not Chinese characters)

Thank you for your work.

>>> unidecode("人生は難しいですね。") 'Ren Sheng haNan shiidesune. '

This is a Japanese sentence with Kanji.
Although the kana(like は and しいですね。) is correctly transliterated, Kanji is not correctly transliterated.

In Chinese, 人生 = Ren Sheng , 難 = Nan.
But it should be 人生=jinsei, 難 = muzuka　here, since it is a sentence in japanese. Can this problem be solved?

Getting following error when trying to instlall with pipenv

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.Hint: try $ pipenv lock --pre if it is a pre-release dependency
Could not find a version that matches Unidecode<0.05,==0.4.21,>=0.04,>=1.0.23
Tried: 0.4.1, 0.4.5, 0.4.6, 0.4.7, 0.4.8, 0.4.9, 0.4.10, 0.4.11, 0.4.12, 0.4.13, 0.4.14, 0.4.16, 0.4.17, 0.4.18, 0.4.19, 0.4.20, 0.4.20, 0.4.21, 0.4.21, 1.0.22, 1.0.22, 1.0.23, 1.0.23
There are incompatible versions in the resolved dependencies.

Truly arbitrary digraphs and trigraphs are less common in non-Latin scripts, but still happen.

unidecode("Բովանդակություն")

actual:
'Bovandakowt`yown'
expected:
'Bovandakut`yun'

The current behaviour is not necessarily wrong, but a bit undesirable because it breaks the roundtrip expectation ('Zulu' == unidecode(transliterate('Zulu', lang='hy'))).

There may be other cases mentioned in https://en.wikipedia.org/wiki/Digraph_(orthography)#Examples that are not covered. Some of them are working as intended though, because for our purposes, logical digraphs like дж is not a true arbitrary digraph because it is still two chars in Latin and even three in Ascii, so mapping from each char without context still works.

Version 1.3.1 is Python only, but when it was published the python_requires information did not get sent to PyPI, and therefore Python 2 versions of pip are trying to download it but failing. I don't know how unidecode gets published to PyPI, but tools like twine will send the python_requires information automatically.
Please could version 1.3.1 be yanked and a version 1.3.2 published with the python_requires information available.

I would like to ask for @alonbl feedback/greenlight before preparing my PR. I am interested in addressing several issues I see in the current Hebrew transliteration:

1. 05ef (triple yod)- can now be transliterated as YYY
2. seems inconsistent to me to have raffe as - and dagensh to '. if we are going by the graphics then dagesh should be . (dot). but i think a more useful choice would be to ignore both of them (as is currently done for the Shin-dots)
3. Better alignment with Hebrew Language Academy rules (https://hebrew-academy.org.il/wp-content/uploads/taatik-ivrit-latinit-1-1.pdf):
   a. 05d7 ח is never transliterated as KH - a more standard-compliant version would be H (to differ from h) or h
   b. it is inconsistent to transliterate א as A and ע as back-tic. ע could be A or 'A or A'. but mind you all these choices including for א are non standard. also back-tic for ע is from the "exact standard", but we are otherwise following here the "simple standard" which uses '. I am really not sure what is the right thing to do here. we could also follow other languages and use the letter name in these cases: ALEPH and AYIN.
   c. using @ for schwa is consistent with the IPA symbol but it is not useful and not part of the hebrew standard which ignores schwa in transliteration (or in some cases uses e)
   d. ק should be k as in the simple standard (q is used in the exact standard)
4. i am not sure what are 05f5, 05f6, 05f7 as they are not part of unicode afaict
5. fixes in hebrew presentation forms (https://www.unicode.org/charts/PDF/UFB00.pdf)
   a. fb4f should be EL not l
   b. fb4e should be f not p
   c. fb4d should be KH not k
   d. fb4c should be v not b
   e. fb4b should be o not vo, similarly fb1d should by i not yi
   f. fix eg sh, ts to be SH, TS as done in regular letters
   g. fb47 should be k not ts (this is a mistake)
   h. fb41 should be s not n (this is a mistake)
   i. fb3e should be m not l (this is a mistake)
   j. fb30 currently missing should be i
   k. fb27 should be r not m (this is a mistake)
   l. add fb21, fb20 similar to the choices decided on for regular א, ע
6. graphically sof-pasuk looks like : but for nlp tasks would be more useful to use "." or even ". " as this is the meaning of the punctuation.

Unidecode cannot currently be moved/packaged inside another project, as it uses absolute __import__ in __init__.py.

The easy fix is to set level=x where x is the recursion level from the top package of the parent project, in my case it had to be level=2.

Could you try to make unidecode more "packaging-friendly"?

Thank you for this great library!

Hi,

I am one of the maintainers of Apache Airflow. Indirectly we make use of unidecode through a dependency (python-nvd3 -> python-slugify -> unidecode). The Apache Software Foundation considers the GPLv2 incompatible with the APL2.

We didn't find out about the issue until now due to the fact that nvd3 and python-slugify maintain compatible licensed. Python-slugify is MIT licensed which might be incompatible with GPLv2 (I am not a lawyer).

We could ask python-slugify to use an alternative library or implement our own. Both are obviously a hassle as we need to convince package maintainers to switch. Also there is not technical reason to do so for us.

As the original Perl package was also available under the Perl Artistic License, would you consider dual licensing your derivative? That would allows us to continue to use the very useful libraries and just make us very happy :-).

Thanks for your consideration in advance!
Bolke

First of all thank you for this project.

I wonder is there specific reason why this

unidecode('Dr. Kargˋs')

converted to Dr. Karg\s and not to Dr. Karg's?

Hello Sir

As you said in your documentation
I used the code, and the code is working
It successfully decode unicode escape characters.

>> from unidecode import unidecode
>> value="\u0048\u0065\u006c\u006c\u006f"
>> print(unidecode(value))
Hello

Problem

When i used input as a value
unidecode prints the same value as i entered.

>> from unidecode import unidecode
>> value=input("Input: ")
Input: \u0048\u0065\u006c\u006c\u006f
>> print(unidecode(value))
\u0048\u0065\u006c\u006c\u006f