avian2 / unidecode Goto Github PK
View Code? Open in Web Editor NEWASCII transliterations of Unicode text - GitHub mirror
Home Page: https://pypi.python.org/pypi/Unidecode
License: GNU General Public License v2.0
ASCII transliterations of Unicode text - GitHub mirror
Home Page: https://pypi.python.org/pypi/Unidecode
License: GNU General Public License v2.0
Hi.
I've noticed that the Unicode character, "⁊", a Tironian note standing for Latin "et", is being changed to the numeral, "7". While they look alike, this doesn't seem like the best way to decode it, as it is never used to represent a numeral.
In Old Irish and Latin texts it was used as short-hand for the full word "et", so perhaps it should be replaced with "et". Alternatively, it is still used as an ampersand in modern Irish, so, maybe "&" would be a better replacement option?
Le meas,
Adrian.
Version 1.3.0 dropped support for Python <3.5, include Python 2. But the setup.cfg didn't change the "universal=1" line, so wheels are still being created with names like this: "Unidecode-1.3.0-py2.py3-none-any.whl", which claim Python 2 support when this is not true.
Line 2 in 9c2dc90
That line should probably also removed, a new 1.3.1 built, and a PyPI"yank" request done on the current 1.3.0, since as long as it is there pip will incorrectly consider it the most recent Python 2 compatible version, instead of the proper 1.2.0?
I'm using unidecode to remove accent from french words, it work perfect if the word is declared as string like accented_string ="Málaga" but doesn't work as expected if i read the word from text file!
this code working well
from unidecode import unidecode
accented_string = 'Málaga'
unaccented_string = unidecode.unidecode(accented_string)
print(unaccented_string)
the output is "Malaga"
now i want to do the same, but by reading text file
from unidecode import unidecode
import fileinput
fr_txt="fr.txt"
for name in fileinput.input([fr_txt]):
clean = name.replace("\n", "")
unaccented_string = unidecode.unidecode(clean)
print(unaccented_string )
the output is "M',laga" !!! so what's wrong?
i tried also this code
import io
from unidecode import unidecode
fr_txt="txt.txt"
f = io.open(fr_txt, mode="r", encoding="utf-8")
f.read()
for name in f:
clean = name.replace("\n", "")
line = unidecode(clean)
print(line)
the output is "M',laga" !!!
any idea?
import io
from unidecode import unidecode
fr_txt="1.txt"
f = io.open(fr_txt, mode="r", encoding="utf8")
f.read()
for name in f:
clean = name.replace("\n", "")
line = unidecode(clean)
print(line)
There is a problem with the case of accented char
Á → a, it should be Á → A
The character u"\xad"
(http://www.fileformat.info/info/unicode/char/00ad/index.htm) looks exactly the same as a normal hyphen (http://www.fileformat.info/info/unicode/char/2010/index.htm).
If I try to normalize it to ascii, it ends up as '' however;
unidecode.unidecode(u"\xad")
''
Other nasty characters, like the HORIZONTAL ELLIPSIS (http://www.fileformat.info/info/unicode/char/2026/index.htm) is normalized correctly.
unidecode.unidecode(u"\u2026")
'...'
Can the SOFT HYPHEN also be normalized into an -
?
I recently upgraded unidecode, and saw some failing test.
The test in question:
"Pickup 65” TV from Platform 9¾, Kingʹs Cross Station."
The result:
- Pickup 65" TV from Platform 93/4, King's Cross Station.
+ Pickup 65" TV from Platform 9 3/4 , King's Cross Station.
I think that separating the 9 from the 3/4 is a good idea, so as to distinguish it from the possibility of 93 / 4
(which the original is not), however there is also a space placed between the 3/4
and the comma which does not read well.
Not a major issue but probably something that will bug people.
Hi,
Planning to use the library
licensed under GNU Library General Public License v2
we are not doing any modifications in the source code of the library
GPL, or our application can be made proprietary
can you please more insights into this...
Regards
Manju
>>> import unidecode
>>>
>>> unidecode.unidecode("Balabəyov")
'Balab@yov'
>>>
Expected: "Balabeyov"
, instead of "Balab@yov"
When trying to transliterate
"马云"
I receive
"Ma Yun " (notice the space in the end) instead of
"Ma Yun"
Here's the code you can use to replicate this issue:
import unittest
import unidecode
class TestStrings(unittest.TestCase):
def test_replace_non_ascii_letters_with_chinese_name(self):
self.assertEquals(unidecode.unidecode("马云"), "Ma Yun")
The test fails with the following error:
AssertionError: 'Ma Yun ' != 'Ma Yun'
- Ma Yun
? -
+ Ma Yun
Run on Python 3.8.5
EDIT:
Google Translate seems to be doing this with no issue, but perhaps Google Translate has the faulty transliteration. Chinese speakers welcome to correct me.
pip install unidecode
Collecting unidecode
Downloading https://files.pythonhosted.org/packages/31/39/53096f9217b057cb049fe872b7fc7ce799a1a89b76cf917d9639e7a558b5/Unidecode-1.0.23-py2.py3-none-any.whl (237kB)
echo de Bernières | unidecode
Unable to decode input: ordinal not in range(128), start: 8, end: 9
Thanks for making the library, it's really helpful in my case for cleaning social media texts.
Here are some cases where the transliteration/conversion was not correct (Version 1.3.2):
>>> from unidecode import unidecode
>>> unidecode("ᕼᗩᑭᑭIᗴᗴ")
'hpokikiIgaga'
>>> unidecode("🇦🇷🇮")
''
>>> unidecode("ωεłł")
'oell'
>>> unidecode("RᗅIPႮ")
'RghoIPP'
>>> unidecode("ғʀᴇᴇ")
"g'REE"
I will update this issue with more examples as I come across. Thanks!
Edit:
It looks like most of the issues is because they are characters of some other scripts, not fancy characters.
So, is there some way to do appearance-based conversion rather than approximate-phonetic conversion?
The tests in 1.1.0 release fail with PyPy:
======================================================================
FAIL: test_encoding_error (tests.test_utility.TestUnidecodeUtility)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/portage/dev-python/unidecode-1.1.0/work/Unidecode-1.1.0/tests/test_utility.py", line 43, in test_encoding_error
self.assertEqual(err, expected)
AssertionError: u'Unable to decode input: invalid utf-8, start: 0, end: 1\n' != 'Unable to decode input: invalid start byte, start: 0, end: 1\n'
----------------------------------------------------------------------
Hi there,
I find myself in a particuliar situation where I deal with manuscripts transcriptions. Of course, some char are not valid in unicode, but some seems to be from my understanding :
unidecode.x0f1
unidecode.x0e6
unidecode.x0f0
ꝛ unidecode.x0a7
unidecode.x0f0
unidecode.x0f0
unidecode.x0e8
unidecode.x0f0
unidecode.x0e5
unidecode.x0e4
ꝓ unidecode.x0a7
unidecode.x0f1
unidecode.x0f0
unidecode.x0f7
unidecode.x0e6
unidecode.x0e7
unidecode.x0e5
unidecode.x0ee
ꝰ unidecode.x0a7
ꝙ unidecode.x0a7
ꝯ unidecode.x0a7
unidecode.x0f1
unidecode.x0f1
unidecode.x0f1
unidecode.x0f1
ꝑ unidecode.x0a7
unidecode.x0e8
unidecode.x0f1
Should I pull request for x0a7 and x0f7 ?
The library is used in wagtail cms, when setting up the search, I noticed that the Cyrillic text is converted to transliteration. For the spelling dictionary this was unacceptable.
Hi Tomaž,
can you clarify the license of the data/mapping files?
The original Text-Unidecode was released as Clarified Perl Artistic License.
I am currently working on a transliteration library under MIT license and would love to make use of your excellent updates/refinements to the mapping files.
Would you be willing to release your updated mappings under a dual Clarified Perl Artistic / GPL license? I have also contacted the original Perl author, whether he's willing to release his code under Perl Artistic License 2.0...
Sorry for all the back and forth.
Regards,
Johannes
unidecode("a"+chr(2000)+"b") == "a[?]b"
unidecode("a"+chr(2000)+"b") == "a b"
Unicode spaces naturally correspond to the usual ascii spaces.
I'd like to see unidecode turn ℉ into something else, although I'm not sure what the best option for replacing the degree symbol is. degF maybe? Just F?
Always transliterating punctuation marks as regular letters could be an issue for some applications. While the paragraph sign ¶ is transliterated to P, I would like to have an option to treat it as unknown.
(I started this issue following 81f938d and regarding the exotic inverted nun ׆ that was changed to be transliterated into n as the regular nun נ. but the inverted one is an editorial/punctuation mark)
Hello @avian2. Would it be possible to add a feature that would take as input a set of one or more Unicode codepoints' and/or ranges to leave unchanged? This means that unidecode
would effectively ignore such ranges of characters and leave them in place while transliterating everything else.
Thanks,
Diogo
Unidecode gives different values if your string is encoded in NFC (short form) or NFD mode (where a single letter may be represented by the letter, followed by a separate unicode accent mark).
unidecode.unidecode(unicodedata.normalize('NFC', 'Ѐ')) = 'Ie'
unidecode.unidecode(unicodedata.normalize('NFD', 'Ѐ')) = 'E'
Is this expected behavior?
\u30fb
and \u30fc
are currently both mapped to an empty character. Is there a reason for this?
If not, \u30fc
(ー) should probably be mapped to -
, and \u30fb
(・) to maybe .
, although I'm slightly less sure about the latter.
pip install unidecode
in a new py file
from unidecode import unidecode
unidecode('ko\u017eu\u0161\u010dek')
ImportError: cannot import name 'unidecode' from partially initialized module 'unidecode' (most likely due to a circular import)
Example code to test japanese letters in Hiragana, full-width Katakana and half-width Katakana using dakuten and handakuten:
import unidecode as ud
hiragana = "はひほへほ ばびぶぼべ ぱぴぷぺぽ"
katakana_fw = "ハヒフヘホ バビブベボ パピプペポ"
katakana_hw = "ハヒフヘホ バビブベボ パピプペポ"
print(ud.unidecode(hiragana))
print(ud.unidecode(katakana_fw))
print(ud.unidecode(katakana_hw))
The result for Hiragana and Katakana full width is correct, however the result for half-width Katakana is not:
hahihoheho babibubobe papipupepo
hahihuheho babibubebo papipupepo
hahihuheho ha:hi:hu:he:ho: ha;hi;hu;he;ho;
Instead of the correct transliteration "ba" it returns "ha:" and instead of "pa" it returns "ha;".
Please, support localization for Unicode block and character description translations from https://github.com/unicode-table/unicode-table-data/tree/master/loc If possible, use a gettetxt
approach similar to https://pypi.org/project/pycountry/ Implementing this feature will allow users to read Unicode descriptions in their own language, other than English.
I'd like to suggest that the project_urls meta data be added to this project to allow users coming from PyPI easier discovery of that urls available, including submission of Bugs, Location of Documentation, and location of source code.
See Packaging and distributing projects for more details
I have a case where most transliterations only have to deal with ASCII chars, with only a few taking non-ASCII ones (Spanish).
See iki/unidecode#11 (comment)
PR will follow.
Hi, I updated my server today with latest and I now get the following error :
File "/usr/local/lib/python3.5/dist-packages/unidecode/__init__.py", line 151
__import__(f'unidecode.{file.stem}', globals(), locals(), ['data']).data,
^
SyntaxError: invalid syntax
I am running python 3.5 on debian stretch.
Forcing previous version 1.3.4 solves the issue.
I am not sure if this means that Python 3.5 is not supported anymore on newest version or if there is an error in latest Unidecode version ?
👋 Could you add this repo as the "homepage" for the pypi listing for unidecode? https://pypi.org/project/Unidecode/
Spotted an issue in a Dependabot pull request upgrading unidecode where the dependency url goes to the wrong repo: https://github.com/kmike/text-unidecode
Example pull request: metabrainz/listenbrainz-server#1219
For reference, Dependabot tries to find the "homepage" for the dependency from the pypi json response, first looking at any homepage entries and if none exist scanning the description for any source repo links. In this case the description for Unidecode on pypi includes a link to text-unidecode before the correct github link under the license section Unidecode should use a permissive license such as MIT or the BSD license.
.
This is a feature request to add a library function that can be called to set custom character replacement mappings. The library function could be called in a similar manner to as follows:
unidecode.setReplacement('™', "(TM)")
after which subsequent calls to unidecode would replace '™' with '(TM)' rather than the default of "(tm).
For my particular use case, this would be useful for standardizing names across data sources. For instance, if one source has the name "Acme™" and the other has the name "Acme", I could make a call to replace all instances of '™' with "", which allows me to easily compare across sources.
However, I could see a large number of other use cases for this as well. For example, in the FAQ, it is mentioned that localization is not supported and that certain german characters translate to english text rather than german phonetics. Having a call like this could allow programmers to make their own "localization" libraries, allowing them to whatever mappings they feel like.
Hi,
in __init__.py:78
: if codepoint > 0xeffff:
Is that the correct codepoint. I can see a private use area 0xe000-0xefff
.
Is the above a typo, or am I missing something re: utf-16?
I'm trying to understand what's happening here. I have the following text in a file:
"\u003e"
But decoding this has no effect:
dom = open('dom.txt', encoding='utf-8').read()
unidecode(dom)
"\u003e"
But "\u003e" -> ">"
Why is this not decoded as expected?
Traditionally Python Unidecode followed the versioning scheme used by the original Perl module. This doesn't make sense any more because:
Next release should be numbered something like 1.0.22
.
My code is as follow:
from unidecode import unidecode
s = "中文字café"
output = unidecode(s)
print(output)
It outputs Zhong Wen Zi cafe
. Can the function only transliterate European accent characters only, but leave Chinese characters as it is?
Desired output: 中文字cafe
Thanks.
Hi there !
I hope this is really a bug and not me not being sure of what's going on... Anyway.
unidecode==0.4.21
from unidecode import unidecode
print(unidecode("y")) # prints y
print(unidecode("ў")) # prints u
print(unidecode("ў"))
should print y
as well
I'll try to see if I can understand the reason why it works this way in the codebase but at least the issue is open...
First of all, thanks for this tool.
I'm trying to decode UTF-16
escaped strings to no avail.
For instance, \u00c1
is converted to A instead of Á:
>>> import unidecode
>>> unidecode.unidecode('\u00c1')
'A'
>>> unidecode.unidecode(u'\u00c1')
'A'
(I've used https://www.branah.com/unicode-converter to verify)
Is there an option to specify the UTF-16
encoding ?
Thanks.
Major scripts like Latin, Cyrillic and Arabic are used to write many languages. Right now, the lib is implicitly unidecoding all Cyrillic as if it were Russian specifically.
For example:
unidecode('Халид Бешлић, Цеца, Жељко Јоксимовић') # sr
actual:
'Khalid Beshlitsh, Tsetsa, Zheljko Joksimovitsh'
expected:
'Halid Beslic, Ceca, Zeljko Joksimovic'
That's because there should be an implicit initial conversion to Latin ('Halid Bešlić, Ceca, Željko Joksimović'
). Passing that to unidecode
works as expected.
Similar is true for Latin. For example:
unidecode('Kadıköy') # tr
unidecode('Schönheimer') # de
actual:
'Kadikoy'
'Schonheimer'
expected:
'Kadikoy'
'Schoenheimer'
The most straightforward fix is an optional parameter lang
.
Is there any chance this could be released with a weak copyleft licence (e.g. BSD or MIT)?
The PyPI wheels contain __init__.pyi
(checked 1.3.6). Is this intended? This commit removed __init__.pyi
two years ago: f877539. It looks like the file is for type annotations, but type annotations have since been added in __init__.py
. Thank you!
呆 Ai -> Dai (Pronunciation in Chinese)
I found that unidecode will transform all characters to its pronunciation in Chinese even those the text is actually written in Japanese Kanji.
While transforming Chinese characters to their pronunciations, a trailing whitespace is always added to the end of the text. 阿傍 -> A Bang (There's is a trailing whitespace at the end)
While transforming text composed of Chinese characters and Japanese katakana/hiragana, the whitespace is left out. 阿呆の足下使い -> A Ai noZu Xia Shi i -> A Ai no Zu Xia Shi i (And, these are Japanese characters, not Chinese characters)
Thank you for your work.
>>> unidecode("人生は難しいですね。") 'Ren Sheng haNan shiidesune. '
This is a Japanese sentence with Kanji.
Although the kana(like は and しいですね。) is correctly transliterated, Kanji is not correctly transliterated.
In Chinese, 人生 = Ren Sheng , 難 = Nan.
But it should be 人生=jinsei, 難 = muzuka here, since it is a sentence in japanese. Can this problem be solved?
Getting following error when trying to instlall with pipenv
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.Hint: try $ pipenv lock --pre if it is a pre-release dependency
Could not find a version that matches Unidecode<0.05,==0.4.21,>=0.04,>=1.0.23
Tried: 0.4.1, 0.4.5, 0.4.6, 0.4.7, 0.4.8, 0.4.9, 0.4.10, 0.4.11, 0.4.12, 0.4.13, 0.4.14, 0.4.16, 0.4.17, 0.4.18, 0.4.19, 0.4.20, 0.4.20, 0.4.21, 0.4.21, 1.0.22, 1.0.22, 1.0.23, 1.0.23
There are incompatible versions in the resolved dependencies.
Truly arbitrary digraphs and trigraphs are less common in non-Latin scripts, but still happen.
unidecode("Բովանդակություն")
actual:
'Bovandakowt`yown'
expected:
'Bovandakut`yun'
The current behaviour is not necessarily wrong, but a bit undesirable because it breaks the roundtrip expectation ('Zulu' == unidecode(transliterate('Zulu', lang='hy'))
).
There may be other cases mentioned in https://en.wikipedia.org/wiki/Digraph_(orthography)#Examples that are not covered. Some of them are working as intended though, because for our purposes, logical digraphs like дж
is not a true arbitrary digraph because it is still two chars in Latin and even three in Ascii, so mapping from each char without context still works.
Version 1.3.1 is Python only, but when it was published the python_requires information did not get sent to PyPI, and therefore Python 2 versions of pip are trying to download it but failing. I don't know how unidecode gets published to PyPI, but tools like twine will send the python_requires information automatically.
Please could version 1.3.1 be yanked and a version 1.3.2 published with the python_requires information available.
$ git clone https://www.tablix.org/~avian/git/unidecode.git
Cloning into 'unidecode'...
fatal: unable to access 'https://www.tablix.org/~avian/git/unidecode.git/': Could not resolve host: www.tablix.org
I would like to ask for @alonbl feedback/greenlight before preparing my PR. I am interested in addressing several issues I see in the current Hebrew transliteration:
Unidecode cannot currently be moved/packaged inside another project, as it uses absolute __import__
in __init__.py
.
The easy fix is to set level=x
where x is the recursion level from the top package of the parent project, in my case it had to be level=2
.
Could you try to make unidecode more "packaging-friendly"?
Thank you for this great library!
Hi,
I am one of the maintainers of Apache Airflow. Indirectly we make use of unidecode through a dependency (python-nvd3 -> python-slugify -> unidecode). The Apache Software Foundation considers the GPLv2 incompatible with the APL2.
We didn't find out about the issue until now due to the fact that nvd3 and python-slugify maintain compatible licensed. Python-slugify is MIT licensed which might be incompatible with GPLv2 (I am not a lawyer).
We could ask python-slugify to use an alternative library or implement our own. Both are obviously a hassle as we need to convince package maintainers to switch. Also there is not technical reason to do so for us.
As the original Perl package was also available under the Perl Artistic License, would you consider dual licensing your derivative? That would allows us to continue to use the very useful libraries and just make us very happy :-).
Thanks for your consideration in advance!
Bolke
First of all thank you for this project.
I wonder is there specific reason why this
unidecode('Dr. Kargˋs')
converted to Dr. Karg\s
and not to Dr. Karg's
?
As you said in your documentation
I used the code, and the code is working
It successfully decode unicode escape characters.
>> from unidecode import unidecode
>> value="\u0048\u0065\u006c\u006c\u006f"
>> print(unidecode(value))
Hello
When i used input as a value
unidecode
prints the same value as i entered.
>> from unidecode import unidecode
>> value=input("Input: ")
Input: \u0048\u0065\u006c\u006c\u006f
>> print(unidecode(value))
\u0048\u0065\u006c\u006c\u006f
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.