urduhack / urdu-characters Goto Github PK

📄 Complete collection of Urdu language characters & unicode code points.

License: MIT License

Python 100.00%

urdu-alphabet letters urdu-hack nlp-machine-learning unicode character python characters words urdu

urdu-characters's Introduction

Urduhack: A Python NLP library for Urdu language

Urduhack is a NLP library for urdu language. It comes with a lot of battery included features to help you process Urdu data in the easiest way possible.

You can reach out core contributor Mr Ikram Ali @ https://github.com/akkefa

Our Goal

Academic users Easier experimentation to prove their hypothesis without coding from scratch.
NLP beginners Learn how to build an NLP project with production level code quality.
NLP developers Build a production level application within minutes.

🔥 Features Support

🛠 Installation

Urduhack officially supports Python 3.6–3.7, and runs great on PyPy.

Installing with tensorflow cpu version.

$ pip install urduhack[tf]

Installing with tensorflow gpu version.

$ pip install urduhack[tf-gpu]

Usage

import urduhack

# Downloading models
urduhack.download()

nlp = urduhack.Pipeline()
text = ""
doc = nlp(text)

for sentence in doc.sentences:
    print(sentence.text)
    for word in sentence.words:
        print(f"{word.text}\t{word.pos}")

    for token in sentence.tokens:
        print(f"{token.text}\t{token.ner}")

🔗 Documentation

Fantastic documentation is available at https://urduhack.readthedocs.io/

Documentation
Installation	How to install Urduhack and download models
Quickstart	New to Urduhack? Here's everything you need to know!
API Reference	The detailed reference for Urduhack's API.
Contribute	How to contribute to the code base.

👍 Contributors

Special thanks to everyone who contributed to getting the Urduhack to the current state.

Backers

Thank you to all our backers! 🙏 [Become a backer]

📝 Copyright and license

Code released under the MIT License.

urdu-characters's People

Contributors

Stargazers

Watchers

Forkers

muhammadfahid51 fadi212 gitter-badger ppptechunit eng-rsmy muhammadyaseen

urdu-characters's Issues

There is no difference between Urdu and Arabic characters in Unicode

I came here with with very high hopes after seeing the report on propakistani.pk. This could be the start of something great and I admire your initiative. You are doing great work.

However, your claims about different 'Arabic' and 'Urdu' range, and the same in propakistani.pk is flawed. This range: 0600-06FF is different from what you call 'Arabic' range in the sense that this one contains characters that are used in normal text, while the second one are presentation forms. (Imagine trying to write out in a child's qaeda how many different shapes 'Bay' can be in. Or to force a certain form of letter somewhere) FB50-FBFF are presentation forms. Since this range is supposed to be used as standalone characters - it won't work with fonts that change shape/height/size of character based on context. This effect can be easily seen in any modern non-monospaced font but it is very clearly visible in Nastaleeq type fonts. The usual range for Arabic text (as you can verify from any Arabic source on the web) is 0600-06FF.

Unicode doesn't have separate code points for Urdu and Arabic. Urdu, Arabic, Persian, Sindhi, all occupy the same code points i.e. 0600-06FF.

I really want a separate range for languages written in nastaleeq scripts, and this got my hopes too high. This makes some sense when you see how much Urdu text is mixed with Arabic, and how many Urdu language users also use Arabic but won't understand Urdu in Naskh or Arabic in Nastaleeq. Separate code ranges would allow fonts that can use Naskh for Arabic and Nastaleeq for Urdu and Persian in a single font, but they don't exist. Such fonts won't require Apple to have region specific font settings (or however they do the nastaleeq bit, idk) and sites like medium.com can have a font that defaults to serif in English, naskh in Arabic and nastaleeq in Urdu/Persian/South Asian languages.

Verification required that we are using correct Urdu characters|unicode points

We need verification for different resources that we are using correct Urdu Unicode points.

Currently, We are using the code points provided by the Unicode organization.
https://www.unicode.org/cldr/charts/34/collation/ur.html

Verification required that Urdu, Arabic and other languages are using same unicode code points.

Need to verify this statement.

Unicode doesn't have separate code points for Urdu and Arabic. Urdu, Arabic, Persian, Sindhi, all occupy the same code points i.e. 0600-06FF.