starwing / luautf8 Goto Github PK

View Code? Open in Web Editor NEW

398.0 23.0 67.0 676 KB

a utf-8 support module for Lua and LuaJIT.

License: MIT License

C 87.63% Lua 12.16% Makefile 0.21%

luautf8's Introduction

UTF-8 module for Lua 5.x

This module adds UTF-8 support to Lua.

It use data extracted from Unicode Character Database, and tested on Lua 5.2.3, Lua 5.3.0 and LuaJIT.

parseucd.lua is a pure Lua script generate unidata.h, to support convert characters and check characters' category.

It mainly used to compatible with Lua's own string module, it passed all string and pattern matching test in lua test suite2.

It also adds some useful routines against UTF-8 features, such as:

a convenient interface to escape Unicode sequence in string.
string insert/remove, since UTF-8 substring extract may expensive.
calculate Unicode width, useful when implement e.g. console emulator.
a useful interface to translate Unicode offset and byte offset.
checking UTF-8 strings for validity and removing invalid byte sequences.

Note that to avoid conflict with the Lua5.3's buitin library 'utf8', this library produce a file like lua-utf8.dll or lua-utf8.so. so use it like this:

local utf8 = require 'lua-utf8'

in your codes :-(

LuaRocks Installation

luarocks install luautf8

It's now full-compatible with Lua5.3's utf8 library, so replace this file (and headers) with lua5.3 source's lutf8lib.c is also okay.

Usage

Many routines are same as Lua's string module:

utf8.byte
utf8.char
utf8.find
utf8.gmatch
utf8.gsub
utf8.len
utf8.lower
utf8.match
utf8.reverse
utf8.sub
utf8.upper

The document of these functions can be find in Lua manual3.

Some routines in string module needn't support Unicode:

string.dump
string.format
string.rep

They are NOT in utf8 module.

Some routines are the compatible for Lua 5.3's basic UTF-8 support library:

utf8.offset
utf8.codepoint
utf8.codes

See Lua5.3's manual to get usage.

Some routines are new, with some Unicode-spec functions:

utf8.escape(str) -> utf8 string

escape a str to UTF-8 format string. It support several escape format:

%ddd - which ddd is a decimal number at any length: change Unicode code point to UTF-8 format.
%{ddd} - same as %nnn but has bracket around.
%uddd - same as %ddd, u stands Unicode
%u{ddd} - same as %{ddd}
%xhhh - hexadigit version of %ddd
%x{hhh} same as %xhhh.
%? - '?' stands for any other character: escape this character.

Examples:

local u = utf8.escape
print(u"%123%u123%{123}%u{123}%xABC%x{ABC}")
print(u"%%123%?%d%%u")

utf8.charpos(s[[, charpos], index]) -> charpos, code point

convert UTF-8 position to byte offset. if only index is given, return byte offset of this UTF-8 char index. if both charpos and index is given, a new charpos will be calculated, by add/subtract UTF-8 char index to current charpos. in all cases, it returns a new char position, and code point (a number) at this position.

utf8.next(s[, charpos[, index]]) -> charpos, code point

iterate though the UTF-8 string s. If only s is given, it can used as a iterator:

for pos, code in utf8.next, "utf8-string" do
   -- ...
end

if only charpos is given, return the next byte offset of in string. if charpos and index is given, a new charpos will be calculated, by add/subtract UTF-8 char offset to current charpos. in all case, it return a new char position (in bytes), and code point (a number) at this position.

utf8.insert(s[, idx], substring) -> new_string

insert a substring to s. If idx is given, insert substring before char at this index, otherwise substring will concat to s. idx can be negative.

utf8.remove(s[, start[, stop]]) -> new_string

delete a substring in s. If neither start nor stop is given, delete the last UTF-8 char in s, otherwise delete char from start to end of s. if stop is given, delete char from start to stop (include start and stop). start and stop can be negative.

utf8.width(s[, ambi_is_double[, default_width]]) -> width

calculate the width of UTF-8 string s. if ambi_is_double is given, the ambiguous width character's width is 2, otherwise it's 1. fullwidth/doublewidth character's width is 2, and other character's width is 1. if default_width is given, it will be the width of unprintable character, used display a non-character mark for these characters. if s is a code point, return the width of this code point.

utf8.widthindex(s, location[, ambi_is_double[, default_width]]) -> idx, offset, width

return the character index at given location in string s. this is a reverse operation of utf8.width(). this function return a index of location, and a offset in in UTF-8 encoding. e.g. if cursor is at the second column (middle) of the wide char, offset will be 2. the width of character at idx is returned, also.

utf8.title(s) -> new_string

utf8.fold(s) -> new_string

convert UTF-8 string s to title-case, or folded case used to compare by ignore case. if s is a number, it's treat as a code point and return a convert code point (number). utf8.lower/utf8.upper has the same extension.

utf8.ncasecmp(a, b) -> [-1,0,1]

compare a and b without case, -1 means a < b, 0 means a == b and 1 means a > b.

utf8.isvalid(s) -> boolean

check whether s is a valid UTF-8 string or not.

utf8.clean(s[, replacement_string]) -> cleaned_string, was_valid

replace any invalid UTF-8 byte sequences in s with the replacement string. if no replacement string is provided, the default is "�" (REPLACEMENT CHARACTER U+FFFD). note that any number of consecutive invalid bytes will be replaced by a single copy of the replacement string. the 2nd return value is true if the original string was already valid (meaning no replacements were made).

utf8.invalidoffset(s[, init]) -> offset

return the byte offset within s of the first invalid UTF-8 byte sequence. (1 is the first byte of the string.) if s is a valid UTF-8 string, return nil. the optional numeric argument init specifies where to start the search; its default value is 1 and can be negative.

utf8.isnfc(s) -> boolean

check whether s is in Normal Form C or not. "Normal Form C" means that whenever possible, combining marks are combined with a preceding codepoint. For example, instead of U+0041 (LATIN CAPITAL LETTER A) U+00B4 (ACUTE ACCENT), an NFC string will use U+00C1 (LATIN CAPITAL LETTER A WITH ACUTE). Also, some deprecated codepoints are converted to the recommended replacements. since the same sequence of characters can be represented in more than one way in Unicode, it is better to ensure strings are in Normal Form before comparing them. an error may be raised if s is not a valid UTF-8 string.

utf8.normalize_nfc(s) -> normal_string, was_nfc

convert s to Normal Form C. the 2nd return value is true if the original string was already in NFC (meaning no modifications were made). an error will be raised if s is not a valid UTF-8 string.

utf8.grapheme_indices(s[, start[, stop]]) -> iterator

return an iterator which yields the starting and ending byte index of each successive grapheme cluster in s. This range of bytes is inclusive of the endpoints, so the yielded values can be passed to string.sub to extract the grapheme cluster. if you provide start and stop byte indices, then the iterator will only cover the requested byte range. start and stop should fall on character boundaries, since an error will be raised if the requested byte range is not a valid UTF-8 string.

local i = 1
for from,to in utf8.grapheme_indices(s) do
  print("grapheme cluster "..i.." is from byte "..from.." to byte "..to)
  i = i + 1
end

Improvement needed

add Lua 5.3 spec test-suite.
more test case.
grapheme-compose support, and affect in utf8.reverse and utf8.width

License

It use same license with Lua: http://www.lua.org/license.html

luautf8's People

Contributors

Stargazers

Watchers

Forkers

formating brianuuuu zhuxiaomengjj cheyilin sysnajar dannote antoniomoder jakezhaojb newboy2004 lihuibin coronalabs caidongyun iamalbert rst256 summitn kerwinzxc infinityhacks jayhello yangdaxu bigcrush archoncap nonchip kn666 timusketeers mnemnion cooboyno111 ytvwld mpeterv hengle cherishpf mosjin slysven jansure lqt5 shmuz komainu8 lsx124 data-man see3p00 0xcds4r crackercat fsource zhaojun2066 johnd0e kibernet tekenlight zhaoguohao nathanrpage97 xiaolitongxue666 hyee cybersys enquora yolwoocle neon1ks aceyin alexdowad blogdron ebobrow dseeni sizolity sst-03 meowboy326 elv13 morew4rd rpg3d

luautf8's Issues

Function to "clean" input string from any invalid UTF-8 sequences?

There are cases where an application receives some arbitrary input from the 'outside world' (maybe from a network or read in from a file) and needs to pass it on to an upstream system which only accepts valid UTF-8. In such cases, it is useful to have a function which goes through the input and "cleans up" any invalid UTF-8 sequences, perhaps replacing them with an error marker like U+FFFD (REPLACEMENT CHARACTER).

Would you like to include something like that in this library? If so, I can send you a PR.

Erorr with LuaJit 2.1.0-beta3

Build:

gcc -O2 -fPIC -I/usr/include -c lutf8lib.c -o lutf8lib.o
gcc  -shared -o lua-utf8.so lutf8lib.o

Tested:

lua5.1 test.lua
OK

luajit test.lua

luajit: error loading module 'lua-utf8' from file './lua-utf8.so':
        ./lua-utf8.so: undefined symbol: lua_callk
stack traceback:
        [C]: at 0x7f0bd1841050
        [C]: in function 'require'
        test.lua:1: in main chunk
        [C]: at 0x56115b10acd0

Invalid byte sequence treating

I found that luautf8 in some cases treats some invalid sequences as correct utf8.
For example, utf8 string cannot consist of single char in range 128..255.
So when we execute following code in lua 5.3 we get no output:

for i=128,255 do
  local c = string.char(i)
  if utf8.len(c) then
    print(i)
  end
end

But luautf8 considers all such strings as valid.

Even more: luautf8 treats some invalid chars as punctuation:

for i=128,255 do
  local c = string.char(i)
  print(
    i,
    utf8.match(c,'%p'),
    string.match(c,'%p')
  )
end

Strange output with multi width emojis

Using LUA 5.1 on macOS

local utf8 = require 'utf8'

local t = {}
local str = "text 😅 👨‍💻 🇧🇷 🏳️‍🌈"
for i=1, utf8.len(str) do
	t[i]= (utf8.sub(str,i,i))
end
for k , v in pairs(t) do
	print(k,v)
end

Output:

utf8.find bug when init==0

string.find('1234','.*',0) -- 1,4 as expected
utf8.find('1234','.*',0)   -- 0,3 bug!

https://www.lua.org/manual/5.1/manual.html#pdf-string.find

FR: support for extended character class escapes in patterns

Just an idea. I need to match a sequence of letters and non-spacing marks, which can't be expressed in Lua patterns even with the extension of the meaning of escapes like %a of this module. Now it occurs to me that a possible solution would be if this module supported some extended character class escapes. I would love to do a PR but I don't do C.

Perhaps the most straight forward would be if %x{hhh} and the other escapes from utf8.escape could be used in patterns, including inside character classes so that [%a%x{300}-%x{36f}]+ would match letters followed by characters from the Combining Diacritical Marks block (although there are many non-spacing marks outside that block!)

A perhaps somewhat more key-hole-surgery solution would be a character class escape %m which matches any character with General Category M and its complement %M.

Somewhat more generally perhaps an escape pattern %g{Gc} (and complement %G{Gc}) where Gc is a one- or two-letter General Category abbreviation like L, Lu, Lo, M, Mn, P, Ps, Pe matching any character which does/doesn't belong to that General Category. The curlies would of course have to be required so that one can still use the regular character class %g including %g%{ with a following curly, or perhaps %k{Gc} as if "Kategory"!

The use case is a function for titlecasing words

-- Helper function
local function ul (u, l)
  return utf8.upper(u) .. utf8.lower(l)
end

local function title_case (s)
  -- Add flanking non-word chars so frontier assertion works at start/end
  s = '(' .. s .. ')'
  s = utf8.gsub(s,'%f[%w](%a)([^%s%d%p%c]*)%f[%W]', ul)
  -- Remove dummy parens
  return utf8.sub(s, 2, -2)
end

That [^%s%d%p%c]* has worked so far for my data but it's ugly, it works by accident and there may be things which it matches which it shouldn't although it seems this module includes GC S in %p.

utf8.offset behavior doesn't match standard Lua 5.3 library

Testcase:

local lua_utf8 = require "lua-utf8"
print(utf8.offset("foo", 1))
print(lua_utf8.offset("foo", 1))

Output:

1
2	111

Lua version: Lua 5.3.5 Copyright (C) 1994-2018 Lua.org, PUC-Rio
luautf8 version: luautf8 0.1.1-1 - A UTF-8 support module for Lua

lutf8.so: undefined symbol: luaL_checkint

Hello Xavier

I'm using lua 5.3 and luarocks 2.2.2 both compiled from sources. After installing lutf8 from sources and requiring it into my lua sources I'm getting the following error:

lutf8.so: undefined symbol: luaL_checkint

utf8 module was built from this rock src
https://luarocks.org/manifests/xavier-wang/lutf8-0.1-1.src.rock

Require string

Reported by @nadako

A require string that contains "-" undergoes a translation to "_" when looking for a C symbol.

luautf8 should use a require string and .so name of just "utf8" instead of the current "lua-utf8"

utf8.find not working on android in russian

local test_remove = "АБВГДИКЛМНОПРСТ"
local remove_word = "ГДИК"

local word_start, word_end = utf8.find(test_remove, remove_word,1, true)
--word_start and word_end are nil

on Mac OS all OK

utf8.remove no arguments doesn't work

###utf8.remove(s[, start[, stop]]) -> new_string
delete a substring in s. If neither start nor stop is given, delete the
last UTF-8 char in s

> print(utf8.remove("мясо"))
stdin:1: bad argument #2 to 'remove' (number expected, got no value)
stack traceback:
	[C]: in function 'remove'

Release 0.1.2

I would highly appreciate a new release (also uploaded to luarocks), since some important things have been fixed & improvement since release 0.1.1 from 5/2015.

charpattern is incompatible with lua 5.1

The hardcoded pattern UTF8PATT contains a zero byte, which is incompatible with Lua 5.1: "A pattern cannot contain embedded zeros. Use %z instead."

Is there a next release schedule for luautf8 ?

Hello,

luautf8 of the current version occurs compile error on CentOS6.
This problem had already resolved by 1c3148d.
However, there is not a release version that includes this commit yet.

Would you tell me when you will release the next version?

Thanks,

Efficient use of find/sub

Consider such example:

  repeat
    pos = utf8.find(str,"\\",pos+1)
  until not pos or utf8.sub(str,1,pos) ~= utf8.sub(str2,1,pos)

I have some doubt about efficiency of this code.

find in loop
pos value have to be translated to byte offset, and this get even worse with every iteration as we move further from beginning.
So question: is there some internal optimization for loop usage?
sub after find
It has to repeat exactly the same translation of pos, which already was done in find.
So question: how to rewrite above example in more efficient way?

undefined symbol: lua_tointeger

Default interpreter lua5.3,
Using snippet:

  local file = assert(io.open(fname, 'r'))
  local fdata = file:read("*all")
  print(utf8.lower(fdata))

produces the following error:

[tastyminerals@dell-tasty sandbox]$ lua test.lua news
lua: error loading module 'lua-utf8' from file '/usr/lib/lua/5.3/lua-utf8.so':
    /usr/lib/lua/5.3/lua-utf8.so: undefined symbol: lua_tointeger
stack traceback:
    [C]: in ?
    [C]: in function 'require'
    test.lua:3: in main chunk
    [C]: in ?

Can't pass pattern matching tests

Hi I got following errors: (LuaJIT 2.1 git HEAD)

$ luajit-2.1.0-alpha test_pm.lua
testing pattern matching
+
... luajit-2.1.0-alpha: test_pm.lua:14: invalid capture index %-47
stack traceback:
    [C]: in function 'match'
    test_pm.lua:14: in function 'f1'
    test_pm.lua:91: in main chunk
    [C]: at 0x00404250

test.lua and test_compat.lua are all passed without errors.

Is it a known issue or a bug? Should I concerned it? Thanks.

Implementing grapheme cluster segmentation support

Hey, @starwing... I am thinking of implementing grapheme cluster support for this library. I might do it very soon, or maybe later, but first it will be good to hear from you: Do you have any ideas what the API should look like?

Unicode how-much?

From the 0.1.3 release notes:

update Unicode version to 14

I'd be curious where you got your hands on that, it would give a lot of us a big head start! As far as I know 13 is the current version, announced on March 10th, 2020.

Also, this package on LuaRocks.org needs updating ;-)

Can I use it without installing via luarocks?

Hello can I use the library by downloading the folder but not installing it with luarocks?
I already tried putting the folder near to the lua file and doing require "luautf8" but this doesn't work.

I have much problems with luarocks. I cant install anything via luarocks.

Syntax error near char(144)

Nevermind, I've got it.

parseucd.lua is incompatible with lua 5.1

Unfortunately line 74 of parseucd.lua uses the goto command. 😱

This feature 🙄 was only introduced in Lua 5.2 so is not supported by 5.1 ...

This is awkward because the unidata.h file is likely to be way out of date by now - the last commit to touch it was entitled "update UCD to 8.0.0" and Unicode is currently at 12.1.0 - importantly some codepoints changed their width in Unicode 9 and I am suspecting this is affecting my project (with an embedded lua interpreted which is fixed to Lua 5.1) and I want to ensure all parts are using the same reference data...

utf8_encode() not implemented properly?

The following example

local utf8 = require("lua-utf8")
local ustr = utf8.char(0x20000)
print(string.format("U+%05x", utf8.codepoint(ustr)))

gives the result

U+003c0

I think there is something wrong in utf8_encode() in treating Unicode >= U+10000.
I am using luautf8 0.1.0-1 installed via LuaRocks on Lua5.2.

Access violation in multithreaded app

I'm trying to use your dll (which I sucessfully compiled) in multithreaded C/C++/Lua app. And I'm getting Access violation error.
I'm new to this, so my question is: Does your dll supports this? Is there a way to fix this error?

utf8.len() never return false when UTF8 has an invalid sequence

Hi,
I want to use utf8.len() on lua 5.1 to validate a UTF8 sequence but it does not do the job.

When I put a good sequence:

> s = '\207\180'
> len, inv = utf8.len(s)
> print(len, inv)
1       nil
> print(utf8.codepoint(s,1,-1))
1012

.. function works as expected, it prints the right codepoint and length, but when I put an invalid sequence:

> s = '\255\180'
> len, inv = utf8.len(s)
> print(len, inv)
2       nil
> print(utf8.codepoint(s,1,-1))
510     180

.. i t does not return the false value and the position of the first invalid byte, like explained here
Or maybe this module does not implement utf8 support like LUA 5.3 ?

Thanks.

utf8 decode

hello how to use it like I do it in python
s=b"\xD0\x9C\xD0\xBE\xD1\x81\xD0\xBA\xD0\xB2\xD0\xB0"

print(s.decode("utf-8"))
Москва
lua 5.2 works similary
lua5.2
Lua 5.2.4 Copyright (C) 1994-2015 Lua.org, PUC-Rio
s="\xD0\x9C\xD0\xBE\xD1\x81\xD0\xBA\xD0\xB2\xD0\xB0"
print(s)
Москва

s="\xD0\x9C\xD0\xBE\xD1\x81\xD0\xBA\xD0\xB2\xD0\xB0"
print(s)
xD0x9CxD0xBExD1x81xD0xBAxD0xB2xD0xB0

how to make similar as in python or lua 5.2 with this lib?
what function I need to use?

match failing

ut = require("lua-utf8")
print(ut.match("北京奥贝克电子股份有限公司（补证）",".-%(.-%)%s*$"))   -- Returns nil
print(string.match("Absadlhsfjh (jas) ",".-%(.-%)%s*$"))   -- Returns Absadlhsfjh (jas)

Provide a release tarball

It would be nice if you would consider adding proper source release tarballs,
as GitHub auto-generated tarballs are not immutable. This is an issue [1] for
operating systems creating packages for luautf8. Thanks in advance.

[1] https://marc.info/?l=openbsd-ports&m=151973450514279&w=2

-- deleted --

module 'lua-utf8' not found

Hi, I am using lua 5.4, i installed the luarocks and luarocks luautf8 package, but studio code searching the following locations for .so library
Desktop/server/lua-apps/lua.lua:3118: module 'lua-utf8' not found: no field package.preload['lua-utf8'] no file '/home/konstantinos/Desktop/server/lua-apps/lua-utf8.lua' no file '/home/konstantinos/Desktop/server/lua-apps/lua-utf8.lua' no file '/home/konstantinos/Desktop/server/lua-apps/lua-utf8.so' no file '/home/konstantinos/Desktop/server/lua-apps/lua-utf8.so'
I required as docs say in my code. Any suggestion?

rockspec 0.1.5-1 refers to the tarball 0.1.4

see https://github.com/starwing/luautf8/blob/master/rockspecs/luautf8-0.1.5-1.rockspec#L4-L5

MacOS luarockes install luautf8 module 'lua-utf8' not found

ERROR: upyun.lua:8: module 'lua-utf8' not found:
no field package.preload['lua-utf8']
no file '/usr/local/Cellar/openresty/1.19.3.1_1/site/lualib/lua-utf8.ljbc'
no file '/usr/local/Cellar/openresty/1.19.3.1_1/site/lualib/lua-utf8/init.ljbc'
no file '/usr/local/Cellar/openresty/1.19.3.1_1/lualib/lua-utf8.ljbc'
no file '/usr/local/Cellar/openresty/1.19.3.1_1/lualib/lua-utf8/init.ljbc'
no file '/usr/local/Cellar/openresty/1.19.3.1_1/site/lualib/lua-utf8.lua'
no file '/usr/local/Cellar/openresty/1.19.3.1_1/site/lualib/lua-utf8/init.lua'
no file '/usr/local/Cellar/openresty/1.19.3.1_1/lualib/lua-utf8.lua'
no file '/usr/local/Cellar/openresty/1.19.3.1_1/lualib/lua-utf8/init.lua'
no file './lua-utf8.lua'
no file '/usr/local/Cellar/openresty/1.19.3.1_1/luajit/share/luajit-2.1.0-beta3/lua-utf8.lua'
no file '/usr/local/share/lua/5.1/lua-utf8.lua'
no file '/usr/local/share/lua/5.1/lua-utf8/init.lua'
no file '/usr/local/Cellar/openresty/1.19.3.1_1/luajit/share/lua/5.1/lua-utf8.lua'
no file '/usr/local/Cellar/openresty/1.19.3.1_1/luajit/share/lua/5.1/lua-utf8/init.lua'
no file '/usr/local/Cellar/openresty/1.19.3.1_1/site/lualib/lua-utf8.so'
no file '/usr/local/Cellar/openresty/1.19.3.1_1/lualib/lua-utf8.so'
no file './lua-utf8.so'
no file '/usr/local/lib/lua/5.1/lua-utf8.so'
no file '/usr/local/Cellar/openresty/1.19.3.1_1/luajit/lib/lua/5.1/lua-utf8.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
upyun.lua:8: in function 'file_gen'
init_worker_by_lua:45: in function <init_worker_by_lua:43>
[C]: in function 'xpcall'
init_worker_by_lua:52: in function <init_worker_by_lua:50>