Code Monkey home page Code Monkey logo

rust-onig's Introduction

Rust Onig

Cargo Documentation CI Build status dependency status

Rust bindings for the Oniguruma regex library, a powerful and mature regular expression library with support for a wide range of character sets and language syntaxes. Oniguruma is written in C. This repository provides two crates: onig-sys which provides the raw Rust FFI bindings, and onig, which provides a safe Rust wrapper around them.

Documentation

Check out the module documentation to find out all the features that are available. To see some example usage of this crate take a look a the examples folder. The examples can be run from the command line with cargo run --example <examplename>.

Getting Started

Add the following to your Cargo.toml file:

[dependencies]
onig = "6"

Add the following extern to your crate root if you are not using edition 2018:

extern crate onig;

You can compile simple regular expressions with Regex::new, check if the pattern matches an entire &str with Regex::is_match and find matches within a &str with Regex::find. The onig crate also supplies more powerful versions of these methods which expose the wide range of options Oniguruma provides.

use onig::*;

let regex = Regex::new("e(l+)").unwrap();
for (i, pos) in regex.captures("hello").unwrap().iter_pos().enumerate() {
    match pos {
         Some((beg, end)) =>
             println!("Group {} captured in position {}:{}", i, beg, end),
         None =>
             println!("Group {} is not captured", i)
    }
}

Linking

If a version of Oniguruma can be found by pkg-config then that will be used. If not then Oniguruma will be compiled from source and linked to the onig-sys crate.

By default rust-onig will be statically linked to libonig. If you would rather that dynamic linking is used then the environment variables RUSTONIG_STATIC_LIBONIG and RUSTONIG_DYNAMIC_LIBONIG can be set. On *nix:

$ RUSTONIG_DYNAMIC_LIBONIG=1 cargo build

Or Windows:

> set RUSTONIG_DYNAMIC_LIBONIG=1
> cargo build

Build errors caused by libclang/llvm

By default onig uses bindgen to generate bindings for libonig. If you plan to only use the bundled version of libonig, you can make compilation faster and more reliable by disabling the default generate feature:

[dependencies]
onig = { version = "6", default-features = false }

Debugging

Sometimes it's useful to debug how Oniguruma parses, compiles, optimizes or executes a particular pattern.

When activating the print-debug feature for this crate, Oniguruma is compiled with debugging. Note that it's a compile-time setting, so you also need to make rust-onig not use the system Oniguruma by using RUSTONIG_SYSTEM_LIBONIG.

With all that combined, here's an example command to debug the pattern a|b:

RUSTONIG_SYSTEM_LIBONIG=0 cargo run --features print-debug --example capturedump 'a|b'

Supported Rust Versions

Rust Onig supports Rust 1.50.0 or later for Windows, Linux, and macOS. If the minimum supported rust version (MSRV) is changed then the minor version number will be increased. That is v6.4.x should always compile with the same version of the compiler.

Rust-Onig is Open Source

The contents of this repository are distributed under the MIT license. See LICENSE for more details. If you'd like to contribute take a look at our open easy issues.

rust-onig's People

Contributors

aomader avatar cogitri avatar cuviper avatar decathorpe avatar defuz avatar devurandom avatar est31 avatar ignatenkobrain avatar iwillspeak avatar kappa avatar kc1212 avatar keats avatar kevinnio avatar kornelski avatar lopopolo avatar lu-zero avatar messense avatar oronsh avatar pthariensflame avatar rivy avatar robinst avatar sethp avatar techcable avatar trishume avatar tyoverby avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

rust-onig's Issues

Implement `Iterator::size_hint` for our Iterator Implementations

We should implement the size_hint method when we know what it is. This is possible for things like captures and capture names where we know bounds.

We currently have the following iterators:

  • SubCaptures<'t>
  • SubCapturesPos<'t>
  • FindMatches<'r, 't>
  • FindCaptures<'r, 't>
  • RegexSplits<'r, 't>
  • RegexSplitsN<'r, 't>
  • CaptureNames<'r>
  • CaptureTreeNodeIter<'t>

Provide some way to compile Oniguruma with debug defines

I recently investigated why Oniguruma was fast/slow for some regexes with catastrophic backtracking.

What was really useful was uncommenting the ONIG_DEBUG_PARSE, ONIG_DEBUG_COMPILE defines in regint.h.

With the library statically compiled, I could then do a cargo run --example capturedump 'a+' and see Oniguruma's debug output:

Compiling 'a+'

PATTERN: /a+/
<quantifier:0x7fa3f6500380>{1,-1}
   <string:0x7fa3f6500340>a
optimize: EXACT
  anchor: []
  sub anchor: []

exact: [a]: length: 1
bt_mem_start: 0x0, bt_mem_end: 0x0
code-length: 15
   0: exact1:a
   2: push:{7/14}
   7: exact1:a
   9: jump:{-12/2}
  14: end

The reason I'm creating this issue is because I'm wondering if you're interested in making this even easier? Would it be possible to add a feature (or something similar) so that modifying the oniguruma source code isn't even necessary? A single cargo command to get that output would be really neat.

(Also, I'm willing to do a PR if you think that's a good idea.)

Re-Create the Oniguruma Samples

The oniguruma samples directory contains some useful examples of the API. We should convert these into runnable examples in this project.

  • sample/simple.c
  • sample/names.c (@b52)
  • sample/encode.c (@iwillspeak)
  • sample/listcap.c
  • sample/scan.c
  • sample/sql.c
  • sample/user_property.c
  • sample/syntax.c
  • sample/crnl.c

Anyone is free to shout up and claim any of these.

Properly Fix Encoding and Syntax Type Enumerations in `onig_sys`

Currently these enumerations are just collections of constants in their own modules. This seems a bit of a hack. They should probably be replaced by real rust enums.

Once this is done the enums should be re exported from the parent crate, and the dead code attributes removed.

onig_sys doesn't build with musl anymore

With v2.0.2, try to build in the clux/muslrust Docker image:

docker run -v "$PWD:/volume" -w /volume -t clux/muslrust:1.21.0-2017-10-12 cargo build

The following error occurs:

    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading lazy_static v0.2.9
 Downloading libc v0.2.33
 Downloading bitflags v1.0.1
 Downloading pkg-config v0.3.9
 Downloading cmake v0.1.27
 Downloading cc v1.0.3
   Compiling cc v1.0.3
   Compiling bitflags v1.0.1
   Compiling pkg-config v0.3.9
   Compiling lazy_static v0.2.9
   Compiling libc v0.2.33
   Compiling cmake v0.1.27
   Compiling onig_sys v66.1.0 (file:///volume/onig_sys)
error: failed to run custom build command for `onig_sys v66.1.0 (file:///volume/onig_sys)`
process didn't exit successfully: `/volume/target/debug/build/onig_sys-ec1c5b4d89d47781/build-script-build` (exit code: 101)
--- stdout
running: "cmake" "/volume/onig_sys/oniguruma" "-DCMAKE_MACOSX_RPATH=NO" "-DCMAKE_INSTALL_PREFIX=/volume/target/x86_64-unknown-linux-musl/debug/build/onig_sys-2364d8247dd85822/out" "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64 -static" "-DCMAKE_C_COMPILER=/usr/bin/musl-gcc" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64 -static" "-DCMAKE_CXX_COMPILER=musl-g++" "-DCMAKE_BUILD_TYPE=Debug"
-- The C compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/musl-gcc
-- Check for working C compiler: /usr/bin/musl-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Looking for alloca
-- Looking for alloca - not found
-- Looking for include file alloca.h
-- Looking for include file alloca.h - found
-- Looking for include file stdarg.h
-- Looking for include file stdarg.h - found
-- Looking for include file stdint.h
-- Looking for include file stdint.h - found
-- Looking for include file stdlib.h
-- Looking for include file stdlib.h - found
-- Looking for include file strings.h
-- Looking for include file strings.h - found
-- Looking for include file string.h
-- Looking for include file string.h - found
-- Looking for include file sys/times.h
-- Looking for include file sys/times.h - found
-- Looking for include file sys/time.h
-- Looking for include file sys/time.h - found
-- Looking for include file sys/types.h
-- Looking for include file sys/types.h - found
-- Looking for include file unistd.h
-- Looking for include file unistd.h - found
-- Looking for include file inttypes.h
-- Looking for include file inttypes.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of int
-- Check size of int - done
-- Check size of long
-- Check size of long - done
-- Check size of short
-- Check size of short - done
-- Looking for 4 include files stdlib.h, ..., float.h
-- Looking for 4 include files stdlib.h, ..., float.h - found
-- Configuring done
-- Generating done
-- Build files have been written to: /volume/target/x86_64-unknown-linux-musl/debug/build/onig_sys-2364d8247dd85822/out/build
running: "cmake" "--build" "." "--target" "install" "--config" "Debug" "--"
Scanning dependencies of target onig
[  1%] Building C object CMakeFiles/onig.dir/src/regerror.c.o
[  3%] Building C object CMakeFiles/onig.dir/src/regcomp.c.o
[  5%] Building C object CMakeFiles/onig.dir/src/regparse.c.o
[  7%] Building C object CMakeFiles/onig.dir/src/regext.c.o
[ 11%] Building C object CMakeFiles/onig.dir/src/regexec.c.o
[ 11%] Building C object CMakeFiles/onig.dir/src/reggnu.c.o
[ 13%] Building C object CMakeFiles/onig.dir/src/regenc.c.o
[ 15%] Building C object CMakeFiles/onig.dir/src/regsyntax.c.o
[ 19%] Building C object CMakeFiles/onig.dir/src/regtrav.c.o
[ 19%] Building C object CMakeFiles/onig.dir/src/regversion.c.o
[ 21%] Building C object CMakeFiles/onig.dir/src/st.c.o
[ 23%] Building C object CMakeFiles/onig.dir/src/regposix.c.o
[ 25%] Building C object CMakeFiles/onig.dir/src/regposerr.c.o
[ 27%] Building C object CMakeFiles/onig.dir/src/onig_init.c.o
[ 29%] Building C object CMakeFiles/onig.dir/src/unicode.c.o
[ 31%] Building C object CMakeFiles/onig.dir/src/ascii.c.o
[ 33%] Building C object CMakeFiles/onig.dir/src/utf8.c.o
[ 35%] Building C object CMakeFiles/onig.dir/src/utf16_be.c.o
[ 37%] Building C object CMakeFiles/onig.dir/src/utf16_le.c.o
[ 39%] Building C object CMakeFiles/onig.dir/src/utf32_be.c.o
[ 41%] Building C object CMakeFiles/onig.dir/src/utf32_le.c.o
[ 43%] Building C object CMakeFiles/onig.dir/src/euc_jp.c.o
[ 45%] Building C object CMakeFiles/onig.dir/src/sjis.c.o
[ 47%] Building C object CMakeFiles/onig.dir/src/iso8859_1.c.o
[ 49%] Building C object CMakeFiles/onig.dir/src/iso8859_2.c.o
[ 50%] Building C object CMakeFiles/onig.dir/src/iso8859_3.c.o
[ 52%] Building C object CMakeFiles/onig.dir/src/iso8859_4.c.o
[ 54%] Building C object CMakeFiles/onig.dir/src/iso8859_5.c.o
[ 56%] Building C object CMakeFiles/onig.dir/src/iso8859_6.c.o
[ 58%] Building C object CMakeFiles/onig.dir/src/iso8859_7.c.o
[ 60%] Building C object CMakeFiles/onig.dir/src/iso8859_8.c.o
[ 62%] Building C object CMakeFiles/onig.dir/src/iso8859_9.c.o
[ 64%] Building C object CMakeFiles/onig.dir/src/iso8859_10.c.o
[ 66%] Building C object CMakeFiles/onig.dir/src/iso8859_11.c.o
[ 68%] Building C object CMakeFiles/onig.dir/src/iso8859_13.c.o
[ 70%] Building C object CMakeFiles/onig.dir/src/iso8859_14.c.o
[ 72%] Building C object CMakeFiles/onig.dir/src/iso8859_15.c.o
[ 74%] Building C object CMakeFiles/onig.dir/src/iso8859_16.c.o
[ 78%] Building C object CMakeFiles/onig.dir/src/euc_tw.c.o
[ 78%] Building C object CMakeFiles/onig.dir/src/euc_kr.c.o
[ 80%] Building C object CMakeFiles/onig.dir/src/big5.c.o
[ 82%] Building C object CMakeFiles/onig.dir/src/gb18030.c.o
[ 84%] Building C object CMakeFiles/onig.dir/src/koi8_r.c.o
[ 86%] Building C object CMakeFiles/onig.dir/src/cp1251.c.o
[ 88%] Building C object CMakeFiles/onig.dir/src/euc_jp_prop.c.o
[ 90%] Building C object CMakeFiles/onig.dir/src/sjis_prop.c.o
[ 92%] Building C object CMakeFiles/onig.dir/src/unicode_unfold_key.c.o
[ 94%] Building C object CMakeFiles/onig.dir/src/unicode_fold1_key.c.o
[ 96%] Building C object CMakeFiles/onig.dir/src/unicode_fold2_key.c.o
[ 98%] Building C object CMakeFiles/onig.dir/src/unicode_fold3_key.c.o
[100%] Linking C shared library libonig.so
CMakeFiles/onig.dir/build.make:1368: recipe for target 'libonig.so' failed
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/onig.dir/all' failed
Makefile:149: recipe for target 'all' failed

--- stderr
DIST_NAME: onig
DIST_VERSION: 6.6.1
DIST_LICENSE: BSD
DIST_AUTHOR: K.Kosako
DIST_MAINTAINER: K.Kosako
DIST_URL: https://github.com/kkos/oniguruma
DIST_DESC: Oniguruma is a regular expressions library.
DIST_DEPENDS:
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_CXX_COMPILER
    CMAKE_CXX_FLAGS


/usr/bin/ld: /usr/lib/x86_64-linux-musl/libc.a(__stack_chk_fail.o): relocation R_X86_64_32 against `__stack_chk_guard' can not be used when making a shared object; recompile with -fPIC
/usr/lib/x86_64-linux-musl/libc.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
make[2]: *** [libonig.so] Error 1
make[1]: *** [CMakeFiles/onig.dir/all] Error 2
make: *** [all] Error 2
thread 'main' panicked at '
command did not execute successfully, got: exit code: 2

build script failed, must exit now', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.27/src/lib.rs:627:4
note: Run with `RUST_BACKTRACE=1` for a backtrace.

Same problem with v2.0.0. But it worked with v1.6.1 (note that you have to clean up things when switching: rm -rf Cargo.lock onig_sys/Cargo.lock target onig_sys/target).

Latest onig-sys fails to build on Travis

See https://travis-ci.org/trishume/syntect/builds/250590654 for the build log.

It's mostly a bunch of errors like this:

Scanning dependencies of target onig
[  2%] [  4%] Building C object CMakeFiles/onig.dir/src/regerror.c.o
[  6%] [  8%] Building C object CMakeFiles/onig.dir/src/regparse.c.o
In file included from /usr/include/stdlib.h:320:0,
                 from /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/onig_sys-64.0.0/oniguruma/src/regint.h:144,
                 from /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/onig_sys-64.0.0/oniguruma/src/regerror.c:30:
/usr/include/x86_64-linux-gnu/sys/types.h:66:17: error: two or more data types in declaration specifiers
/usr/include/x86_64-linux-gnu/sys/types.h:81:17: error: two or more data types in declaration specifiers

@iwillspeak

names::tests::test_regex_names Crashes on Windows 64 Bit

On the AppVeyor build the 64 bit tests fail. This looks to be due to a segfault.

The test which fails appears to be names::tests::test_regex_names. This enumerates capture names iterator. I suspect this is because back_ref1 and back_refs in NameEntry areu32 rather than c_int.

32 bit Windows Build Failures

For some reason the windows i686 build started failing after updating to Oniguruma 6.9.1 (3deb557). Not sure if this is due to an API change on Oniguruma's part. Guessing that the test failure is due to a segfault or something as it isn't listed properly in the build output.

Create and Search with Byte-Based Strings

Currently we're missing a major feature of Oniguruma by not exposing the ability to create regular expressions in arbitrary encodings and to search strings in arbitrary encodings.

We can either add a new set of functions to the API to parallel the &str based one, or we could create a new trait OnigStr which exposes the encoding, and bytes that is being searched and implement it for &str. We could then provide another implementation where you specify an encoding and byte slice. This would be a breaking change to the exported interface though.

Build breaks on Windows with nightly rustc

Trying to build rust-onig on Windows with the nightly toolchain gives this for me:

error: failed to run custom build command for `onig_sys v63.0.2`
process didn't exit successfully: `c:\Users\Prin\build\oxidoc\target\debug\build
\onig_sys-165165d91a6214d3\build-script-build` (exit code: 101)
--- stderr
thread 'main' panicked at 'Build error:
STDERR:
Microsoft (R) Program Maintenance Utility Version 14.00.24210.0
Copyright (C) Microsoft Corporation.  All rights reserved.

NMAKE : fatal error U1065: invalid option '-'
Stop.

STDOUT:
C:\Users\Prin\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-63.0.2\on
iguruma>cd src

C:\Users\Prin\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-63.0.2\on
iguruma\src>copy config.h.win64 config.h
        1 file(s) copied.

C:\Users\Prin\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-63.0.2\on
iguruma\src>nmake -f Makefile.windows
', C:\Users\Prin\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-63.0.2
\build.rs:57
note: Run with `RUST_BACKTRACE=1` for a backtrace.

After looking around, I noticed the issue might have to do with the MAKEFLAGS environment variable. I made sure to unset it before running cargo build but the error still happens. However, running an env | grep MAKE just before the build command, it shows that MAKEFLAGS is set to this:

MAKEFLAGS=--jobserver-fds=__rust_jobserver_semaphore_2345838727 --jobserver-auth
=__rust_jobserver_semaphore_2345838727

These are messing up the nmake command. Unsetting them just before the nmake command in make_win64.bat lets the build run normally. Apparently this commit added some arguments to MAKEFLAGS that confuse nmake.

> cl
Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24215.1 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

> rustc --version
rustc 1.19.0-nightly (fe7227f6c 2017-06-16)

Document the `tree` Module

The tree module (src/tree.rs) Isn't properly documentaed and is currently marked with an #[allow(missing_docs)] attribute.

Add documentation for each item in the module and remove the #[allow(missing_docs)] attribute from lib.rs.

I'm willing to provide help and support if anyone would like to give this a go. Even documenting part of the module would be a great help.

is_match() and find().is_some() give different results

When working on Advent of code examples, I came across this issue. is_match method gives different results from find followed by is_some although I think it should return the same.

For instance:

extern crate onig;

use onig::Regex;

fn main() {
    let ipv7 = "abba[mnop]qrst";
    let abba = Regex::new(r#"(\w)(?!\1)(\w)\2\1"#).unwrap();
    if abba.find(ipv7).is_some() {
        println!("Found pattern");
    } else {
        println!("Pattern not found");
    }
    if abba.is_match(ipv7) {
        println!("Found pattern");
    } else {
        println!("Pattern not found");
    }

}

Gives this output:

$ cargo run
Found pattern
Pattern not found

Is this a bug or am I using it wrong?

Add New API Surface from Oniguruma 6.7.0

The 6.7.0 version of Oniguruma adds a new synatx type declaration (ONIG_SYNTAX_ONIGURUMA). We should add this to the onig_sys crate too.

This is up for grabs if anyone is interested before I get round to it.

Make Static Linking the default for _all_ Platforms

Following on from #38, #63 and the discussion around the removal of feature-flag linking.

At the moment the default link type is static for MUSL and MSVC, and dynamic for all other platforms. There is an argument that dynamic linking goes against the default expectations of Rustaceans, with crates normally being statically linked into the final binary. The counter argument is that we should try to link to libraries in the way that most fits in with the target platform's social mores.

So, the question is "should static linking be the default?"

For those in favour of changing static linking to be the default for all platforms, not just MUSL and MSVC react with 👍 . For those in favour of dynamic linking react with 👎 .

The proposal to merge the two projects

Hello!

A month ago, I needed binding for oniguruma library in Rust, and I found your project. Unfortunately, it did not implement everything that I needed, and looked as if you do not maintain it anymore. So, I decided to use your achievements to make a more comprehensive binding and put it in crates (repo, crate). I hope you do not mind this. I wanted to contact you, but unfortunately did not find your email.

Today I noticed that you continue to develop your project. I believe that we have common goals, so I propose to merge them into one, so as not to perform the same work twice. My main vision is to realize the full binding, which reproduces API of regex library as much as possible, but at the same time provides all the advantages of the Oniguruma library (using Regions, capture tree traversing, etc)

I will be glad to hear what you think about it.

include LICENSE into onig_sys crate

It would be nice if you could include license file into onig_sys directory so that archive on crates.io contains it. If you don't care about git repo on Windows, then you can do simple symlink and publish new version.

Thanks!

onig_sys doesn't build with musl

muslrust is a Docker image for building Rust projects using musl libc. The nice thing about it is that it can produce statically linked binaries that don't have any dependencies (not even to libc).

It looks like rust-onig currently fails to link in muslrust:

$ docker run -v $PWD:/volume -w /volume -t clux/muslrust cargo build --release
    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading onig v1.1.0
 Downloading libc v0.2.16
 Downloading onig_sys v61.1.0
 Downloading bitflags v0.7.0
 Downloading lazy_static v0.2.1
 Downloading pkg-config v0.3.8
 Downloading cmake v0.1.17
 Downloading gcc v0.3.35
   Compiling bitflags v0.7.0
   Compiling lazy_static v0.2.1
   Compiling pkg-config v0.3.8
   Compiling gcc v0.3.35
   Compiling libc v0.2.16
   Compiling cmake v0.1.17
   Compiling onig_sys v61.1.0
error: failed to run custom build command for `onig_sys v61.1.0`
process didn't exit successfully: `/volume/target/release/build/onig_sys-1a89ba0e5167b8c5/build-script-build` (exit code: 101)
--- stdout
running: "cmake" "/root/.cargo/registry/src/github.com-1ecc6299db9ec823/onig_sys-61.1.0/oniguruma" "-DCMAKE_MACOSX_RPATH=NO" "-DCMAKE_INSTALL_PREFIX=/volume/target/x86_64-unknown-linux-musl/release/build/onig_sys-1a89ba0e5167b8c5/out" "-DCMAKE_C_FLAGS= -O0 -ffunction-sections -fdata-sections -m64 -fPIC -static" "-DCMAKE_C_COMPILER=/usr/bin/musl-gcc" "-DCMAKE_CXX_FLAGS= -O0 -ffunction-sections -fdata-sections -m64 -fPIC -static" "-DCMAKE_CXX_COMPILER=musl-g++" "-DCMAKE_BUILD_TYPE=Release"
-- The C compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/musl-gcc
-- Check for working C compiler: /usr/bin/musl-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Looking for alloca
-- Looking for alloca - not found
-- Looking for include file alloca.h
-- Looking for include file alloca.h - found
-- Looking for include file stdarg.h
-- Looking for include file stdarg.h - found
-- Looking for include file stdint.h
-- Looking for include file stdint.h - found
-- Looking for include file stdlib.h
-- Looking for include file stdlib.h - found
-- Looking for include file strings.h
-- Looking for include file strings.h - found
-- Looking for include file string.h
-- Looking for include file string.h - found
-- Looking for include file sys/times.h
-- Looking for include file sys/times.h - found
-- Looking for include file sys/time.h
-- Looking for include file sys/time.h - found
-- Looking for include file sys/types.h
-- Looking for include file sys/types.h - found
-- Looking for include file unistd.h
-- Looking for include file unistd.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of int
-- Check size of int - done
-- Check size of long
-- Check size of long - done
-- Check size of short
-- Check size of short - done
-- Looking for 4 include files stdlib.h, ..., float.h
-- Looking for 4 include files stdlib.h, ..., float.h - found
-- Configuring done
-- Generating done
-- Build files have been written to: /volume/target/x86_64-unknown-linux-musl/release/build/onig_sys-1a89ba0e5167b8c5/out/build
running: "cmake" "--build" "." "--target" "install" "--config" "Release" "--" "-j4"
Scanning dependencies of target onig
[  1%] Building C object CMakeFiles/onig.dir/src/regerror.c.o
[  3%] Building C object CMakeFiles/onig.dir/src/regparse.c.o
[  5%] Building C object CMakeFiles/onig.dir/src/regext.c.o
[  7%] Building C object CMakeFiles/onig.dir/src/regcomp.c.o
[  9%] Building C object CMakeFiles/onig.dir/src/regexec.c.o
[ 11%] Building C object CMakeFiles/onig.dir/src/reggnu.c.o
[ 13%] Building C object CMakeFiles/onig.dir/src/regenc.c.o
[ 15%] Building C object CMakeFiles/onig.dir/src/regsyntax.c.o
[ 17%] Building C object CMakeFiles/onig.dir/src/regtrav.c.o
[ 19%] Building C object CMakeFiles/onig.dir/src/regversion.c.o
[ 21%] Building C object CMakeFiles/onig.dir/src/st.c.o
[ 23%] Building C object CMakeFiles/onig.dir/src/regposix.c.o
[ 25%] Building C object CMakeFiles/onig.dir/src/regposerr.c.o
[ 27%] Building C object CMakeFiles/onig.dir/src/onig_init.c.o
[ 29%] Building C object CMakeFiles/onig.dir/src/unicode.c.o
[ 31%] Building C object CMakeFiles/onig.dir/src/ascii.c.o
[ 33%] Building C object CMakeFiles/onig.dir/src/utf8.c.o
[ 35%] Building C object CMakeFiles/onig.dir/src/utf16_be.c.o
[ 37%] Building C object CMakeFiles/onig.dir/src/utf16_le.c.o
[ 39%] Building C object CMakeFiles/onig.dir/src/utf32_be.c.o
[ 41%] Building C object CMakeFiles/onig.dir/src/utf32_le.c.o
[ 43%] Building C object CMakeFiles/onig.dir/src/euc_jp.c.o
[ 45%] Building C object CMakeFiles/onig.dir/src/sjis.c.o
[ 47%] Building C object CMakeFiles/onig.dir/src/iso8859_1.c.o
[ 49%] Building C object CMakeFiles/onig.dir/src/iso8859_2.c.o
[ 50%] Building C object CMakeFiles/onig.dir/src/iso8859_3.c.o
[ 52%] Building C object CMakeFiles/onig.dir/src/iso8859_4.c.o
[ 54%] Building C object CMakeFiles/onig.dir/src/iso8859_5.c.o
[ 56%] Building C object CMakeFiles/onig.dir/src/iso8859_6.c.o
[ 58%] Building C object CMakeFiles/onig.dir/src/iso8859_7.c.o
[ 60%] Building C object CMakeFiles/onig.dir/src/iso8859_8.c.o
[ 62%] Building C object CMakeFiles/onig.dir/src/iso8859_9.c.o
[ 64%] Building C object CMakeFiles/onig.dir/src/iso8859_10.c.o
[ 66%] Building C object CMakeFiles/onig.dir/src/iso8859_11.c.o
[ 68%] Building C object CMakeFiles/onig.dir/src/iso8859_13.c.o
[ 70%] Building C object CMakeFiles/onig.dir/src/iso8859_15.c.o
[ 72%] Building C object CMakeFiles/onig.dir/src/iso8859_16.c.o
[ 74%] Building C object CMakeFiles/onig.dir/src/iso8859_14.c.o
[ 76%] Building C object CMakeFiles/onig.dir/src/euc_tw.c.o
[ 78%] Building C object CMakeFiles/onig.dir/src/euc_kr.c.o
[ 80%] Building C object CMakeFiles/onig.dir/src/big5.c.o
[ 82%] Building C object CMakeFiles/onig.dir/src/gb18030.c.o
[ 84%] Building C object CMakeFiles/onig.dir/src/koi8_r.c.o
[ 86%] Building C object CMakeFiles/onig.dir/src/cp1251.c.o
[ 88%] Building C object CMakeFiles/onig.dir/src/euc_jp_prop.c.o
[ 90%] Building C object CMakeFiles/onig.dir/src/sjis_prop.c.o
[ 92%] Building C object CMakeFiles/onig.dir/src/unicode_unfold_key.c.o
[ 94%] Building C object CMakeFiles/onig.dir/src/unicode_fold1_key.c.o
[ 96%] Building C object CMakeFiles/onig.dir/src/unicode_fold2_key.c.o
[ 98%] Building C object CMakeFiles/onig.dir/src/unicode_fold3_key.c.o
[100%] Linking C shared library libonig.so
CMakeFiles/onig.dir/build.make:1368: recipe for target 'libonig.so' failed
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/onig.dir/all' failed
Makefile:149: recipe for target 'all' failed

--- stderr
DIST_NAME: onig
DIST_VERSION: 6.1.1
DIST_LICENSE: BSD
DIST_AUTHOR: K.Kosako
DIST_MAINTAINER: K.Kosako
DIST_URL: https://github.com/kkos/oniguruma
DIST_DESC: Oniguruma is a regular expressions library.
DIST_DEPENDS: 
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_CXX_COMPILER
    CMAKE_CXX_FLAGS


/usr/bin/ld: /usr/lib/x86_64-linux-musl/libc.a(__stack_chk_fail.o): relocation R_X86_64_32 against `__stack_chk_guard' can not be used when making a shared object; recompile with -fPIC
/usr/lib/x86_64-linux-musl/libc.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
make[2]: *** [libonig.so] Error 1
make[1]: *** [CMakeFiles/onig.dir/all] Error 2
make: *** [all] Error 2
thread 'main' panicked at '
command did not execute successfully, got: exit code: 2

build script failed, must exit now', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.17/src/lib.rs:463
note: Run with `RUST_BACKTRACE=1` for a backtrace.

You should be able to reproduce it using this test repo.

Would be very cool to make this work :).

Can not build in MSVC

It seemed CMake is not working in oniguruma with MSVC, but use make_win64.bat works.

Linker error on Windows

When I was starting to use syntect again on Windows 8.1, I was getting a bunch of linker errors against onig that I hadn't seen before.

   Compiling oni v0.1.0 (file:///C:/Users/Prin/build/oni)
error: linking with `link.exe` failed: exit code: 1120
  |
  = note: libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_region_copy referenced in function _ZN4onig6
region6Region14clone_from_raw17h90a2926bd02a4b2eE
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_region_clear referenced in function _ZN4onig
6region6Region5clear17h0943019933f017d6E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_region_resize referenced in function _ZN4oni
g6region6Region7reserve17h2e21b8eba1c7c698E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_get_capture_tree referenced in function _ZN4
onig6region6Region4tree17haa92f78c975224a1E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_region_free referenced in function _ZN62_$LT
$onig..region..Region$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdb00ed37907068
3bE
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_number_of_names referenced in function _ZN4o
nig5names29_$LT$impl$u20$onig..Regex$GT$17capture_names_len17h0d23453bea6026a4E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol OnigSyntaxASIS referenced in function _ZN4onig6sy
ntax6Syntax4asis17h07051850d7ac02f8E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol OnigSyntaxPosixBasic referenced in function _ZN4o
nig6syntax6Syntax11posix_basic17h8f94c6a1d7d0fd27E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol OnigSyntaxPosixExtended referenced in function _Z
N4onig6syntax6Syntax14posix_extended17hefc2c6558a4cfe6bE
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol OnigSyntaxEmacs referenced in function _ZN4onig6s
yntax6Syntax5emacs17hc0011254d42ea637E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol OnigSyntaxGrep referenced in function _ZN4onig6sy
ntax6Syntax4grep17h23b28b15c2e57fd9E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol OnigSyntaxGnuRegex referenced in function _ZN4oni
g6syntax6Syntax9gnu_regex17haffc7df923584aacE
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol OnigSyntaxJava referenced in function _ZN4onig6sy
ntax6Syntax4java17h03df64086a6bd097E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol OnigSyntaxPerl referenced in function _ZN4onig6sy
ntax6Syntax4perl17ha62f1f23ef287413E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol OnigSyntaxPerl_NG referenced in function _ZN4onig
6syntax6Syntax7perl_ng17h98a400f2eb9a7246E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol OnigSyntaxRuby referenced in function _ZN4onig6sy
ntax6Syntax4ruby17heae472d73ebe8cd9E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol OnigDefaultSyntax referenced in function _ZN4onig
6syntax6Syntax7default17h19f130e4c5d89b88E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_get_syntax_op referenced in function _ZN4oni
g6syntax6Syntax9operators17h9cbbd6619bb7b7fbE
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_get_syntax_op2 referenced in function _ZN4on
ig6syntax6Syntax9operators17h9cbbd6619bb7b7fbE
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_set_syntax_op referenced in function _ZN4oni
g6syntax6Syntax13set_operators17h0b644e6f76214633E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_set_syntax_op2 referenced in function _ZN4on
ig6syntax6Syntax13set_operators17h0b644e6f76214633E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_get_syntax_behavior referenced in function _
ZN4onig6syntax6Syntax8behavior17h9273e5477819f7bfE
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_set_syntax_behavior referenced in function _
ZN4onig6syntax6Syntax12set_behavior17h8168d4ec28e4ada5E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_get_syntax_options referenced in function _Z
N4onig6syntax6Syntax7options17ha03142a5d6915c1fE
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_set_syntax_options referenced in function _Z
N4onig6syntax6Syntax11set_options17hbbd66aef47c49219E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_set_meta_char referenced in function _ZN4oni
g6syntax6Syntax13set_meta_char17haff62794d01ca9beE
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_version referenced in function _ZN4onig5util
s7version17hc4f4c7df30b43442E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_copyright referenced in function _ZN4onig5ut
ils9copyright17hd41cf1c5e8327d73E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_unicode_define_user_property referenced in f
unction _ZN4onig5utils20define_user_property17hae57edc7cfaf4c09E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol OnigEncodingUTF8 referenced in function _ZN4onig7
buffers12EncodedChars8encoding17habb540862bafa864E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol OnigEncodingASCII referenced in function _ZN4onig
7buffers12EncodedBytes5ascii17h6d1d7a00f651e984E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_error_code_to_str referenced in function _ZN
4onig5Error3new17h6ee23316d404a59bE
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_new referenced in function _ZN4onig5Regex25w
ith_options_and_encoding17h319c120f6d3f1061E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_match referenced in function _ZN4onig5Regex1
9match_with_encoding17heade9c40a2d3c3e7E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_search referenced in function _ZN4onig5Regex
20search_with_encoding17h481a573cb3a515d5E
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_get_encoding referenced in function _ZN4onig
5Regex8encoding17hfe75399f662b5e3fE
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_number_of_captures referenced in function _Z
N4onig5Regex12captures_len17h8090d0cb009b54caE
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_number_of_capture_histories referenced in fu
nction _ZN4onig5Regex21capture_histories_len17hde151228c0d2622bE
          libonig-84ca367bb0b16366.rlib(onig-84ca367bb0b16366.0.o) : error LNK20
19: unresolved external symbol onig_free referenced in function _ZN53_$LT$onig..
Regex$u20$as$u20$core..ops..drop..Drop$GT$4drop17hc1663ba6a945dd84E
          C:\Users\Prin\build\oni\target\debug\deps\oni-5fdd0388e1ac1608.exe : f
atal error LNK1120: 39 unresolved externals

Code used:

extern crate onig;

use onig::*;

fn main() {
    Regex::new("");
}
> cl
Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24215.1 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

> rustc --version
rustc 1.18.0 (03fc9d622 2017-06-06)

git clone && cargo build Fails

git clone repo
cargo build

Updating registry https://github.com/rust-lang/crates.io-index
Compiling libc v0.2.26
Compiling bitflags v0.7.0
Compiling lazy_static v0.2.8
Compiling gcc v0.3.51
Compiling pkg-config v0.3.9
Compiling cmake v0.1.24
Compiling onig_sys v64.0.0 (file:///home/yash/Projects/Fresh/gui/research-sources/rust-onig/onig_sys)
error: failed to run custom build command for onig_sys v64.0.0 (file:///home/yash/Projects/Fresh/gui/research-sources/rust-onig/onig_sys)
process didn't exit successfully: /home/yash/Projects/Fresh/gui/research-sources/rust-onig/target/debug/build/onig_sys-7db7e7810c98c6f1/build-script-build (exit code: 101)
--- stdout
running: "cmake" "/home/yash/Projects/Fresh/gui/research-sources/rust-onig/onig_sys/oniguruma" "-DBUILD_SHARED_LIBS=OFF" "-DCMAKE_INSTALL_PREFIX=/home/yash/Projects/Fresh/gui/research-sources/rust-onig/target/debug/build/onig_sys-4629b99d11a8ec71/out" "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_C_COMPILER=/usr/bin/cc" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_CXX_COMPILER=/usr/bin/c++" "-DCMAKE_BUILD_TYPE=Debug"

--- stderr
CMake Error: The source directory "/home/yash/Projects/Fresh/gui/research-sources/rust-onig/onig_sys/oniguruma" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.
thread 'main' panicked at '
command did not execute successfully, got: exit code: 1

build script failed, must exit now', /home/yash/.cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.24/src/lib.rs:593
note: Run with RUST_BACKTRACE=1 for a backtrace.

onig v4.3.1 fails to build with rust 1.27.0

Compilation is broken for rust v1.27.0 with v4.3.1...

   Compiling onig v4.3.1
error[E0658]: access to extern crates through prelude is experimental (see issue #44660)
   --> C:\Users\appveyor\.cargo\registry\src\github.com-1ecc6299db9ec823\onig-4.3.1\src\find.rs:180:43
    |
180 |                 (&mut region.raw) as *mut onig_sys::OnigRegion,
    |                                           ^^^^^^^^
error: aborting due to previous error
For more information about this error, try `rustc --explain E0658`.
error: Could not compile `onig`.

A month ago, v4.2.1 compiled and worked without error.

I think a minimum rust compiler version change should mean a semver major version increment (not just a minor), so that builds don't just arbitrarily break on re-build.

I also can't find any indication for minimum rust version in your code, docs, or CI.

Thanks for your attention.

Define missing functions in onig_sys

There are some functions that are not documented, but still are part of the public API. Here is the full list:

  • onig_copyright
  • onig_recompile
  • onig_recompile_deluxe
  • onig_reg_init
  • onig_region_init
  • onig_region_set
  • onigenc_get_default_encoding
  • onigenc_get_right_adjust_char_head_with_prev
  • onigenc_init
  • onigenc_set_default_caseconv_table
  • onigenc_set_default_encoding
  • onigenc_step_back

Signatures can be found in oniguruma.h

Create Better Panic Message when Search Fails

At the moment a generic "Onig: Internal error during regex match" panic message is raised if a match doesn't complete successfully. Update this to convert the error code to a string properly.

This should be a case of updating the following line in src/lib.rs:
https://github.com/rust-onig/rust-onig/blob/master/src/lib.rs#L452

The panic message should use onig_sys::onig_error_code_to_str with a null error info in a similar way to the Error struct here:

https://github.com/rust-onig/rust-onig/blob/master/src/lib.rs#L148

Add Support for Scan-Based Searching

In the latest Oniguruma API a new function has been added which allows a user to specify a callback which is executed each time a given regex matches within a buffer. We should expose this to users of the onig and onig_sys crates. We could also look at updating the FindCaptures and FindMatches to use the scan API to iterate through the matches in a given buffer.

I imagine the API would look something like this:

impl Regex {
  pub fn scan_with_region<CB, UD>(&self, to_search: &str, region: &mut Region, callback: CB, user_data: UD)
     where CB : Fn(i32, i32, &mut Region, UD) -> bool {
     // ...
  }

  pub fn scan<CB>(&self, to_search: &'t str, callback: CB)
    where CB : Fn(i32, Captures<'t>) -> bool {
    // ...
  }
}

Consider Switching to Onigmo

The k-takata/Onigmo library was forked from Ongiuruma and seems to have fixes and new features. We could move to using this rather than the Oniguruma tarball that we currently compile.

Run `cargo fmt`

The codebase is looking like it could benefit from a rustfmt again.

test for version is unreliable

We are not using bundled version of oniguruma in onig_sys which means that it might be different from what test expects.

---- utils::tests::utils_get_version_returns_expected_version stdout ----
thread 'utils::tests::utils_get_version_returns_expected_version' panicked at 'assertion failed: `(left == right)`
  left: `"6.8.2"`,
 right: `"6.9.0"`', src/utils.rs:54:9

cargo doc fails on onig_sys.

Rust version 1.22.1 and 1.23.0. The lines that have the language specifier after the 3 accent graves appear to be the problem.

error: unknown start of token: `
 --> <stdin>:1:1
  |
1 | ```c
  | ^
  |
help: unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it's not
 --> <stdin>:1:1
  |
1 | ```c
  | ^

error: Could not document `onig_sys`.

Caused by:
  process didn't exit successfully: `rustdoc --crate-name onig_sys /Users/paul/.cargo/registry/src/github.com-1ecc6299db9ec823/onig_sys-65.0.1/src/lib.rs -o /Users/paul/dev/x/target/doc --cfg feature="static_onig" -L dependency=/Users/paul/dev/x/target/debug/deps --extern libc=/Users/paul/dev/x/target/debug/deps/liblibc-37eea92b7cbeea63.rlib` (exit code: 101)

Option to statically link?

One of my favorite features of Rust is that I can bundle up my program in a single executable without any hassle by statically linking every dependency. Would it be possible for rust-onig to statically link (either by default, or through a feature)?

Is it possible to use this with lazy_static?

Can't get it compile.

error[E0277]: the trait bound `*mut onig_sys::OnigRegexType: std::marker::Sync` is not satisfied in `onig::Regex`
  --> src/lib.rs:14:1
   |
14 | / lazy_static! {
15 | |     static ref ROLES_FIND_RE: Regex = {
16 | |         let mut roles: Vec<String> = include_str!("../litigant/data/roles.txt")
17 | |             .lines()
...  |
52 | |     static ref ENDSWITH_COMPANY_RE: Regex = Regex::new("company$").unwrap();
53 | | }
   | |_^ `*mut onig_sys::OnigRegexType` cannot be shared between threads safely
   |
   = help: within `onig::Regex`, the trait `std::marker::Sync` is not implemented for `*mut onig_sys::OnigRegexType`
   = note: required because it appears within the type `onig::Regex`
   = note: required by `lazy_static::lazy::Lazy`
   = note: this error originates in a macro outside of the current crate

onig version: 1.4.1

Failed in another project

/home/yash/.cargo/registry/src/github.com-1ecc6299db9ec823/onig_sys-64.0.0/oniguruma/src/regint.h:667:9: error: unknown type name ‘intptr_t’
typedef intptr_t OnigStackIndex;

trishume/syntect#89

I'm not sure if it's rust-onig issue or syntect but since it's clearly failing on onig I included a report here. For details please click on above link.

Make Static Linking Default for MSVC

Update the MSVC toolchain to enable static linking by default.

  • Set static linking to true in the build script
  • Allow overriding of default with RUSTONIG_STATIC_LIBONIG=0

Stop Using Feature Flags for Static Linking

Apparently using Cargo's features to control the link type of the library is bad:

Also, please don't use cargo features incorrectly. A cargo feature to select whether to link a library statically or dynamically is wrong. Cargo features should be used exclusively for API additions. Consider using an environment variable instead to configure how onig is linked.

We should add a new environment variable that controls the behaviour of the build script and deprecate the old way of statically linking.

  • Update the build script to check RUSTONIG_STATIC_LIBONIG for static linking.
  • Change the .travis.yaml and appveyor.yaml to use this new variable to specify the link type.
  • Add documentation for the new static linking flag and deprecate the old feature flag.

Thread safety for Regex compilation

From oniguruma docs:

In order to make thread safe, which of (A) or (B) must be done.

   (A) Oniguruma Layer

       Define the macro below in oniguruma/regint.h.

       USE_MULTI_THREAD_SYSTEM
       THREAD_ATOMIC_START
       THREAD_ATOMIC_END
       THREAD_PASS

       THREAD_SYSTEM_INIT
       THREAD_SYSTEM_END


   (B) Application Layer

       The plural threads should not do simultaneously that making 
       new regexp objects or re-compiling objects or freeing objects,
       even if these objects are differ.

We should make global lock for calling onig_regex_new

Expose all Oniguruma API Functions in `onig_sys`

  • Add extern declarations for all Oniguruma functions to the onig_sys crate.
  • Add documentation comments for all exported functions in the crate.
  • Add structure definitions for all remaining C structures
  • Add constant declarations for all C constants.
  • Add extern declaration for undocumented functions. #8

Investigate Windows Build Failures

Recent windows builds seem to have started failing. This appears to be due to a symbol missing when compiling against oniguruma 6.8 and later (for accessing the standard file descriptors).

There are two things to work on here:

  • Add more MSVC targets to test on beta & nightly as well as stable.
  • Look into getting the right toolchain so that the gnu builds don't fail.

From investigations of Rust's Appveyor config it appears that they pull a custom snapshot rather than using one that's already installed. We might not need to go that drastic. There are quite a few MinGW and MSYS versions available to choose from.

Out-of-Tree Windows MSVC Builds of Oniguruma

Currently the onig_s and onig.dll are build in the source directory itself. This is in the cargo cache and is generally considered bad form. The Oniguruma makefile for NMAKE doesn't appear to support out-of-tree builds. The proposed solution is:

  • Create our own makefile in onig_sys which can do out-of-tree builds.
  • When happy with it submit a pull request against Oniguruma.

Replace with captured groups?

I'm not sure if it good idea, but I noticed that there is lack of replacing with capturing groups.
I.e.

let re = Regex::new("(.*)\s+(.*)").unwrap();
let result = re.replace_all("first second", "$2 $1");

Result will be just "$2 $1".

I tried to look into inguruma itself but I didn't find replacement functionality there so I guess rust-onig implements it on its own?
If so then I would just like to highlight that this feature may be useful, but if it is not worth it to implement, then the request can be closed :)

onig 3.0.1 / onig_sys fails to build on windows

There appears to be a difference in the onig 3.0.1 on crates.io and the 3.0.1 on git hub - the former has onig_sys 66.1.4 and the latter has onig_sys 67.0.0 which has a change in build.rs - I don't know if this is what's causing the build failure... but maybe?

error: failed to run custom build command for `onig_sys v66.1.4`
process didn't exit successfully: `C:\Gitlab\builds\.\0\y\x\target\debug\build\onig_sys-074ef9fa23663e81\build-script-build` (exit code: 101)
--- stderr

Microsoft (R) Program Maintenance Utility Version 14.12.25831.0
Copyright (C) Microsoft Corporation.  All rights reserved.

NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.12.25827\bin\Hostx64\x64\cl.EXE"' : return code '0x2'
Stop.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { repr: Custom(Custom { kind: Other, error: StringError("command [\"cmd\", \"/c\", \"C:\\\\Windows\\\\system32\\\\config\\\\systemprofile\\\\.cargo\\\\registry\\\\src\\\\github.com-1ecc6299db9ec823\\\\onig_sys-66.1.4\\\\make_win64.bat\"] exited with code 2") }) }', src\libcore\result.rs:906:4
stack backtrace:
   0: std::sys_common::backtrace::_print
             at C:\projects\rust\src\libstd\sys_common\backtrace.rs:91
   1: std::panicking::default_hook::{{closure}}
             at C:\projects\rust\src\libstd\panicking.rs:380
   2: std::panicking::default_hook
             at C:\projects\rust\src\libstd\panicking.rs:397
   3: std::panicking::rust_panic_with_hook
             at C:\projects\rust\src\libstd\panicking.rs:577
   4: std::panicking::begin_panic<alloc::string::String>
             at C:\projects\rust\src\libstd\panicking.rs:538
   5: std::panicking::begin_panic_fmt
             at C:\projects\rust\src\libstd\panicking.rs:522
   6: std::panicking::rust_begin_panic
             at C:\projects\rust\src\libstd\panicking.rs:498
   7: core::panicking::panic_fmt
             at C:\projects\rust\src\libcore\panicking.rs:71
   8: core::result::unwrap_failed<std::io::error::Error>
             at C:\projects\rust\src\libcore\macros.rs:23
   9: core::result::Result<alloc::string::String, std::io::error::Error>::unwrap<alloc::string::String,std::io::error::Error>
             at C:\projects\rust\src\libcore\result.rs:772
  10: build_script_build::compile
             at .\build.rs:102
  11: build_script_build::main
             at .\build.rs:120
  12: panic_unwind::__rust_maybe_catch_panic
             at C:\projects\rust\src\libpanic_unwind\lib.rs:101
  13: std::rt::lang_start
             at C:\projects\rust\src\libstd\rt.rs:51
  14: main
  15: __scrt_common_main_seh
             at f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283
  16: BaseThreadInitThunk

warning: build failed, waiting for other jobs to finish...
error: build failed

Run `cargo fmt`

I'm pretty sure some of the formatting will need fixing up afterwards. Things like checking where comments end up and that arrays are still readable. This should be a good place to get started though.

Should we use the original names for the types?

I'm a little concerned that the names of some struct's in onig_sys do not equal to the original names.

Original names:

  • OnigCaseFoldType
  • OnigOptionType
  • OnigMetaCharTableType
  • OnigEncodingType
  • OnigSyntaxType

We use:

  • OnigCaseFold
  • OnigOptions
  • OnigMetaCharTable
  • OnigEncoding
  • OnigSyntax

On the one hand, I do not see much sense to use Type word in the names of structures. On the other hand, compliance with the original API may be more important than our sense of beauty and traditions in Rust. @iwillspeak what you think?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.