baygeldin / tantiny Goto Github PK

View Code? Open in Web Editor NEW

202.0 2.0 7.0 52 KB

Tiny full-text search for Ruby powered by Tantivy

License: MIT License

Ruby 61.13% Shell 0.11% Rust 38.76%

ruby rust search-engine tantivy

tantiny's People

Contributors

Stargazers

Watchers

Forkers

mfkp takahashim morygonzalez dut3062796s anna328p a-chris

tantiny's Issues

hello everyone, thanks for your work on this gem.
I'm looking forward to use it but our use-case is a bit different: we need to filter ~100 contents at a time, then we delete the index and go with the other 100 contents, and so on for thousands of times a day.

I was wondering, is it possible to use in memory indexing instead of writing on file?

I saw that tantivy supports this use-case
https://docs.rs/tantivy/latest/tantivy/struct.Index.html#method.create_in_ram

I've never worked with Ruby and Rust extensions but I can dive into a solution if you tell me that's feasible

UPDATE: I've been playing a bit with the code and it seems that this change does most of the job
a-chris@9c47fc0

Is it something you would like to have on tantiny? Maybe making it configurable while creating a new Index object?

Error loading library

Hello! My attempt to require the gem causes a LoadError:

$ gem install tantiny

Fetching tantiny-0.3.1.gem
Building native extensions. This could take a while...
Successfully installed tantiny-0.3.1
Parsing documentation for tantiny-0.3.1
Installing ri documentation for tantiny-0.3.1
Done installing documentation for tantiny after 0 seconds
1 gem installed

$ irb

irb(main):001:0> require "tantiny"
/Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/gems/tantiny-0.3.1/lib/tantiny.rb:18:in `require_relative': cannot load such file -- /Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/gems/tantiny-0.3.1/lib/tantiny.so (LoadError)
	from /Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/gems/tantiny-0.3.1/lib/tantiny.rb:18:in `<top (required)>'
	from <internal:/Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:160:in `require'
	from <internal:/Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:160:in `rescue in require'
	from <internal:/Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
	from (irb):1:in `<main>'
	from /Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/gems/irb-1.4.1/exe/irb:11:in `<top (required)>'
	from /Users/alex/.asdf/installs/ruby/3.1.0/bin/irb:25:in `load'
	from /Users/alex/.asdf/installs/ruby/3.1.0/bin/irb:25:in `<main>'
<internal:/Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- tantiny (LoadError)
	from <internal:/Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from (irb):1:in `<main>'
	from /Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/gems/irb-1.4.1/exe/irb:11:in `<top (required)>'
	from /Users/alex/.asdf/installs/ruby/3.1.0/bin/irb:25:in `load'
	from /Users/alex/.asdf/installs/ruby/3.1.0/bin/irb:25:in `<main>'

tantiny.so actually exists:

ls -al /Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/gems/tantiny-0.3.1/lib/tantiny.so
-rwxr-xr-x  1 alex  staff  3706829 Apr 18 21:46 /Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/gems/tantiny-0.3.1/lib/tantiny.so

I'm running Ruby on MacBook Pro M1. Rust is installed from Homebrew (brew install rust).

Returning the match indexes along with results?

Hello, first of all, thanks for publishing this, looks to be a very interesting gem.

I'm wondering if it would be possible to return the matched text (or index of matched characters) along with the results? This would be useful in cases where a fuzzy match returns results, and I would want to highlight the matching text (or just show a snippet of text around the matching text for context) in the results.

In the readme it says:

"You may have noticed that search method returns only documents ids. This is by design. The documents themselves are not stored in the index."

So maybe this is not possible, but I figured I would ask anyway:

From your example:

brother = {
  imdb_id: "tt0118767",
  type: "/crime/Russia",
  title: "Brother",
  description: "An ex-soldier with a personal honor code enters the family crime business in St. Petersburg, Russia.",
  duration: 99,
  rating: 7.9,
  release_date: Date.parse("December 12, 1997")
}

Right now, this is how the search returns:

index.search('bersonal coder', fuzzy_distance: 1)
=> ["tt0118767"]

It would be great to return something like:

index.search('bersonal coder', fuzzy_distance: 1)
=> 
[{
  "tt0118767": {
    match_ranges: [[21, 28], [36, 39]]
  }
}]

That way I could display the results in my search listing like:

An ex-soldier with a personal honor code enters the family crime business in St. Petersburg, Russia.

I haven't dug into the source yet to see if this is possible, but I figure you'd know the limitations better and might be able to provide some input on if this is feasible.

Aggregations

Hi @baygeldin, this is an awesome project! Do you have any plans to add support for aggregations?

Term types

Are there plans to support term queries for types other than strings?

Custom tokenizer

I want to use Tantiny with Japanese. There are several Tantivy tokenizers for Japanese language. I'm now considering lindera-tantivy which supports not only Japanese but also Chinese and Korean. Is it possible to use these custom tokenizers with Tantivy via Tantiny?

How do I release Lock?

I got this error:

Tantiny::TantivyError (Failed to acquire Lockfile: LockBusy. Some("Failed to acquire index lock. If you are using a regular directory, this means there is already an IndexWriter working on this Directory, in this process or in a different process."))

How can I release the lock?

baygeldin / tantiny Goto Github PK

tantiny's People

Contributors

Stargazers

Watchers

Forkers

tantiny's Issues

Failed to build gem

in memory indexing

Error loading library

Returning the match indexes along with results?

Aggregations

Term types

Custom tokenizer

How do I release Lock?

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent