Comments (6)
Okay, I see. The body
field gets sanitized by removing non-alphabetic characters and lowercasing every word. Right now, this is what gets ingested into the bloom filter. I'd like to try fst and a 3-gram of the sanitized body to begin with and see how the package size changes if we go from a 3-gram to a 4-gram for example.
From my prior tests, I found that there are indeed a lot of common prefixes (at least for the sample dataset from my blog). I didn't even consider suffixes, but potentially exploiting that would be really nice.
Seeing that it's not an entirely weird idea, I'll go ahead and give it a shot (or somebody else that might find the time). Will report back here.
from tinysearch.
Great! Would love to hear how it goes and bounce more ideas around.
from tinysearch.
Not sure I know how to answer that without more details on how you're building the index. "fuzzy search" can refer to many different things... I'd be happy to say more if you shared more of your plan. :-)
from tinysearch.
Sure, sorry. Here's a writeup. It's basically using a bloom filter now but only full words can be searched. What I like to have is an optional, alternative engine that could provide prefix matches and "typo correction" (e.g. configurable Levenshtein). From what I can tell, all of this is supported by fst and it was recommended to me by two independent people, so I thought about giving it a try. My only concern is that the output size of the final (wasm) package, which contains the index will be too big for serving it directly to the clients.
from tinysearch.
I guess what I meant was, what specifically would be the keys in your FST? I see your JSON file, but there are many different things there. Is it every word in the body
field? Every 3-gram in the body
field? And what are the values that your keys map to?
Either way, it's hard for me to say up front. Just think of an FST as a finite state machine. If there are a lot of common prefixes or suffixes, then an FST should be able to exploit that. I would, however, expect a bloom filter to give you more control over the size of your index. e.g., You can make trade offs like "give me a smaller filter perhaps at the expense of a higher false positive rate" more easily than you can with an FST.
But yes, if your FST keys are words, then you could use levenshtein to find all matching words within an edit distance of 1, for example. And that's a pretty easy and well supported thing you can do via the fst
crate.
from tinysearch.
Thanks for the quick response and the encouragement. Stay tuned for updates.
from tinysearch.
Related Issues (20)
- Search results seems a bit random HOT 9
- Add an option to only generate the wasm HOT 1
- Consider replacing failure crate with anyhow or thiserror ? HOT 3
- Consider replacing structopt with argh crate ? HOT 7
- Rename project? HOT 14
- Benchmark against competition HOT 7
- building docker image gives error from wasm-pack in step 19/28 HOT 5
- Could tiny search run in IOS and Android WebView browsers? HOT 2
- Error: No such file or directory (os error 2) HOT 19
- Too many false positives HOT 7
- Error: failed to execute "wasm-pack" "build" HOT 21
- Error: Engine directory could not be created HOT 8
- bug HOT 1
- Update Cargo build for tinysearch HOT 1
- Does tinysearch support stemming, stopwords, and CJK? HOT 2
- Is there a way to return the page description or body in the results? HOT 19
- Github action HOT 2
- Please tag release 0.7.0 HOT 3
- Failing to find Cargo.toml in temp directory? HOT 10
- For Zola sites, the tinysearch json index gets included in the sitemap.xml file. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tinysearch.