The blocking from ncn-foreigners

Full `RcppAnnoy` support

Support for RcppAnnoy in:

deduplication (the same dataset, given by x)
record blocking with two datasets given by x, y
blocking from saved index
saving and reading index

Full `RcppHNSW` support

Support for RcppHNSW in:

deduplication (the same dataset, given by x)
record blocking with two datasets given by x, y
blocking from saved index
saving and reading index

Approximate nearest neighbours algorithms

List of backends for approximate nearest neighbours (ANN) search:

Release 0.2.0

Plans:

Support for rnndescent as it will be shipped to CRAN.

standard evaluation of blocking procedures

Evaluations based on the standard metrics as in klsh::confusion.from.blocking or in [blockr]

Improvement of performance

Ideas for improving performance:

if a large dataset is present for index then index should be created iteratively as converting sparse to dense matrix is a bottleneck
if a large query data is present the same procedure should be applied.

Consider using:

sparse matrix (Matrix) as an input for x, y
bigmemory::big.matrix as an input for x, y

add quality metrics

Quality metrics about blocking. This would require specifying new argument: true_block

`pair_ann` does not work with `data.table`

> pair_ann(x = df_example, on = "txt")
  First data set:  8 records
  Second data set: 8 records
  Total number of pairs: 10 pairs
  Blocking on: 'txt'

       .x    .y block
    <int> <int> <num>
 1:     1     2     1
 2:     1     3     1
 3:     1     4     1
 4:     2     3     1
 5:     2     4     1
 6:     5     6     2
 7:     5     7     2
 8:     5     8     2
 9:     6     7     2
10:     6     8     2
> pair_ann(x = setDT(df_example), on = "txt")
Error: j (the 2nd argument inside [...]) is a single symbol but column name 'on' is not found. If you intended to select columns using a variable in calling scope, please try DT[, ..on]. The .. prefix conveys one-level-up similar to a file system path.

vignette with basic examples

Vignette with examples:

Codes from tinytest dir can be used

Full `rnndescent` support

deduplication (the same dataset, given by x)
record blocking with two datasets given by x, y
blocking from saved index
saving and reading index

One may consider using result from the package as an input for reclin2 package. This may be done by creating an instance of the pairs class. See https://github.com/djvanderlaan/reclin2/blob/master/R/pair_minsim.R.

This would require a new function as.pairs to create an an object of this type or function pair_ann as the function reclin2::pair_minsim.