microsoft / elfie-arriba Goto Github PK

open source search projects

License: MIT License

C# 78.39% C++ 0.95% C 0.05% JavaScript 18.85% CSS 0.40% HTML 0.02% Batchfile 0.30% PowerShell 0.13% XQuery 0.02% SCSS 0.87%

elfie-arriba's Introduction

Arriba

Arriba is a C# data engine designed for instant structured search, free text search, and data exploration across large single machine datasets.

Give Arriba CSVs or simple C# arrays and it will automatically create tables with appropriately typed columns.

Arriba features a simple, elegant query syntax so that users can write anything from a web-style query "louvau -Closed" to a fully structured query ([Assigned To] = "Scott Louvau" AND ([State] != "Closed" OR [Remaining Work] > 0)).

Arriba has a beautiful website to make search and exploration easy, with comprehensive query suggestions, a configurable listing, customizable item details, and a Grid for quick analytics. Query suggestions go beyond showing just column names and the search syntax by adding "Inline Insights", showing query-specific top values and distributions for columns and showing which columns word searches are matching to answer questions directly and help users construct the queries they really intend.

Arriba exposes a comprehensive HTTP service you can use to programmatically run queries and aggregations, get query suggestions, and add/decorate/update/delete rows.

You can even host the Arriba engine directly in your C# process, creating tables in-memory and making custom column types, queries, and aggregations by implementing simple interfaces.

See the Arriba QuickStart to get Arriba and the Website running with sample CSV data in 15 minutes.

Elfie

Elfie is a library which makes it easy to build memory-efficient, extremely fast item sets providing search and traversal. Elfie uses the "structure of arrays" layout model for performance. A set class contains multiple columns of primitive types, enums, or a replacement string type, String8. Items of the set are structs which point to the set and a specific index. This gives the performance of structs (no allocations) with the convenience of classes (updates change all references of the item without copying). Elfie has primitives to provide text search (MemberIndex), define hierarchies (ItemTree), and define graphs (ItemMap). It also provides very fast read and write of CSV, TSV, and JSON via a consistent interface.

As an example, a set of ~5M Active Directory items with five columns and ~25M links in a graph fit in ~800MB, loads in ~600ms, and can be traversed at a rate of ~15M links per second, measured on a Surface Book i7.

Contributing

Arriba and Elfie are not owned by a dedicated team, so while fixes and small changes are welcome, our ability to include contributions and comment on design changes is limited. For larger fixes and design change ideas, please contact us so that we can comment on the design or suggest working in another fork.

Please:

Follow the .NET Foundation Coding Guidelines
Sign a Contributor License Agreement so that the community may freely use your contributions
Ensure a Release build and Unit Test run is clean
Run the Code Formatter passing "/rule+:UsingOrder,BraceNewline Arriba.sln"

Arriba and Elfie performance depend on minimizing allocations, boxing, and indirect method calls, so compare performance of real-life scenarios involving your code to avoid regressions. They were created by Microsoft to enable great internal tools, and we've opened them hoping they will enable you to create great search and analytics tools in your favorite language. =)

elfie-arriba's People

Contributors

Stargazers

Watchers

elfie-arriba's Issues

Consuming Elfie via NuGet?

I've only found a very old unlisted package for Elfie used by Roslyn:
https://www.nuget.org/packages/Microsoft.CodeAnalysis.Elfie

Is there a newer NuGet package that I can consume that is published anywhere?

Improve logging capabilities

The current code base has a mix of logging styles and still leave much to be desired. Improve the logging to use and enforce structured logging as well as enable a cloud backed solution such as application insights.

Add IPRange column type

Requested by JBL for storing single IPs and IP ranges.
Has a start and end address, inclusive, translated to integers from dotted notation.
Searches for a single IP should translate to an integer and find values in range.
Searches for an IP prefix (10.120) should translate to a range (10.120.0.0 - 10.120.255.255) and check for any intersection from the column ranges.

We could also use a column capable of containing IPv6 addresses, though they might be separate column types.

XForm: Null and Empty handling issues

Several Issues:

It looks like 'where [Column] = ""' isn't working correctly.
Add a function to allow checking for null or empty string.
Add an easy way to turn "NULL" the string into a null.

Document process for running services under docker

Ideas for this one

Implement a data simulator
document how to configure for auth
document how to run without auth

Move to using the common aspnet controller pattern

The present implementation is a custom request handling decorator pattern, that while it works is unnecessary and complex. It increases the amount of time that someone new to the code base needs to ramp up.

Unifi on a single dependency injection service

The current implementation uses a mix of dependency injection practices. Update the code base to unify on a single on to help simplfiy the development and debugging experience.

Race condition in AddOrUpdate when adding and updating an item in the same block

Because ChooseSplit is done as a parallel operation, an item that is added and then updated in the same call to AddOrUpdate might be evaluated in random order. To fix this, if we detect an update to an add, we should first sort the incoming data and then evaluate it in the order presented (the same as if each add or update were made individually)

XForm: Whitespace isn't allowed after compare operators

Reported by @michaelcfanning.

read TableName
where [Column] =="Value"

What's Elfie package licence?

https://www.nuget.org/packages/Microsoft.CodeAnalysis.Elfie
We've been using the NuGet package in our project, but since it's been unlisted the licence info is gone and we don't know if we can still use it or not. Can you help?

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Provide alternate hashing/distribution schemes

Allow support for custom hash application.

For ordered, integer keys, a round robin (modulus) is faster and more balanced than the MurMur3 hash. For time ordered data, fill-then-spill (fill a partition before moving on) is best.

Support cross platform and eliminate a strong dependency on the Registry

As part of moving to .NET Core a strong dependency on the registry was commented out creating a bug. Please investigate abstracting the requirement away so they code is not dependent on the registry.

Commit

Edge Browser: Column Hover Distribution Graphic Doesn't Appear on Edge

When hovering over columns with relatively small domain, a graph appears showing the %percent of the distribution for each distinct value.

On edge, this does not appear.

Create the deployment scripts and/or automated pipeline

Make it very easy to get up and running in the cloud though an automated, click once pipeline.

Role Based Access for elfie-arriba

Hi,
Does Elfie-arriba supports role based access? If not, can that feature be provided?

Enable running in Azure using by using persistent cloud storage

Investigate what it would take to remove the hard dependency on the file system and move to a cloud storage provider. The first pass of this would be a naïve implementation, however a future version may enable versioning of the data and global replication.

Code sample for quick substring search in a large list of strings

Hello folks!

I have a million strings (file paths) in an array. When user enters a search pattern I need to quickly display a list of all file paths from my array that have that search pattern as a substring. I only need the top N results.

What's the minimal code sample using Elfie to achieve my scenario? It's OK to take time to prepare any data structures in advance, but after that searches should be fast.

Thanks!

Improve identity support cross-platform and cloud based identities

As part of the migration to .NET Core to enable running in Azure the strong dependency on the WindowsIdentity was commented out introducing a bug and security vulnerability.

Breaking Commit - 89a5421

Instant update after UI actions

Hello folks,

Thanks a lot for this awesome project.

I'm noticing that most of UI actions are only applied after changing the current view to another one. Examples of that;

Click on a lisiting result header to sort result
Show / hide recent queries list
Delete an item from the recent queries list
Mark search query as a favourite one

I find this behavior consistent and this left me wondering; is that on purpose? or am I missing something?

Select OrderBy is not stable if a non-unique column is used for ordering

If a non-unique column is used for ordering, compute returns the top N matches. But those can be out of order w/rt to a secondary sort ordering (ID column is implied for now). Merge assumes these are in order. This was made worse by parallel/cascading merges because now merges happen in arbitrary order instead of partition order.