So far I've been playing with Zinc and it's been working great.
It would be cool if we could have a log system like Logtash for zinc.
I've always mainly been using Elastic for a log system for multiple machines.

When starting zinc I see the following:

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:	export GIN_MODE=release
 - using code:	gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET    /healthz                  --> github.com/prabhatsharma/zinc/pkg/meta/v1.GetHealthz (3 handlers)
[GIN-debug] GET    /                         --> github.com/prabhatsharma/zinc/pkg/meta/v1.GUI (3 handlers)
[GIN-debug] GET    /ui/*filepath             --> github.com/gin-gonic/gin.(*RouterGroup).createStaticHandler.func1 (3 handlers)
[GIN-debug] HEAD   /ui/*filepath             --> github.com/gin-gonic/gin.(*RouterGroup).createStaticHandler.func1 (3 handlers)
[GIN-debug] POST   /api/login                --> github.com/prabhatsharma/zinc/pkg/handlers.ValidateCredentials (3 handlers)
[GIN-debug] PUT    /api/user                 --> github.com/prabhatsharma/zinc/pkg/handlers.CreateUpdateUser (4 handlers)
[GIN-debug] DELETE /api/user/:userID         --> github.com/prabhatsharma/zinc/pkg/handlers.DeleteUser (4 handlers)
[GIN-debug] GET    /api/users                --> github.com/prabhatsharma/zinc/pkg/handlers.GetUsers (4 handlers)
[GIN-debug] PUT    /api/index                --> github.com/prabhatsharma/zinc/pkg/handlers.CreateIndex (4 handlers)
[GIN-debug] GET    /api/index                --> github.com/prabhatsharma/zinc/pkg/handlers.ListIndexes (4 handlers)
[GIN-debug] PUT    /api/:target/document     --> github.com/prabhatsharma/zinc/pkg/handlers.UpdateDocument (4 handlers)
[GIN-debug] POST   /api/:target/_search      --> github.com/prabhatsharma/zinc/pkg/handlers.SearchIndex (4 handlers)
[GIN-debug] PUT    /es/:target/_doc/:id      --> github.com/prabhatsharma/zinc/pkg/handlers.UpdateDocument (4 handlers)
[GIN-debug] POST   /es/:target/_doc          --> github.com/prabhatsharma/zinc/pkg/handlers.UpdateDocument (4 handlers)
[GIN-debug] PUT    /es/:target/_create/:id   --> github.com/prabhatsharma/zinc/pkg/handlers.UpdateDocument (4 handlers)
[GIN-debug] POST   /es/:target/_create/:id   --> github.com/prabhatsharma/zinc/pkg/handlers.UpdateDocument (4 handlers)
[GIN-debug] POST   /es/:target/_update/:id   --> github.com/prabhatsharma/zinc/pkg/handlers.UpdateDocument (4 handlers)
[GIN-debug] POST   /es/_bulk                 --> github.com/prabhatsharma/zinc/pkg/handlers.BulkHandler (4 handlers)
[GIN-debug] POST   /es/:target/_bulk         --> github.com/prabhatsharma/zinc/pkg/handlers.BulkHandler (4 handlers)

Does this mean zinc doesn't support deleting documents or indices? I tried some curls modified from ElasticSearch tutorials I found online and had no success with those either

I'd love to be able to use the same ID between my SQL DB and zinc. However, when I upload with the _id field set, zinc panics:

$ curl -u admin:Complexpass#123 -XPUT -d '{"author":"Prabhat Sharma", "_id": 1}' http://localhost:4080/api/myshinynewindex/document

2021/12/16 20:07:50 [Recovery] 2021/12/16 - 20:07:50 panic recovered:
PUT /api/myshinynewindex/document HTTP/1.1
Host: localhost:4080
Accept: */*
Authorization: *
Content-Length: 37
Content-Type: application/x-www-form-urlencoded
User-Agent: curl/7.77.0


interface conversion: interface {} is float64, not string
/usr/local/go/src/runtime/iface.go:261 (0x1009f94)
/Users/prabhat/projects/raelabs/zinc/pkg/handlers/UpdateDocument.go:25 (0x1571566)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x156bbc7)
/Users/prabhat/projects/raelabs/zinc/pkg/auth/AuthMiddleware.go:21 (0x156ba0e)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x1497121)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:99 (0x149710c)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x1495fdd)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:489 (0x1495c65)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:445 (0x14957c4)
/usr/local/go/src/net/http/server.go:2878 (0x126e47a)
/usr/local/go/src/net/http/server.go:1929 (0x1269b27)
/usr/local/go/src/runtime/asm_amd64.s:1581 (0x10634a0)

Is there a more detailed explanation of the search types somewhere?
Are the named search types already implemented in a functional way?

Please see
https://github.com/sbt/zinc

It would be great if there was an option in the time settings for a search which would let you set the search time frame to all time.

Login pages shows OK when accessing to http://192.168.56.112:4080, but API_ENDPOINT is configured to localhost (index.js:6) and pressing "Sing in" button can get the response.
Any way to modify this without recompiling?

Good job!!! Thank you in advanced

After staring zinc on Windows 10, loading the data from the README python script, and choosing the games3 index, no results are returned - despite choosing 1 day (or longer) as the time scale (the test was done an hour or so ago).

There are no logs (except a successful authentication).

The data is in because the available labels are displayed in the left drawer.

When adding "_source" in query request, it still returns all the fields.

Since #16 is now merged, it would be good to register the zinc name over at snapcraft.io and have a CI job to publish builds of zinc automatically. Opening this to keep track! :)

First of all, this is a really cool project, with really impressive response times in searches.

According to the docs there are currently a thew ES APIs implemented.
I'd like to ask if compatibility, I mean in regards of a drop-in replacement for workloads where ES is the backend, is intended?

Are there already experiences with deployments where ES is normally used as a backend like Jaeger, OpenCTI or other projects?

I would like to use Magento 2 open source ecommerce platform, which requires elasticsearch as a prerequisite for the installation. Would it be possible to swap ES with zinc?

Thank you in advance!

https://github.com/johannesboyne/gofakes3

To allow testing

Hello, in the meantime, thank you for the constantly evolving product. Very useful for my projects. One question though: from version 0.1.3 onwards, I don't understand how to enable the limitation of 20 "records" on the "ZincSearch" web application. Thank you. I only ever see 20 records.

Would be lovely to have "how to contribute" section, a way to gain insights into your backlog and/or any working group meet up(s). So we can quickly identify opportunities to render our assistance.

Happy to get involved ~ 10hrs/week (go/vuejs)

Would be great to have a cursor pagination for the search results.

Hey, i've just recently stumbled across this project and before i invest time setting this up, i want to ask first: Is zinc a real logserver alternative? Like can i setup filebeat/whatever and push my logs to it or is zinc designed for other stuff?

I was in the process of installing ELK but its quite complex and there's no free user management so i was searching for other options.

Apologies if that's not the right place to ask those questions.

@prabhatsharma

This is wonderful stuff ..

I was playing with Hey and GUI tools to test benchmarks and was thinking about a more solid way to get test data into the system.

I was wondering if you had any preferences / plans for this:
A faker like counterfeiter: https://github.com/maxbrunsfeld/counterfeiter
Example usage:
https://github.com/cloudfoundry/cf-networking-release/search?q=counterfeiter

Hello, zinc uses GoReleaser, which offers excellent Homebrew integration. Would you be open to adding this integration? Here's what I do for reference

This tool looks promising and it would be great to start planning SDK, so everyone can build tools on top of it.

Maybe this is more observation, but still - when particular value is null, I receive 500. I do not know what would be correct behaviour, ESS survives exactly this scenario.

When for particular key "null" value is assigned on indexing.
As:

185	3.377123217	127.0.0.1	127.0.0.1	HTTP	505	POST /es/product_v2/_doc HTTP/1.1  (application/json)

{"product_id":"eabf69fbe53f57a4ef6fabd6af8430ba","product_type":"car","car_manufacturer":null,"car_model":"\"VOLUNTEER 4X2\"","car_year":"2011","car_gear_type":null,"weight":0}

I received back: 500 Internal error with following output from zinc:

2021/12/28 18:00:38 [Recovery] 2021/12/28 - 18:00:38 panic recovered:
POST /es/product_v2/_doc HTTP/1.1
Host: localhost:4080
Accept: application/json
Authorization: *
Content-Length: 176
Content-Type: application/json
User-Agent: elasticsearch-php/6.7.1 (Linux 5.11.0-43-generic, PHP 7.4.3)

interface conversion: interface {} is string, not float64
/usr/local/go/src/runtime/iface.go:261 (0x40a374)
/Users/prabhat/projects/zinc/zinc/pkg/core/Index.go:77 (0x957df4)
/Users/prabhat/projects/zinc/zinc/pkg/core/UpdateDocument.go:5 (0x95a1a6)
/Users/prabhat/projects/zinc/zinc/pkg/handlers/UpdateDocument.go:47 (0x96de96)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x967c27)
/Users/prabhat/projects/zinc/zinc/pkg/auth/AuthMiddleware.go:21 (0x967a6e)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x891fa1)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:99 (0x891f8c)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x890e5d)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:489 (0x890ae5)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:445 (0x890644)
/usr/local/go/src/net/http/server.go:2879 (0x668eda)
/usr/local/go/src/net/http/server.go:1930 (0x664587)
/usr/local/go/src/runtime/asm_amd64.s:1581 (0x463580)

I started an instance on Windows 10 and used the exact example from the README (just adapting the admin password). When the data was sent (probably successfully - there is a successful auth message but nothing about data ingestion, and the script ran fine) and I pressed the Search button, zinc panicked with

2021/12/04 10:03:52 [Recovery] 2021/12/04 - 10:03:52 panic recovered:
POST /api//_search HTTP/1.1
Host: localhost:4080
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,fr;q=0.8,pl;q=0.7,de;q=0.6
Authorization: *
Connection: keep-alive
Content-Length: 145
Content-Type: application/json
Dnt: 1
Origin: http://localhost:4080
Referer: http://localhost:4080/ui/
Sec-Ch-Ua: " Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36


runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/runtime/panic.go:221 (0x459ebc)
/usr/local/go/src/runtime/signal_windows.go:254 (0x459e8c)
/Users/prabhat/projects/raelabs/zinc/pkg/core/Search.go:61 (0x979af9)
/Users/prabhat/projects/raelabs/zinc/pkg/handlers/Search.go:23 (0x98d0c4)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x987e87)
/Users/prabhat/projects/raelabs/zinc/pkg/auth/AuthMiddleware.go:21 (0x987cce)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x8a5a81)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:99 (0x8a5a6c)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x8a493d)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:489 (0x8a45c5)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:445 (0x8a4124)
/usr/local/go/src/net/http/server.go:2878 (0x67d35a)
/usr/local/go/src/net/http/server.go:1929 (0x678a07)
/usr/local/go/src/runtime/asm_amd64.s:1581 (0x4749c0)

Would it be possible to have an armv6 Docker image also? Thank you.

mkdir data
docker run -v $(pwd)/data:/data  -e DATA_PATH="/data" -p 4080:4080 -e FIRST_ADMIN_USER=admin -e FIRST_ADMIN_PASSWORD=Complexpass#123 -p 4081:4081 public.ecr.aws/m5j1b6u0/zinc:v0.1.1

• output:

WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested
{"level":"debug","time":"2021-12-07T03:39:24Z","message":"Loading indexes..."}
{"level":"debug","time":"2021-12-07T03:39:24Z","message":"Loading system indexes..."}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xa299e4]

goroutine 1 [running]:
github.com/prabhatsharma/zinc/pkg/core.LoadZincSystemIndexes(0xc000402f00, 0x0, 0x0)
	/go/src/github.com/prabhatsharma/zinc/pkg/core/LoadIndexes.go:26 +0x1a4
github.com/prabhatsharma/zinc/pkg/core.init()
	/go/src/github.com/prabhatsharma/zinc/pkg/core/types.go:10 +0x3e

I try to insert data (from a little file of 37M) but zinc get killed by OOM.
It eat more than 8GB before be killed by kernel.

oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/[email protected],task=zinc,pid=129244,uid=1000
[13291.887950] Out of memory: Killed process 129244 (zinc) total-vm:12128240kB, anon-rss:11309292kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:22328kB oom_score_adj:0
[13292.196257] oom_reaper: reaped process 129244 (zinc), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

[GIN-debug] Listening and serving HTTP on :4080
{"level":"debug","time":"2021-12-15T18:49:58+01:00","message":"auth: success"}
Killed

It would be nice if there could be a discord server for zinc,
Plenty of OSS projects got a chat group somewhere.

Adding the "DELETE" option to the Zinc API would allow users to "clean up" unused/old indices. This should clean up the disk space and metadata around the indexes.

curling (or simply clicking in a browser) https://github.com/prabhatsharma/zinc/releases/download/v0.1.1/https://github.com/prabhatsharma/zinc/releases/download/v0.1.1/olympics.ndjson.gz returns a 404 Not Found error

I like to try installing on FreeBSD. How can I build from code?

Refer to the Es-use-document to paging query，but it didn't work
Es-use-document: https://www.elastic.co/guide/cn/elasticsearch/guide/2.x/_empty_search.html

GET /_search
{ "from": 30, "size": 10 }

@prabhatsharma

dda7ec0 broke things becuase the default port changed to 8080

Might be best to make the test match the defaults here https://github.com/prabhatsharma/zinc/blob/main/main_test.go#L14

?

Thanks for your work on this!

It would be great to be able to specify a subpath, or path prefix to be used, for hosting behind a reverse proxy.

For example. if I wanted to proxy :4080/ui, and have it accessible as /zinc/ui

Add summary aggregation function for zinc, which can help summarize the number of certain keyword in the search results

This project looks promising. Any benchmark that compares ES? Thanks!

Good afternoon. Are there any plans to integrate with filebeat?

One of my indices is getting bigger and Zinc crashes every time i search anything using that index. It's the systemdlog index and it's ~560Mi on the filesystem

[root@lb02 tmp]# du -sch data/*
16K	data/games3
116K	data/_index_mapping
104K	data/journal
46M	data/syslog
558M	data/systemdlog
12K	da

Free memory before a search:

[root@lb02 tmp]# free -h
              total        used        free      shared  buff/cache   available
Mem:          7.5Gi       592Mi       6.1Gi        21Mi       843Mi       6.5Gi
Swap:   ta/_users

After triggering a search on that index and just before the crash:

[root@lb02 tmp]# free -h
              total        used        free      shared  buff/cache   available
Mem:          7.5Gi       7.0Gi       158Mi        21Mi       298Mi        90Mi
Swap:            0B          0B          0B

Journal log after crash:

Dec 10 13:27:18 lb02.localdomain kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-12.scope,task=zinc,pid=59785,uid=0
Dec 10 13:27:18 lb02.localdomain kernel: Out of memory: Killed process 59785 (zinc) total-vm:8521368kB, anon-rss:7148568kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:14596kB oom_score_adj:0

Is this expected behavior with such memory usage?

Is it possible to search in all indices?

and

is it possible to search without start and end date?

I save the data into zinc through a python script, and when I search for a complete word, the result is correct. But when searching for a part of the word, no results were found. Is this problematic?

The short of the problem is that I can't get Zinc in a container to stay running without pre-existing indexes to load. It seems that without an existing index in your $DATA_PATH directory, Zinc will panic and the docker container stops instantly before anyone would have the chance to have Zinc ingest any data.

I built an image from the Dockerfile in this repository using the command docker build -t test .;

I ran the following command: docker run -d -v /server/zinc:/data -e DATA_PATH="/data" -e FIRST_ADMIN_USER=admin -e FIRST_ADMIN_PASSWORD=admin --name zinc test

I observed that the container had stopped near-instantly, and so I ran docker logs zinc which returned the following:

{"level":"debug","time":"2021-12-03T20:09:04Z","message":"Loading indexes..."}
{"level":"debug","time":"2021-12-03T20:09:04Z","message":"Loading system indexes..."}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xa299e4]

goroutine 1 [running]:
github.com/prabhatsharma/zinc/pkg/core.LoadZincSystemIndexes(0xc0003f6180, 0x0, 0x0)
        /go/src/github.com/prabhatsharma/zinc/pkg/core/LoadIndexes.go:26 +0x1a4
github.com/prabhatsharma/zinc/pkg/core.init()
        /go/src/github.com/prabhatsharma/zinc/pkg/core/types.go:10 +0x3e

It appears you need to do more to start Zinc without pre-existing data that your documentation does not mention.

Hi @prabhatsharma

First thanks for creating Zinc. For now, its amazing for small applications and I am sure more features are coming.

I was playing with Zinc after reading your article. I tested with data with email addresses. There are no issues with writing/reading documents but on search it does not seems to consider email. I tried to encode email address and tried few ways and found that search does not work as soon as I add dot "." any data.

I tried to debug from source but my skill is limited with go lang so its not much help.

Thanks

I wish I could mashup this project and quickwit:

Quickwit allows true decoupled compute and storage. We designed it to search straight from object storage like AWS S3 in a stateless manner.

I love that zinc comes with a Kibana replacement -- a built in UI!

If zinc could also store all its data directly in S3 and decouple storage and compute, you'd have replaced the 99% percent use case of ES/Kibana with a vastly superior technology that is both free as in beer and tshirts, easy to operate, hard to lose data on, and effortless to deploy.

Tried several options, but cat get it to go up and running from a command schell

$ FIRST_ADMIN_USER=admin FIRST_ADMIN_PASSWORD=Complexpass 123 zinc.exe
'FIRST_ADMIN_USER' is not recognized as an internal or external command,
operable program or batch file.

other try:

zinc.exe FIRST_ADMIN_USER=admin FIRST_ADMIN_PASSWORD=QNHadmin
{"level":"debug","time":"2021-12-06T16:19:12+01:00","message":"Loading indexes..."}
{"level":"debug","time":"2021-12-06T16:19:12+01:00","message":"Loading system indexes..."}
{"level":"debug","time":"2021-12-06T16:19:12+01:00","message":"Index loaded: _users"}
{"level":"debug","time":"2021-12-06T16:19:12+01:00","message":"Index loaded: _index_mapping"}
2021/12/06 16:19:12 FIRST_ADMIN_USER and FIRST_ADMIN_PASSWORD must be set on first start. You should also change the credentials after first login.

goroutine 1 [running]:
github.com/prabhatsharma/zinc/pkg/core.LoadZincSystemIndexes()
/go/src/github.com/prabhatsharma/zinc/pkg/core/LoadIndexes.go:25 +0x137
github.com/prabhatsharma/zinc/pkg/core.init()
/go/src/github.com/prabhatsharma/zinc/pkg/core/types.go:10 +0x3c
{"level":"debug","time":"2022-01-04T14:23:59Z","message":"Loading indexes..."}
{"level":"debug","time":"2022-01-04T14:23:59Z","message":"Loading system indexes..."}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x958bd7]

https://github.com/opensearch-project/opensearch-go is an elastic alternaitve.

Seems perfect match ?

Are there any plans for implementation of relevance tuning?

Here's something similar with elastic to show what I'm talking about.

Whenever I try to post data I get the message "404 page not found". This is with version 0.1.1 on Windows 10. I can access the web UI without problems.

This is what I did:

1. Downloaded Zinc 0.1.1.
2. In a command prompt, cd'd to the directory where I unpacked zinc.exe
3. Set an admin user environment variable: set FIRST_ADMIN_USER=admin
4. Set an admin password environment variable: set FIRST_ADMIN_PASSWORD=Pass123
5. Created a data subdirectory
6. Started zinc.exe
7. Opened the web UI in my browser successfully at http://localhost:4080/ui/

I unsuccessfully tried to post data to the following URLs:

1. http://localhost:4080/api/_bulk
2. http://localhost:4080/api/myindexname/_bulk
3. http://localhost:4080/api/myindexname/document

Am I missing something?

In some cases it's required to have a string indexed as keyword instead of just text.

Could be solved like this in a single record:

data = {
    "Athlete": "DEMTSCHENKO, Albert",
    "Country": {
        "mapping": "keyword",
        "value": "RUS"
    },
    // ..
  }

Or like this in a bulk ingestion file:

{ "index" : { "_index" : "olympics" } }
{"Athlete": "HAJOS, Alfred", "Country": {"mapping": "keyword", "value": "HUN"}, /* .. */}