synchronize data from MongoDb to ElasticSearch about monstache HOT 4 CLOSED

@hassen2017 thanks for trying monstache.

First, you will want to make sure that your settings for namespace-regex and namespace-exclude-regex do not match. The former is what your want to include in sync and the latter is what you want to exclude from sync. In the case above you are excluding documents from the testdb11.commentaires collection which I'm pretty sure is not what you want. I should make that more clear in the docs. You do not need to configure either of those filters if you just want to sync all changes.

Next, you start monstache with the -verbose flag you should see activity in your console anytime you make changes to MongoDB that match your filtering (if any). This activity will be the bulk indexing API requests to Elasticsearch. Note the responses to these requests and make sure that they are successful.

Finally, to view your data in Elasticsearch you can use something like curl or install Kibana to visualize your indexes. See mapping for information on how monstache determines the index and type of documents.

For your config you can try a curl request like below to view the data in Elasticsearch.

curl -XGET 'localhost:9200/testdb11.commentaires/_search?pretty'

from monstache.

@rwynn thanks for your response that's helped to understand the concepts

I have cloned and installed git://github.com/mobz/elasticsearch-head.git and I succeeded to see my data into the Cluster Overview

from monstache.

Hello @rwynn
I used Kibana to visualise the data indexed to elasticsearch & I was runnig Monstache with this cmd in Ubentu 16.4 : ./bin/monstache -f config.toml --verbose
But i cannot visualise the data in realTime when i make changes to MongoDB
Please what can I doing for that ?

from monstache.

@hassen2017

Please read the info at https://www.elastic.co/guide/en/kibana/current/discover.html

You will want to make sure you have configured a valid index pattern in Kibana for the data you want to view. For example if you wanted to view the Mongo data from the database test and the collection users you would set the index pattern in Kibana to test.users. When you type the index pattern, Kibana should let you create it when it detects matching indexex in Elasticsearch. Also, setup the index pattern to be time based. You can choose use the _timestamp field, or tell monstache to add a date field by using the following configuration settings -> https://rwynn.github.io/monstache-site/config/#index-oplog-time. Set index-oplog-time to true and you will get a date field added by monstache that you can use when creating the index-pattern. Finally, make sure you have auto-refresh turned on in Kibana and that you are not filtering the data with a search term.

from monstache.