seznam / elasticlient Goto Github PK
View Code? Open in Web Editor NEWC++ Elasticsearch client library
Home Page: https://seznam.github.io/elasticlient/
License: MIT License
C++ Elasticsearch client library
Home Page: https://seznam.github.io/elasticlient/
License: MIT License
to many errors...
Is there interest in adding the cpr async methods (e.g GetCallback)? It appears all calls are blocking in this lib. I have an event based async runtime that I'd like to add this libraries functionality on to, but it can't use blocking calls for HTTP requests given that they may take a few seconds to execute. Thanks for your thoughts and this lib!
Dear maintainer(s),
it seems "scroll API" client does not meet elasticsearch expectations.
As described in https://www.elastic.co/guide/en/elasticsearch/reference/current/scroll-api.html, "scroll_api" shall be a JSON param to provide in the payload.
However, the scroll lib "<elasticlient/scroll.h>" posts next scoll request using "scroll_id" directly in request body without encapsulating it in {"scroll_id": "VALUE"}.
Same case for clear function.
scroll.cc (changes):
diff --git a/src/scroll.cc b/src/scroll.cc
index c939c8d..685dcf0 100644
--- a/src/scroll.cc
+++ b/src/scroll.cc
@@ -183,7 +183,7 @@ bool Scroll::next(Json::Value &parsedResult) {
std::ostringstream urlPart;
urlPart << "_search/scroll?scroll=" << impl->scrollTimeout;
LOG(LogLevel::INFO, "Scroll (next) on %s.", urlPart.str().c_str());
- if (impl->run(urlPart.str(), scrollParameters.scrollId, parsedResult)) {
+ if (impl->run(urlPart.str(), "{\"scroll_id\": \"" + scrollParameters.scrollId + "\"}", parsedResult)) {
return true;
}
}
@@ -202,7 +202,7 @@ void Scroll::clear() {
} else {
try {
const cpr::Response r = impl->client->performRequest(
- Client::HTTPMethod::DELETE, "_search/scroll/", scrollParameters.scrollId);
+ Client::HTTPMethod::DELETE, "_search/scroll", "{\"scroll_id\": \"" + scrollParameters.scrollId + "\"}");
if (r.status_code / 100 != 2) {
LOG(LogLevel::WARNING, "Scroll delete failed response text: %s", r.text.c_str());
}
This modification works fine for me.
Do not hesitate if you need more inputs / discuss's
Regards,
Laurent.
Are there plans to support username/password / jwt authentication methods?
Hello. Thank you for the pretty good library for work with ElasticSearch from C++.
But integration into C++ project this library isn't so easy. I suugest to prepare Conan package for the library. What do you think?
Hi,
elasticlient lib version: version-02
elastic-search version: 7.17.0
I'm bulk perform fail and without error.
Checking log found a msg: Unsupported 'action' found at bulk response.
deatail log:
Host response text: {
"took": 0,
"errors": true,
"items": [
{
"create": {
"_index": ".ds-game-of-thrones-a-2022.04.06-000032",
"_type": "_doc",
"_id": "docId_c",
"status": 409,
"error": {
"type": "version_conflict_engine_exception",
"reason": "[docId_c]: version conflict, document already exists (current version [1])",
"index_uuid": "1pbbHdwvQyKv55pxEqUjjg",
"shard": "0",
"index": ".ds-game-of-thrones-a-2022.04.06-000032"
}
}
}
]
}
Host response size: 406
Unsupported 'action' found at bulk response.
Can we achieve ACID transactional support using Elasticlient (i.e., commit and rollback operations) .?
For eg :- when a document is added into a index which is in a transaction, we need to add/update the subset of that in the index for real time search. In case any error happens in table persist or elastic search index persist, the entire transaction should roll back. Is that case possible with Elasticlient .? If not, any other way to achieve this .?
I'm facing a problem of coherence in the answers of elasticlient::Client::search() versus thoses displayed by Kibana . Same problem when I use elasticlient::Client::Scroll
On my personal debug config (docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.2 and docker.elastic.co/kibana/kibana-oss:6.2.2) , I observed that the number of total _shards strongly differs in the results of elasticlient::Client::search() and Kibana : 5 _shards with search(), vs 21 in Kibana.
Hi, when I run cmake ..
in build, it appear error like this:
CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: MHD_INCLUDE_DIR (ADVANCED) used as include directory in directory /data/denny/github/elasticlient/external/httpmockserver/src used as include directory in directory /data/denny/github/elasticlient/external/httpmockserver/src used as include directory in directory /data/denny/github/elasticlient/external/httpmockserver/src used as include directory in directory /data/denny/github/elasticlient/external/httpmockserver/src used as include directory in directory /data/denny/github/elasticlient/external/httpmockserver/src used as include directory in directory /data/denny/github/elasticlient/external/httpmockserver/src used as include directory in directory /data/denny/github/elasticlient/external/httpmockserver/src used as include directory in directory /data/denny/github/elasticlient/external/httpmockserver/src used as include directory in directory /data/denny/github/elasticlient/external/httpmockserver/test used as include directory in directory /data/denny/github/elasticlient/external/httpmockserver/test used as include directory in directory /data/denny/github/elasticlient/external/httpmockserver/test used as include directory in directory /data/denny/github/elasticlient/external/httpmockserver/test used as include directory in directory /data/denny/github/elasticlient/external/httpmockserver/test used as include directory in directory /data/denny/github/elasticlient/external/httpmockserver/test used as include directory in directory /data/denny/github/elasticlient/external/httpmockserver/test used as include directory in directory /data/denny/github/elasticlient/external/httpmockserver/test MHD_LIBRARY (ADVANCED) linked by target "test-server" in directory /data/denny/github/elasticlient/external/httpmockserver/test linked by target "test-server" in directory /data/denny/github/elasticlient/external/httpmockserver/test linked by target "tests-elasticlient" in directory /data/denny/github/elasticlient/test
Could you tell me how to fix it, thank you
I noticed that _scroll_id
is parsed only if response from ES is absolutely correct (https://github.com/seznam/elasticlient/blob/master/src/scroll.cc#L111). Maybe it should be parsed whenever _scroll_id
arrived in ES response. ES documentation says: only the most recently received _scroll_id
should be used.
I have followed every step from the README file but I get the following error:
cmake -DUSE_ALL_SYSTEM_LIBS=YES ..
-- Checking compiler C++11 support.
-- ELASTICLIENT_VERSION_MAJOR: 2
-- ELASTICLIENT_VERSION_MINOR: 1
-- ELASTICLIENT_VERSION_PATCH: 0
-- BUILD_ELASTICLIENT_TESTS: ON
-- BUILD_ELASTICLIENT_EXAMPLE: ON
-- USE_ALL_SYSTEM_LIBS: YES
-- CMAKE_MODULE_PATH: /home/vlaicu/ELASTIC_TEST/elasticlient/cmake/Modules/
-- Could NOT find HTTPMOCKSERVER (missing: HTTPMOCKSERVER_LIBRARY HTTPMOCKSERVER_INCLUDE_DIR)
CMake Error at external/CMakeLists.txt:8 (message):
HTTPMockServer library not found.
-- Configuring incomplete, errors occurred!
I have also done step 2: git submodule update --init --recursive
make error
/usr/bin/ld: not found -lgtest
collect2:error: ld returned 1 exit status
at current tip of master
externals/elasticlient0.1.0-1_src/elasticlient $ git rev-parse HEAD
3adb172a26baae1a995e810e49fee1688ea44df5
a duplicate case statement...
-- The CXX compiler identification is Clang 6.0.1
-- Check for working CXX compiler: /externals/clang6.0-0/bin/clang++
-- Check for working CXX compiler: /externals/clang6.0-0/bin/clang++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Checking compiler C++11 support.
-- ELASTICLIENT_VERSION_MAJOR: 2
-- ELASTICLIENT_VERSION_MINOR: 1
-- ELASTICLIENT_VERSION_PATCH: 0
-- BUILD_ELASTICLIENT_TESTS: ON
-- BUILD_ELASTICLIENT_EXAMPLE: ON
-- USE_ALL_SYSTEM_LIBS: OFF
-- USE_SYSTEM_JSONCPP: OFF
-- USE_SYSTEM_CPR: OFF
-- USE_SYSTEM_HTTPMOCKSERVER: OFF
-- USE_SYSTEM_GTEST: OFF
-- CMAKE_MODULE_PATH: /externals/elasticlient0.1.0-1_src/elasticlient/cmake/Modules/
-- Checking compiler C++11 support.
-- httpmockserver CMake Options
-- =======================================================
-- USE_ALL_SYSTEM_LIBS: OFF
-- USE_SYSTEM_CPR: OFF
-- USE_SYSTEM_GTEST: OFF
-- =======================================================
-- Found mhd: /usr/include
-- Found CURL: /usr/lib64/libcurl.so (found version "7.66.0")
-- Checking compiler flags for C++11 support.
-- Performing Test COMPILER_SUPPORTS_CXX11
-- Performing Test COMPILER_SUPPORTS_CXX11 - Success
-- Performing Test COMPILER_SUPPORTS_CXX0X
-- Performing Test COMPILER_SUPPORTS_CXX0X - Success
-- C++11 is supported.
-- C++ Requests CMake Options
-- =======================================================
-- USE_SYSTEM_CURL: YES
-- BUILD_CPR_TESTS: NO
-- GENERATE_COVERAGE: OFF
-- CPR_CURL_NOSIGNAL: OFF
-- USE_SYSTEM_GTEST: OFF
-- CMAKE_USE_OPENSSL: ON
-- =======================================================
-- Set CURL_FOUND to TRUE.
-- Set CURL_LIBRARIES to /usr/lib64/libcurl.so.
-- Set CURL_INCLUDE_DIRS to /usr/include.
-- Using CURL_INCLUDE_DIRS: /usr/include.
-- Using CURL_LIBRARIES: /usr/lib64/libcurl.so.
-- The C compiler identification is Clang 6.0.1
-- Check for working C compiler: /externals/clang6.0-0/bin/clang
-- Check for working C compiler: /externals/clang6.0-0/bin/clang -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found PythonInterp: /usr/bin/python (found version "2.7.17")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- JsonCpp Version: 1.8.4
-- Looking for C++ include clocale
-- Looking for C++ include clocale - found
-- Looking for localeconv
-- Looking for localeconv - found
-- Looking for C++ include sys/types.h
-- Looking for C++ include sys/types.h - found
-- Looking for C++ include stdint.h
-- Looking for C++ include stdint.h - found
-- Looking for C++ include stddef.h
-- Looking for C++ include stddef.h - found
-- Check size of lconv
-- Check size of lconv - done
-- Performing Test HAVE_DECIMAL_POINT
-- Performing Test HAVE_DECIMAL_POINT - Success
-- Configuring done
-- Generating done
-- Build files have been written to: /externals/elasticlient0.1.0-1_src/elasticlient/build
INFO - stderr: CMake Warning:
Manually-specified variables were not used by the project:
BOOST_ROOT
INFO -
DEBUG - 32 processor(s) detected, using -j31
DEBUG - run_env: {'TERM': 'xterm', 'SHLVL': '2', 'OLDPWD': '/', 'HOSTNAME': '1fdf0a7a43b6', 'MAKELEVEL': '1', 'PWD': '/externals', 'MAKE_TERMERR': '/dev/pts/0', '_': './build.py', 'HOME': '/root', 'PATH': '/usr/local/rvm/rubies/ruby-2.6.5/bin:/usr
/local/rvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'GPG_TTY': '/dev/pts/0', 'MFLAGS': '', 'GEM_HOME': '/usr/local/rvm/rubies/ruby-2.6.5', 'MAKEFLAGS': '', 'MAKE_TERMOUT': '/dev/pts/0'}
INFO - running: cd build && make, unsafe_shell=True, check_rc=build failed, retries=0
INFO - stdout: make[1]: Entering directory '/externals/elasticlient0.1.0-1_src/elasticlient/build'
make[2]: Entering directory '/externals/elasticlient0.1.0-1_src/elasticlient/build'
make[3]: Entering directory '/externals/elasticlient0.1.0-1_src/elasticlient/build'
make[3]: Leaving directory '/externals/elasticlient0.1.0-1_src/elasticlient/build'
make[3]: Entering directory '/externals/elasticlient0.1.0-1_src/elasticlient/build'
[ 2%] Building CXX object external/httpmockserver/external/cpr/cpr/CMakeFiles/cpr.dir/error.cpp.o
make[3]: Leaving directory '/externals/elasticlient0.1.0-1_src/elasticlient/build'
make[2]: Leaving directory '/externals/elasticlient0.1.0-1_src/elasticlient/build'
make[1]: Leaving directory '/externals/elasticlient0.1.0-1_src/elasticlient/build'
INFO - stderr: /externals/elasticlient0.1.0-1_src/elasticlient/external/httpmockserver/external/cpr/cpr/error.cpp:41:14: fatal error: duplicate case value 'CURLE_PEER_FAILED_VERIFICATION'
case CURLE_SSL_CACERT:
^
/usr/include/curl/curl.h:623:26: note: expanded from macro 'CURLE_SSL_CACERT'
#define CURLE_SSL_CACERT CURLE_PEER_FAILED_VERIFICATION
/externals/elasticlient0.1.0-1_src/elasticlient/external/httpmockserver/external/cpr/cpr/error.cpp:25:14: note: previous case defined here
case CURLE_PEER_FAILED_VERIFICATION:
^
1 error generated.
make[3]: *** [external/httpmockserver/external/cpr/cpr/CMakeFiles/cpr.dir/build.make:115: external/httpmockserver/external/cpr/cpr/CMakeFiles/cpr.dir/error.cpp.o] Error 1
make[2]: *** [CMakeFiles/Makefile2:191: external/httpmockserver/external/cpr/cpr/CMakeFiles/cpr.dir/all] Error 2
make[1]: *** [Makefile:141: all] Error 2
INFO -
ERROR - build failed
fixed in the meantime with this local edit... but i'm sure there's a better way
externals/elasticlient0.1.0-1_src/elasticlient/external/httpmockserver/external/cpr/cpr $ git describe --tags
1.3.0-57-g3d14e61
externals/elasticlient0.1.0-1_src/elasticlient/external/httpmockserver/external/cpr/cpr $ git diff
diff --git a/cpr/error.cpp b/cpr/error.cpp
index 713cb10..7a7f9d6 100644
--- a/cpr/error.cpp
+++ b/cpr/error.cpp
@@ -38,8 +38,8 @@ ErrorCode Error::getErrorCodeForCurlError(std::int32_t curl_code) {
return ErrorCode::SSL_LOCAL_CERTIFICATE_ERROR;
case CURLE_SSL_CIPHER:
return ErrorCode::GENERIC_SSL_ERROR;
- case CURLE_SSL_CACERT:
- return ErrorCode::SSL_CACERT_ERROR;
+// case CURLE_SSL_CACERT:
+// return ErrorCode::SSL_CACERT_ERROR;
case CURLE_USE_SSL_FAILED:
return ErrorCode::GENERIC_SSL_ERROR;
case CURLE_SSL_ENGINE_INITFAILED:
JsonCpp library used to parse scroll results is extremely slow. Scrolling through larger indices show the delay much more. In numbers: what the current implementation iterates in 85 seconds, scroll implemented using rapidjson did in 25 seconds.
Unfortunately JsonCpp is used in Scroll header interface, so the change would break API/ABI. But the speed is worth it.
Possible implementation with rapidjson (as it is header only library):
First option means that parsing has to be done twice. Internally and then for data parsing itself. Otherwise there is a risk of using different versions (binary incompatible) of rapidjson.
The second option allows shared parsing (internall and client's data) without possibility of using different versions of rapidjson (one compiled into elasticlient, another version in client code).
elasticlient/src/bulk.cc:167:18: warning: ‘Json::Reader::Reader()’ is deprecated: Use CharReader and CharReaderBuilder instead [-Wdeprecated-declarations]
167 | Json::Reader reader;
elasticlient/src/scroll.cc:62:18: warning: ‘Json::Reader::Reader()’ is deprecated: Use CharReader and CharReaderBuilder instead [-Wdeprecated-declarations]
62 | Json::Reader reader;
| ^~~~~~
Environment
I did this :
README.md
but, not build my solution. errors are
Command line error D8021 : invalid numeric argument '/Wextra' -> when I tried no.2 (pull request)
c4430 error ( If I erase /wextra from the build option, c4430 appears countless times, resulting in c1003. )
Compiling with clang
(Apple LLVM version 10.0.0) gives several warnings:
.../elasticlient/src/client.cc:79:11: warning: moving a temporary object prevents copy elision [-Wpessimizing-move]
return std::move(impl->performRequest(method, urlPath, body));
note: remove std::move call here
return std::move(impl->performRequest(method, urlPath, body));
Was building fine with CPR v1.9.3 - might be a CXX v17 issue?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.