Comments (6)
Looks like in DoCommand a bulk request will give a 200 status code and return the errors that came back from the bulk request so this code doesn't take that into consideration
httpStatusCode, body, err = req.Do(&response)
if err != nil {
return body, err
}
if httpStatusCode > 304 {
from elastigo.
I have an update to the BulkSend function which gets errors put on the channel. Example code below, let me know if you want it as a 2.0 pull request. (I'd move the struct out of the function, just there for brevity of code example).
The idea is that the bulk response is 200 OK but contains a JSON object that indicates how many errors that were returned. You have to unmarshall the JSON blob and check to see if the errors == true. This correctly now puts an error message on the ErrorsChannel.
// This does the actual send of a buffer, which has already been formatted
// into bytes of ES formatted bulk data
func BulkSend(buf *bytes.Buffer) error {
type responseStruct struct {
Took int64 `json:"took"`
Errors bool `json:"errors"`
Items []map[string]interface{} `json:"items"`
}
response := responseStruct{}
body, err := api.DoCommand("POST", "/_bulk", nil, buf)
if err != nil {
BulkErrorCt += 1
return err
}
// check for response errors, bulk insert will give 200 OK but then include errors in response
jsonErr := json.Unmarshal(body, &response)
if jsonErr == nil {
if response.Errors {
BulkErrorCt += uint64(len(response.Items))
return fmt.Errorf("Bulk Insertion Error. Failed item count [%d]", len(response.Items))
}
}
return nil
}
from elastigo.
You might consider using
return fmt.Errorf("Bulk Insertion Error... [%d]", len(response.Items))
from elastigo.
updated based on schleppy's comment
from elastigo.
@mattbaird looks like you added the code example in commit:
dbd553d
want me to close this issue?
from elastigo.
yes please.
from elastigo.
Related Issues (20)
- Issue with reusing connection in another go procedure.. HOT 1
- Testing package is being included due to lib/testsetup.go HOT 2
- tracking individual messages successes or failures in bulkindexer HOT 1
- not support bool query and bool filter? HOT 2
- Race Conditions in corebulk.go
- Index documents to a whole cluster?
- Adding support for the 2.x Query API (and maybe other stuff)
- strings not escaped in WriteBulkBytes
- Support json.UseNumber option HOT 2
- Error Handling
- Possible file descriptor leaks? HOT 2
- making bulkChannel buffer size configurable?
- Amazon ElasticSearch support HOT 1
- Occasional Failed to derive xcontent in bulk index
- MAX_SHUTDOWN_SECS make me get into trouble
- DELETE /_query endpoint removed
- Unmarshal error of parsing shards failures in SearchResult
- gzipped requests omit content-type header
- requests are sent without content-type header HOT 3
- No timeout set for search function in file "coresearch.go"
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elastigo.