MapperParsingException on Bulk Index about elasticsearch-net HOT 10 CLOSED

The Index(IEnumerable) method doesn't partition for you so it will try to do a single call. Its best to partition yourself and index for every 1000 to 10,000 items going by how many data each items contains.

To speed up indexing i'd use the IndexAsync(IEnumerable) which returns a task you can wait on while it processes asynchronously in the background. There's a demo project that indexes a hacker news which might be helpful.

If that doesn't resolve your issue please paste the response you get from ElasticSearch (query ConnectionStatus.Response and ConnectionStatus.Request or use fiddler).

from elasticsearch-net.

It didn't solve it, I even lowered my batches down to 2 in size. Batching code at the bottom as well.

response: {"error":"MapperParsingException[Malformed content, must start with an object]","status":400}
request: [{"divisions":[{"id":{"timestamp":1334773504,"machine":8097553,"pid":6552,"increment":8473475,"creationTime":"2012-04-18T18:25:04Z"}}],"type":1,"primaryDemo":{"id":533},"years":[2011],"name":"2Q'10 x.com week 3/28","harborTeamId":44,"advertiser":{"id":10643},"agency":{"id":31065},"agencyOffice":{"id":21035},"salesSplit":{"id":16537},"salesPlanner":{"id":3248},"spendingGoal":0.0,"cpmGoal":0.0,"id":{"timestamp":1338490981,"machine":14860174,"pid":4452,"increment":936677,"creationTime":"2012-05-31T19:03:01Z"},"brand":{"id":20862,"advertiserId":0},"calendar":0},{"divisions":[{"id":{"timestamp":1334773504,"machine":8097553,"pid":6552,"increment":8473471,"creationTime":"2012-04-18T18:25:04Z"}}],"type":0,"primaryDemo":{"id":36},"years":[2011,2012],"name":"11/12 Upfront","harborTeamId":3,"advertiser":{"id":2513},"agency":{"id":20660},"agencyOffice":{"id":10645},"salesSplit":{"id":21284},"salesPlanner":{"id":3700},"spendingGoal":0.0,"cpmGoal":0.0,"id":{"timestamp":1338490981,"machine":14860174,"pid":4452,"increment":960000,"creationTime":"2012-05-31T19:03:01Z"},"brand":{"id":1,"advertiserId":0},"calendar":0}]

ElasticClient.DeleteIndex();

var repo = Dependency.Container.Resolve<IRepository>();
var deals = repo.GetAll();
var batchSize = 2;

var current = 0;
var results = deals.Skip(0).Take(batchSize).ToList();
var tasks = new List<Task>();

while (results.Any() && current < 6)
{
var copy = results.ToArray();
tasks.Add(ElasticClient.IndexAsync(copy));
tasks.Last().ContinueWith(i => Console.WriteLine(i.Result.Result));
current += batchSize;
results = deals.Skip(current).Take(batchSize).ToList();
}

Task.WaitAll(tasks.ToArray());

from elasticsearch-net.

Ahh i think i know whats going wrong here. C# infers the wrong generic method here.

Try explicit typing the index call tasks.Add(ElasticClient.IndexAsync<Division>(copy));

The copying itself should not be neccessary either.

Should i rename the IEnumerable overloaded index methods to IndexMany so c# will infer them better?

from elasticsearch-net.

I think that makes sense, most people will not know to use the explicit version. That said, I'm now getting this exception with the change:

System.InvalidCastException occurred
  HResult=-2147467262
  Message=Object must implement IConvertible.
  Source=mscorlib
  StackTrace:
       at System.Convert.ChangeType(Object value, Type conversionType, IFormatProvider provider)
       at Nest.ElasticClient.b__37[T](T object) in c:\Users\driggins\Documents\GitHub\NEST\src\Nest\ElasticClient-Inferno.cs:line 36
  InnerException: 

from elasticsearch-net.

Try get the latest version from Nuget (or update your checkout).

It looks like its using an older version of ElasticClient-Inferno.cs
which is a couple commits behind.

For reference though, as i've never hit that error before, exactly what type is your Id property here ?

Thanks for your input! I'll update the calls to IndexMany.

from elasticsearch-net.

It's a Mongo ObjectId type. I'll try updating now.

from elasticsearch-net.

Ok in the update it should work with anything that implements a ToString() which makes sense to use as Id.

Curious if you have better luck now with Mongo's ObjectId here, let me know.

from elasticsearch-net.

Just pushed 0.9.1.0 to nuget which contains the rename from Index(IEnumerable) to IndexMany(IEnumerable)

from elasticsearch-net.

Works perfectly, also the new IndexMany worked great, was able to index everything without batching as well. Thanks for all the help!

from elasticsearch-net.

Awesome, glad to know everything works a lot smoother now :)

from elasticsearch-net.