at OsmSharp.Osm.PBF.Streams.PBFOsmStreamSource.MoveNext(Boolean ignoreNodes, Boolean ignoreWays, Boolean ignoreRelations)
at OsmSharp.Osm.Streams.Complete.OsmSimpleCompleteStreamSource.MoveNext()
at IsraelHiking.API.Services.OsmDataService.<UpdateElasticSearchFromFile>d__11.MoveNext() in D:\Github\IsraelHikingMap\Site\IsraelHiking.API\Services\OsmDataService.cs:line 82
I'm using a machine with 16Gb RAM.
I tried to use it without the complete stream and I think it didn't had this exception, but then I needed to cache the data into dictionaries and I think the issue was there when the dictionaries got too big.
the code I'm writing is something like so:
var source = new PBFOsmStreamSource(stream);
await _elasticSearchGateway.DeleteAll();
var completeSource = new OsmSimpleCompleteStreamSource(source);
var converter = new OsmGeoJsonConverter();
var list = new List<Feature>(PAGE_SIZE);
int page = 0;
foreach (var completeOsmGeo in completeSource)
{
var geoJson = converter.ToGeoJson(completeOsmGeo);
if (geoJson?.Properties == null || !geoJson.Properties.Keys.Any(k => k.Contains("name")))
{
continue;
}
list.Add(geoJson);
if (list.Count == PAGE_SIZE)
{
page++;
_logger.Info($"Indexing {PAGE_SIZE * page} records");
_elasticSearchGateway.UpdateData(list).Wait();
list.Clear();
}
}
basically what it does is convert the osm to geojson and index in bulks in elastic search.
Any advise?