Comments (15)
@nilesh-patil To run it on EC2, you would need to install the Git tools, Maven, and a JDK on the EC2 instance, clone the repository, and build it via Maven per https://github.com/awslabs/amazon-neptune-tools/tree/master/neptune-export#building-neptune-export.
from amazon-neptune-tools.
@SivaVijayakumar You may need to increase the size of your Neptune instance. With order().by(...) all of the results for the query need to be materialized before the results are ordered. The limit(...) step does not reduce the amount of data that needs be returned. You can test by removing the order by and trying with just the limit.
from amazon-neptune-tools.
Hey @SivaVijayakumar we did end up using a larger instance to which query was pointed. Overall - our use case was more of a data export rather than analytics on the Neptune instance - and we just directly export the whole graph to S3.
from amazon-neptune-tools.
This worked out for us, I'll add a few details for anyone who lands up here.
from amazon-neptune-tools.
@nilesh-patil hi Nilesh. I am trying to do bulk export using this module. I am getting the below error. Please let me know, how to resolve this issue?
Query:
gremlin> g.V().hasLabel('lableName').order().by('lastVisitedDate').limit(20).valueMap(true)
Response:
{"requestId":"23761783-c777-407c-bebf-ffd55fd55ff3","code":"MemoryLimitExceededException","detailedMessage":"Query cannot be completed due to memory limitations."}
from amazon-neptune-tools.
@beebs-systap Thank you very much. let me try this step.
from amazon-neptune-tools.
Running on EC2 is giving following stacktrace:
./bin/neptune-export.sh export-pg -e <cluster_endpoint> -d /home/ec2-user/output
Creating nodes metadata
Completed creating nodes metadata from graph scan in 15 seconds
Completed export-pg in 17 seconds
An error occurred while exporting from Neptune:
java.lang.IllegalStateException: org.apache.tinkerpop.gremlin.process.remote.RemoteConnectionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
at org.apache.tinkerpop.gremlin.process.remote.traversal.step.map.RemoteStep.promise(RemoteStep.java:98)
at org.apache.tinkerpop.gremlin.process.remote.traversal.step.map.RemoteStep.processNextStart(RemoteStep.java:65)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:128)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:38)
at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.next(DefaultTraversal.java:205)
at org.apache.tinkerpop.gremlin.process.traversal.Traversal.forEachRemaining(Traversal.java:272)
at com.amazonaws.services.neptune.propertygraph.NodesClient.queryForMetadata(NodesClient.java:61)
at com.amazonaws.services.neptune.propertygraph.metadata.ExportSpecification.scan(ExportSpecification.java:48)
at com.amazonaws.services.neptune.propertygraph.metadata.CreateMetadataFromGraphScan.execute(CreateMetadataFromGraphScan.java:36)
at com.amazonaws.services.neptune.ExportPropertyGraph.run(ExportPropertyGraph.java:97)
at com.amazonaws.services.neptune.NeptuneExportCli.main(NeptuneExportCli.java:54)
Caused by: org.apache.tinkerpop.gremlin.process.remote.RemoteConnectionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
at org.apache.tinkerpop.gremlin.driver.remote.DriverRemoteConnection.submitAsync(DriverRemoteConnection.java:226)
at org.apache.tinkerpop.gremlin.process.remote.traversal.step.map.RemoteStep.promise(RemoteStep.java:89)
... 10 more
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
at org.apache.tinkerpop.gremlin.driver.Client$AliasClusteredClient.submitAsync(Client.java:571)
at org.apache.tinkerpop.gremlin.driver.remote.DriverRemoteConnection.submitAsync(DriverRemoteConnection.java:224)
... 11 more
Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
at org.apache.tinkerpop.gremlin.driver.Client.submitAsync(Client.java:371)
at org.apache.tinkerpop.gremlin.driver.Client$AliasClusteredClient.submitAsync(Client.java:589)
at org.apache.tinkerpop.gremlin.driver.Client$AliasClusteredClient.submitAsync(Client.java:569)
... 12 more
Caused by: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
at org.apache.tinkerpop.gremlin.driver.Client$ClusteredClient.chooseConnection(Client.java:495)
at org.apache.tinkerpop.gremlin.driver.Client$AliasClusteredClient.chooseConnection(Client.java:628)
at org.apache.tinkerpop.gremlin.driver.Client.submitAsync(Client.java:366)
... 14 more
from amazon-neptune-tools.
Can you validate that you can reach the cluster endpoint from our client, curl https://<cluster_endpoint>:8182/status
?
from amazon-neptune-tools.
Can you validate that you can reach the cluster endpoint from our client,
curl https://<cluster_endpoint>:8182/status
?
curl https://<cluster_endpoint>:8182/status
{"status":"healthy","startTime":"Wed Jul 22 17:52:40 UTC 2020","dbEngineVersion":"1.0.2.2.R2","role":"writer","gremlin":{"version":"tinkerpop-3.4.3"},"sparql":{"version":"sparql-1.1"},"labMode":{"ObjectIndex":"disabled","ReadWriteConflictDetection":"enabled"}}
from amazon-neptune-tools.
Can you validate that you can reach the cluster endpoint from our client,
curl https://<cluster_endpoint>:8182/status
?curl https://<cluster_endpoint>:8182/status
{"status":"healthy","startTime":"Wed Jul 22 17:52:40 UTC 2020","dbEngineVersion":"1.0.2.2.R2","role":"writer","gremlin":{"version":"tinkerpop-3.4.3"},"sparql":{"version":"sparql-1.1"},"labMode":{"ObjectIndex":"disabled","ReadWriteConflictDetection":"enabled"}}
With debug log level enabled:
[main] DEBUG org.apache.tinkerpop.gremlin.driver.Connection - Error opening connection on ws://<cluster_endpoint>:8182/gremlin
[main] DEBUG org.apache.tinkerpop.gremlin.driver.ConnectionPool - Could not initialize connections in pool for Host{address=<cluster_endpoint>/<cluster_IP>:8182, hostUri=ws://<cluster_endpoint>:8182/gremlin} - pool size at 0
[main] WARN org.apache.tinkerpop.gremlin.driver.Host - Marking Host{address=<cluster_endpoint>/<cluster_IP>:8182, hostUri=ws://<cluster_endpoint>:8182/gremlin} as unavailable. Trying to reconnect.
[main] INFO org.apache.tinkerpop.gremlin.driver.ConnectionPool - Opening connection pool on Host{address=<cluster_endpoint>/<cluster_IP>:8182, hostUri=ws://<cluster_endpoint>:8182/gremlin} with core size of 2
[main] DEBUG org.apache.tinkerpop.gremlin.driver.Client - Submitted RequestMessage{, requestId=24f9872d-28e5-4040-8d88-66b71d6f55b2, op='bytecode', processor='traversal', args={gremlin=[[], [V(), valueMap(true)]], aliases={g=g}}} to - connection not initialized
from amazon-neptune-tools.
Wondering if you have SSL-only. Does curl http://<cluster_endpoint>:8182/status
work?
from amazon-neptune-tools.
Wondering if you have SSL-only. Does
curl http://<cluster_endpoint>:8182/status
work?
curl http://<cluster_endpoint>:8182/status
curl: (52) Empty reply from server
from amazon-neptune-tools.
Please try adding the --use-ssl
option to the neptune_export command. See [1].
from amazon-neptune-tools.
Please try adding the
--use-ssl
option to the neptune_export command. See [1].
Thanks. It did the trick!
from amazon-neptune-tools.
This worked out for us, I'll add a few details for anyone who lands up here.
Can you share detailed steps of how it was done?
from amazon-neptune-tools.
Related Issues (20)
- [export-neptune-to-elasticsearch] Encrypted databases not supported
- csv-gremlin should trim whitespace in the header row
- CSV file having URL data HOT 8
- Add vertex label to Neptune stream records when a vp is updated HOT 1
- Neptune export with AWS Batch has restrictive config length limit
- Need 2 csv files in opencypher data load format HOT 1
- [Java] Localhost host used on transaction in bytecode queries HOT 5
- log4j 1.2.8, 1.2.17 HOT 1
- [python-utils] Example for multi-valued properties HOT 4
- [csv-to-neptune-bulk-format] : upsert HOT 1
- [csv-to-neptune-bulk-format] : Multiple relational tables as source CSV files HOT 3
- problems with streaming to kinesis with neptune-export tool HOT 15
- Uploading exported data to s3 without storing at the local disk HOT 1
- Upsert raising "edge already existing" error. HOT 1
- support OpenSearch 1.x in non-compatibility mode and 2.x
- ReservedConcurrentExecutions error when trying to deploy the cloudformation
- neptune streams-to-firehose build.sh HOT 1
- When running neptune-python-utils in Glue 4, the job fails due to urllib3 conflict
- Issues with Importing GraphML File in AWS Neptune Engine 1.2.1.0 HOT 2
- Implementing Bidirectional Edge Creation in csv-to-neptune-bulk-format
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amazon-neptune-tools.