How to setup neptune export? about amazon-neptune-tools HOT 15 CLOSED

@nilesh-patil To run it on EC2, you would need to install the Git tools, Maven, and a JDK on the EC2 instance, clone the repository, and build it via Maven per https://github.com/awslabs/amazon-neptune-tools/tree/master/neptune-export#building-neptune-export.

from amazon-neptune-tools.

@SivaVijayakumar You may need to increase the size of your Neptune instance. With order().by(...) all of the results for the query need to be materialized before the results are ordered. The limit(...) step does not reduce the amount of data that needs be returned. You can test by removing the order by and trying with just the limit.

from amazon-neptune-tools.

Hey @SivaVijayakumar we did end up using a larger instance to which query was pointed. Overall - our use case was more of a data export rather than analytics on the Neptune instance - and we just directly export the whole graph to S3.

from amazon-neptune-tools.

This worked out for us, I'll add a few details for anyone who lands up here.

from amazon-neptune-tools.

@nilesh-patil hi Nilesh. I am trying to do bulk export using this module. I am getting the below error. Please let me know, how to resolve this issue?

Query:
gremlin> g.V().hasLabel('lableName').order().by('lastVisitedDate').limit(20).valueMap(true)

Response:
{"requestId":"23761783-c777-407c-bebf-ffd55fd55ff3","code":"MemoryLimitExceededException","detailedMessage":"Query cannot be completed due to memory limitations."}

from amazon-neptune-tools.

@beebs-systap Thank you very much. let me try this step.

from amazon-neptune-tools.

Running on EC2 is giving following stacktrace:

./bin/neptune-export.sh export-pg -e <cluster_endpoint> -d /home/ec2-user/output
Creating nodes metadata

Completed creating nodes metadata from graph scan in 15 seconds

Completed export-pg in 17 seconds
An error occurred while exporting from Neptune:
java.lang.IllegalStateException: org.apache.tinkerpop.gremlin.process.remote.RemoteConnectionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
at org.apache.tinkerpop.gremlin.process.remote.traversal.step.map.RemoteStep.promise(RemoteStep.java:98)
at org.apache.tinkerpop.gremlin.process.remote.traversal.step.map.RemoteStep.processNextStart(RemoteStep.java:65)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:128)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:38)
at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.next(DefaultTraversal.java:205)
at org.apache.tinkerpop.gremlin.process.traversal.Traversal.forEachRemaining(Traversal.java:272)
at com.amazonaws.services.neptune.propertygraph.NodesClient.queryForMetadata(NodesClient.java:61)
at com.amazonaws.services.neptune.propertygraph.metadata.ExportSpecification.scan(ExportSpecification.java:48)
at com.amazonaws.services.neptune.propertygraph.metadata.CreateMetadataFromGraphScan.execute(CreateMetadataFromGraphScan.java:36)
at com.amazonaws.services.neptune.ExportPropertyGraph.run(ExportPropertyGraph.java:97)
at com.amazonaws.services.neptune.NeptuneExportCli.main(NeptuneExportCli.java:54)
Caused by: org.apache.tinkerpop.gremlin.process.remote.RemoteConnectionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
at org.apache.tinkerpop.gremlin.driver.remote.DriverRemoteConnection.submitAsync(DriverRemoteConnection.java:226)
at org.apache.tinkerpop.gremlin.process.remote.traversal.step.map.RemoteStep.promise(RemoteStep.java:89)
... 10 more
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
at org.apache.tinkerpop.gremlin.driver.Client$AliasClusteredClient.submitAsync(Client.java:571)
at org.apache.tinkerpop.gremlin.driver.remote.DriverRemoteConnection.submitAsync(DriverRemoteConnection.java:224)
... 11 more
Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
at org.apache.tinkerpop.gremlin.driver.Client.submitAsync(Client.java:371)
at org.apache.tinkerpop.gremlin.driver.Client$AliasClusteredClient.submitAsync(Client.java:589)
at org.apache.tinkerpop.gremlin.driver.Client$AliasClusteredClient.submitAsync(Client.java:569)
... 12 more
Caused by: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
at org.apache.tinkerpop.gremlin.driver.Client$ClusteredClient.chooseConnection(Client.java:495)
at org.apache.tinkerpop.gremlin.driver.Client$AliasClusteredClient.chooseConnection(Client.java:628)
at org.apache.tinkerpop.gremlin.driver.Client.submitAsync(Client.java:366)
... 14 more

from amazon-neptune-tools.

Can you validate that you can reach the cluster endpoint from our client, curl https://<cluster_endpoint>:8182/status?

from amazon-neptune-tools.

Can you validate that you can reach the cluster endpoint from our client, curl https://<cluster_endpoint>:8182/status?

curl https://<cluster_endpoint>:8182/status
{"status":"healthy","startTime":"Wed Jul 22 17:52:40 UTC 2020","dbEngineVersion":"1.0.2.2.R2","role":"writer","gremlin":{"version":"tinkerpop-3.4.3"},"sparql":{"version":"sparql-1.1"},"labMode":{"ObjectIndex":"disabled","ReadWriteConflictDetection":"enabled"}}

from amazon-neptune-tools.

Can you validate that you can reach the cluster endpoint from our client, curl https://<cluster_endpoint>:8182/status?

curl https://<cluster_endpoint>:8182/status
{"status":"healthy","startTime":"Wed Jul 22 17:52:40 UTC 2020","dbEngineVersion":"1.0.2.2.R2","role":"writer","gremlin":{"version":"tinkerpop-3.4.3"},"sparql":{"version":"sparql-1.1"},"labMode":{"ObjectIndex":"disabled","ReadWriteConflictDetection":"enabled"}}

With debug log level enabled:

[main] DEBUG org.apache.tinkerpop.gremlin.driver.Connection - Error opening connection on ws://<cluster_endpoint>:8182/gremlin
[main] DEBUG org.apache.tinkerpop.gremlin.driver.ConnectionPool - Could not initialize connections in pool for Host{address=<cluster_endpoint>/<cluster_IP>:8182, hostUri=ws://<cluster_endpoint>:8182/gremlin} - pool size at 0
[main] WARN org.apache.tinkerpop.gremlin.driver.Host - Marking Host{address=<cluster_endpoint>/<cluster_IP>:8182, hostUri=ws://<cluster_endpoint>:8182/gremlin} as unavailable. Trying to reconnect.
[main] INFO org.apache.tinkerpop.gremlin.driver.ConnectionPool - Opening connection pool on Host{address=<cluster_endpoint>/<cluster_IP>:8182, hostUri=ws://<cluster_endpoint>:8182/gremlin} with core size of 2
[main] DEBUG org.apache.tinkerpop.gremlin.driver.Client - Submitted RequestMessage{, requestId=24f9872d-28e5-4040-8d88-66b71d6f55b2, op='bytecode', processor='traversal', args={gremlin=[[], [V(), valueMap(true)]], aliases={g=g}}} to - connection not initialized

from amazon-neptune-tools.

Wondering if you have SSL-only. Does curl http://<cluster_endpoint>:8182/status work?

from amazon-neptune-tools.

Wondering if you have SSL-only. Does curl http://<cluster_endpoint>:8182/status work?

curl http://<cluster_endpoint>:8182/status
curl: (52) Empty reply from server

from amazon-neptune-tools.

Please try adding the --use-ssl option to the neptune_export command. See [1].

from amazon-neptune-tools.

Please try adding the --use-ssl option to the neptune_export command. See [1].

Thanks. It did the trick!

from amazon-neptune-tools.

This worked out for us, I'll add a few details for anyone who lands up here.

Can you share detailed steps of how it was done?

from amazon-neptune-tools.