Spaial Matching query performance about magellan HOT 3 CLOSED

can you share the cars and radars dataset?

On Tue, May 10, 2016 at 8:36 AM, Mustafa [email protected] wrote:

Hi,

I am trying to execute a within query with Magellan since two weeks,
and the query always executes finitely ..

The first dataframe is created using GeoJSON type loader .. A record of
the dataset is shown below :-

{"type": "FeatureCollection", "features": [{"geometry": {"type":
"Polygon", "coordinates": [[[-83.472, 42.71800120413723], [-83.472,
42.7202513456291]
, [-83.47152052359189, 42.72024826025985], [-83.47157381414216,
42.717998461684836], [-83.472, 42.71800120413723]]]}, "type": "Feature",
"id": 0, "pro
perties": {"startTime": 1365674580000.0, "endTime": 1365674940000.0,
"colorIndex": 6, "value": 0.2171378}}

The second dataframe contains data from a CSV file .. A case class is
created before loading the data using Spark textFile() .. The case class is
found below :-

case class CarTraces(deviceId: Long, trip: String, gpsUtcTime: Double,
point: Point)

I am executing the query on AWS EC2 m4.xlarge instance .. The two datasets
are only a sample, geoJson file is 200 MB, and the other one is only 80
MB ..

the query executed is the following :-

cars.join(radars).where($"point".within($"polygon)).cache()

Although the show() call is working fine on the resulting dataframe,
whenever I call collect() or write(), the execution kept going and
never ends ..

I will appreciate any suggestion to resolve the problem, since I am stuck
in this issue since two weeks.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#60

Ram Sriharsha
Product Manager, Apache Spark
PPMC Member and Committer, Apache Spark
Databricks
San Francisco, CA
Ph: 408-510-8635
email: [email protected]

[image: https://www.linkedin.com/in/harsha340]
https://www.linkedin.com/in/harsha340 https://twitter.com/halfabrane
https://github.com/harsha2010/

from magellan.

Thanks for prompt reply .. The data can be found here

https://www.dropbox.com/sh/34uhf0lu8xosqdj/AAAuSmrc-X_TAEgjxcf0J3-Ga?dl=0

from magellan.

Any updates regaring this issue ?

from magellan.