Setting StorageLevel to MEMORY_AND_DISK about sedona HOT 4 CLOSED

Hi @ignasva ,

You understand the code correctly. I hardcoded it because spilling onto disk actually is not very time efficient.

To solve this, you don't have to make your own GeoSpark build. Here is an easy workaround (but not perfect):

You can directly use GeoSpark Maven Coordinate. Since you checked the code, you may notice that all SpatialRDD attributes are "public". You can directly call the attribute in your code and set to the persist level you want. For example,

myPointRDD.rawPointRDD.persist(MEMORY_AND_DISK_SER);

myPointRDD.indexedRDD.persist(MEMORY_AND_DISK_SER);

Of course, making your own build may give you the perfect control.

Thanks,
Jia

from sedona.

Hi @jiayuasu,

At first this sounded like a good idea, however, apparently you cannot change the storage level after a Spark RDD is cached. This is the resulting exception:

java.lang.UnsupportedOperationException: Cannot change storage level of an RDD after it was already assigned a level

Anyway, it would be awesome to be able to control the storage level in future releases. Keep up the good work!

from sedona.

Hi @ignasva ,

Thanks for pointing this out. I should give the granular control back to the user. I will change the cache strategy in GeoSpark.

Thanks,
Jia

from sedona.

Solved in GeoSpark 0.4.0

from sedona.