Comments (1)
There is no direct way to fix this if you use Databricks. Sedona's in-memory raster computation engine is not intended to load large GeoTiff in-memory. It is designed to deal with a massive amount of small geotiff images.
The correct way to do this is, split this huge image to small tiffs files on S3, then load them to Sedona.
SedonaDB from Wherobots (https://wherobots.com/) offers a new raster processing mode called out-db
mode (https://docs.wherobots.services/latest/references/havasu/raster/out-db-rasters/). It can solve this exact problem.
df = sedona.read.format("binaryFile")
.load("s3a://XXX/*.tif")
.drop("content").withColumn("rast", expr("RS_FromPath(path)"))
df.selectExpr("RS_TileExplode(rast) as (x, y, rast)").show()
If you are interested, please try it on Wherobots Cloud (https://www.wherobots.services/)
from sedona.
Related Issues (20)
- AttributeError: 'sedona' has no attribute 'read' HOT 2
- St_isempty(geometry) finds non null geometries but does not find null geometries. HOT 2
- ST_Snap example code does not work HOT 2
- Flink Sedona,geomTbl.execute().print() happen error: HOT 2
- Sedona fails to write Delta Lake on Databricks 15.3 Beta: ClassCastException HOT 7
- ST_IsPolygonCW, ST_IsPolygonCCW, ST_ForcePolygonCW and ST_ForcePolygonCCW fails on Polygons without interior ring
- Breaking change between 1.5.3 and 1.6.0 affecting RASTER functions java.lang.NoSuchMethodError: void org.geotools.coverage.grid.GridGeometry2D HOT 5
- There was garbled code when reading Chinese from shp file HOT 3
- sedona's docker image can not run successfully on k8s HOT 2
- Cannot run sedona examples into spark-shell HOT 3
- Using setMaster(“ spark://master:7077) An error occurred during operation, but there is no problem running locally HOT 3
- Sedona website homepage layout bug in mobile view
- Sedona website favicon checker issues and missing files
- An error occurred when submitting the file. The actual directory does exist.
- Initialization of sedona 1.6.0 is quite slower than sedona 1.5.1 HOT 2
- 1.6.0 documentation needs update? HOT 6
- st_union - understanding difference with PostGIS HOT 6
- Issue with st_dump? HOT 2
- ShapefileReader with Unity Catalogue on Databricks HOT 14
- ST_ClusterDBSCAN Feature Request HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sedona.