Comments (8)
fvaleye
commented on May 18, 2024
2
@lucienfregosi could you share your environment with us? including kernel and libc (or musl) version. We recently implemented concurrent multi-writer support for the filesystem backend using
renameat2system (#126). Here is the version requirements for this system call:
- kernel >= 3.15
- glibc >= 2.28
The CI used Ubuntu 16.04 LTS distribution by default.
glibc version is 2.23
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial
We changed the distribution to Ubuntu 20.04.2 LTS and it is working fine!
glibc version is 2.31
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal
from delta-rs.
houqp
commented on May 18, 2024
@lucienfregosi could you share your environment with us? including kernel and libc (or musl) version. We recently implemented concurrent multi-writer support for the filesystem backend using renameat2 system (#126). Here is the version requirements for this system call:
- kernel >= 3.15
- glibc >= 2.28
from delta-rs.
houqp
commented on May 18, 2024
double checked the manylinux spec, looks like it does use a very old version of glibc: https://www.python.org/dev/peps/pep-0571/.
I am going to yank that release out of pypi until we lands a fallback for other linux kernel/userspace configurations.
from delta-rs.
houqp
commented on May 18, 2024
relevant discussion in the libc project on what's the minimal supported glibc version: rust-lang/libc#1412. Right now they require >= 2.26, but has already added features that would only work for 2.27 and 2.28. renameat2 is one of the feature that would only work for 2.28 even though we are not using the libc crate for syscall invocation at the moment.
from delta-rs.
lucienfregosi
commented on May 18, 2024
All good with ubuntu 20.04 :)
We can close the issue I guess !
from delta-rs.
jeisinge
commented on May 18, 2024
FYI - I'm seeing a similar issue on Databricks Runtime 7.6.
from delta-rs.
houqp
commented on May 18, 2024
@jeisinge will be fixed with a new release after #148, hopefully by end of tomorrow.
from delta-rs.
houqp
commented on May 18, 2024
FYI - I'm seeing a similar issue on Databricks Runtime 7.6.
As a workaround for now, please manually install a version lower than 0.4.2
from delta-rs.
Related Issues (20)
- Add `UNSET TBLPROPERTIES` HOT 6
- Add equivalent of ConfigurationKey in Python as helper to set config settings
- _internal.DeltaError when merging HOT 19
- Unexpected high costs on Google Cloud Storage HOT 5
- Cant create empty table with write_deltalake HOT 2
- After dt.optimize.compact(), commitInfo.operationMetrics.filesAdded is a JSON map when other readers (e.g. Databricks) expect a string HOT 3
- ALTER table, ALTER Column, Add/Modify Comment, Add/remove/rename partitions, Set Tags, Set location, Set TBLProperties HOT 1
- Partitions not pushed down HOT 1
- Change Data Feed in Delta HOT 2
- convert to delta, add support for overwrite existing Table HOT 1
- MERGE logical plan vs execution plan schema mismatch HOT 7
- `WriteBuilder::with_input_execution_plan` does not apply the schema to the log's metadata fields
- Merge execution time grows exponetially with the number of column HOT 4
- Updating Table log retention configuration with `write_deltalake` silently changes nothing HOT 4
- test_repair_on_update broken in main HOT 1
- Is this print necessary? Can we remove this. HOT 1
- Pandas import error when merging tables HOT 2
- Possibly expose a DeltaTable#query method HOT 11
- Add AWS Credentials parsing from file HOT 9
- Expose the ability to compile DataFusion with SIMD HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from delta-rs.