Comments (2)
roman-khimov
commented on August 17, 2024
Whoa. Testnet:
Apr 19 20:03:44 titan4 neofs-node[419988]: 2024-04-19T20:03:44.575Z info log/log.go:12 local object storage operation {"shard_id": "2xcLzuvvNzKg8jELk1mn4q", "address": "cWaXnFSPr5teUdtUu8MFwsm2LJ3W7NanzBpWWmCCmeF/3PVjGxCNEyTCuhinjVR3iBmp5qhGzSAArkQWxwe6VThC", "op": "DELETE", "type": "peapod", "storage_id": "peapod"}
Apr 19 20:03:44 titan4 neofs-node[419988]: panic: runtime error: invalid memory address or nil pointer dereference
Apr 19 20:03:44 titan4 neofs-node[419988]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x138 pc=0x9a14a9]
Apr 19 20:03:44 titan4 neofs-node[419988]: goroutine 221 [running]:
Apr 19 20:03:44 titan4 neofs-node[419988]: go.etcd.io/bbolt.(*DB).page(...)
Apr 19 20:03:44 titan4 neofs-node[419988]: go.etcd.io/[email protected]/db.go:1078
Apr 19 20:03:44 titan4 neofs-node[419988]: go.etcd.io/bbolt.(*Tx).page(0xfd0da0?, 0xc075b75260?)
Apr 19 20:03:44 titan4 neofs-node[419988]: go.etcd.io/[email protected]/tx.go:533 +0x49
Apr 19 20:03:44 titan4 neofs-node[419988]: go.etcd.io/bbolt.(*Bucket).pageNode(0xc00904b680?, 0x1ef?)
Apr 19 20:03:44 titan4 neofs-node[419988]: go.etcd.io/[email protected]/bucket.go:748 +0x85
Apr 19 20:03:44 titan4 neofs-node[419988]: go.etcd.io/bbolt.(*Cursor).search(0xc005089160, {0xc012199440, 0x40, 0x40}, 0x4866acd85d38d679?)
Apr 19 20:03:44 titan4 neofs-node[419988]: go.etcd.io/[email protected]/cursor.go:272 +0x48
Apr 19 20:03:44 titan4 neofs-node[419988]: go.etcd.io/bbolt.(*Cursor).seek(0xc005089160, {0xc012199440?, 0x682e1ef6bbc51809?, 0x35f0a6fc0a89190c?})
Apr 19 20:03:44 titan4 neofs-node[419988]: go.etcd.io/[email protected]/cursor.go:159 +0x2e
Apr 19 20:03:44 titan4 neofs-node[419988]: go.etcd.io/bbolt.(*Bucket).Get(0x682e1ef6bbc51809?, {0xc012199440, 0x40, 0x4866acd85d38d679?})
Apr 19 20:03:44 titan4 neofs-node[419988]: go.etcd.io/[email protected]/bucket.go:263 +0x58
Apr 19 20:03:44 titan4 neofs-node[419988]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/peapod.(*Peapod).Delete.func1(0xc00904b680)
Apr 19 20:03:44 titan4 neofs-node[419988]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/peapod/peapod.go:434 +0x65
Apr 19 20:03:44 titan4 neofs-node[419988]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/peapod.(*Peapod).batch(0xc0001f1b90, {0x13d6830, 0x1e57220}, 0xc0050892c8)
Apr 19 20:03:44 titan4 neofs-node[419988]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/peapod/peapod.go:464 +0x113
Apr 19 20:03:44 titan4 neofs-node[419988]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/peapod.(*Peapod).Delete(0x11839a8?, {{{0x9, 0x18, 0xc5, 0xbb, 0xf6, 0x1e, 0x2e, 0x68, 0xc, ...}, ...}, ...})
Apr 19 20:03:44 titan4 neofs-node[419988]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/peapod/peapod.go:432 +0x8b
Apr 19 20:03:44 titan4 neofs-node[419988]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor.(*BlobStor).Delete(0xc0001f1d50, {{{0x9, 0x18, 0xc5, 0xbb, 0xf6, 0x1e, 0x2e, 0x68, 0xc, ...}, ...}, ...})
Apr 19 20:03:44 titan4 neofs-node[419988]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/delete.go:34 +0x159
Apr 19 20:03:44 titan4 neofs-node[419988]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard.(*Shard).delete(0xc00169f9a0, {{0xc075856000?, 0x1d2fab44c39eaa94?, 0xcb5ff16b99ac1623?}, 0x80?})
Apr 19 20:03:44 titan4 neofs-node[419988]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard/delete.go:108 +0xc18
Apr 19 20:03:44 titan4 neofs-node[419988]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard.(*Shard).removeGarbage(0xc00169f9a0)
Apr 19 20:03:44 titan4 neofs-node[419988]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard/gc.go:210 +0x331
Apr 19 20:03:44 titan4 neofs-node[419988]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard.(*gc).tickRemover(0xc001e206c0)
Apr 19 20:03:44 titan4 neofs-node[419988]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard/gc.go:170 +0xb3
Apr 19 20:03:44 titan4 neofs-node[419988]: created by github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard.(*gc).init in goroutine 1
Apr 19 20:03:44 titan4 neofs-node[419988]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard/gc.go:107 +0x13c
from neofs-node.
roman-khimov
commented on August 17, 2024
Preceeding the testnet case:
Apr 19 20:03:44 titan4 neofs-node[419988]: 2024-04-19T20:03:44.565Z info neofs-node/grpc.go:138 stopping gRPC server... {"name": "NeoFS Control API"}
Apr 19 20:03:44 titan4 neofs-node[419988]: 2024-04-19T20:03:44.566Z info neofs-node/grpc.go:154 gRPC server stopped successfully {"name": "NeoFS Control API"}
Apr 19 20:03:44 titan4 neofs-node[419988]: 2024-04-19T20:03:44.566Z info neofs-node/storage.go:76 closing components of the storage engine...
Apr 19 20:03:44 titan4 neofs-node[419988]: 2024-04-19T20:03:44.571Z info shard/gc.go:181 waiting for GC workers to stop...
Apr 19 20:03:44 titan4 neofs-node[419988]: 2024-04-19T20:03:44.571Z warn shard/gc.go:117 stop event listener by closed channel
Apr 19 20:03:44 titan4 neofs-node[419988]: 2024-04-19T20:03:44.571Z info log/log.go:12 local object storage operation {"shard_id": "2xcLzuvvNzKg8jELk1mn4q", "address": "cWaXnFSPr5teUdtUu8MFwsm2LJ3W7NanzBpWWmCCmeF/3GH4MNDCMXpkd8rWw56Q8eGPF3ogxyVqSkYXzXiCnPDd", "op": "DELETE", "type": "peapod", "storage_id": "peapod"}
Apr 19 20:03:44 titan4 neofs-node[419988]: 2024-04-19T20:03:44.574Z info shard/gc.go:181 waiting for GC workers to stop...
Apr 19 20:03:44 titan4 neofs-node[419988]: 2024-04-19T20:03:44.574Z warn shard/gc.go:117 stop event listener by closed channel
Apr 19 20:03:44 titan4 neofs-node[419988]: 2024-04-19T20:03:44.575Z info log/log.go:12 local object storage operation {"shard_id": "2xcLzuvvNzKg8jELk1mn4q", "address": "cWaXnFSPr5teUdtUu8MFwsm2LJ3W7NanzBpWWmCCmeF/3PVjGxCNEyTCuhinjVR3iBmp5qhGzSAArkQWxwe6VThC", "op": "DELETE", "type": "peapod", "storage_id": "peapod"}
It's a concurrent DB Close() issue, GC runs in its own thread and tries to do things after the DB is closed. Not critical for now (we may deprecate peapods after #2814).
from neofs-node.
Related Issues (20)
- Endless deletion of the same object
- Dropping object via control service leaves traces in the metabase
- GC stops on metabase errors
- Run neofs-testcases for linux-arm64 HOT 5
- Contract deployment with custom zone failed in tests
- Fetching strategy if object is not found locally
- Policer error for REP 1 CBF 1 container HOT 1
- Replicator can't fetch local object HOT 1
- Evaluate tree service performance HOT 7
- Handle server-side crypto failures gracefully
- Range test failure for complex objects HOT 2
- Deleted object is displayed in the list of container object HOT 2
- List objects via control service
- FS chain deployment stucks on error in forming a role assignment transaction
- GC has not collected an expired object HOT 2
- Evacuate tests fail HOT 2
- remove neogo_version metric HOT 2
- Ersatz-chain for object headers HOT 6
- Improve GC routine HOT 1
- Do not skip shards that are busy
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from neofs-node.