neofs-cli needs to work with Storage policy in human friendly format. There should be:

• policy module provinding a way to write a policy in human format (SQL-like, JSON) and apply it netmap (current or pre-constructed)
• container module has to accept human friendly format as storage policy on container creation
• as a best effort, it would be nice to have the ability to translate storage policy of already existing containers from protobuf format into human-readable form.

Inner ring nodes periodically transfer sidechain GAS to storage nodes so they can send sidechain txs: create container, bootstrap, etc.

According to #139 (comment) there is a possible case when network map has more nodes than available GAS to transfer. We can solve this issue with several approaches:

Do not do anything

This situation is possible when there are too many nodes in network map. If we set available GAS amount to 2(Fixed8), then network map should contain more than 200 000 000 nodes, which is HIGHLY unlikely.

However event if there are less nodes in network map, each node can get insufficient amount of GAS to operate with sidechain. In this case it should wait several epochs to get enough GAS. It is quite unpleasant for the users and node holders so we might think about alternatives.

Increase GAS distribution amount

• adjust GAS distribution value locally on each node,
• use global config value in sidechain or mainnet contracts.

In first case we calculate GAS distribution value in runtime automatically, which is good. But it should stay in some limits so it won't be abused and inner ring node won't spend all it's remaining GAS.

Second option is more explicit and controllable, but takes out-of-chain coordination of inner ring node holders.

Send GAS to a subset of nodes

We can choose a set of nodes that inner ring will transfer a GAS. E.g. inner ring node sends GAS to network map nodes where storageNodeListIndex % innerRingListIndex == 0. Inner ring nodes stays in the GAS distribution amount limit, but some storage node still can get insufficient amount of GAS, see first option.

Both object and util commands use cmd.Println / cmd.Printf. However if output destination is not explicitly set, then cobra produces output into stderr which is not nice.

We have to either set output to os.Stdout or use fmt.Println in those cases.

Some smart-contracts have strings as method arguments. While VM can interpret any []byte as strings in contracts, this can be changed later with stronger typing.

toStackParamether method converts native go types to smartcontract.Parameter and it should support strings.

Hash range with non-zero offset returns invalid value for non-split object. I compare neofs-node results with combination of xxd and sha256 output:

xxd -s <offset> -l <length> <filepath> | xxd -r | sha256sum

Current Behavior

✔️ Whole file

$ neofs-cli -c ../config.yml object hash --cid <> --oid <>
83f19b8a49e9655e7d704f5ee6547fa53dc7f5a524035ca589f365cb4e4be3d9
$ xxd -s 0 -l 1626 ../go.mod | xxd -r | sha256sum
83f19b8a49e9655e7d704f5ee6547fa53dc7f5a524035ca589f365cb4e4be3d9  -

✔️ Chunk with zero offset

$ neofs-cli -c ../config.yml object hash --cid <> --oid <> --range 0:500
Offset=0 (Length=500)   : 1984b3d081b204dadf0b36eb35a37b58b0d5395be57af1450038bfc4c11c5b51
$ xxd -s 0 -l 500 ../go.mod | xxd -r | sha256sum
1984b3d081b204dadf0b36eb35a37b58b0d5395be57af1450038bfc4c11c5b51  -

❌ Chunk with non zero offset

$ neofs-cli -c ../config.yml object hash --cid <> --oid <> --range 1:500
Offset=1 (Length=500)   : d5a54a6b06539f1b6f58216b8176f83ac62d604452fc154b9869af1f573273fb
$ xxd -s 1 -l 500 ../go.mod | xxd -r | sha256sum
46609c92a267c792a12961b5760767184fad5e2a5d3f538ce163350b964fc9d9  -

Your Environment

• Version used: 0.12.0-rc3-22-g54818d5

Object search with --root flag in CLI returns tombstone object ID but it shouldn't.

Steps to Reproduce

1. Upload non-split object
2. Remove non-split object
3. Wait for garbage collector to remove original object
4. Search in container for root objects.

Your Environment

• Version used: 0.12.0-rc3-22-g54818d5

Add neofs-cli option --ttl in the search operation to give a possibility of the search operation on the selected node only.

QL parsers are quite useful for applications such as protocol gates or GUI apps. It provides human readable way to set up container policies. So it will be nice to have it in SDK library.

Basic set of StorageGroup operations to implement:

• Create
• List StorageGroups for container
• Get StorageGroup by ID
  There should be an option to save SG to a file or print in a desired format
• Delete StorageGroup by ID

Add github actions similar to neofs-api-go at pull requests:

• check sign-of line in commits,
• run unit tests,
• run linter.

neofs-cli needs to have a module for container operations. As a first step the minimal required feature set would be:

• Create container with Storage Policy, BasicACL and Attributes
• Delete container
• List containers by owner
• List objectIDs in container
• Get container structure and display parsed information of dump to file or stdout
• Get list of nodes serving container by ID or from file dump for netmap from file dump or stdin

Tombstone body contains object ID that should be removed. This is helpful when object was split into several parts, so tombstone contains ID's of every smaller part. However it does not include removed object ID itself.

Now meta-base stores virtual object IDs of big objects in separate index. If object ID is not presented in tombstone, then it is not marked on meta-base, then it will be present in search responses after delete which is incorrect.

neofs-cli needs to have --rpc-endpoint option in root module to set NeoFS node to connect to.

Node address should be accepted in both multiaddr and plain traditional IPv4/IPv6/DNSname:port formats.

With #117 CLI sets object filename header but it uses value provided from command line. This value is actually a path to the file, so we have to extract filename first.

Expected

$ ./bin/neofs-cli object put --cid <> --file ../2_5422526881285015969.pdf
$ ./bin/neofs-cli object head --cid <> --oid <>
...
Attributes:
  FileName=2_5422526881285015969.pdf

Got

$ ./bin/neofs-cli object put --cid <> --file ../2_5422526881285015969.pdf
$ ./bin/neofs-cli object head --cid <> --oid <>
...
Attributes:
  FileName=../2_5422526881285015969.pdf

Most of the services of neofs-storage do not set response meta header in responses. Response meta header should contain at least default TTL value, node's epoch counter value and used API version.

Both inner ring and storage node configurations have NEO endpoint fields. It would be nice to support pool of addresses.

# Pure YAML:
morph:
  endpoint: 
    - https://morph1.nspcc.ru
    - https://morph2.nspcc.ru
    - https://morph3.nspcc.ru

# With ENV support:
morph:
  endpoint_0: https://morph1.nspcc.ru
  endpoint_1: https://morph2.nspcc.ru
  endpoint_2: https://morph3.nspcc.ru

# With priorities:
morph:
  endpoint_0: 
    url: https://morph1.nspcc.ru
    priority: 1.0
  endpoint_1: 
    url: https://morph2.nspcc.ru
    priority: 0.5
  endpoint_2: 
    url: https://morph3.nspcc.ru
    priority: 0.5

Neofs-node can choose random address and use it in runtime if it is active. This way we can uniformly distribute load of RPC nodes that will be present in preinstalled configuration.

What's wrong with runtime connection pool?

Instead of choosing one active address before startup, neofs-node can keep pool of active connections, healthcheck it in runtime and rotate connections whenever it needs. This increases stability of RPC node failures but there is a tricky part.

While connection pool works fine with clients, neofs-node also uses event listener to fetch new notifications from blockchain. Therefore neofs-node should multiplex several listeners to provide single stable channel of events. This is not trivial task and requires connections to several RPC nodes simultaneously. The other option: rotate listeners the same way as client connection, but we can have a bit of downtime during rotation, which is bad.

So the convenient way to address this problem: choose one endpoint and work with it. As soon as there are problems listener will close connection and client will return an error. In this case we can restart / reinitialize application with new single RPC endpoint.

It would be great to add a tombstone id in the neofs-cli output.

For example:

$ neofs-cli --rpc-endpoint s01.neofs.devenv:8080 --key L1XsFNrFqyB9VA6dbus6UtAcv8yJMdfpfqCcteHAahpz8rqin9K3 object delete --cid 7KjvGB3geDDUkpXkyV7E8mj6n3x5eb2h6kXm6ZfUgYMN --oid 8pw9MagFHW7DDWhS4qYMToENv65HhEag5ds7AbDTF5BT 
Object removed successfully.
  ID: 8pw9MagFHW7DDWhS4qYMToENv65HhEag5ds7AbDTF5BT
  CID: 7KjvGB3geDDUkpXkyV7E8mj6n3x5eb2h6kXm6ZfUgYMN
Тombstone ID: 5s2XS8uJPZsThBjjv6D8Qo8pmTePneEB35U7W585bYuF

Container has salt (nonce) field to avoid container id collisions. This field set via MessageID field in container put request. This field ignored by neofs-node, since container put response has different container id.

If you get container with returned container id, it will return with empty salt (nonce) field. This marks container as invalid.

Expected Behavior

Container put response has the same container id as your locally created structure.
Container get response returns valid container with non empty salt (nonce) field.

Current Behavior

Container put response has different container id from locally created container structure.
Container get response has empty salt (nonce) field, when MessageID was not empty.

Possible Solution

Do not regenerate salt (nonce) at server side. Return error if MessageID field is not set. Check if salt field did not missed during conversion from proto-defined container structure into neofs-node defined container structure.

Steps to Reproduce

1. Create container.Container{} structure from neofs-api-go and fill it.
2. Create container.PutRequest{} and fill the fields according to container structure.
3. Check if PutResponse.CID field has different container id than container id structure.
4. Wait container to be stored in morph chain.
5. Get container with container id from PutResponse.
6. Check if ID() function returns error because salt(nonce) field is empty.

Your Environment

• Version used: neofs-node v0.11.0 at latest neofs-dev-env.

invoke.SetConfig() function sends key-value pair to the contract, however the contract also has ID argument.

https://github.com/nspcc-dev/neofs-contract/blob/72574e56eb7ac7b67be8427ce7100ceafea0ccd0/netmap/netmap_contract.go#L275

func SetConfig(id, key, val []byte) bool {
	// check if it is inner ring invocation
	innerRing := getIRNodes(ctx)
	threshold := len(innerRing)/3*2 + 1
	...

neofs-cli search operation with --filter does not operate with object id.

For example for object FM4vHDMDGN2ESBwFgK9cBfrgp2F26xPfA8ysBgtnaVjS with headers:
{"objectID":{"value":"1Sep9/aMFvx9y/kn4x+vy29sWpqPYLzVBd8iM7L5tiU="},"signature":{"key":"AgsHtCnrUUyZDtJMMUXqDuSNl1pNNaqjVgJ8fJ9kme2N","signature":"BIXS1CLrPUlsmot96xory2o0fBxTMBU62ULcenlgcffzJ3H3wl/CxlJygdm7QDAL2cf6ECl3TqTUix72rLBopJg="},"header":{"version":{"major":2,"minor":0},"containerID":{"value":"5UCvtrxiFLRZshssllLlzYJL8LPLAv8ubJDMTcnaUZ4="},"ownerID":{"value":"NeMOrwz/s6pJpbz3iuxvjRAb2ulLdNSB4w=="},"creationEpoch":"46","payloadLength":"20000000","payloadHash":{"type":"SHA256","sum":"4uAFRMGh+kff29mO/rbFPmfqz7bBxkVN/xAxZvJv+cY="},"objectType":"REGULAR","homomorphicHash":{"type":"TZ","sum":"frFCPV2/1YDrABHsg3maDn061aCbbXK1+AhK6cP5jeooYqGYV/O72FxTMBkQjj+rUpeksQIc6mPrZpsC/ePQnQ=="},"sessionToken":{"body":{"id":"6DreJiJBTzqW0gbt9ZdYew==","ownerID":{"value":"NeMOrwz/s6pJpbz3iuxvjRAb2ulLdNSB4w=="},"lifetime":{"exp":"0","nbf":"0","iat":"0"},"sessionKey":"AgsHtCnrUUyZDtJMMUXqDuSNl1pNNaqjVgJ8fJ9kme2N","object":{"verb":"PUT","address":{"containerID":{"value":"5UCvtrxiFLRZshssllLlzYJL8LPLAv8ubJDMTcnaUZ4="},"objectID":null}}},"signature":{"key":"AkfmbTvwi++QggnbMcifHtQpW4fkaB1KlblI0/scf6gC","signature":"BL1y0JgSD6BJdBQqth1mvFXeHbMWB2vAiqpkAuS6/njfXlk6f3p6puRgyFXl8yaw48B9lxh+kniivbQsB9V6hng="}},"attributes":[{"key":"key1","value":"1"},{"key":"key2","value":"abc"},{"key":"FileName","value":"b8c2cbc8-48db-418e-ab25-438553013bb8"},{"key":"Timestamp","value":"1605657650"}],"split":null},"payload":""}

neofs-cli --rpc-endpoint s03.neofs.devenv:8080 --key L2GseHKsbhXZFa9xXaUPXErhxPRQs2vgnH5ie6aFVP1kG2keNCHV  --ttl 1 object search --root --cid GRuaANbC7pyiG7qEsyK1odTAnk1qN4muTECRPJeogV6u --filters objectID=9uk6MGNQ87LiH2JhtEyLYrTGR3keGj2NZtC1roVyEsjP
Found 0 objects.

neofs-cli --rpc-endpoint s03.neofs.devenv:8080 --key L2GseHKsbhXZFa9xXaUPXErhxPRQs2vgnH5ie6aFVP1kG2keNCHV  --ttl 1 object search --root --cid GRuaANbC7pyiG7qEsyK1odTAnk1qN4muTECRPJeogV6u --filters key1=1
Found 1 objects.
FM4vHDMDGN2ESBwFgK9cBfrgp2F26xPfA8ysBgtnaVjS

Network map is stored in blockchain as a array of binary encoded NodeInfo structures. Neo-go can invoke Netmap() method to return these structures. On top of that we can build whole network map structure represented in JSON.

To do that we should be able to convert base64 encoded NodeInfo into JSON. CLI can work as a unix filter:

echo "RXhhbXBsZQ==" | neofs-cli decode node-info | jq

neofs-cli need to support eACL operations in container module.
Minimal required feature set should be:

• CRUD for eACL table for particular container
• eACL table converter from/to binary protobuf/JSON formats

When Neo RPC node fails, event listener generates huge amount of empty events that hang whole application. This case should be gracefully handled by switching node to passive state (for inner ring) and making reconnects. Or just shut down application.

It would be nice to have GitHub issue template for neofs-node repository.

neofs-cli needs to have accounting->balance command implementing Balance request.

Any correct OwnerID must be accepted as argument. By default OwnerID is calculated for the key in use.

Inner ring can't calculate precise fee for contract invocation with signature collection. While most of the nodes only register signature, one node will trigger method execution, that requires much more gas to spend. Amount of extra fee depends on method complexity. Some methods depends on inner ring size, e.g. container creation triggers nep-5 transfers to every single inner ring nodes. The more inner ring nodes we have, the more gas method spend.

So there should be some heuristic evaluation before each invocation for extra fee.

There are also different solution approaches for this issue. For example inner ring nodes can repeat tx that not accepted in block: tx that trigger method execution lacks gas after testinvoke evaluation. But next time testinvoke evaluation will be precise. However this approach can slower all chain operations.

After nspcc-dev/neofs-api-go#172 will be implemented in SDK, there will be well-known attributes for containers and objects. We should adopt it in CLI.

Container attributes

• Timestamp attribute can be set up automatically by CLI. Consider adding --disable-timestamp flag to not set it.
• Name of the container should be provided with --name argument, that can be omitted.

Object attributes

• Timestamp can be set up the same as in container. Consider --disable-timestamp flag.
• FileName attribute can be set up based on --file argument automatically. Consider adding --disable-filename flag to not set it.

When searching for objects from particular container, split objects are ignored, because they do not exist as a single object.

Expected Behavior

Container listing returned all objects: regular objects, split objects, split object parts.

Current Behavior

Split object are ignored

Possible Solution

On each node serving search request for container listing, additionally list all OIDs from Parent fields and merge those lists in response.

Now all delete operations happen with some sort of delay. object.Delete invocation generates tombstone and node broadcast it to all container nodes. Some of container nodes store tombstone but every container nodes put list of removed objects from tombstone into a GC queue. After a some time GC takes object IDs from the queue, marks them as removed in meta-storage and removes them from object storage.

With new meta-storage it can be implemented without GC timeout. By receiving tombstone node can mark all removed object IDs in meta-storage index. Then GC will take batch of object IDs asynchronously from index of removed objects, remove it from object storage and rebuild index.

Replicator service has to PUT objects to other nodes according to container's storage policy. ObjectID's are taken from a queue. Replicator's request queue is processed in a such way that it would not harm the main object service performance.

To avoid unnecessary network load, Replicator should take into account hints provided in it's queue items, including the list of nodes known to already store the object under replication and nodes known not to store it.

There is an issue with object.GetRange that leads to errors in object.Get. This issue happens only with split object, non-split objects are fetched correctly. object.Get returns an error (formatted for better readability):

Error: can't put object: could not receive Get response: rpc error: code = Unknown desc = 
 could not receive response: 
   could not receive response message for signing: 
     (*object.getStreamResponser) could not receive response: 
       could not receive response message for signing: 
         (*getsvc.streamer) could not receive get response: 
           (*getsvc.Streamer) could not receive range response: 
             (*rangesvc.streamer) incomplete get payload range

Steps to Reproduce

1. Upload object with size > 1 megabyte (max object size in dev-env)
2. Get that object

Your Environment

• Node version used: 0.12.0-rc3-22-g54818d5
• Container with public basic ACL

With nspcc-dev/neofs-api-go#185 JSON encoders of neofs structures now returns errors, therefore we should update neofs-cli code.

Neofs-cli operation to implement:

• Netmap request to dump latest netmap

Policer service goes through locally known objects and makes sure they are stored in the NeoFS network according to container's storage policy. To check if object is stored on a particular node the HEAD request is used. Policer should rely on MetaBase requests without producing additional load to object storage subsystem.

If the object is not stored properly, it's address is passed to Replicator service's request queue. Request queue item should have additional information including the list of nodes known to store the object and a list of nodes known to miss the object despite being supposed to store it.

Because other storage nodes may not be trusted, additional checks will be implemented when Reputation and Data Audit subsystems will be ported.

Some placement policies do not specify selection attributes e.g. "put two copies in container of 6 nodes with SSD". This can be presented as

REP 2
SELECT 6 FROM F
FILTER StorageType EQ SSD as F

Take generic policy "put two copies in container of 6 nodes":

REP 2
SELECT 6 FROM *

Expected Behavior

These queries parsed with selectors:

SelectorExample1 {           SelectorExample2 {
  name:     ""                 name:     ""
  count     6                  count     6
  clause    SAME               clause    SAME
  attribute ""                 attribute ""
  filter    "F"                filter    "*"
}                            }

Current Behavior

Placement policy parser grammar requires IN token in selectors to specify bucket, that makes all these queries invalid.

2:10: unexpected token "FROM" (expected "IN")

Possible Solution

Make IN token optional in placement policy parser grammar.

cc: @realloc @fyrchik

Expected Behavior

In case of trusted object placement within opened session between user and trusted node session token must be written to object body.

Current Behavior

Session token is not written to the objects during trusted placement.

Possible Solution

1. write session token to object in format transformer;

2. take session token from the request and write to object before transformations.

Your Environment

• jindo branch

Neofs-node repository contains neofs-storage, neofs-ir and neofs-cli applications. At least first two applications use config files. Both of them use viper as a config reader and both share some configuration parameters, such as:

• blockchain endpoint,
• smart-contract script hases,
• ...

It will be great if both apps, and even neofs-cli, will be able to share sections of config file.

With nspcc-dev/neofs-api-go#147 we've decided to encode object ID, owner ID and container ID as base58 strings. These strings used in search filters and EACL filters.

Now match functions for these filter use hex encoded representation:

func idValue(id *objectSDK.ID) string {
	return hex.EncodeToString(id.ToV2().GetValue())
}

func cidValue(id *container.ID) string {
	return hex.EncodeToString(id.ToV2().GetValue())
}

func ownerIDValue(id *owner.ID) string {
	return hex.EncodeToString(id.ToV2().GetValue())
}

func idValue(id *objectSDK.ID) string {
	return hex.EncodeToString(id.ToV2().GetValue())
}

func cidValue(id *container.ID) string {
	return hex.EncodeToString(id.ToV2().GetValue())
}

func ownerIDValue(id *owner.ID) string {
	return hex.EncodeToString(id.ToV2().GetValue())
}

After nspcc-dev/neofs-api-go#147 they should adopt stringers from neofs-api-go.

Basic set of operations to implement:

• Dump netmap for latest Epoch or for Epoch number set in option, if available.
  Should be saved to file or printed to stdout in different formats including JSON and protobuf.

• Get latest Epoch number

• Request LocalNodeInfo from particular host

• Filter netmap from file or stdin using set of filters. List of filters can
  be set in binary, JSON or SQL-like format

In the previous NeoFS version attempt to PUT/GET an object into a private container from the Inner Ring key was impossible.

In the current version of the NeoFS ACL, it is possible to PUT not only for the container nodes (SYS group) but also for the inner ring nodes.

Do we think this behavior is correct for the current version of NeoFS?

For example for container with basic ACL 0x1C8C8CCC:

$ neofs-cli --rpc-endpoint s01.neofs.devenv:8080 --key <INNER RING KEY> object put --file 31bec1f6-9d84-4d92-b9ca-1ed97b91319a --cid 6fWrinptxkQWfNr7qFExmHViz4Wb5ctVMdZFBQ1QhAVy 
[31bec1f6-9d84-4d92-b9ca-1ed97b91319a] Object successfully stored
  ID: 4mo44oJ4KeG6w3YM7HA9XmrLXBoEaT8mN8wAqgabCbi6
  CID: 6fWrinptxkQWfNr7qFExmHViz4Wb5ctVMdZFBQ1QhAVy

Expected Behavior

New objects have non zero creation_epoch field.

Current Behavior

Objects have zero in creation_epoch field.

Steps to Reproduce

1. Put object.
2. Head object.
3. Check creation_epoch field.

To select the key to be used for operation neofs-cli has to support --key option in root module. File path has to be used as argument.

In nspcc-dev/neofs-api#80 and nspcc-dev/neofs-api-go#182 there are new well-known filters CHILDFREE and split.parent. Adopt in neofs-storage.

Right now object delete scheme for split objects looks like this:

• Node looks for linking object;
• If it finds it, then node fills tombstone with ID's from linking object;
• If linking object was not found, then node looking for most-right child and traverse linked list to the most-left child, then put all ID's into tombstone.

There are a few things that can go wrong:

1. Linking object exists but unavailable to node right now (e.g. network issues)
2. Both one of the child and linking object are lost forever.

We already examined (1) due to bug in the implementation (#164). In this case linking object will be stored forever since it is not presented in a tombstone. I think this issue is more likely to happen than (2) and needs to be addressed.

In case (2) we can't properly build tombstone itself so we will delete at best only part of the object.

In both cases left objects are not ROOT objects and probably they will be removed after there won't be any payment for them.
However in the other environment without payment subsystem, these objects will be stored forever.

It would be nice if node could make sanity checks on container fields at container.Put request:

• do not accept containers without owner ID or with invalid owner ID,
• do not accept containers without nonce or with invalid nonce,
• do not accept containers without placement policy,
• do not accept containers with unknown version,
• optional: check container signature (smart-contract won't accept containers with invalid signature anyway, but we can make one more check).

Max object size as other global system preferences stored at netmap contract. At node initialization, storage node has to read configuration parameters and store it locally.

To do that it is proposed to create method in morph/netmap/client to get configuration parameters. Then implement getter in morph/netmap/client/wrapper such as GetObjectMaxSize() that will call client method with MaxObjectSize argument.

ACL service in neofs-storage should check bearer token if it is present and use it for extended ACL check instead of on-chain extended ACL table.

Also bearer token should be attached to all generated request from the node the same way as session token.

neofs-cli needs to have a module for object operations.

Minimal required feature set:

• Get object by address
  Header is ignored by default. Using an option it should be printed or saved to file in JSON or protobuf format
  Payload should be streamed to stdout or saved to file
  There should be an option to dump the requested object 'as-is' in binary form
  Use GetRange request if --range option is set

• Head object by address
  --main and raw options must be supported
  Header should be printed or saved in JSON or protobuf format to file

• Get object's payload hash
  Return full payload hash if --range option is not set

• Delete object by address

• Search
  Search query should be accepted in binary, JSON or SQL-like format

All CLI commands that output structures should support --json flag as a version command. It will be easier to implement after nspcc-dev/neofs-api#73