Currently only a few Volume types works. Add support to all Volume types supported by GlusterFS.

File system crawler to list the files from each directory from Storage unit and issue the setxattr for each file. Stat the file and collect the size of the file and update the total size crawled and rebalanced.

Ex: setlxattr(file, "trusted.distribute.migrate-data", "1")

Status: Percentage of completion: Size of the files crawled or rebalanced / Total size of Storage Unit

Start rebalance:

Start the above crawler in all the Storage units. Start in any one Storage unit from each distribute group.

kadalu volume rebalance-start PROD/vol1

Stop rebalance:

Stop the rebalance service that is started earlier.

kadalu volume rebalance-stop PROD/vol1

Rebalance Status

• Show the health of each service.
• Show the percentage of completion.

kadalu volume rebalance-status PROD/vol1

kadalu volume start <pool_name>/<name>

Example:

kadalu volume start mypool/vol1

• Starts the Kadalu Storage Volume
• If already started, only starts the pending storage units

Add support for Volume Statedump

moana-node expects the following directories to be exist in runtime. Make sure these directories are available before starting the service than failing runtime.

• ENV.fetch("WORKDIR", "/var/lib/moana")
• <workdir>/volfiles
• /var/run/moana
• /var/log/moana

To manage brick services, systemd unit template file is required in one of the systemd directory

/lib/systemd/system/[email protected]

Brick service wrapper script

/usr/sbin/kadalu-brick

(kadalu-brick should be in PATH)

Steps documented to start moana-node here https://github.com/kadalu/moana/tree/devel/moana-node

Hint: $SRC/moana-node/src/moana-node.cr is the starting point.

Similar to mount-glusterfs, add mount script for Kadalu Storage.(Moana based)

$ # mount -t kadalu <moana-server>:<volume-name> <mount-point>
$ mount -t kadalu moana.example.com:gvol1 /mnt/gvol1

• Download the Client Volfile.
• Execute GlusterFS command to mount the Volume using the downloaded Volfile.
• Keep Websocket connection with Moana Server to get notification if any Volfile changes.
• Reload glusterfs when Client Volfile changes.(GlusterFS change is required for this use case)

Example:

$ kadalu cluster status
Cluster Name: mycluster
Cluster ID: f594dd01-43da-4f56-a974-49f3f6e8cd0a

  Node                   State
* server1.example.com    Up
  server4.example.com    Up
  server5.example.com    Up

* prefix indicates it is a Meta Server

Inspired by zfs snapshot commands.

kadalu snapshot create <cluster_name>/<volname>@<snapname>

Example:

kadalu snapshot create mycluster/vol1@snap1

Introduce Server APIs for importing and Exporting the Cluster data.

Import

POST /api/v1/clusterdata/:cluster_id

Export

GET /api/v1/clusterdata/:cluster_id

Import/Export format TBD.

Option Set

kadalu volume set <name> <optname> <optvalue> ...

Option Reset

kadalu volume reset <name> <optname1> <optname2>...

• Notification should be sent to affected Server nodes
• Notification should be sent to affected Client nodes(TBD or as a separate issue)

mount -t kadalu [OPTIONS] <host>:<volname> <mount-point>

Also add support for readonly mount for Snapshot Volume.(May be as a separate issue)

Tests available for Cluster and Node database operations(https://github.com/kadalu/moana/blob/devel/server/spec/db/db_spec.cr). Tests required for all other entities.

On successful completion of task by node agent, it reports back with State=Completed. Update the respective table once the task is Completed. For example, if Volume Create task is complete, then add new entry to Volumes table.

Two variants

kadalu volume create <cluster_name>/<volume-name> \
    [replica <count>|disperse <count> redundancy <count>]
    storage_unit1 storage_unit2...

Storage unit in the form of <host>:<path> or <host>:<device>

Example: (Replica 3 Volume)

kadalu volume create mycluster/myvol \
    replica 3 \
    node1.example.com:/exports/myvol/s1 node2.example.com:/exports/myvol/s2 node3.example.com:/exports/myvol/s3

kadalu volume create <cluster_name>/<volume-name> \
    [replica storage_unit1 storage_unit2... | disperse storage_unit1,storage_unit2.. redundancy storage_unit3..]

Example: (Replica 3 Volume)

kadalu volume create mycluster/myvol \
    replica node1.example.com:/exports/myvol/s1 node2.example.com:/exports/myvol/s2 node3.example.com:/exports/myvol/s3

• Add support for mirror alias for replica
• Save the Volume create request in the Volume so that a few customizations are possible while showing the Volume info/status(Example: Show Mirrored instead of Replicate if mirror was used in create command)

Other flags required are:

• --brick-fs=xfs|ext4|zfs This is useful when storage unit is provided as <host>:<device>
• --fs-opts=<opts> To use with mkfs
• --use-lvm To use Thin Volume to use LVM Snapshot feature. Not applicable for zfs
• --allow-rootdirs To allow creating Storage units in the Root directories.
• --no-start Do not start the Volume automatically after create.

A few more options required in this command. I will add the list of flags in comments or create a new issue.

• Infra to start Heal daemons on Replica family volume start.
• Heal commands to view the status
• Heal command to trigger heal

While handling requests cluster wide, two or more parallel requests may corrupt or cause undefined behaviour without locks.

Implement Cluster wide lock framework using fcntl and local lock framework(To handle parallel requests in single Manager instance).

Current implementation is without locks with TODOs where lock is necessary.

Ref:

• https://crystal-lang.org/api/1.1.1/IO/FileDescriptor.html#flock_exclusive(blocking=true,&)-instance-method
• https://crystal-lang.org/api/1.1.1/Mutex.html

Add support for Simple Quota based Sub volumes.

Create

kadalu subvol create <volname>/<subvol-name> size <size>

Example:

kadalu subvol create myvol/photos size 10G

• Mount the Volume if not mounted.(Predictable directory say /mnt/<volname>)
• Create a directory and Set Simple Quota
• Store info within the Volume

Delete

kadalu subvol delete <volname>/<subvol-name>

Archive

kadalu subvol archive <volname>/<subvol-name>

Expand

kadalu subvol expand <volname>/<subvol-name> size <new-size>

Info

kadalu subvol info [--status] [<volname>]

Mount

Enhance the mount script to support mounting the subvol.

• Mount the Main Volume if not mounted
• Bind mount the subvol to requested path

mount -t kadalu <host>:/subvols/<volname>@<subvol-name> <mount-path>

Currently, Crystal language bindings available in $SRC/client directory. Add Python bindings.

Start with below two backends.

• LVM
• ZFS

Intitialize the Cluster.

• Install and Start the Storage Manager
• Run kadalu cluster init <name> --meta-servers=<meta-server1,meta-server2...>

Behaviour:

• If the current node is part of Meta Servers list
  • Start the Storage unit(Brick) process
  • Start the Automatic heal process
  • Start the Client process
  • Start the ReST Server
  • Start the Monitoring service
• If the current node is not part of Meta Servers list
  • Start the Client process
  • Start the ReST Server
  • Start the Monitoring Service

Install moana-server, moana-node and cli in a VM and Test the functionality using the CLIs. (Using Binnacle)

Add support for Block Subvols(Create a Block file and mount as loop dev)

Create

kadalu subvol create <volname>/<subvol-name> size <size> --virtblock

Example:

kadalu subvol create myvol/photos size 10G --virtblock

• Mount the Volume if not mounted.(Predictable directory say /mnt/<volname>)
• Truncate the file to the required size.
• Store info within the Volume
• Create xfs filesystem

Delete

kadalu subvol delete <volname>/<subvol-name>

Archive

kadalu subvol archive <volname>/<subvol-name>

Expand

kadalu subvol expand <volname>/<subvol-name> size <new-size>

Info

kadalu subvol info [--status] [<volname>]

Mount

Enhance the mount script to support mounting the subvol.

• Mount the Main Volume if not mounted
• Mount the block device to the requested path

mount -t kadalu <host>:/subvols/<volname>@<subvol-name> <mount-path>

Create a tool to migrate Cluster configurations from Glusterd to Moana. For example run the below tool in any one of storage nodes.

sudo moana-importer <moana-url> [args...]

The tool runs gluster volume info --xml and other required commands, Parses the Cluster, Volume information and converts to the format supported by Moana import(#21)

Steps:

1. Install Moana Server in a highly available setup or use Hosted service by Kadalu.io
2. Install Moana node agents in all Storage nodes of the Cluster.
3. Create User and Cluster using moana CLI
4. Request all nodes to join the newly created Cluster using moana node join command.
5. Run below command from any one storage node to import the Cluster State from Glusterd.

   sudo moana-import <moana-url> [args...]
   

6. Thats it. Run moana health to see if all is OK.

kadalu volume expand <pool_name>/<volname> \
    <new-subvolume>

Example:

kadalu volume create DEV/myvol                  \
    replica node1.example.com:/exports/myvol/s1 \
            node2.example.com:/exports/myvol/s2 \
            node3.example.com:/exports/myvol/s3

kadalu volume expand DEV/myvol                  \
    replica node4.example.com:/exports/myvol/s4 \
            node5.example.com:/exports/myvol/s5 \
            node6.example.com:/exports/myvol/s6

Steps to Reproduce:

Create a Volume using

$ moana volume create gvol1 node1.example.com:/bricks/b1
Volume creation request sent successfully.
Task ID: 0f8ea18a-bbd2-403a-b752-ea3fce74e8c6

But moana-node fails with the following traceback since parent directory /bricks doesn't exist.

Exception: Unable to create directory: '/bricks/b1': No such file or directory (File::NotFoundError)
  from ../../../../usr/share/crystal/src/crystal/system/unix/dir.cr:60:7 in 'create'
  from ../../../../usr/share/crystal/src/dir.cr:234:5 in 'mkdir'
  from ../../../../usr/share/crystal/src/dir.cr:233:3 in 'mkdir'
  from src/task/brick_utils.cr:186:5 in 'mount'
  from src/task/brick_utils.cr:214:3 in 'create_brick'
  from src/task/volume.cr:27:11 in 'run'
  from src/routes.cr:14:5 in '->'
  from ../../../../usr/share/crystal/src/primitives.cr:255:3 in '->'
  from ../../../../usr/share/crystal/src/primitives.cr:255:3 in 'process_request'
  from lib/kemal/src/kemal/route_handler.cr:17:7 in 'call'
  from ../../../../usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/websocket_handler.cr:13:14 in 'call'
  from ../../../../usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/static_file_handler.cr:11:11 in 'call'
  from ../../../../usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/exception_handler.cr:8:7 in 'call'
  from ../../../../usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/log_handler.cr:8:37 in 'call'
  from ../../../../usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/init_handler.cr:12:7 in 'call'
  from ../../../../usr/share/crystal/src/http/server/request_processor.cr:50:11 in 'process'
  from ../../../../usr/share/crystal/src/http/server.cr:513:5 in 'handle_client'
  from ../../../../usr/share/crystal/src/http/server.cr:468:13 in '->'
  from ../../../../usr/share/crystal/src/primitives.cr:255:3 in 'run'
  from ../../../../usr/share/crystal/src/fiber.cr:92:34 in '->'
  from ???

2020-12-28 07:27:29 UTC 500 POST /api/v1/tasks/volume_create 221.5ms

Add support for Volume Profile

• Test all the ReST routes using https://kemalcr.com/guide/#testing
• Add Unit tests for the functions which can be tested independently

Implement the invite command to generate the token and other necessary information. Run that join command in the respective node to join to the Cluster.

$ kadalu cluster invite
Run the following command from the node that needs to be added to the Cluster. Invite code is valid
only for next 2 hours.

kadalu node join --invite 2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824 \
    --invited-by node1.example.com \
    --meta-servers="node1.example.com,node2.example.com,node3.example.com"

node join command will call the ReST API of invited-by node. If valid, API will add this node to allowed list for mount. (Or some other Token mechanism TBD)

Add support for Volume Top

• Install Crystal lang in a VM
• Run crystal spec in server, node and cli directories. (TODO: All tests are not yet available)

kadalu volume stop <pool>/<name>

• If any node is offline while running this command then store that as pending operation for that node. Once that node comes back it will stop the brick when the Storage manager starts.

Snapshot auto delete, Activate on create etc.

Add tests to $SRC/volgen package

Systemd unit file for managing moana-server. Also set required env variables

[Unit]
Description=Moana Server
After=network.target

[Service]
PIDFile=/var/run/moana/server.pid
ExecStart=/usr/bin/moana-server

[Install]
WantedBy=multi-user.target

Systemd unit file for managing moana-node. Also set required env variables

[Unit]
Description=Moana Node agent
After=network.target

[Service]
PIDFile=/var/run/moana/node.pid
ExecStart=/usr/bin/moana-node

[Install]
WantedBy=multi-user.target

• Add support for set/reset/remove Quota to directories
• Start and Manage the Quota monitor processes(To update the Quota usage in regular intervals)

More details TBD.

• Test all the ReST routes using https://kemalcr.com/guide/#testing
• Add Unit tests for the functions which can be tested independently

Since Moana supports multiple Clusters, user management is required.

Usage:

Register a User using(POST /api/v1/users)

$ moana user register <email>
Password:
User created Successfully.

Login to Moana Server using, (POST /api/v1/login)

$ moana user login <email>
Password:
User logged in Successfully. Token saved in `~/.moana/token.json`

Now user can create Cluster or assign new user roles.

Database table for Roles:

cluster_id   |  volume_id  |  user_id  | name

For example,

cluster_id | volume_id | user_id | name
c1         | all       | u1      | admin
c1         | v1        | u2      | maintainer
c1         | v1        | u3      | viewer
c2         | all       | u1      | viewer

To start with, Roles management can be Cluster level. Volume level granularity can be added later.(volume_id=all means all volumes within the cluster).

In all the Server API routes, check if a particular action is allowed or not. For example:

post "/api/v1/volumes" do |env|
    if cluster_maintainer?(env)
        # Volume create implementation
    else
        env.response.status_code = 403
        {"error": "Forbidden"}.to_json
    end
end