stratis-storage / stratis-docs Goto Github PK

Create a document to describe the Stratis-cli command-line arguments and options.

See: stratis-storage/stratisd#744

• add document version
• section 2 Overview and 2.1.1 this feels like implementation details, which I don't know if should go in an API reference. Can this go in the design doc, maybe section 9?
• 2.2.2 maybe this also goes in design doc?
• more detail on methods: document in/out parameters & types, etc.

I see that storaged is mentioned in stratis-interactions.svg diagram, where it uses Stratis D-Bus API, but there is no other explanation of how storaged relate to Stratis in the document.

Do I understand it right that you expect Storaged/udisks to implement support for Stratis managed storage for Cockpit, Gnome and other components which uses storaged? Moreover since Storaged is a daemon which provides D-Bus API, would it make sense to compare it's design with Stratis?

• I wish MDAA and MDAB were something like MDAl and MDAr (for left and right).
• I wish I knew where the Stratis signauture came from.
• I think the length of a sector should be defined in the text, where it is used to specify the length of the areas. (It is defiend in the table).
• I think purpose of unused should be explained.
• I do not know why flags and unused are grouped in the item that says both are zeroed, especially as that is their only mention. Flags for what?
• You're a bit inconsistant in naming.
  • CRC should probably be CRC32 everywhere.
  • Sometimes you use MDA to stand for one half of the MDA, and sometimes you use it to stand for the whole thing.
• A diagram of the arrangement of metadata on the blockdev would be nice, but maybe better for later.
• What is reserved space reserved for?
• Keep the offsets in the relevant tables! They are helpful.
• It might be nice to group the MDAB and MDAA items in the Signautre block table visualyl, perhaps with a double line.

We have agreed that since devicemapper is heavily used and also authored by the Stratis team, it is a good thing to split it out from the other external use declarations. The style doc should document this decision. cc @mulkieran

API will not be stable for 1.0.

• Use list comprehensions instead of for-loops. for-loops are super-general, obviously, while list comprehensions are more restricted. This makes list-comprehensions clearer in the case where they can be used at all.
• Do not use boolean overloading in if conditions unless it is the case that the type of the object is unknown. If the type of the object is known, then do an explicit comparison as is None, == [], etc. This way, when an object of an unexpected type arrives, you get a failure at the comparison, not a subtle bug which may hang around for literally years before being discovered.
• Use os.path methods when assembling or deconstructing paths. This is just for clarity, to make it clear that it's a path, not a string that is being manipulated. It would be important if we were cross-platform, which we are not.

Even if we're not stabilizing the API yet, let's document them for our own benefit for the time being even so.

XFS doesn't support shrinking at the moment. And in the design document, you state that:

Stratis requires the filesystem used have online resize (or at least online
grow) capabilities

Would it make sense to explain in more detail why lack of shrinking is not a problem for Stratis? As I went through the design document, it seems that Stratis would grow the filesystem automatically while removing block devices would be supported.

References

Seems like "Implementation Details" is better.

One of the limitations of the current LVM2 + XFS or MD + XFS lash-ups is that the block layer cannot talk up the stack to the filesystem, and the filesystem treats the block layer as a dumb pile of storage.

I'm not familiar with btrfs in this respect, but with ZFS, one of the huge advantages you get from a unified filesystem and volume manager is that when all redundant copies of a given block are unreadable or corrupt, it doesn't just mark the pool as degraded and demand a disk swap. Instead, it tells you exactly which file is affected. If you remove or replace that file, the problem instantly goes away, because those damaged blocks are no longer involved with that file.

This is partly thanks to CoW, but independent of it, since even in a filesystem that overwrites blocks in place, that overwrite operation may be able to repair the damage. If it's just soft corruption, simply overwriting the blocks fixes it. If it's on-rust corruption, the drive's sector remapper is likely to swap in a fresh sector when the write occurs, and if not, the filesystem can potentially react to a failed write and mark that sector out in the same way badblocks does.

Will Stratis have this tight interoperation between the filesystem and the block layer, or will the fact that it is built atop several "foreign" code bases mean it will remain unable to diagnose and repair itself in the way ZFS can?

Updated Note: problem is easy to reproduce on dell-r730-011.dsal.lab.eng.rdu2.redhat.com - need to create at least 25 filesystems.

The attached script causes the following on the 3rd filesystem:

stratis filesystem create pool1 24
Execution failed: 5: Command failed: cmd: "/usr/sbin/mkfs.xfs" "-f" "-q" "/dev/dm-16" "-m" "uuid=1cf50752-f0e3-4648-b107-31a3c0761ccb", stdout: stderr: mkfs.xfs: libxfs_device_zero write failed: No space left on device

#!/bin/bash
set -x
set -e
DEVS1="<list of block devices here>"

FS_COUNT=25
POOL_NAME=pool1

stratis pool create $POOL_NAME $DEVS1

for((fs=1;fs<=$FS_COUNT;fs++))
do
   echo "create $POOL_NAME-fs-$fs"
   stratis filesystem create $POOL_NAME "$fs"
done 

Problem happens every time on dell-r730-011. Have not reproduced on other systems.

see stratis-storage/stratisd#814

Current plan/goal is to maintain symlinks to filesystems under /dev/stratis/poolname/filesystemname.

I infer by your use of XFS that Stratis does not support copy-on-write today, one of the key advantages of ZFS and btrfs relative to the current RHEL storage stack, being the near-manual LVM2 + MD + XFS lash-up.

I understand that CoW-on-XFS is coming in some unspecified future version of XFS.

I would like the document to clarify this situation:

1. State that CoW-on-XFS is expected to arrive sometime during the Stratis development plan, and say which versions are expected to coincide. That is, will we have it for Stratis 1.0, 2.0, 3.0...?

2. If you cannot make such a claim, this should be called out in the document as a known limitation of Stratis relative to ZFS and btrfs.

stratis-storage/stratisd#978 has revealed that the treatment of these sectors is not properly specified.

• Properties

  • State listed as a s, it is actually a q or uint16
  • InitializationTime documented as a s string, it's actually a t or uint64
  • Tier not documented, currently it is a q

• Method

  • SetUserId is actually SetUserInfo

See stratis-storage/stratisd#370.

We need version 37 for dm event poll support.

see stratis-storage/stratisd#6

• Initialization time missing from header
• CRC does not clarify Castagnoli, aka CRC-32C

• 10.2.5 Progressively throttling writes, is this implemented?

• 10.2.7 Will Stratis support changing the pass phrase for a Snapshot of an
  encrypted FS?

• 10.2.8 What if a FS has no idle periods, will stratis eventually force a
  fstrim or will the fstrim starve? Will dm stats be used for the historical
  system I/O activity levels?

• 10.3.4 Should the BDA format include a version field?
  Note: It would be good to have a little more detail on the write sequence when
  multiple devices exist. More detail on the algorithm would be helpful.

• 10.3.5 Should the MDA JSON format include a version field?

• 10.3.7 Would it be useful to have a background or periodic task that updates
  the physical device(s) meta data to make them current? When a box comes up with
  the current design you are required to read every physical disk to ensure you have
  found the latest metadata, correct?

We should specify the scheme we are using for dm names for all devices we are creating in the design doc.

The "References" section would have to be newly created.

Document the DBus API, since it is not in the design doc any more.

There are madly out of date and will be very hard to maintain.

It might make sense to dump in the current, most up-to-date introspection information, and then maybe have a little further information about princiiples, if any.

Demo of this approach now available in docs.

Part I contains a requirements section, but Part II and III should describe how Stratis works, rather than saying what requirements a solution must meet.

This really just means searching for requirements words like "must" and "should", and replacing them with present-tense words that describe Stratis's implementation. It already does this partially so it's just a matter of fixing a few spots.

The scheduling section at the end of the document worries me. 1.0 looks like it is scheduled for the first release of RHEL 8, simply based on how long it's been since RHEL 7 came out. That section says we won't have any form of RAID until Stratis 2.0. Is this reading correct?

Without at least mirroring RAID, I cannot see Stratis as any kind of replacement for btrfs.

Is there any possibility of moving the RAID bits alone into Stratis 1.0, or 1.1? My main concern is that Stratis-with-RAID be available in some version of RHEL 8, preferably the first version.

General

• Potentially long running operations, eg. creating many filesystems in
  one call. How sure are we that the method will complete
  before the timeout between the client and service expires? Or is the client
  to set the dbus client timeout to infinity?

• When you supply a list of things to be created, what happens if some are
  successful and some fail? eg. CreateFileSystems, DestroyFileSystems? Or is the
  contract all or none?

• Clarification, if you repeatedly set the name of a resource to the same name,
  is the result = False, 0, "" ?

Specific

• 1.1.2 Typically in dbus calls to provide for an optional argument you use a
  hashmap, so that the existence of the key indicates it's presence and of course
  it's associated value. What is the convention if a method takes an optional
  argument for the value of the second argument to be even though it's to be
  ignored?

• 3.0 Would it be better to be consistent with the return value being always
  in element 0, message in element 1 and then the result in element 2 if
  present. Perhaps I'm missing a benefit to utilizing the documented approach.

After we determine the usage model for snapshots that we'll be implementing, document this in the design doc, along with implementation details.

We are calling many functions that can fail. We should document guidelines for when a failure should result in:

• stratisd failing to start
• stratisd exiting
• The DBus method failing, and if this affects subsequent DBus method calls (e.g. is the pool now in a bad/failed state so nothing can succeed until a restart?)
• continuing if other resources can be tried, and only failing if all resources fail

as well as when debug, warn, and critical debug messages should be emitted.

When should one leave debug!() and other logging levels in the code? This issue is not just that it would be good to put this in the style guide so we all are consistent, but also coming up with the policy itself.

See: stratis-storage/stratisd#382.

Some documents seem to say that it is, and other's disagree. It may be a question of the particular version of the spec.

We should remove it from the appendix and use it to actually validate our metadata, somehow, in a test of stratisd.

We've chosen to use LyX to generate our design doc and to-be-written user's guide, but we want to make these easily readable via HTML. One possible way might be to generate HTML output off of a commit hook, and then publish them to our project's GitHub Pages (see https://pages.github.com/), and then point people there in the README.md or from wherever.

In moving away from Google Docs, we also have lost its nifty comment-adding feature. We should document what readers should do if they have comments or questions about our docs. (I guess open an issue?)

We have an initial size requirement for meta_dev, data_dev (for thinpool) and also for mdv, but I can not find anything about these in the design doc.

stratis-storage/stratisd#641

We should have a page on the website that answers common questions about using Stratis, such as why it shows 1 TiB for filesystems in df.

Right now, we know this:

1. When a BDA is first initialized, it has no variable length metadata. That is correct. We assume that after setup, the BDA has variable length metadata, at least one, and that that is correct.
2. If saving of both areas succeeds, the MDAHeaders are correctly updated, i.e., the designated saved one is updated with correct new information, timestamp, size of metadata written, metadata crc.
3. If the size of the data is too big for the metadata region, saving will fail immediately, w/out making an attempt to write or to update the headers. This will also be true for a save time that is actually less recent than the timestamp of the data recorded. In each of these cases, an EngineError with value Invalid will be returned, which allows to distinguish between these conditions and I/O errors. This gets back to the question of how to select which blockdev's to write the metadata to in the first place. It makes no sense to try to write data that is too big, but we have no reason to believe that data is always going to increase in size. If we trim our pool right down, the data could actually decrease in size.
4. If saving the first or both fails, the corresponding MDAHeader will not be updated and an error result will be returned. This is a good, simple decision. It has one interesting consequence, though. The next time it is needed to write metadata to the disks, that region will be attempted again. If it was a non-transient failure, that particular blockdev will fail to have the metadata correctly written over and over again. We should consider the consequences of this.

What this leaves open is understanding the probability of getting wrong data when reading metadata during setup.

The documentation talks about the stratisd monitoring daemon but does not talk about SMART at all. Will stratisd functionally replace smartd?

For example, is stratisd expected to make a decision about a rapidly-climbing reallocated sector count on a drive and mark the drive as troubled, maybe even automatically swap in a spare when that functionality lands?

docs/faq/FAQ.md.

• Why and when should I use Stratis?
• Is Stratis a filesystem?
• Does Stratis work with LVM?
• How does Stratis handle hard drive or other hardware failures?
• Does Stratis provide storage management?
• What happens when disks become full?
• Does Stratis help manage Docker containers?
• How does Stratis compare to ZFS and BTRfs?
• Can I use Stratis with tools like Puppet and Chef?
• Does Stratis have any storage limits?
• What is a flexible file system?
• Can I use Stratis with VM etc.?
• Does Stratis provide SNMP messages to use with my monitoring software?
• Why did you write Stratis in Rust?
• How can I contribute to Stratis?
• How do I submit feature requests?
• I found a bug in Stratis, what do I do?