jigarius / drall Goto Github PK

Requirements

• Say, there are 200 sites.
• The user wants to run 5 instances of drush, thereby running commands on 5 sites at a time.
  • User defines number of parallel processes, e.g. --drall-processes=5.
• Ensure intelligible output.
  • If we run a command like foo &; bar & the lines of output of foo and bar get mixed up.
  • We need to ensure that the output of each drush command appears together and is readable.

Solution

• If --drall-processes, say n, is greater than 1, then continue, otherwise, there's nothing special to do.
• Create a unique ID for the command, e.g. a UUID.
• Get a list of all the targeted sites.
• Create a JSON file in the tmp directory to track command progress.
  • Let's call this the task file.
• Launch n drall commands (let's call these workers) with a reference to the task file.
  • Let's call these Drall workers.
  • E.g. drall --drall-task=/path/to/task.json ... &;.
• Each Drall worker picks up a site,
  • Takes a task from the list (if it's not empty).
  • Does the task.
  • Obtains a lock on the list.
  • Updates the list.
  • Prints output.
  • Releases lock on the list.

Tasks

• Decide task file specifications.
  • Decide task file name and location.
  • Decide task file structure.
• Create an internal command exec:task.
  • Takes a task path as an argument.
  • Implement task file locking mechanism, i.e. only one process can write to the task file.
  • Implement task execution mechanism, i.e. execute a task/command from the task file and mark it as done.
• Update the exec:drush command.
  • Support a --drall-processes=n parameter.
  • If n > 1, then generate a task file and launch Drall workers to execute the tasks.
• Update tests
• Update documentation

Tasks

• drall st – Shows status of all sites
• drall st --fields=site ... works correctly
  • Infinite arguments are sent to Drush correctly

Requirements

• If someone wants to issue multiple commands on each site, they need to write a custom script at the moment.
• If we have two separate commands, drush and exec, it'll be great!
  • drall exec:drush ... Run a single Drush command.
  • drall exec:shell ... Run one or more shell command.
• In fact, the drall exec:drush ... command would be like an alias to drall exec drush ...

Requirements

• Currently, Drall uses site's subdir name as --uri=@@uri.
• However, certain commands work better if the --uri is a proper hostname
  • e.g. drush uli generates the correct URL with the correct --uri.
• Thus, it'll be good to have:
  • The @@uri placeholder that uses the keys in $sites.
  • A @@dir placeholder that uses the values in $sites.

Tasks

• Revisit and decide placeholder nomenclature
• @@key represents the keys in $sites.
  • Special $sites keys which include port numbers and paths. See example.sites.php.
  • Use existing libraries to convert these keys into a URI if possible?
• @@ukey represents unique keys in $sites.
• @@dir represents the site directory, i.e. replaces old @@uri.
• Update documentation.

Requirements

• Currently, the dev workflow assumes that Drupal is always installed, i.e. drush site:install.
• In reality, we don't need the Drupal database to develop or test drall unless we really want to test commands that interact with Drupal's database.
• Thus, a good solution will be to make the database setup optional.
• This will speed up build times for the Docker setup and in turn, speed up GitHub actions.

Tasks

• Make drush site:install optional.
• Do not require the database service in CI builds.

Please create a launcher like drush or composer has, which is installed globally, then the drall is installed per project.

Drall currently depends on consolidation/site-alias:3.x but Drupal 10 requires consolidation/site-alias:4.x, which means drall can't be updated to support newer versions of Drupal.

Requirements

• Run linter
• Run tests

Requirements

All sites in a multi-site don't always have site aliases. Many projects rely on drush's --uri option to run the command on the correct site. Thus, we need to allow users to choose whether they want to use the URI or Drush aliases.

Todo

• If the user runs a command with --uri=@@uri, we replace the @@uri with individual site URIs before sending the command to Drush.
• If the user runs a command with @@site, we replace the @@site with a site alias's name part, i.e. excluding the env part.
• If the user runs a command without any of the above, we use --uri=@@uri by default.

When I simple give the command $ ./vendor/bin/drall exd uli
I receive URLs which does not contain the domains:

Running: drush --uri=eaad uli
http://eaad/en/user/reset/1/1658916912/XtxmYUYDfYWCNDiF2Jf43DCXuQmpeNo6JbRW1NFIFhQ/login
Running: drush --uri=escaide uli
http://escaide/en/user/reset/1/1658916913/nFrSxO18O9BEHqC5E9cYO13pnJpp_2_28_qo5Ai0Rc4/login
Running: drush --uri=portal uli
http://portal/en/user/reset/1/1658916913/ugMuDeCVgV0Q5LgUJWTl7fPJ3N7lnDPebipyeVGtoeQ/login
Running: drush --uri=vaccine uli
http://vaccine/en/user/reset/1/1658916913/zzinbUGo-UrOm4izTtFtSByJqQQofUl6AHGIzIxgaMY/login

To have it correctly I need to run ./vendor/bin/drall exd @@@uri uli

Running: drush @eaad uli
http://ecdc-eaad.lh:8080/en/user/reset/1/1658916788/TpF3oxM2c5b3xuyisYbgfdzJqkBSRvNMgP27JjX4KlA/login
Running: drush @escaide uli
http://ecdc-escaide.lh:8080/en/user/reset/1/1658916789/xrZNk927t2Yalo1n3fpPrdRcMY3ZfEN9j_ylpAgjq9o/login
Running: drush @portal uli
http://ecdc-portal.lh:8080/en/user/reset/1/1658916789/gIvVSelKq1VyWkXExzQ7kjWFZ_5ZCrYcXI8FKeal6Gk/login
Running: drush @vaccine uli
http://ecdc-vaccine.lh:8080/en/user/reset/1/1658916789/NXtzQYeDinF04dF4nU5DyZR3zWMrKp-f4ZjHF4bAKDI/login

I would expect to have the correct url with the simple exd uli command too.

NOTE: here for us the @@@uri does work as the directory and the drush alias name is actually the same, otherwise it won't work.

Motivation

Depending on where we write --drall-* options, Symfony Console, Drall commands do not receive the --drall-* options. We need to figure out exactly what's going on and document it in the README.

Requirements

• Passing the --root option should make Drall do site/alias discovery in the correct directory instead of just using PWD.
• Works with Drupal document root
• Works with Composer project root

There are some features and fixes that will need to be ported to v2.x. Once that is done, all efforts will be concentrated on Drall 3.x.

Motivation

• Currently, Drall uses the following placeholders in it’s commands:
  • @@site for site alias' name part, e.g. The @foo part in @foo.env.
  • @@uri for site's subdir under sites.
  • @@host for the site's key in $sites.
• Maybe the names could be better?
  • @@dir instead of @@uri.
  • @@uri instead of @@host.

Need to think over it before jumping to conclusions. This will be required once @@host support is added.

This will make Drall wait n seconds between tasks.
This will also require the number of workers to be set as 1.

I have run "composer require global jigarius/drall" and then when I run the command drall I get the error "drall: command not found"

How do I run drall from the command line after installing this way? Do I instead need to run "composer global require jigarius/drall"? Or is there a different / proper way?

Giving the command: ./vendor/bin/drall exd cr --drall-workers=4 on a site where there are 4 installations, it should do the command under one command time but it's visibly and measurable that it runs serial still, instead of parallel.

Motivation

• When running a drall command with -vvv, the -vvv gets passed to Drush.
• Drush output can get very verbose with -vvv and it makes it hard to read the Drall log messages.

Proposed solution

• Use something like --drall-verbose and --drall-debug to control Drall's verbosity.
• These options should not be forwarded to Drush commands.

Motivation

• Many times, users forget to include exec:drush or exd after drall.
• By default, Symfony Console recommends the command drall list in such cases.
• In our case, we want to recommend exec:drush.
  • Symfony Console doesn't allow overriding the Application::findAlternatives() command, so we'll have to cheat.

Requirements

Allow user to define site groups and then run drush on those groups. Not sure how this will work with site aliases though.

Proposed solution

• User creates DRUPAL/sites/sites.*.php to define groups.
  • sites.php always contains all the sites and it is the default site group, default.
• Running drall exec --drall-group=foo runs the command on all sites in sites.foo.php.
• The site:directories command should support this option.
• The site:aliases command should support this option.

Allow the user to run commands on a list of sites. Say, there are 4 sites, but I want to run the command only on abc.com and def.com.

Possible syntaxes

# This one has a safe namespace and takes multiple sites at once.
drall --drall-uris="abc.com def.com"
# Similarly, we can have something for aliases.
drall --drall-sites="@abc @def" st

Motivation

• Before running a command for real, sometimes it makes sense to preview what will be executed.
  • Say, you're running Drall with --drall-filter, you might want to see exactly which sites will be affected by the filtering.
• For such cases, it'd be good to just see the list of commands that'd be run and not really run them.

Tasks

• Create option: --drall-no-execute for drall exec
  • When this is present, all the commands that'd be run will just be displayed and not actually executed.
• Update tests
• Update documentation

Motivation

• Currently, Drall uses amphp/process libraries to execute commands in parallel.
• This means the output of every command is buffered and then displayed when the command finishes.
• In some scenarios, say, when running with only 1 worker, the user might be interested in seeing the command output in realtime instead.
  • This also helps in debugging Drall.
• Maybe we could use a parameter, say, --drall-no-buffer that would force Drall to throw all output to the console instead of collecting it in a variable.

Tasks

• Add commands to show a list of sites present in the multi-site installation
  • Detect sites using sites/*/settings.php
  • Detect sites using drush alias definitions
  • Preferably, the list of sites should be iterable in shell scripts
• Use a SiteDetector service to make the code re-usable.

Notes

• Only support D8+.
• Use PHP 8 or higher.

Requirements

• Currently, Drall discovers sites defined in sites.php.
  • In this approach, the devs have to actively maintain the sites.php.
• In many cases, the sites/* directories have meaningful names and they don't need to be explicitly defined in sites.php.
• It'll be cool to have an alternate site discovery mechanism using sites/*/settings.php.

If I get 15 thumbs up on the issue, I'll consider starting development.

Not only Bitbucket, it seems Drall gets stuck without any message when it is executed on aliases that refer to remote environments, e.g.

drall exd @@site.dev core:status

Maybe there's some kind of dialog that's present in there that we can't see? This seems like a dealbreaker.

Motivation

• Say, you have a Drupal installation with a 100 sites.
• Say, Drall stops abruptly after 30 sites.
• There should be a way to resume the Drall operation without having to repeat the 30 sites that are already done.

Proposed solution

• Add an option --with-resume.
• When using resume, store Drall's execution state in a drall.state.json file.
• If a state file already exists, then resume the previous operation.

Requirements

• Count and display the number of drush commands that returned a non-zero exit code.
• Return a non-zero exit code if one or more drush commands return a non-zero exit code.

Motivation

• A few versions ago, 2 commands were introduced: exec:shell and exec:drush.
• The exec:drush command was used to ensure that we use the correct drush executable, i.e. the one in the project.
• Thinking about it, this complicates the application by giving us more code to maintain
• If we can unify these 2 commands, we can have drall exec ... in all cases, which is easier for the users as well and makes more sense because after all drush is nothing but a command just like any other.

Proposed solution

• Remove exec:drush command in favor of exec:shell.
  • exec:drush can then be performed with: drall exec:shell drush @@site.local st
• By default the drush executable detected by the system will be used.
• If the user wants to force the Drush that's in vendor/bin/drush, they can write @@drush instead of drush?

Requirements

Currently, drall simply issues the drush command in the shell. If the drush binary isn't somewhere in the $PATH, then we get unknown command drush. Since drall depends on drush/drush, it should be able to detect and use vendor/bin/drush instead of simply issuing the drush command.

Tasks

• Run vendor/bin/drush instead of just drush.
  • Should the runtime output say Running: drush ... to keep it neat? Or should it show the entire /path/to/drush?

I'm using Drall v2.1.0.

Running the command vendor/bin/drall exec drush @@site.T cr inside a container derived from php:8.1.18-cli spawns workers, but they don't return the result or exit code:

My version has the code committed for #71, so I'm not sure what the issue could be.

Requirements

Tried using a quick Ddev setup, but it made life difficult to a great extent. Here's what would help:

• Create a docker-compose.yml
  • Use the official Drupal docker image
  • Use the official MariaDB docker image
• Expose files/directories using volumes
  • composer.json
  • sites.php
  • drush directory
• Create working multi-site setup
  • All sites should have drush aliases.
  • All sites should have a separate database.
• Optionally, all sites should be accessible using the browser?
  • This will probably need modification of the /hosts file.
• Mount drall at /opt/drall
  • It should be possible to run drall from anywhere inside the container.

Motivation

When running a command on multiple sites, the user might want to cancel the operation. Currently, when Ctrl + C is pressed, Drall stops immediately. It might make sense to finish at least the current operation before terminating.

Tasks

• When SIGINT is received,
  • Display a message: Stopping...
  • Do not execute any commands.
• If SIGINT is received more than once, stop immediately.

Motivation

In sites.php the values are not sorted. If you monitor the output, it seems a bit odd. You see a site with z first and then a site with a when you're reading the logs.

Proposed solution

• Sort these records alphabetically by default.
• If someone doesn't want the sorting, they can say --no-sort?

Requirements

Allow the user to choose verbosity of output by using a nicer way of logging with something like the LoggerAwareTrait and ConsoleLogger instead of just writing to $output.

Requirements

• Drall should work on Windows
  • This will mainly involve using the Windows directory separator \.

If I get 25 👍🏽 on this issue, I'll start working on it.

The problem

• Execute drall ex drush cr
• The command runs successfully, but Drall cannot display the [success] Cache rebuild complete. message generated by Drush.

Requirements

• Sometimes, the --drall-group is fixed for specific environments/servers.
• It might be useful to support a DRALL_GROUP env var which will automatically decide the --drall-group value.

Requirements

• Allow users to view the report in different formats.
• Some basic formats I to start with are shown below.

Tasks

• Use Symfony Console's Progress Bar Helper
• Show progress bar by default.
• Implement option to disable progress bar; --drall-no-progress.
• Write tests?
• Update documentation.

Requirements

• Show deprecation warning for the exec command.
• Introduce an exec:drush command.
• Update README
• In the future, an exec:shell command will be added alongside.

This will demotivate people from using the exec command which will eventually be renamed to exec:drush in v2.x.