Feature: allow glob-based recursive interactive search about dua-cli HOT 23 CLOSED

I have implemented approach with additional root for the glob search: https://github.com/gosuwachu/dua-cli/tree/glob_pattern_search - everything works much better! I think this version in terms of UX is pretty close to the final product.

/ to enter glob search, enter to search, esc from main or the glob pane to cancel the search, you can navigate while in glob search and all the normal controls like sorting work as expected.

from dua-cli.

Right, feature level 24 was released yesterday on the 24th :).

from dua-cli.

Sometimes I also wish for having this feature.

For implementing it that would definitely need some fleshing out, but that can probably be done by the one picking it up.

from dua-cli.

I think better option may be to let the user run a command and use its output to the mark files.

It would be a considerable effort to implement fully featured search, and even then dua would be competing with tools better designed to do it.

E.g. you can run:

$ find . -name "target" -type d
./closure-mut/target
./alacritty/target
./dua-cli/target
./rust-hello/target

from dua-cli.

@gosuwachu globs tend to be far easier than remembering the complex syntax of the find or fd commands. If someone is proficient with the CLI tools and remembers all the flags by heart, they are less likely to use an interactive tool to do the same. On the other hand, many users of a rich UI tool are less likely to remember the CLI command parameters.

So yes, I see value in arbitrary list, e.g. supplying a file with a list of files or dynamically generating it from a command output, but I think the lower hanging fruit that would cover 80% would be to support a simple glob pattern.

from dua-cli.

I agree that integrating simple globs would be the way to go here, and think that the glob crate would do the job wonderfully.
There is also https://github.com/helix-editor/nucleo , which might be an even better choice as it's optimized to apply a single expression to a whole set of items/paths.

I think the UX of this feature would also be interesting to flash out.

I imagine it to be invoked with /, which automatically activates the mark pane, which receives all matches as the user types an expression. The expression will be applied to all paths reachable from the current visible folder in the entries list. Some cleanup should be applied to display the shortest viable path, i.e. target should match target/ and foo/target, but not everything within target/. It's also the question what's better here, apply the search to a flattened list of the directory tree, or to leverage the hierarchy. Maybe an iterative approach is possible, too, and one can start on this level of the hierarchy, use matches as is, while going deeper only where there is no match in the hope to find one then. For this case, I believe glob would be better, which then matches one expression against one path at a time.

Definitely an interesting problem to solve and to experiment with.

from dua-cli.

Here is a proposition for the UX side of things:

• Press / to activate the mark pane and display a prompt line
  • below the yellow warning line ("hit Ctrl + r to delete") if it's on top
  • above the yellow warning line if it's on the bottom
• Typing will be interpreted as glob (let's use the glob crate) and all matching entries are displayed (live) in the mark pane
  • It think it's a requirement to show the shortest match only, i.e. target matches target/, but not its children
  • Hitting Ctrl + r|t now would trigger deletion like usual
• Hitting <tab> goes back to the entries pane like before, and the user can add or remove entries like before
  • The prompt line will be removed in that moment.
• Hitting <tab> goes back to the mark pane allowing the user to hit Ctrl + r to trigger a deletion, or interact with it normally, after all we are not in mark-by-prompt anymore.
• Hitting / again (while in the mark-pane) will bring back the previously entered filter term and refresh the mark pane entries according to it (possibly discarding manual changes done previously)
• If the filter does not match any entry, hitting <tab> will go back to the entries pane and auto-close the mark pane, as it generally is only open if there are entries in it.
  • This implies that the new mark-by-prompt mode is able to keep it open even it is empty, but only while the prompt is being entered (i.e. right after / was pressed).

Probably ESC should also be usable to go back to the entries pane (just like it does now if the mark pane is focussed), which will naturally get the user out of the mark pane and deactivate the prompt as per the rules above.

To me this makes sense at least while thinking about the interaction, but of course it might be that when trying it, flaws are discovered. In the end, it's all about feeling natural while being practical.

I'd also recommend to keep the implementation simple for now and it's OK to let the live view be blocking, i.e. if the search takes long, the user will perceive the lag when typing. It's something that can certainly be improved later if it should be necessary.

I'd also out-scope anything smart related to how glob patterns are applied. git for instance allows to write glob patterns that end in /, indicating they should match directories only. Also dealing with case folding should be left for later.

from dua-cli.

Typing will be interpreted as glob (let's use the glob crate) and all matching entries are displayed (live) in the mark pane

I would wait for the user to press enter to filter based on the glob pattern. It is going to be too slow and clunky to display results live on large number of entries.

Hitting / again (while in the mark-pane) will bring back the previously entered filter term and refresh the mark pane entries according to it (possibly discarding manual changes done previously)

If you leave the prompt and add entries based on the pattern then you shouldn't modify your previously entered entries by entering the prompt again. This doesn't sound very intuitive. I think every prompt should be treated as a new search.

I think what you are describing is closer to a glob search preview. I was planning to show a popup in the centre of the screen, with the prompt and list of matches - rather than reusing mark pane for that purpose. I think it will be especially weird to mix manually added entries with the preview in the glob search if we use the mark pane.

from dua-cli.

I would wait for the user to press enter to filter based on the glob pattern. It is going to be too slow and clunky to display results live on large number of entries.

Maybe this is something to try out - live-previews should definitely be the goal, and if necessary one has to offload the computation for a lag-free typing experience. Alternatively, nucleo is made for exactly that. Doing so would cost a lot of memory though and I haven't quite given up on the idea of traversing the tree yet either.

If you leave the prompt and add entries based on the pattern then you shouldn't modify your previously entered entries by entering the prompt again. This doesn't sound very intuitive. I think every prompt should be treated as a new search.

It's a very valid concern that starting another prompt (and typing) would nuke the possibly previously altered list. To me it seemed acceptable especially if this is somehow clarified in the UI. After all, one can play with the search term interactively, then do final manual modifications, and then either delete the files or start over. I think this covers typical use.

I think what you are describing is closer to a glob search preview.

Thanks for the visualisation! However, I do have a distaste for popups and never liked to see them in any TUI that I used or tried. Thus I'd do my best to keep them out of dua as well and think that's possible, too.

I think it will be especially weird to mix manually added entries with the preview in the glob search if we use the mark pane.

You can think of the glob search like an action, that is applied with the result being kept in the mark pane. It's a good point that pressing <enter> should probably confirm the result, which would also mean that pressing <tab> or ESC would instead discard the result. This also leads to the question as to what happens if there were already manually added entries - these should probably be kept and it would be most natural if the search results would be added additively. That way, aborting the search will naturally keep what's there.
Also also, then pressing / again, with some entries already marked, one would probably not want to see the previous search, but start over for another round of additions to what's already there.

After all, one can easily press a oneself to clear all marked entries, so the / logic has no reason to ever do it (as originally suggested by me).

I fully understand if you wouldn't want to tackle this if you are not on board my 'vision' - the more I keep thinking about it, there more intrigued I am to implement it myself 😅 (I have no plans to do that now though).

from dua-cli.

I think I have even better idea. Instead of modifying the mark pane or doing separate popup for the preview it would be better to just filter the entries pane based on the glob pattern. From that point you can work with the filtered entries as usual selecting which ones you want to add to the mark pane.

It would look like this:

• You start with your normal entries pane.
• You press / to show the prompt from which you can enter your glob pattern.
• cycles between focused panes and the glob pattern prompt.
• closes the glob pattern prompt and returns entries pane to normal.

E.g. entries pane with the prompt at the bottom:

And after you press enter you only see the filtered entries list:

from dua-cli.

That's a great idea! It's a possibility that I completely ignored as I considered it unfeasible from the get go, but technical issues aside, it could work. Maybe it all gets easy if the original children of that node would be saved, and the results of the filter operation would be re-linked into the tree location while allowing their names to span multiple path components.

Something that I was initially sceptical towards is to treat the filter prompt as its own pane, even though it would also mean that it's possible to interact with the entries pane naturally using the usual hotkeys.
It's still unclear to me how the prompt can be cleared, maybe hitting <ESC> can do the trick, while retaining selections and markings. It would probably link the original children of that node back to restore the original tree.

Pressing <enter> to filter isn't where I want it to be, but doing it live could be seen as optimisation, and as something that requires more complexity in order to at least be lag-free, even though it won't always be instant no matter how fast it is. So ultimately, that could be omitted for the very first version to keep things simpler.

I hope the implementation of this won't give you too much trouble, but if it works it will be good I am sure! Good luck!

from dua-cli.

We would need some "pending" indicator, like a little spinner character, to indicate that the filtering (searching) is still in progress, and the shown results are incomplete. Changing the content of the filter box would restart the search. But I really like this approach.

from dua-cli.

Here is very rough version if you want to try it: https://github.com/gosuwachu/dua-cli/tree/glob_pattern_search. You can activate it with / shortcut.

It still has some unsolved problems:

• what should happen when user goes into one of the filtered directories? (maybe it shouldn't be possible in v1)
• the UX still has some rough edges (e.g. cancelling a search is a bit painful)
• cursor in glob search entry field is off when deleting characters for some reason

from dua-cli.

This is awesome, thank you!!!

Thinking out loud:

• if a user hits Enter, we could ignore it for now
• We could also show some warning like this:
  • Tree navigation is disabled when filtering. Hit "Enter" again to clear the filter and open directory
• Allow "navigation in filtered mode". The "top level" in a filtered view is a list of files and dirs - i.e. a new "virtual root" of the program, which can be navigated just like filtering was not enabled. The only difference with the regular "starting root" is that the list names have prefixed paths instead of just the directory entry names, there is a glob search at the bottom.
  • The only problem I can see here is if the user tries to modify filter while inside a directory. Possible solutions: disable filter modification in subdirs, or on filter change immediately jump to the top "virtual root". The second option is a bit more surprising, so perhaps disable with a message like filter can only be modified from the root, hit "Home" to jump to the top

from dua-cli.

Thanks so much for bringing the feature this far and for sharing the preview :)!

It's also now obvious (even to me) how great it is to use the entries view for displaying the glob results as it now allows to drill down and select from there, which gives a lot of choice to the user.

Here are my thoughts.

Interestingly I always thought of the workflow as:

• type /
• type the glob, see the search results life
• hit ESC to cancel and Enter to accept the list

Now the tree stays as is.

I truly love @nyurik's idea of treating glob matches as virtual root. What if one could go up to see the following two entries.

• listing (absolutely open for better words here)
  • below this entry is the original tree
• matches (or whatever it's called in the UI, should be consistent)
  • contains all the matches of the previous search

As the underlying data structure expects to have only a single parent for each node to enable proper navigation, I don't think it's possible to actually drill into the entries that are generated from matches. Otherwise, if you go 'down' and 'up' again, there is a choice which parent to pick. Of course, this could be solved by keeping track of the path so far and it would be a cool feature.

In theory, this adjusted tree of two nodes, listing and matches, could remain throughout the runtime of the application and matches get updated each time a new search is made.

Doing it like this has an advantage of being natural to the UI and there is no special case for these two lists, no extra keys to learn or to wire up. I also like the idea of the prompt list being much like a modal - while you are in, it intercepts everything and action needs to be taken to get back to the 'normal' mode of the UI.

from dua-cli.

Thx @Byron!

hit ESC to cancel and Enter to accept the list

Why would you want to have the accepted/unaccepted states here? I think that would introduce unneeded extra complexity. I was thinking the "virtual root" gets updated on each filter change, and the user can already move the cursor up/down through that list, e.g. if it is too long, as well as select/di-select items. IMO, Enter should always be reserved for entering sub-dirs.

from dua-cli.

Thanks for chiming in as well.

Did you know that o also enters subdirs? And j and k move around in the list? That's why I only ever see the glob entry as modal, it has to catch everything. And from there, everything I shared seemed logical.

The only one who truly knows though is @gosuwachu, and I leave it to their experience to come up with what feels best.

from dua-cli.

Thx @Byron, I never realized letters were used in that way... The problem with this approach is that it introduces extra states to the users who might never need it... to the point of me even wondering if it should be possible to have two UI systems: vim - like (with data-entry vs non-data-entry modes) and mode-less (special-keys and letters serve different purpose, letters are for typing only). I am a big fan of the later (mostly because Dvorak keyboard layout is not that great for letter-based navigation), but I know a lot of people love their vim, and who am I to say they are wrong :)

from dua-cli.

P.S. Note that the requirement to treat letters as special keys also breaks the expectation of the non-vim GUI users to quickly jump to the needed file/dir, similar to most GUI file lists (on all platforms).

from dua-cli.

I was going to implement virtual root, but ran into problems which would require bigger refactor. So I have thought to validate what we have so far. Thanks for your feedback, I think we are on the right path.

My plan was to introduce some sort of "entry tree provider/source" which would allow us to switch between completely separate trees: normal and matches. I don't think it is possible to do it without copying all the entries or introducing reference counting to allow entries to live in both trees.

from dua-cli.

My plan was to introduce some sort of "entry tree provider/source" which would allow us to switch between completely separate trees: normal and matches. I don't think it is possible to do it without copying all the entries or introducing reference counting to allow entries to live in both trees.

I think this would be a good first step if it's moderately easy to implement. My feeling is that if this was done, and if one would deem copying everything unacceptable, one could live with the shortcomings of effectively implementing two completely separate trees, and loose the ability of drilling down into the matched patterns. A side-effect of this would probably be that the markings can't easily be shared anymore if they refer remember the node index (which I think they do but didn't check).

Of course, I did hope there was a way to 'link' up the matches accordingly, but also hope the alternative doesn't end up being complex in a different way along with other shortcomings.

@nyurik It's probably not quite easy to convey ones thought exactly, and I am hopeful that I just have trouble getting the idea across. In any case, I think it's best to wait for the PR to be opened to see how it really feels.

P.S. Note that the requirement to treat letters as special keys also breaks the expectation of the non-vim GUI users to quickly jump to the needed file/dir, similar to most GUI file lists (on all platforms).

This is an interesting note as dua really only has to work for me - that's the reason it exists and also is the reason there aren't any new features unless they are contributed. However, I'd also hope that there is a reasonably large intersection between 'works for me' and 'is generally matching expectations of a large group of users' :).

from dua-cli.

TODO at this stage:

• code clean up
• long paths should be shortened to "root/.../leaf" so that not go out of screen
• probably some smaller bugs

from dua-cli.

I believe this is now done and available in the recent release v2.24.0! 🎉

from dua-cli.