Code Monkey home page Code Monkey logo

shell-functools's Introduction

shell-functools

Unit tests

A collection of functional programming tools for the shell.

This project provides higher order functions like map, filter, foldl, sort_by and take_while as simple command-line tools. Following the UNIX philosophy, these commands are designed to be composed via pipes. A large collection of functions such as basename, replace, contains or is_dir are provided as arguments to these commands.

Contents

Demo

Quick start

If you want to try it out on your own, run:

pip install shell-functools

If you only want to try it out temporarily, you can also use:

git clone https://github.com/sharkdp/shell-functools /tmp/shell-functools
export PATH="$PATH:/tmp/shell-functools/ft"

Documentation and examples

Usage of map

The map command takes a function argument and applies it to every line of input:

> ls
document.txt
folder
image.jpg

> ls | map abspath
/tmp/demo/document.txt
/tmp/demo/folder
/tmp/demo/image.jpg

Usage of filter

The filter command takes a function argument with a Boolean return type. It applies that function to each input line and shows only those that returned true:

> find
.
./folder
./folder/me.jpg
./folder/subdirectory
./folder/subdirectory/song.mp3
./document.txt
./image.jpg

> find | filter is_file
./folder/me.jpg
./folder/subdirectory/song.mp3
./document.txt
./image.jpg

Usage of foldl

The foldl command takes a function argument and an initial value. The given function must be a binary function with two arguments, like add or append. The foldl command then applies this function iteratively by keeping an internal accumulator:

Add up the numbers from 0 to 100:

> seq 100 | foldl add 0
5050

Multiply the numbers from 1 to 10:

> seq 10 | foldl mul 1
3628800

Append the numbers from 1 to 10 in a string:

> seq 10 | map append " " | foldl append ""
1 2 3 4 5 6 7 8 9 10

Usage of foldl1

The foldl1 command is a variant of foldl that uses the first input as the initial value. This can be used to shorten the example above to:

> seq 100 | foldl1 add
> seq 10 | foldl1 mul
> seq 10 | map append " " | foldl1 append

Usage of sort_by

The sort_by command also takes a function argument. In the background, it calls the function on each input line and uses the results to sort the original input. Consider the following scenario:

> ls
a.mp4  b.tar.gz  c.txt
> ls | map filesize
7674860
126138
2214

We can use the filesize function to sort the entries by size:

> ls | sort_by filesize
c.txt
b.tar.gz
a.mp4

Chaining commands

All of these commands can be composed by using standard UNIX pipes:

> find
.
./folder
./folder/me.jpg
./folder/subdirectory
./folder/subdirectory/song.mp3
./document.txt
./image.jpg

> find | filter is_file | map basename | map append ".bak"
me.jpg.bak
song.mp3.bak
document.txt.bak
image.jpg.bak

Lazy evaluation

All commands support lazy evaluation (i.e. they consume input in a streaming way) and never perform unnecessary work (they exit early if the output pipe is closed).

As an example, suppose we want to compute the sum of all odd squares lower than 10000. Assuming we have a command that prints the numbers from 1 to infinity (use alias infinity="seq 999999999" for an approximation), we can write:

> infinity | filter odd | map pow 2 | take_while less_than 10000 | foldl1 add
166650

Working with columns

The --column / -c option can be used to apply a given function to a certain column in the input line (columns are separated by tabs). Column arrays can be created by using functions such as duplicate, split sep or split_ext:

> ls | filter is_file | map split_ext
document	txt
image	jpg

> ls | filter is_file | map split_ext | map -c1 to_upper
DOCUMENT	txt
IMAGE	jpg

> ls | filter is_file | map split_ext | map -c1 to_upper | map join .
DOCUMENT.txt
IMAGE.jpg

Here is a more complicated example:

> find -name '*.jpg'
./folder/me.jpg
./image.jpg

> find -name '*.jpg' | map duplicate
./folder/me.jpg   ./folder/me.jpg
./image.jpg       ./image.jpg

> find -name '*.jpg' | map duplicate | map -c2 basename
./folder/me.jpg   me.jpg
./image.jpg       image.jpg

> find -name '*.jpg' | map duplicate | map -c2 basename | map -c2 prepend "thumb_"
./folder/me.jpg	  thumb_me.jpg
./image.jpg       thumb_image.jpg

> find -name '*.jpg' | map duplicate | map -c2 basename | map -c2 prepend "thumb_" | map run convert
Running 'convert' with arguments ['./folder/me.jpg', 'thumb_me.jpg']
Running 'convert' with arguments ['./image.jpg', 'thumb_image.jpg']

Get the login shell of user shark:

> cat /etc/passwd | map split : | filter -c1 equal shark | map index 6
/usr/bin/zsh

Available function arguments

You can call ft-functions, to get an overview of all available arguments to map, filter, etc.:

File and Directory operations

abspath             :: Path   → Path
dirname             :: Path   → Path
basename            :: Path   → Path
is_dir              :: Path   → Bool
is_file             :: Path   → Bool
is_link             :: Path   → Bool
is_executable       :: Path   → Bool
exists              :: Path   → Bool
has_ext ext         :: Path   → Bool
strip_ext           :: Path   → String
replace_ext new_ext :: Path   → Path
split_ext           :: Path   → Array

Logical operations

non_empty           :: *      → Bool
nonempty            :: *      → Bool

Arithmetic operations

add num             :: Int    → Int
sub num             :: Int    → Int
mul num             :: Int    → Int
even                :: Int    → Bool
odd                 :: Int    → Bool
pow num             :: Int    → Int

Comparison operations

eq other            :: *      → Bool
equal other         :: *      → Bool
equals other        :: *      → Bool
ne other            :: *      → Bool
not_equal other     :: *      → Bool
not_equals other    :: *      → Bool
ge i                :: Int    → Bool
greater_equal i     :: Int    → Bool
greater_equals i    :: Int    → Bool
gt i                :: Int    → Bool
greater i           :: Int    → Bool
greater_than i      :: Int    → Bool
le i                :: Int    → Bool
less_equal i        :: Int    → Bool
less_equals i       :: Int    → Bool
lt i                :: Int    → Bool
less i              :: Int    → Bool
less_than i         :: Int    → Bool

String operations

reverse             :: String → String
append suffix       :: String → String
strip               :: String → String
substr start end    :: String → String
take count          :: String → String
to_lower            :: String → String
to_upper            :: String → String
replace old new     :: String → String
prepend prefix      :: String → String
capitalize          :: String → String
drop count          :: String → String
duplicate           :: String → Array
contains substring  :: String → Bool
starts_with pattern :: String → Bool
startswith pattern  :: String → Bool
ends_with pattern   :: String → Bool
endswith pattern    :: String → Bool
len                 :: String → Int
length              :: String → Int
format format_str   :: *      → String

Array operations

at idx              :: Array  → String
index idx           :: Array  → String
join separator      :: Array  → String
split separator     :: String → Array
reverse             :: Array  → Array

Other operations

const value         :: *      → *
run command         :: Array  → !
id                  :: *      → *
identity            :: *      → *

shell-functools's People

Contributors

cristiancantoro avatar dboyliao avatar franklingu avatar guilhermeleobas avatar lexofleviafan avatar sharkdp avatar vegarsti avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

shell-functools's Issues

Offering: Desending sort_by

sort_by currently only supports ascending sorts. The following allows descending sorts:

For README.md:

### Usage of `sort_by`

The `sort_by` command also takes a [function argument](#available-function-arguments).
It calls the function on each *line of input*, and uses the results to sort
the *original input* in __ascending__ order.  The `--descending` / `-d` option
will sort the results in __descending__ order.  (There is a `--ascending` / `-a`
option for completeness.)

For ft/ft/commands.sort_by.py:

diff --git a/ft/ft/commands/sort_by.py b/ft/ft/commands/sort_by.py
index b0d0e99..d5e63c0 100644
--- a/ft/ft/commands/sort_by.py
+++ b/ft/ft/commands/sort_by.py
@@ -6,6 +6,19 @@ class SortBy(Command):
     def __init__(self):
         super().__init__("sort_by")
         self.arr = []
+        self.reverse = False
+
+    def add_command_arguments(self, parser):
+        parser.add_argument("--ascending", "-a", dest="reverse",
+                action="store_false", default=False, 
+                help="sort in ascending order (default)")
+        parser.add_argument("--descending", "-d", dest="reverse",
+                action="store_true",
+                help="sort in descending order")
+        return parser
+
+    def parse_additional_command_arguments(self, args):
+        self.reverse = args.reverse
 
     def handle_input(self, value):
         if value.fttype == T_ARRAY and self.column is not None:
@@ -16,5 +29,5 @@ class SortBy(Command):
         self.arr.append((value, result))
 
     def finalize(self):
-        arr = sorted(self.arr, key=lambda x: x[1])
+        arr = sorted(self.arr, key=lambda x: x[1], reverse=self.reverse)
         list(map(lambda x: self.print_formatted(x[0]), arr))

Predicates negation

Hi,
Is there some way to negate predicates in filter? How can I express filtering out things that do not contain some string?

Test breaks: test_add_dynamic_type in ft/ft/test_command.py

__________________________________________________________________________________________ test_add_dynamic_type ___________________________________________________________________________________________

    def test_add_dynamic_type():
        assert add_dynamic_type("True").fttype == T_BOOL
>       assert add_dynamic_type("-1223").fttype == T_INT
E       AssertionError: assert String == Int
E        +  where String = TypedValue('-1223', String).fttype
E        +    where TypedValue('-1223', String) = add_dynamic_type('-1223')

ft/test_command.py:7: AssertionError

I got this error because I am using py.test auto discovery now and it executes test based on certain patterns. This is a failure.

Custom functions

How about custom functions support (or plugins)
Like:

seq 5 | filter is_odd

will return:

1
3
5

where is_odd is custom function (e.g. in ~/.config/shell-functools/plugins/main.py directory)

@functools.func("Int", "Bool") #or @functools.func(int, bool)
def is_odd(a: int) -> bool:
    return a % 2 == 1

but, yes, it looks very complicated...

Offering: Allow `join` function to work on a single column

The current join function fails if a line of input only has one column (i.e., not an array). The update below allow it to function in that case:

@register("join")
@typed(None, T_STRING)
def join(separator, inp):
    if type(inp) == list:
        separator = dynamic_cast(T_STRING, separator).value
        vals = map(lambda x: T_STRING.create_from(x).value, inp)
        return separator.join(vals)

    return inp   # Only one 'column'

Maintenance status of this project

I am not using this project actively myself. I still like the general idea, but I also think this project has a few limitations (e.g. #19) which I am not sure how to solve. In the meantime, there are also other projects like https://www.nushell.sh/ that aim in a similar direction.

Maintenance of this project is therefore also put on hold. If someone wants to take over maintainership of this project, please let me know.

is_link does not detect links

When tried to get started with shell-functools, I noticed that is_link does not work as I expected it to do.

I am on Debian stable. I executed the following steps:

$ touch /tmp/ft-source
$ ln -s /tmp/ft-source /tmp/ft-destination
$ ls /tmp/ft-* | map is_link
False
False

I would expect that one output is True. Indeed, if I execute os.path.islink("/tmp/ft-destination") in IPython, it returns True.

I tried to produce a Docker to reproduce the error. Interestingly, there everything works as I expect it to do.

FROM python:3.9.8-slim-bullseye

RUN python -m pip install shell-functools
RUN touch /tmp/ft-source
RUN ln -s /tmp/ft-source /tmp/ft-destination
CMD ls /tmp/ft-* | map is_link

Am I missing something about how is_link is supposed to work? Or is this a bug?

Support for function composition

It would be great if we could compose functions before we hand them over to map or filter.

Suppose that : would be the reverse composition operator (the flipped version of .):

find | filter extension : equals "jpg" | map prepend "thumbnail_" : replace_ext "png"

Arrays with ints is broken

> echo -e '1\t2' | filter -c1 odd
Traceback (most recent call last):
  File "/home/jdhedden/bin/filter", line 6, in <module>
    Filter().run()
  File "/mnt/data/Computer/Linux/shell-functools/ft/ft/command.py", line 101, in run
    self.handle_input(value)
  File "/mnt/data/Computer/Linux/shell-functools/ft/ft/commands/filter.py", line 34, in handle_input
    self.print_formatted(value)
  File "/mnt/data/Computer/Linux/shell-functools/ft/ft/command.py", line 78, in print_formatted
    formatted = ftformat(result)
  File "/mnt/data/Computer/Linux/shell-functools/ft/ft/internal.py", line 17, in ftformat
    return "\t".join(map(ftformat, val.value))
TypeError: sequence item 0: expected str instance, int found

Offered fix:

diff --git a/ft/ft/internal.py b/ft/ft/internal.py
index 33ee227..b8a981f 100644
--- a/ft/ft/internal.py
+++ b/ft/ft/internal.py
@@ -13,7 +13,7 @@ def colored(inp, col):
 
 def ftformat(val):
     if val.fttype == T_ARRAY:
-        return "\t".join(map(ftformat, val.value))
+        return "\t".join(map(str, map(ftformat, val.value)))
     elif val.fttype == T_PATH:
         return colored(val.value, "cyan")
     elif val.fttype == T_STRING:

Result:

> echo -e '1\t2' | filter -c1 odd
1	2

Categorize (and document) functions

We could add new decorators:

@category("filesystem")
@help("Returns the absolute path")
@register("abspath")
@typed(T_PATH, T_PATH)
def abspath(inp):
  ...

Problem installing in Python3.6 env

When I try to pip install shell-functools in a Python3.6 env I get the following:

Downloading https://files.pythonhosted.org/packages/71/89/8badd56d0264f7a38e3832dac59ad1026ef72f642638b3249ae58b87325a/shell-functools-0.3.0.tar.gz Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-build-aobq6968/shell-functools/setup.py", line 34, in
LONG_DESC = ifile.read()
File "/opt/conda/envs/py3.6/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 6071: ordinal not in range(128)

Numeric file name

When I have a file named 1 among others

ls | filter is_file

Command gives

[functools error] Incompatible input type: expected 'Path', got 'Int'

Add a way to negate the `filter` criterion.

... in order to say something like find | filter --not is_dir

Some options:

  • add a --not/-n option to filter
  • provide a new command reject
  • Add a composition operator (:?) in order to write find | filter is_dir : negate.

Support for 'flip'

It would be great if we could apply flip to a function before we hand it over to a command. For example (suppose that % is a flip-operator):

> seq 3 | map %append "-"
-1
-2
-3

> seq 3 | filter %contains "12"
1
2

Add foldl1

Like foldl but without the initial accumulator argument:

> seq 10 | foldl1 mul
3628800

Handling of type-inference errors (file named `3`)?

As we always read strings via stdin, we have to guess the type of each input line (via add_dynamic_type). This can lead to errors:

> touch a b
> ls | map filesize  # okay
> touch 3
> ls | map filesize  # Incompatible input type: expected 'Path', got 'Int'

How should we handle this? Add explicit type annotations? Always assume everything to be a string?

filtering in a stream doesn't work as expected

if i look in the stream of syslog for a word, the output is as expected

>tail -f /var/log/syslog | map contains Attached  
False
False
False
False
True
False

but if i filter for True nothing comes out

>tail -f /var/log/syslog | map contains Attached | filter equals True 

Strangely it works if the stream is saved to a file before

>tail -f /var/log/syslog > syslog.temp
>cat syslog.temp | map contains Attached | filter equals True
True
True

latest version: 0.3.0

Offering: Update the `run` function to allow for command line arguments

The run function currently only works a single word, so trying to use command line argument with a command doesn't work (e.g., 'chmod 755'). The version adds that capability:

@register("run")
@typed(T_ARRAY, T_VOID)
def run(command, inp):
    command = dynamic_cast(T_STRING, command).value.split()
    args = map(T_STRING.create_from, inp)
    args = list(map(lambda v: v.value, args))

    print("Running '{}' with arguments {}".format(command, args))
    subprocess.call(command + args)

For README.md:

### Usage of `run`

The `run` *function* executes a command over arguments presented:

/usr/bin/ls -1 ft/ft/*.py | map run 'chmod 644'
Running '['chmod', '644']' with arguments ['ft/ft/command.py']
Running '['chmod', '644']' with arguments ['ft/ft/error.py']
Running '['chmod', '644']' with arguments ['ft/ft/functions.py']
Running '['chmod', '644']' with arguments ['ft/ft/init.py']
Running '['chmod', '644']' with arguments ['ft/ft/internal.py']
Running '['chmod', '644']' with arguments ['ft/ft/termcolor.py']
Running '['chmod', '644']' with arguments ['ft/ft/test_command.py']
Running '['chmod', '644']' with arguments ['ft/ft/types.py']
Running '['chmod', '644']' with arguments ['ft/ft/version.py']

Offering: Add array support to `format` function

The current format function only support single-valued inputs. The below permits it to take arrays as inputs:

For ft/ft/functions.py:

@register("format")
@typed(None, T_STRING)
def format(format_str, inp):
    try:
        if type(inp) == list:
            args = list(map(lambda v: v.value, inp))
            return format_str.value.format(*args)
        else:
            return format_str.value.format(inp)
    except ValueError:
        panic("Incorrect format string '{}' for input '{}'.".format(format_str.value, inp))
    except IndexError:
        if type(inp) == list:
            count = len(inp)
        else:
            count = 1
        panic("Not enough arguments (only {} given) for format string '{}'.".format(count, format_str.value))

Example:

> echo -e '1\tone' | map format "'{}' is spelled '{}'"
'1' is spelled 'one'

Commands not installed

I just tried to install shell-functools through pipsi and some commands were not installed.

I only have access to filter, foldl, ft-functions, and map.

I also tried with a regular virtualenv (python 3.5.5) and a pip install, with the same result.

I suspect the Python scripts in the ft folder not to be picked while creating the dist/wheel.

Installation error with pip3

Currently, I can't install with pip3.

$ pip3 install shell-functions 
Collecting shell-functions
Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://pypi.org/simple/shell-functions/

I did verify that the package does not exist in the pypi simple index as well.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.