Code Monkey home page Code Monkey logo

teip's Introduction


Masking tape to help commands "do one thing well"

Latest version Test Status

Taping

Git Animation for Introduction

  • Convert timestamps in /var/log/secure to UNIX time
$ cat /var/log/secure | teip -c 1-15 -- date -f- +%s
  • Replace 'WORLD' with 'EARTH' on lines containing 'HELLO'
$ cat file | teip -g HELLO -- sed 's/WORLD/EARTH/'
  • Make characters upper case in the 2nd field of a CSV (RFC4180)
$ cat file.csv | teip --csv -f 2 -- tr a-z A-Z
  • Edit the 2nd, 3rd and 4th fields of a TSV file
$ cat file.tsv | teip -D '\t' -f 2-4 -- tr a-z A-Z
  • Edit lines containing 'hello' and the three lines before and after
$ cat access.log | teip -e 'grep -n -C 3 hello' -- sed 's/./@/g'

Performance enhancement

teip allows a command to focus on its own task.

Here is a comparison of the processing time to replace approx 761,000 IP addresses with dummy ones in a 100 MiB text file.

benchmark bar chart

See detail at wiki > Benchmark.

Features

  • Taping: Help the command "do one thing well"

    • Passing a partial range of the standard input to any command — whatever you want
    • The targeted command just actions the passed parts of the standard input
    • Flexible methods for selecting a range (Select like awk, cut or grep)
  • High performance

    • The targeted command's standard input/output are written to and read from by multiple teip threads asynchronously.
    • If general UNIX commands in your environment can process a few-hundred MB file in a few seconds, then teip can do the same or better performance.

Installation

macOS (x86_64, ARM64) / Linux (x86_64)

Install Homebrew, and

brew install teip

Linux (x86_64, ARM64)

dpkg

wget https://github.com/greymd/teip/releases/download/v2.3.2/teip-2.3.2.$(uname -m)-unknown-linux-musl.deb
sudo dpkg -i ./teip*.deb

apt

wget https://github.com/greymd/teip/releases/download/v2.3.2/teip-2.3.2.$(uname -m)-unknown-linux-musl.deb
sudo apt install ./teip*.deb

dnf

sudo dnf install https://github.com/greymd/teip/releases/download/v2.3.2/teip-2.3.2.$(uname -m)-unknown-linux-musl.rpm

yum

sudo yum install https://github.com/greymd/teip/releases/download/v2.3.2/teip-2.3.2.$(uname -m)-unknown-linux-musl.rpm

If necessary, check the hash value from the latest release page. Files whose filenames end with sha256 have hash values listed.

Windows (x86_64)

Download installer from here.

See Wiki > Use on Windows for detail.

Other architectures

Check the latest release page for executables for the platform you are using.

If not present, please build teip from source.

Build from source

With Rust's package manager cargo

cargo install teip

To enable Oniguruma regular expression (-G option), build with --features oniguruma option. Please make sure the libclang shared library is available in your environment.

### Ubuntu
$ sudo apt install cargo clang
$ cargo install teip --features oniguruma
### Red Hat base OS
$ sudo dnf install cargo clang
$ cargo install teip --features oniguruma
### Windows (PowerShell) and choco (chocolatey.org)
PS C:\> choco install llvm
PS C:\> cargo install teip --features oniguruma

Usage

USAGE:
  teip -g <pattern> [-Gosvz] [--] [<command>...]
  teip -c <list> [-svz] [--] [<command>...]
  teip -l <list> [-svz] [--] [<command>...]
  teip -f <list> [-d <delimiter> | -D <pattern> | --csv] [-svz] [--] [<command>...]
  teip -e <string> [-svz] [--] [<command>...]

OPTIONS:
    -g <pattern>        Act on lines that match the regular expression <pattern>.
        -o              -g acts on only matched ranges.
        -G              -g interprets Oniguruma regular expressions.
    -c <list>           Act on these characters.
    -l <list>           Act on these lines.
    -f <list>           Act on these white-space separated fields.
        -d <delimiter>  Use <delimiter> for the field delimiter of -f.
        -D <pattern>    Use regular expression <pattern> for the field delimiter of -f
        --csv           -f interprets <list> as field numbers of a CSV according to
                        RFC 4180, instead of whitespace separated fields.
    -e <string>         Execute <string> in another process that will receive identical
                        standard input as the main teip command, emitting numbers to be
                        used as line numbers for actioning.

FLAGS:
    -h, --help          Prints help information.
    -V, --version       Prints version information.
    -s                  Execute a new command for each actioned chunk.
        --chomp         The command spawned by -s receives the standard input without
                        trailing newlines.
    -I  <replace-str>   Replace the <replace-str> with the actioned chunk in <command>,
                        implying -s.
    -v                  Invert the range of actioning.
    -z                  Line delimiter is NUL instead of a newline.

ALIASES:
    -g <pattern>
        -A <number>     Alias of -e 'grep -n -A <number> <pattern>'
        -B <number>     Alias of -e 'grep -n -B <number> <pattern>'
        -C <number>     Alias of -e 'grep -n -C <number> <pattern>'
    --sed <pattern>     Alias of -e 'sed -n "<pattern>="'
    --awk <pattern>     Alias of -e 'awk "<pattern>{print NR}"'

Getting Started

Try this at first:

$ echo "100 200 300 400" | teip -f 3

The result is almost the same as the input, but "300" is highlighted and surrounded by [...], because -f 3 specifies the 3rd field of space-separated input.

100 200 [300] 400

Understand that the area enclosed in [...] is a hole in the masking tape.

Next, put the sed and its arguments at the end.

$ echo "100 200 300 400" | teip -f 3 sed 's/./@/g'

The result is as below. The highlight and [...] will not be present when a command is added.

100 200 @@@ 400

As you can see, the sed command only acted on the input defined by the "hole" and ignored the masked parts. Technically, teip passes only the highlighted part to the sed process, and replaces the highlighted part with the result of the sed command.

Of course, any command you like can be specified. It is called the targeted command in this article.

Let's try cut as the targeted command, to extract the first character only.

$ echo "100 200 300 400" | teip -f 3 cut -c 1
teip: Invalid arguments.

Oops! Why did this fail?

This is because the cutcommand uses the -c option. An option of the same name is also provided by teip, which is confusing.

When specifying a targeted command to teip, it is better to give it after --. Then, teip interprets any arguments after -- as the targeted command and its arguments.

$ echo "100 200 300 400" | teip -f 3 -- cut -c 1
100 200 3 400

Great — the first character 3 is extracted from 300!

Although -- is not always necessary, it is always better to use it. So, -- is used in all the examples from here on.

Now let's double these number with awk. The command looks like the following (Note that the variable to be doubled is not $3).

$ echo "100 200 300 400" | teip -f 3 -- awk '{print $1*2}'
100 200 600 400

OK, the selection in the "hole" went from 300 to 600.

Now, let's change -f 3 to -f 3,4 and run teip.

$ echo "100 200 300 400" | teip -f 3,4 -- awk '{print $1*2}'
100 200 600 800

The numbers in the 3rd and 4th fields were doubled!

As you may have noticed, the argument to -f is compatible with the LIST of cut. You can refer to cut --help to see how it works.

Examples:

$ echo "100 200 300 400" | teip -f -3 -- sed 's/./@/g'
@@@ @@@ @@@ 400

$ echo "100 200 300 400" | teip -f 2-4 -- sed 's/./@/g'
100 @@@ @@@ @@@

$ echo "100 200 300 400" | teip -f 1- -- sed 's/./@/g'
@@@ @@@ @@@ @@@

Select range by character

The -c option allows you to specify a range by character. The below example is specifying the 1st, 3rd, 5th, 7th characters and applying the sed command to them.

$ echo ABCDEFG | teip -c 1,3,5,7
[A]B[C]D[E]F[G]

$ echo ABCDEFG | teip -c 1,3,5,7 -- sed 's/./@/'
@B@D@F@

Like -f, the argument to -c is compatible with cut's LIST.

Processing delimited text like CSV and TSV

The -f option recognizes delimited fields like awk by default.

Any continuous whitespace (all forms of whitespace categorized by Unicode) is interpreted as a single delimiter.

$ printf "A       B \t\t\t\   C \t D" | teip -f 3 -- sed s/./@@@@/
A       B                       @@@@   C         D

This behavior might be inconvenient for the processing of CSV and TSV.

However, the -d option in conjunction with -f can be used to specify a delimiter. You can process a simple CSV file like this:

$ echo "100,200,300,400" | teip -f 3 -d , -- sed 's/./@/g'
100,200,@@@,400

In order to process TSV, the TAB character must be given at the command line. If you are using Bash, type $'\t' which is in the form of ANSI-C Quoting.

$ printf "100\t200\t300\t400\n" | teip -f 3 -d $'\t' -- sed 's/./@/g'
100     200     @@@     400

teip also provides -D option to specify an extended regular expression as the delimiter. This is useful when you want to ignore consecutive delimiters, or when there are multiple types of delimiter.

$ echo 'A,,,,,B,,,,C' | teip -f 2 -D ',+'
A,,,,,[B],,,,C
$ echo "1970-01-02 03:04:05" | teip -f 2-5 -D '[-: ]'
1970-[01]-[02] [03]:[04]:05

The TAB character regular expression (\t) can also be specified with the -D option.

$ printf "100\t200\t300\t400\n" | teip -f 3 -D '\t' -- sed 's/./@/g'
100     200     @@@     400

For the available regular expression notations, refer to regular expression of Rust.

Complex CSV processing

If you want to process a complex CSV file, such as the one below, which has columns surrounded by double quotes, use the -f option together with the --csv option.

Name,Address,zipcode
Sola Harewatar,"Doreami Road 123
Sorashido city",12877
Yui Nagomi,"Nagomi Street 456, Nagomitei, Oishina town",26930-0312
"Conectol Motimotit Hooklala Glycogen Comex II a.k.a ""Kome kome""","Cooking dam",513123

With --csv, teip will parse the input as a CSV file according to RFC4180. Thus, you can use -f to specify column numbers for CSV files with complex structures.

For example, the CSV above will have a "hole" as shown below.

$ cat tests/sample.csv | teip --csv -f2 
Name,[Address],zipcode
Sola Harewatar,["Doreami Road 123]
[Sorashido city"],12877
Yui Nagomi,["Nagomi Street 456, Nagomitei, Oishina town"],26930-0312
"Conectol Motimotit Hooklala Glycogen Comex II a.k.a ""Kome kome""",["Cooking dam"],513123

Because -f2 was specified, there is a hole in the second column of each row. The following command is an example of rewriting all characters in the second column to "@".

$ cat tests/sample.csv  | teip --csv -f2 -- sed 's/[^"]/@/g'
Name,@@@@@@@,zipcode
Sola Harewatar,"@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@",12877
Yui Nagomi,"@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@",26930-0312
"Conectol Motimotit Hooklala Glycogen Comex II a.k.a ""Kome kome""","@@@@@@@@@@@",513123

Notes for the --csv option:

  • Double quotes " surrounding fields are also included in the holes.
  • Escaped double quotes "" are treated as-is; two double quotes "" are given as input to the targeted command.
  • Fields containing newlines will have multiple holes, separated by newlines, instead of a single hole.
    • However, if the -s or -z option is used, such a field is treated as a single hole, and line breaks are included.

Matching with Regular Expressions

You can also use -g to select a specific line matching a regular expression as the hole location.

$ echo -e "ABC1\nEFG2\nHIJ3" | teip -g '[GJ]\d'
ABC1
[EFG2]
[HIJ3]

By default, the entire line containing the pattern is the range of holes. With the -o option, the range of the holes will ony cover the matched range.

$ echo -e "ABC1\nEFG2\nHIJ3" | teip -og '[GJ]\d'
ABC1
EF[G2]
HI[J3]

Note that -og is one of the most useful idioms and is frequently used in this manual.

Here is an example using \d, which matches numbers.

$ echo ABC100EFG200 | teip -og '\d+'
ABC[100]EFG[200]

$ echo ABC100EFG200 | teip -og '\d+' -- sed 's/.*/@@@/g'
ABC@@@EFG@@@

This feature is quite versatile and can be useful for handling files that have no fixed form such as logs, markdown, etc.

What commands are appropriate?

teip passes the strings from the hole line-by-line, so that each hole is one line of input. Therefore, a targeted command must follow the below rule.

  • A targeted command must print a single line of result for each line of input.

In the simplest example, the cat command always succeeds, because the cat prints the same number of lines as it is given in input.

$ echo ABCDEF | teip -og . -- cat
ABCDEF

If the above rule is not satisfied, the result will be inconsistent. For example, grep may fail. Here is an example.

$ echo ABCDEF | teip -og .
[A][B][C][D][E][F]

$ echo ABCDEF | teip -og . -- grep '[ABC]'
ABC
teip: Output of given command is exhausted

$ echo $?
1

teip did not receive results corresponding to the holes of D, E, and F. That is why the above example fails.

If an inconsistency occurs, teip will exit with the error message. Also, the exit status will be 1.

To learn more about teip's behavior, see Wiki > Chunking.

Advanced usage

Solid mode (-s)

If you want to use a command that does not satisfy the condition, "A targeted command must print a single line of result for each line of input", enable "Solid mode" with the -s option.

Solid mode spawns the targeted command multiple times: once for each hole in the input.

$ echo ABCDEF | teip -s -og . -- grep '[ABC]'

In the above example, understand that the following commands are executed by teip internally:

$ echo A | grep '[ABC]' # => A
$ echo B | grep '[ABC]' # => B
$ echo C | grep '[ABC]' # => C
$ echo D | grep '[ABC]' # => Empty
$ echo E | grep '[ABC]' # => Empty
$ echo F | grep '[ABC]' # => Empty

The empty result is replaced with an empty string. Therefore, D, E, and F are replaced with the empty string.

$ echo ABCDEF | teip -s -og . -- grep '[ABC]'
ABC

$ echo $?
0

However, this option is not suitable for processing large files because of its high overhead, which can significantly degrade performance.

Solid mode with placeholder (-I <replace-str>)

If you want to use the contents of the hole as an argument to the targeted command, use the -I option.

$ echo AAA BBB CCC | teip -f 2 -I @ -- echo '[@]'
AAA [BBB] CCC

<replace-str> can be any string, and multiple characters are allowed.

$ seq 5 | teip -f 1 -I NUMBER -- awk 'BEGIN{print NUMBER * 3}'
3
6
9
12
15

Please note that -s is automatically enabled with -I. Therefore, it is not suitable for processing huge files. In addition, the targeted command does not get any input from stdin. The targeted command is expected to work without stdin.

Solid mode with --chomp

If the -s option does not work as expected, --chomp may be helpful.

A targeted command in solid mode always accepts input with a newline (\x0A) at the end. This is because teip assumes the use of commands which return a single line of output in response to a single line of input. Therefore, even if there is no line break in the hole, a line break is added, to ensure it is treated as a single line of input.

However, there are situations where this behavior is inconvenient. For example, when using commands whose behavior changes depending on the presence or absence of a newline.

$ echo AAABBBCCC | teip -og BBB -s
AAA[BBB]CCC
$ echo AAABBBCCC | teip -og BBB -s -- tr '\n' '@'
AAABBB@CCC

The above is an example where the targeted command is: "tr command which converts newlines (\x0A) to @". "BBB" does not contain a newline, but the output is "BBB@", because implicitly-added line breaks have been processed. To prevent this behavior, use the --chomp option. This option gives the targeted command pure input with no newlines added.

$ echo AAABBBCCC | teip -og BBB -s --chomp -- tr '\n' '@'
AAABBBCCC

--chomp is useful whenever using commands which interpret and process input as binary such as tr. Below is an example of "removing newlines from the second column of a CSV which contains newlines.

$ cat tests/sample.csv
Name,Address,zipcode
Sola Harewatar,"Doreami Road 123
Sorashido city",12877

The result is:

$ cat tests/sample.csv | teip --csv -f 2 -s --chomp -- tr '\n' '@'
Name,Address,zipcode
Sola Harewatar,"Doreami Road 123@Sorashido city",12877

Line number (-l)

You can specify a line number and drill holes only in that line.

$ echo -e "ABC\nDEF\nGHI" | teip -l 2
ABC
[DEF]
GHI
$ echo -e "ABC\nDEF\nGHI" | teip -l 1,3
[ABC]
DEF
[GHI]

Overlay teips

Any command can be used with teip, surprisingly, even if it is teip itself.

$ echo "AAA@@@@@AAA@@@@@AAA" | teip -og '@.*@'
AAA[@@@@@AAA@@@@@]AAA

$ echo "AAA@@@@@AAA@@@@@AAA" | teip -og '@.*@' -- teip -og 'A+'
AAA@@@@@[AAA]@@@@@AAA

$ echo "AAA@@@@@AAA@@@@@AAA" | teip -og '@.*@' -- teip -og 'A+' -- tr A _
AAA@@@@@___@@@@@AAA

In other words, by composing multiple functions of teip with AND conditions, it is possible to drill holes in a more complex range. Furthermore, this works asynchronously and in multi-processes, similar to the shell pipeline. Performance will hardly degrade unless the machine reaches the limits of parallelism.

Oniguruma regular expression (-G)

If -G option is given together with -g, the regular expressin is interpreted as an Oniguruma regular expression. For example, "keep" and "look-ahead" syntax can be used.

$ echo 'ABC123DEF456' | teip -G -og 'DEF\K\d+'
ABC123DEF[456]

$ echo 'ABC123DEF456' | teip -G -og '\d+(?=D)'
ABC[123]DEF456

Empty holes

If a blank field exists when the -f option is used, the blank is not ignored and is treated as an empty hole.

$ echo ',,,' | teip -d , -f 1-
[],[],[],[]

Therefore, the following command can work (Note that .* matches empty values as well).

$ echo ',,,' | teip -f 1- -d, sed 's/.*/@@@/'
@@@,@@@,@@@,@@@

In the above example, the sed command reads four newline characters and prints @@@ four times.

Invert match (-v)

The -v option allows you to invert the range of holes. When the -f or -c option is used with -v, holes are made in the complement of the specified field.

$ echo 1 2 3 4 5 | teip -v -f 1,3,5 -- sed 's/./_/'
1 _ 3 _ 5

Of course, -v can also be used with -og.

$ printf 'AAA\n123\nBBB\n' | teip -og '\d+' -- sed 's/./@/g'
@@@
123
@@@

Zero-terminated mode (-z)

If you want to process the data in a more flexible way, the -z option may be useful. This option allows you to use the NUL character (the ASCII NUL character) as a line delimiter, instead of the newline character. It behaves like -z provided by GNU sed or GNU grep, or the -0 option provided by xargs.

$ printf '111,\n222,33\n3\0\n444,55\n5,666\n' | teip -z -f3 -d,
111,
222,[33
3]
444,55
5,[666]

With this option, the standard input is interpreted per each NUL character rather than per each newline character. You should also pay attention to the fact that strings in the hole have the NUL character appended instead of a newline character.

If you use a targeted command that cannot handle NUL characters (and cannot print NUL-separated results), the final result can be unintended.

$ printf '111,\n222,33\n3\0\n444,55\n5,666\n' | teip -z -f3 -d, -- sed -z 's/.*/@@@/g'
111,
222,@@@
444,55
5,@@@

$ printf '111,\n222,33\n3\0\n444,55\n5,666\n' | teip -z -f3 -d, -- sed 's/.*/@@@/g'
111,
222,@@@
@@@
444,55
5,teip: Output of given command is exhausted

This option is useful for treating multiple lines as a single combined input.

$ cat test.html | teip -z -og '<body>.*</body>'
<html>
<head>
  <title>AAA</title>
</head>
[<body>
  <div>AAA</div>
  <div>BBB</div>
  <div>CCC</div>
</body>]
</html>

$ cat test.html | teip -z -og '<body>.*</body>' -- grep -a BBB
<html>
<head>
  <title>AAA</title>
</head>
  <div>BBB</div>
</html>

External execution for match offloading (-e)

-e is the option to use external commands to define pattern matching. Without -e, you must use teip's own functions, such as -c or -g, to control the position of the holes on the masking tape. With -e, however, you can use the external commands you are familiar with to specify the range of holes.

-e allows you to specify a shell pipeline as a string. On a UNIX-like OS, this pipeline is executed via /bin/sh; on Windows via cmd.exe.

For example, given a simple pipeline echo 3, which outputs 3, only the third line will be actioned by teip.

$ echo -e 'AAA\nBBB\nCCC' | teip -e 'echo 3'
AAA
BBB
[CCC]

This works even if the -e output is somewhat "dirty". For example, if spaces or tab characters are included at the beginning of the -e output, they are ignored. Also, once a number is seen, all non-numerical characters to the right of the number are ignored.

$ echo -e 'AAA\nBBB\nCCC' | teip -e 'echo " 3"'
AAA
BBB
[CCC]
$ echo -e 'AAA\nBBB\nCCC' | teip -e 'echo " 3:testtest"'
AAA
BBB
[CCC]

Technically, the first captured group in the regular expression ^\s*([0-9]+) is interpreted as a line number.

-e will also recognize multiple numbers if the pipeline provides multiple lines of numbers. For example, the seq command to display only odd numbers up to 10 is

$ seq 1 2 10
1
3
5
7
9

This means that only odd-numbered rows can be actioned by specifying the following:

$ echo -e 'AAA\nBBB\nCCC\nDDD\nEEE\nFFF' | teip -e 'seq 1 2 10' -- sed 's/. /@/g'
@@@
BBB
@@@
DDD
@@@
FFF

Note that the order of the numbers must be ascending. Now, on its own, this looks like a feature that is just a slight improvement of the -l option.

However, the breakthrough feature of -e is that the pipeline obtains identical standard input as the main teip command. Thus, it generates output using not only seq and echo, but also commands such as grep, sed, and awk, which process the standard input.

Let's look at a more concrete example. The following is a grep command that prints the line numbers of the line containing the string "CCC" and the two lines after it.

$ echo -e 'AAA\nBBB\nCCC\nDDD\nEEE\nFFF' | grep -n -A 2 CCC
3:CCC
4-DDD
5-EEE

If you give this command to -e, you can punch holes in the line containing the string "CCC" and the two lines after it!

$ echo -e 'AAA\nBBB\nCCC\nDDD\nEEE\nFFF' | teip -e 'grep -n -A 2 CCC'
AAA
BBB
[CCC]
[DDD]
[EEE]
FFF

grep is not the only useful command for -e. GNU sed has =, which prints the line number being processed. Below is an example of how to use = to drill from the line containing "BBB" to the line containing "EEE".

$ echo -e 'AAA\nBBB\nCCC\nDDD\nEEE\nFFF' | teip -e 'sed -n "/BBB/,/EEE/="'
AAA
[BBB]
[CCC]
[DDD]
[EEE]
FFF

Of course, similar operations can also be done with awk.

$ echo -e 'AAA\nBBB\nCCC\nDDD\nEEE\nFFF' | teip -e 'awk "/BBB/,/EEE/{print NR}"'

The following is an example of combining the commands nl and tail. You can make holes in only the last three lines of input!

$ echo -e 'AAA\nBBB\nCCC\nDDD\nEEE\nFFF' | teip -e 'nl -ba | tail -n 3'
AAA
BBB
CCC
[DDD]
[EEE]
[FFF]

The argument to -e is a single string. The pipe (|) and other symbols can be used within it.

Alias options (-A, -B, -C, --awk, --sed)

There are several experimental options which are aliases of -e and specific directives. These options may be discontinued in the future since they are only experimental. Do not use them in a script or something that is not a one-off.

-A <number>

This is an alias of -e 'grep -n -A <number> <pattern>'. If it is used together with -g <pattern>, it makes holes in rows matching <pattern>, and <number> rows after the match.

$ cat AtoG.txt | teip -g B -A 2
A
[B]
[C]
[D]
E
F
G

-B <number>

This is an alias of -e 'grep -n -B <number> <pattern>' If it is used together with -g <pattern>, it makes holes in rows matching <pattern>, and <number> rows before the match.

$ cat AtoG.txt | teip -g E -B 2
A
B
[C]
[D]
[E]
F
G

-C <number>

This is an alias of -e 'grep -n -C <number> <pattern>'. If it is used together with -g <pattern>, it makes holes in rows matching <pattern>, and <number> rows before and after the match.

$ cat AtoG.txt | teip -g E -C 2
A
B
[C]
[D]
[E]
[F]
[G]

--sed <pattern>

This is an alias of -e 'sed -n "<pattern>=".

$ cat AtoG.txt | teip --sed '/B/,/E/'
A
[B]
[C]
[D]
[E]
F
G
$ cat AtoG.txt | teip --sed '1~3'
[A]
B
C
[D]
E
F
[G]

--awk <pattern>

This is an alias of -e 'awk "<pattern>{print NR}".

$ cat AtoG.txt | teip --awk '/B/,/E/'
A
[B]
[C]
[D]
[E]
F
G
$ cat AtoG.txt | teip --awk 'NR%3==0'
A
B
[C]
D
E
[F]
G

Environment variables

teip refers to the following environment variables. Add a statement to your default shell's startup file (i.e .bashrc, .zshrc) to change them as you like.

TEIP_HIGHLIGHT

DEFAULT VALUE: \x1b[36m[\x1b[0m\x1b[01;31m{}\x1b[0m\x1b[36m]\x1b[0m

The default format for highlighting holes. It must include at least one {} as a placeholder.

Example:

$ export TEIP_HIGHLIGHT="<<<{}>>>"
$ echo ABAB | teip -og A
<<<A>>>B<<<A>>>B

$ export TEIP_HIGHLIGHT=$'\x1b[01;31m{}\x1b[0m'
$ echo ABAB | teip -og A
ABAB  ### Same color as grep

ANSI Escape Sequences and ANSI-C Quoting are helpful for customizing this value.

TEIP_GREP_PATH

DEFAULT VALUE: grep

The path to the grep command used by the -A, -B, and -C options. For example, if you want to use ggrep instead of grep, set this variable to ggrep.

$ export TEIP_GREP_PATH=/opt/homebrew/bin/ggrep
$ echo -e 'AAA\nBBB\nCCC\nDDD\nEEE\nFFF' | teip -g CCC -A 2
AAA
BBB
[CCC]
[DDD]
[EEE]
FFF

TEIP_SED_PATH

DEFAULT VALUE: sed

The path to the sed command used by the --sed option. For example, if you want to use gsed instead of sed, set this variable to gsed.

TEIP_AWK_PATH

DEFAULT VALUE: awk

The path to the awk command used by the --awk option. For example, if you want to use gawk instead of awk, set this variable to gawk.

Background

Why make this?

See this post.

Why "teip"?

  • tee + in-place.
  • And it sounds similar to Masking-"tape".

License

Modules imported/referenced from other repositories

Thank you so much for these helpful modules!

  • ./src/list/ranges.rs

    • One of the modules used in the cut command is from uutils/coreutils
    • The original source code is distributed under the MIT license
    • The license file is in the same directory
  • ./src/csv/parser.rs

    • Many parts of the source code are referenced from BurntSushi/rust-csv.
    • The original source code is dual-licensed under the MIT and Unlicense

Source code

Teip is available as open source under the terms of the MIT License.

Logo

Creative Commons License
The logo of teip is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

teip's People

Contributors

br1ght0ne avatar garasubo avatar gdetrez avatar greymd avatar jiro4989 avatar k-nasa avatar n4kdoe avatar pandaman64 avatar rolandwalker avatar tranzystorekk avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

teip's Issues

Extended LIST Syntax

echo ABCDE | teip -c '1,3,$-1'
=> [A]B[C][D]E

$-1 supposed to be 4, because $ means the number of field.

Instead of -, .. is used for specifying the range.

1..3 => 1,2,3

can't process (named) pipe

$ teip -og '([0-9]{1,3}\.){3}[0-9]{1,3}' -- sed -n 'i@@@.@@@.@@@.@@@' < test_secure

works

$ teip -og '([0-9]{1,3}\.){3}[0-9]{1,3}' -- sed -n 'i@@@.@@@.@@@.@@@' <(gzip -cd test_secure.gz) 

freezes

but

$ gzip -cd test_secure.gz | teip -og '([0-9]{1,3}\.){3}[0-9]{1,3}' -- sed -n 'i@@@.@@@.@@@.@@@'

works


$ file <(gzip -cd test_secure.gz) 
/dev/fd/63: symbolic link to pipe:[52260780]

$ file -L <(gzip -cd test_secure.gz) -
/dev/fd/63: fifo (named pipe)

$ cat /etc/debian_version 
12.6

$ teip --version
teip 2.3.2

$ ldd $(which teip)
        statically linked

Handles multiple chunks in inter-thread communication

teip degrades the performance rapidly if is has large number of small chunks.

$ yes | tr -d \\n | fold -w 1024 | TEIP_HIGHLIGHT="<{}>" teip -og . | head -n 1
<y><y><y><y><y><y><y><y><y><y><y><y><y><y><y><y><y><y><y><y><y><y><y><y><y><y><y>...
$ yes | tr -d \\n | fold -w 1024 | TEIP_HIGHLIGHT="<{}>" teip -og . | pv >/dev/null
.0MiB 0:00:05 [3.59MiB/s] [              <=>                                                                                                                                     ]
$ yes | tr -d \\n | fold -w 1024 | TEIP_HIGHLIGHT="<{}>" teip -og '.{64}' | pv >/dev/null
30MiB 0:00:04 [31.7MiB/s] [           <=>                                                                                                                                        ]

I believe that this performance degration is caused by the large mount of inter-thread communication.
Actually, futex occupies large ratio of execution time.

$ yes | tr -d \\n | fold -w 1024 | head -n 1024 | TEIP_HIGHLIGHT="<{}>" strace -cf teip -og .
︙
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 92.52    0.582339           7     85927     34598 futex
  5.85    0.036801          36      1026           write
  0.67    0.004226           9       485           sched_yield
  0.49    0.003090          24       130           read
  0.35    0.002197           9       248           brk
  0.06    0.000409         102         4           mmap
  0.01    0.000083          10         8           rt_sigprocmask
  0.01    0.000055          11         5           rt_sigaction
  0.01    0.000047           8         6           sigaltstack
  0.01    0.000039          13         3           munmap
  0.00    0.000028           9         3           mprotect
  0.00    0.000020          20         1           clone
  0.00    0.000016          16         1           poll
  0.00    0.000014          14         1           getrandom
  0.00    0.000013          13         1           ioctl
  0.00    0.000013          13         1           arch_prctl
  0.00    0.000010          10         1           set_tid_address
  0.00    0.000000           0         1           execve
------ ----------- ----------- --------- --------- ----------------
100.00    0.629400                 87852     34598 total

To mitigate the issue, it is good idea to reduce the number of futex calls.
Currently, the PipeIntercepter sends single chunk by tx.send for each time.

https://github.com/greymd/teip/blob/v2.2.0/src/pipeintercepter.rs#L23-L53
https://github.com/greymd/teip/blob/v2.2.0/src/pipeintercepter.rs#L235

This part can be improved more.
If PipeIntercepter can bufferes chunks and sends it to other thread,
number of futex will be reduced and performance improves rapidly under such the particular situation likea above.

フィールドをコマンドに渡す際に、位置を指定して差込むような機能がほしい

日本語ですみません!
すごいいいツールですね!個人マシンに早速入れさせてもらいました!

使ってて欲しいなと思った機能として、フィールドをコマンドにわたす際に標準入力ではなく xargs -I@みたいな形でフィールドを渡せたら便利そうだなぁと思いました。

$ # syslogの日付をdateに渡して変換する例
$ head syslog
Jun 27 15:25:01 BS-PUB-DEVELOP CRON[6002]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jun 27 15:31:32 BS-PUB-DEVELOP systemd[1]: Starting Cleanup of Temporary Directories...
Jun 27 15:31:32 BS-PUB-DEVELOP systemd-tmpfiles[6004]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
Jun 27 15:31:32 BS-PUB-DEVELOP systemd[1]: Started Cleanup of Temporary Directories.
Jun 27 15:35:01 BS-PUB-DEVELOP CRON[6009]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jun 27 15:36:00 BS-PUB-DEVELOP systemd[1]: Starting Daily apt download activities...
Jun 27 15:36:36 BS-PUB-DEVELOP systemd[1]: Started Daily apt download activities.
Jun 27 15:45:01 BS-PUB-DEVELOP CRON[6279]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jun 27 15:55:01 BS-PUB-DEVELOP CRON[6282]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jun 27 16:05:02 BS-PUB-DEVELOP CRON[6286]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

$ # v1.2.0でやる場合
$ cat syslog | teip -og '^([^ ]+ ){3}' -- gdate -f- "+%Y-%m-%d %H:%M:%S "                                                                                                                                                            [2020/06/27 18:02:11 (土) JST]
2020-06-27 15:25:01 BS-PUB-DEVELOP CRON[6002]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd[1]: Starting Cleanup of Temporary Directories...
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd-tmpfiles[6004]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd[1]: Started Cleanup of Temporary Directories.
2020-06-27 15:35:01 BS-PUB-DEVELOP CRON[6009]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:36:00 BS-PUB-DEVELOP systemd[1]: Starting Daily apt download activities...
2020-06-27 15:36:36 BS-PUB-DEVELOP systemd[1]: Started Daily apt download activities.
2020-06-27 15:45:01 BS-PUB-DEVELOP CRON[6279]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:55:01 BS-PUB-DEVELOP CRON[6282]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:05:02 BS-PUB-DEVELOP CRON[6286]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:15:01 BS-PUB-DEVELOP CRON[6289]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:17:01 BS-PUB-DEVELOP CRON[6292]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
2020-06-27 16:25:01 BS-PUB-DEVELOP CRON[6296]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:35:01 BS-PUB-DEVELOP CRON[6299]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:45:01 BS-PUB-DEVELOP CRON[6303]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:55:01 BS-PUB-DEVELOP CRON[6306]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 17:05:01 BS-PUB-DEVELOP CRON[6309]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 17:13:14 BS-PUB-DEVELOP systemd[1]: Started Session 1391 of user blacknon.

$ # こんな感じで指定できると嬉しいかも?
$ cat syslog | teip -og '^([^ ]+ ){3}' -I{} -- gdate -d {} "+%Y-%m-%d %H:%M:%S "                                                                                                                                                            [2020/06/27 18:02:11 (土) JST]
2020-06-27 15:25:01 BS-PUB-DEVELOP CRON[6002]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd[1]: Starting Cleanup of Temporary Directories...
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd-tmpfiles[6004]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd[1]: Started Cleanup of Temporary Directories.
2020-06-27 15:35:01 BS-PUB-DEVELOP CRON[6009]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:36:00 BS-PUB-DEVELOP systemd[1]: Starting Daily apt download activities...
2020-06-27 15:36:36 BS-PUB-DEVELOP systemd[1]: Started Daily apt download activities.
2020-06-27 15:45:01 BS-PUB-DEVELOP CRON[6279]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:55:01 BS-PUB-DEVELOP CRON[6282]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:05:02 BS-PUB-DEVELOP CRON[6286]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:15:01 BS-PUB-DEVELOP CRON[6289]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:17:01 BS-PUB-DEVELOP CRON[6292]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
2020-06-27 16:25:01 BS-PUB-DEVELOP CRON[6296]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:35:01 BS-PUB-DEVELOP CRON[6299]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:45:01 BS-PUB-DEVELOP CRON[6303]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:55:01 BS-PUB-DEVELOP CRON[6306]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 17:05:01 BS-PUB-DEVELOP CRON[6309]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 17:13:14 BS-PUB-DEVELOP systemd[1]: Started Session 1391 of user blacknon.

(このとき、通常だとヒットする箇所ごとにコマンドに渡してると思うんですが、%1 %2 %3...みたいな形で、該当するフィールドを結合して渡せるオプションもあると正規表現で書かないで済みそうなのでとっつきやすいかも思ったんですが、実質的には1個の機能のような気もしてるので、ここで触れさせてください)

$ # さらにこんな感じで指定できると嬉しいかも?
$ cat syslog | teip -f1-3 -I -- gdate -d '\1 \2 \3' "+%Y-%m-%d %H:%M:%S "                                                                                                                                                            [2020/06/27 18:02:11 (土) JST]
2020-06-27 15:25:01 BS-PUB-DEVELOP CRON[6002]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd[1]: Starting Cleanup of Temporary Directories...
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd-tmpfiles[6004]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd[1]: Started Cleanup of Temporary Directories.
2020-06-27 15:35:01 BS-PUB-DEVELOP CRON[6009]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:36:00 BS-PUB-DEVELOP systemd[1]: Starting Daily apt download activities...
2020-06-27 15:36:36 BS-PUB-DEVELOP systemd[1]: Started Daily apt download activities.
2020-06-27 15:45:01 BS-PUB-DEVELOP CRON[6279]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:55:01 BS-PUB-DEVELOP CRON[6282]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:05:02 BS-PUB-DEVELOP CRON[6286]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:15:01 BS-PUB-DEVELOP CRON[6289]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:17:01 BS-PUB-DEVELOP CRON[6292]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
2020-06-27 16:25:01 BS-PUB-DEVELOP CRON[6296]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:35:01 BS-PUB-DEVELOP CRON[6299]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:45:01 BS-PUB-DEVELOP CRON[6303]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:55:01 BS-PUB-DEVELOP CRON[6306]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 17:05:01 BS-PUB-DEVELOP CRON[6309]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 17:13:14 BS-PUB-DEVELOP systemd[1]: Started Session 1391 of user blacknon.

-e option allocates memory aggressively

See https://github.com/greymd/teip/wiki/Memory-Usage-Benchmark

-e requires approx 3 times as much memory as the standard input.
It can be reduced more.

Let me note my brief idea.
One of the reasons why it allocates large mount of memory is standard input is taken "hard copy" in _tee_thread.
https://github.com/greymd/teip/blob/v2.0.0/src/main.rs#L294-L349

Instead of hard copy of text, the text can be possessed in a linked list or a ring buffer.
If so, _tee_thread will not need to pass text through channel, instead, pass the index numbers of buffer.

Then, Main thread and _ex_thread may have to refer to the shared mutex object which indicates the latest index. All the data stored on the indexes smaller than the latest one would be released then.

tokenizationのために外部コマンドを使えるオプション

こんにちは! シェル芸勉強会のLTをみてとても感動しました。

ところで teip は tokenization(? ちゃんと読んだわけではないので誤用だったら申し訳ないです)を自前で実装しているようですが、これに外部コマンド(典型的にはgrepやcutなど?)を用いるオプションがあると便利かなと思いましたので提案してみます。

$ teip -e 'grep <pattern> -A 3' --row

マスキングに用いているコマンドが行指向なのか列指向なのかということを指定してあげる必要がありそうな気がしているのと、ナイーブな実装に比べて遅くなりがちであろうというのは懸念点ですが、現状teipで実装されていないようなマスキングを行いたいときのfallback先として実装しておくのはどうでしょうか? ご意見を伺いたいです

Load CSV file comply with RFC 4180

Example

$ cat file.csv
"AAA","BBB,CCC",DDD
$ cat file.csv | teip --csv -f 2
"AAA","[BBB,CCC]",DDD
$ cat file.csv | teip --csv-ex -f 2
"AAA",["BBB,CCC"],DDD
$ cat file2.csv | teip --csv -f 2
"AAA","[BBB
CCC]",DDD

=> If a record includes the newline, solid mode will automatically be enabled.

$ cat file2.csv | teip --csv --csv-escape-newline -f 2
"AAA","[BBB\nCCC]",DDD

Select range of lines with regex like sed's /AAA/,/BBB/

sed, awk and perl support the range selection with regex.

$ seq 10 | sed -n '/4/,/8/p'
4
5
6
7
8

$ seq 10 | awk '/4/,/8/{ print $0}'
4
5
6
7
8

$ seq 10 | perl -nle 'print $_ if /4/../8/'
4
5
6
7
8

teip should have same feature.
such as..

$ teip -B 'AAA' -E 'BBB'

Came from begin and end.
But in that situation, -s option have to be enabled.

Replace docopt with clap or structopt?

docoptはREADMEにかかれているように、あまり活発に更新されていないので、clapないしstructoptをコマンドライン引数の処理のために使うほうがヘルプなども自動生成されるので、そちらのほうがいいのかもしれないと思い提案させていただきます。
もし、何か特別な理由があってdocoptを選んだのであればcloseしてもらって構いません

Occurrence option

Would be great to be able to do this:

echo $'a\nb\nc\na\nb\nc' | teip -g '^b$' -o 2, -- '' | teip -g '^b$' -- sd '.*' 'B'

To get back:

a
B
c
a

c

FR: man page

It seems that rust folk are allergic to man pages, which is weird because those who are on linux use them all the time. I do understand that it's a linux thing and that writing portable code mean portable documentation but IMVH it's a mistake to not integrate in the man page system; some users will also fine tune their man page system to be more "efficient", I do).

There's also the false impression that creating man pages is tedious, well it can be, but modern solutions exist like converting and installing a markdown file.

Since I plan to use teip more I created one (I do it for my modules when the build system doesn't https://github.com/nkh/ftl/blob/main/config/ftl/man/gen_man_pages)

Screenshot of me searching for solid in your manpage (it's not a default setup but normal man setup will work fine too)

screenshot_2023-09-01_10-08-06

Installation failed on Mac

Hi!
Thank you for the awesome cli command.

I tried to install it on mac using brew but failed.
However, I could install it from source code using cargo.

Thanks in advance.

Brew install errors as follows.

$ brew install greymd/tools/teip
==> Tapping greymd/tools
Cloning into '/usr/local/Homebrew/Library/Taps/greymd/homebrew-tools'...
remote: Enumerating objects: 240, done.
remote: Counting objects: 100% (30/30), done.
remote: Compressing objects: 100% (22/22), done.
remote: Total 240 (delta 16), reused 20 (delta 8), pack-reused 210
Receiving objects: 100% (240/240), 1.79 MiB | 3.14 MiB/s, done.
Resolving deltas: 100% (144/144), done.
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/greymd/homebrew-tools/tmux-xpanes.rb
tmux-xpanes: Calling bottle :unneeded is disabled! There is no replacement.
Please report this issue to the greymd/tools tap (not Homebrew/brew or Homebrew/core):
  /usr/local/Homebrew/Library/Taps/greymd/homebrew-tools/tmux-xpanes.rb:7

Error: Invalid formula: /usr/local/Homebrew/Library/Taps/greymd/homebrew-tools/egzact.rb
egzact: Calling bottle :unneeded is disabled! There is no replacement.
Please report this issue to the greymd/tools tap (not Homebrew/brew or Homebrew/core):
  /usr/local/Homebrew/Library/Taps/greymd/homebrew-tools/egzact.rb:7

Error: Invalid formula: /usr/local/Homebrew/Library/Taps/greymd/homebrew-tools/ewscli.rb
ewscli: Calling bottle :unneeded is disabled! There is no replacement.
Please report this issue to the greymd/tools tap (not Homebrew/brew or Homebrew/core):
  /usr/local/Homebrew/Library/Taps/greymd/homebrew-tools/ewscli.rb:8

Error: Invalid formula: /usr/local/Homebrew/Library/Taps/greymd/homebrew-tools/echo-meme.rb
echo-meme: Calling bottle :unneeded is disabled! There is no replacement.
Please report this issue to the greymd/tools tap (not Homebrew/brew or Homebrew/core):
  /usr/local/Homebrew/Library/Taps/greymd/homebrew-tools/echo-meme.rb:9

Error: Invalid formula: /usr/local/Homebrew/Library/Taps/greymd/homebrew-tools/issagen.rb
issagen: Calling bottle :unneeded is disabled! There is no replacement.
Please report this issue to the greymd/tools tap (not Homebrew/brew or Homebrew/core):
  /usr/local/Homebrew/Library/Taps/greymd/homebrew-tools/issagen.rb:7

Error: Cannot tap greymd/tools: invalid syntax in tap!

My environment

  • System Software Overview:
  System Version:	macOS 12.3.1 (21E258)
  Kernel Version:	Darwin 21.4.0
  Boot Volume:	Macintosh HD
  Boot Mode:	Normal
  Secure Virtual Memory:	Enabled
  System Integrity Protection:	Enabled

  • Hardware Overview:
  Model Name:	MacBook Pro
  Model Identifier:	MacBookPro16,2
  Processor Name:	Quad-Core Intel Core i5
  Processor Speed:	2 GHz
  Number of Processors:	1
  Total Number of Cores:	4
  L2 Cache (per Core):	512 KB
  L3 Cache:	6 MB
  Hyper-Threading Technology:	Enabled
  Memory:	16 GB
  System Firmware Version:	1731.100.130.0.0 (iBridge: 19.16.14243.0.0,0)
  OS Loader Version:	540.100.7~23
  Activation Lock Status:	Disabled

Omit -g if -G enabled

Currently, using oniguruma expression requires -G option and -g.

teip -Gg <pattern>

But it is redundunt.
It is obvious that -g is required when -G is given.
The way like below is better.

teip -G <pattern>

atty potential unalined read

This bug is created from a dependabot alert
Screenshot 2024-01-02 at 19 28 43

Following crates are using atty internally.

  • env_logger 0.7.1
  • structopt v0.3.26
  • criterion v0.3.6
$ cargo tree | grep -C 5 atty
teip v2.3.0 ..
├── cfg-if v0.1.10
├── env_logger v0.7.1
│   ├── atty v0.2.14
│   │   └── libc v0.2.139
│   ├── humantime v1.3.0
│   │   └── quick-error v1.2.3
│   ├── log v0.4.17
│   │   └── cfg-if v1.0.0
--
│           ├── quote v1.0.23 (*)
│           └── unicode-ident v1.0.6
└── structopt v0.3.26
    ├── clap v2.34.0
    │   ├── ansi_term v0.12.1
    │   ├── atty v0.2.14 (*)
    │   ├── bitflags v1.3.2
    │   ├── strsim v0.8.0
    │   ├── textwrap v0.11.0
    │   │   └── unicode-width v0.1.10
    │   ├── unicode-width v0.1.10
--
│   │   ├── predicates-core v1.0.5
│   │   └── termtree v0.4.0
│   └── wait-timeout v0.2.0
│       └── libc v0.2.139
└── criterion v0.3.6
    ├── atty v0.2.14 (*)
    ├── cast v0.3.0
    ├── clap v2.34.0 (*)
    ├── criterion-plot v0.4.5
    │   ├── cast v0.3.0
    │   └── itertools v0.10.5 (*)

I will try to upgrade those crates and see if this alert is gone or not.

Implement -g (grep) option and abolish -r option

teip -g <pattern>

=> compatible with grep

teip -o -g <pattern>

=> only option (current -r option)

teip -Oog <pattern>

=> Oniguruma + only option (current -R option)

teip -g <pattern> -A 3

=> compatible with grep -A -B -C

strip binary file

~ $ uname -a
Linux ip-172-31-9-222 5.4.0-1093-aws #102~18.04.2-Ubuntu SMP Wed Dec 7 00:31:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
~ $ size -A /usr/bin/teip
/usr/bin/teip  :
section                 size      addr
.note.gnu.build-id        36       624
.gnu.hash                 28       664
.dynsym                   24       696
.dynstr                    1       720
.rela.dyn             239160       728
.init                      3    241664
.plt                      16    241680
.plt.got                   8    241696
.text                1654051    241712
.fini                      3   1895763
.rodata               626952   1896448
.eh_frame_hdr          30356   2523400
.gcc_except_table      51984   2553756
.eh_frame             189588   2610800
.tdata                    40   2800392
.tbss                    160   2800432
.init_array                8   2800432
.fini_array                8   2800440
.data.rel.ro          167384   2800448
.dynamic                 384   2967832
.got                    5456   2968216
.data                  20000   2973696
.bss                    9160   2993696
.comment                  60         0
.debug_aranges         52512         0
.debug_pubnames       531616         0
.debug_info           926493         0
.debug_abbrev          15776         0
.debug_line           468849         0
.debug_frame           13624         0
.debug_str           1256784         0
.debug_pubtypes          198         0
.debug_ranges         608736         0
Total                6869458

~ $ du -h /usr/bin/teip
7.4M    /usr/bin/teip
~ $ du  /usr/bin/teip
7532    /usr/bin/teip

Executable file has debug symbols and it occupies big part of the file.
It should be removed.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.