tomnomnom / anew Goto Github PK

A tool for adding new lines to files, skipping duplicates

License: MIT License

Go 53.92% Shell 46.08%

anew's Introduction

anew

Append lines from stdin to a file, but only if they don't already appear in the file. Outputs new lines to stdout too, making it a bit like a tee -a that removes duplicates.

Usage Example

Here, a file called things.txt contains a list of numbers. newthings.txt contains a second list of numbers, some of which appear in things.txt and some of which do not. anew is used to append the latter to things.txt.

▶ cat things.txt
Zero
One
Two

▶ cat newthings.txt
One
Two
Three
Four

▶ cat newthings.txt | anew things.txt
Three
Four

▶ cat things.txt
Zero
One
Two
Three
Four

Note that the new lines added to things.txt are also sent to stdout, this allows for them to be redirected to another file:

▶ cat newthings.txt | anew things.txt > added-lines.txt
▶ cat added-lines.txt
Three
Four

Flags

To view the output in stdout, but not append to the file, use the dry-run option -d.
To append to the file, but not print anything to stdout, use quiet mode -q.

Install

You can either install using go:

go install -v github.com/tomnomnom/anew@latest

Or download a binary release for your platform.

anew's People

Contributors

Stargazers

Watchers

Forkers

bbhunter inosec hworm ice3man543 offsecdawn developer-box elamaran619 76428778fada 3453-315h polling-repo-continua 0xagentsteal manasmbellani jeanfredson shahid1996 ankurvaishley cowlingbanana krouser iiiusky prongedfork rajivraj alonek1 danielmiessler c0de-sin brycereynolds mhmh55516 no-github isgasho lagait nitindhariyal tamersp25 karmaz95 keyur-dasarwar stfuanu rudraimmunefi michaellcader adibdz fabiopw tkmyles normadchief madusec nestorwheelock gprime31 rtak47 dragneel1050 greenmind-sec hakluke vijaypyii jaredumbehr d1nk0993 gsatyadev marlborojamez matheuswm heavykenny 0xcrypto rblancor th3xace jahodaradek storenth biseshbhattarai unashamedgeek n25sec afwu bugbountydude jeromeyoung zlisang guleum cheahengsoon hack404-007 racthack tommalvoriddle securitystuffbackup lieunguyen-tma rayiik iahoeu jonlittle-fun cmonbb doris49383 dingyshark sylviagaytaneh2021 sunhuaqiang77 sobinge raindouble xmardus scriptkkiddie santosh-gundimeda anandjambhulkar g6kg sugar42 papyrusdo cyberfury101 akbruster cwavesoftware molombitoh1ck3r aherrom 8mrblank r0nk sosaurizar secmind9254 rick-vini pipsson

anew's Issues

installation error

its good but provide some more features.
thanks

I can't install

go install -v github.com/tomnomnom/anew@latest No command output is displayed

Entries appended without a new line

I have two files:
first.txt (Doesn't have an empty line in the end)

1
2
3

second.txt

cat second.txt | anew first.txt

cat first.txt

The solution might be to add a new line if previous character wasn't a new line before appending new entries to the file.

cant use properly

it is just working on my home dir, i am not able to use it anywhere alse

/root/go/src/github.com/tomnomnom/anew/main.go:44:3: non-bool lines[line] (type struct {}) used as if condition

getting the above error ... :)

Feature Request: Add a flag to take whitespace into account

Right now, if there's an entry with a trailing space in the file to be appended to, anew will append another entry (without a trailing space) resulting in duplicate entries.

Please consider adding a toggle flag to ignore or not ignore white space when determining if a new entry should be appended.

how to fix it

github.com/tomnomnom/anew

go/src/github.com/tomnomnom/anew/main.go:44:3: non-bool lines[line] (type struct {}) used as if condition

windows`s how to use?

why my anew windows product can`t to sure work?

Silent flag not working

The -q flag still prints all output to stdout.

go1.20.2

Hi, I'm getting compile: version "go1.18.1" does not match go tool version "go1.20.2"

Is there a workaround for this?

Memory usage

Hi,
Thank you very much for providing useful tools like anew.

I came across the following error

fatal error: runtime: out of memory
...
main.main()
	/root/src/github.com/tomnomnom/anew/main.go:27 +0x546 fp=0xc420feaf80 sp=0xc420feacc0 pc=0x495ea6

Checking further, I can see the memory usage increase when running anew.
Example:

echo "test" | anew file.txt

With a file.txt is 376MB and containing 7 364 495 lines, I can see the memory increase from
138MB to 1.2GB. This is around 3x the size of the file.txt used.

With another test file twice the size and lines, (750MB), Memory goes to 1.6GB (2x the size)

So this make anew difficult to use in some cases ( "big" files and/or low memory like RaspberryPI )

I wonder if there are some possible code modifications possible (or a new option use) to reduce/limit the memory usage (Splitting the file in smaller bits first, write temporary to disk etc... ) even if this impact the time of the operation.. ?

Thank's !

anew breaks my cron

I am running some automation. anew is part of it. when running manually it works fine. but somehow, the script breaks when run using cron. I have tried adding and removing anew in the script to make sure the problem. when I add anew, the script breaking after the line where anew is added.
Please check, reproduce and fix :-D

Submission of anew Package for Debian Packaging

Hello everyone,

My name is Marcos R Carvalho (aka oday).

I have initiated the packaging process for anew for Debian. I would like to express my gratitude for the excellent work done by the developers of anew.

I have submitted the package to the mentors repository for review by other Debian Developers (DDs). You can find the package at the following link: https://mentors.debian.net/package/anew/