philyoon / pyp Goto Github PK

pyp should not assume that whitespace is irrelevant. It could do so with a 
--strip option perhaps, but it should not be the default as it leads to 
surprises.

What is the expected output? What do you see instead?
$ echo -e '1\n\n2\n3' | ./pyp
1
2
3

I expect the same as `cat` would do:
$ echo -e '1\n\n2\n3' | cat  
1

2
3

I installed pyp using

    pip3 install pyp

Now if I want to try pyp I get an Syntax Error because of python2 syntax.


  File "/usr/local/bin/pyp", line 542
    print Colors.YELLOW + macro_name , "successfully saved!" + Colors.OFF
               ^
SyntaxError: invalid syntax

Examples:
ls -l |pyp "p[0]|o" => ok
ls -l |pyp "p[0]|pp|o" => wrong, same as "p[0]"
ls -l|pyp "pp[0]|o" => wrong, same as "pp[0]"

I guess modifs on pp are in-place on history and destroy o & h values.

I ported this to cywgin python and included a 'bs' for backslash token.

Keep up the great work!

See attached.

What version of the product are you using? On what operating system?

python 2.6 on windows xp, cygwin

Please provide any additional information below.

'pip install pyp' currently complains that there's no setup.py.  Please add one 
so that it will be more conveniently installable.  Thanks!

pyp --manual --turn_off_color
is colored while it shouldn't

Problem is that reassignment of class Colors to NoColors is done after class 
Docs is defined.
One quite non intrusive solution is to modify Docs.manual as such so that 
manual text is evaluated only when called:

$ diff -u /usr/bin/pyp pyp
--- /usr/bin/pyp    2012-04-04 22:02:16.000000000 +0200
+++ pyp 2012-04-19 00:23:44.183598728 +0200
@@ -1390,7 +1390,7 @@
         '''

         if options.manual:
-            print Docs.manual
+            print Docs().manual()
             sys.exit()

         if options.unmodified_config:
@@ -1476,7 +1476,8 @@


 class Docs():
-    manual = ''' 
+    def manual(self):
+        return ''' 
     ===================================================================================
     PYED PIPER MANUAL

What steps will reproduce the problem?

1. Create test file with 

for j in xrange(50000):
    print ",".join(str(i) for i in [random.choice(xrange(1000)) for i in xrange(8)

2. Test with

time (cat testmedium.txt |~/.local/bin/pyp "mm | p if n==0 else (p[:-2] + 
[(int(x)%12) for x in p[-2:]]) | mm" > /dev/null)

3. See how slow it is and much memory is being used.

What is the expected output? What do you see instead?

I expect it to take under a second and use almost no RAM. Instead it takes 
about 90 seconds on my computer and uses about 730MB of RAM.  If you change 
50000 to 500000 it takes a huge amount of time.

For comparison I tested a perl equivalent 

time (cat testmedium.txt |perl -l -a -F',' -p -e'if ($. > 1) { $F[6] %=12; 
$F[7] %= 12;$_ = join(q{,}, @F[6,7]) }' > /dev/null)

real    0m0.196s
user    0m0.192s
sys 0m0.012s

What version of the product are you using? On what operating system?

pyp 2.11 on ubuntu 12.10

Please provide any additional information below.

Where is the latest beta version?

On page
          http://code.google.com/p/pyp/

the link under ...

"new pythonic REGEX functions added!
02/15/14"                       downloads version 2.13

"Finally, Simple Command Line Statistics Using pyp!  
8/27/12"                        downloads version 2.13

"pyp beta officially released!"             is a brokenlink

...and apparently version 2.13 does _not_ have the "stats" or "math" module 
logic.
Could you provide just _one_ good link on the page?

Thanks

pyp v2.12 (svn rev 238) cannot be run at all on Windows (native Python, not a 
cygwin port) because the temporary files  '/%s/pyp_rerun_%d.txt' for "rerun" 
functionality of pyp are created with an unnecessary leading forward slash in 
the file path. 

I think there is no reason to use '/%s/pyp_rerun_%d.txt' instead of 
'%s/pyp_rerun_%d.txt'.

Python's tempfile.gettempdir() always returns an absolute path from the root of 
the filesystem on all platforms (see Python module source at 
http://hg.python.org/cpython/file/tip/Lib/tempfile.py#l130 ). 

Patch is attached. I have tested that everything works on Linux, and everything 
except "rerun" works on Windows (will post the complete fix for rerun on 
Windows later).


Index: pyp
===================================================================
--- pyp (revision 238)
+++ pyp (working copy)
@@ -1397,12 +1397,12 @@
             print Docs.unmodified_config
             sys.exit()

-        rerun_path = '/%s/pyp_rerun_%d.txt' 
%(tempfile.gettempdir(),os.getppid())        
+        rerun_path = '%s/pyp_rerun_%d.txt' 
%(tempfile.gettempdir(),os.getppid())        

         if options.rerun: #deals with rerunning script with -r flag
             if not os.path.exists(rerun_path):
                 gpid = int(os.popen("ps -p %d -oppid=" % os.getppid()).read().strip())
-                rerun_gpid_path = '/%s/pyp_rerun_%d.txt' 
%(tempfile.gettempdir() ,gpid)
+                rerun_gpid_path = '%s/pyp_rerun_%d.txt' 
%(tempfile.gettempdir() ,gpid)
                 if os.path.exists(rerun_gpid_path):
                     rerun_path = rerun_gpid_path
                 else:

What steps will reproduce the problem?
echo '123' | pyp "p.replace(p.re('BLAH'), '_')"
echo '123' | pyp "p.replace(p.re('[123]'), 'x')"

What is the expected output? What do you see instead?

EXPECTED:
123
xxx

GOT:
_1_2_3_
x23

What version of the product are you using? On what operating system?
2.12, Linux - Fedora 19 (Python 2.7)

Please provide any additional information below.
I tried the above because the docs specifically say: "works great with 
p.replace(p.re(REGEX),STR)".  However, as the above examples indicates, it can 
lead to some unexpected results.  However, since you wish to preserve the 
"normal" behavior of p.re, I propose a new / separate method for doing regexp 
based search-replace, ie, something like:

p.re_replace
or
p.re_sub

...which would simply wrap re.sub

What steps will reproduce the problem?
1. shell out a command to the terminal that uses a systems aliased command
2.
3.

What is the expected output? What do you see instead?

"command not found"


What version of the product are you using? On what operating system?

linux, curent


Please provide any additional information below.

I want to end up with the data in a python shell to manipulate it futher, 
import my own modules etc. The easiest way I've found to do this is:

~/bin$ cat PypCustom.py
class PowerPipeListCustom():
    def pickle(self):
        import cPickle
        return "import cPickle; pp = cPickle.loads(%r)" % (cPickle.dumps(list(self)),)

~/bin$ python -i <(echo 123 | pyp "pp.sort() | pp.pickle()")
>>> pp
['123']

This fails for more complicated examples becaause pyp isn't on python's 
PYTHONPATH. (AttributeError: 'module' object has no attribute 'PypStr')

What steps will reproduce the problem?
1. Install on OSX with Python 2.5.2

What is the expected output? What do you see instead?
running pyp produces error: importError: no module named json

What version of the product are you using? On what operating system?
OSX, python 2.5.2

Please provide any additional information below.

That will make it easier to install.

What steps will reproduce the problem?
1. echo -e "1\n\n3" | pyp -k 'p'
2. observe the output

What is the expected output? What do you see instead?
expected: 1\n\n3\n
observed: 1\n3\n


What version of the product are you using? On what operating system?
version 2.13


Please provide any additional information below.
possible fix, change line 1463 to
    pipe_input = [x.rstrip() for x in sys.stdin.readlines() if options.keep_false or x.strip()]

currently pp.uniq() can only eliminate duplicates as a whole.  But I think it 
would be cooler if you give it a (w[-1]) so that it would eliminate (or merge 
entries) based on a single column, rather than the whole line. 

I just started trying out pyp and one feature that I was very surprised to find 
missing was a wrapper around re.findall. Currently, there doesn't seem to be an 
easy way of getting all instances of a match of a regular expression, and that 
would be really nice for combining with p.kill.

Any plans regarding ^?

I often need to make line-by-line modifications to large files (1,000,000 lines 
or more), but pyp is not currently suited to this as it reads in the complete 
input before producing output. Would it be possible to add a mode for pyp that 
produces line-by-line output without loading the entire file at once? Obviously 
this would prohibit some operations, such as those involving 'pp', but much 
useful functionality would remain.

Using a logic filter,

$ for i in 1 2 3 4 5; do echo "$i $i"; done | pyp "(int(w[0])+int(w[1]))>5"
3 3
4 4
5 5

works as expected: it shows only lines whose columns sum to greater than 5. 
Likewise with "(int(w[0])+int(w[1])>5)". However, removing the outer 
parentheses produces an unexpected result:

$ for i in 1 2 3 4 5; do echo "$i $i"; done | pyp "int(w[0])+int(w[1])>4"
1
2
3
4
55 5

This is equivalent to the output with "int(w[0])+(int(w[1])>4)": I think it is 
treating int(w[1])>4 as a logic filter and concatenating its result with the 
value of int(w[0]). In any case, if this is the intended behavior, it should 
probably be documented because it differs from Python's operator precedence 
rules.

Thanks for that version (issue 25) with "stats", but it doesn't seem to have 
the latest regexp commands (rereplace, etc).  Is there a version that has all 
the latest goodies?   Thanks again.

How do you install pyp?

I did:

$ pip install pyp

But it looks like pyp is not on PyPI. Maybe you guys should upload pyp to PyPI.

I originally borrowed the implementation of pyp's `cmds_split` function for my 
pyp-inspired `piep` tool. I recently found some cases where this function 
doesn't break up strings properly, so I rewrote the function until it passed, 
and thought you might be interested in the results (or at least the test cases).

There is probably a less-dramatic change to get everything working, but I 
suspect it's hard to understand this function until you write it yourself ;)

https://github.com/gfxmonk/piep/commit/0bbdcc4cedd351fb4472945ffcd38633eab1e322

(note the doctests, which don't pass in `pyp`)

I like pyp but the easiest way to edit file in place with pyp I found was:
cat f | pyp "p.replace('e', 'R')" | sponge f

But it's complicated and sponge isn't standard program (moreutils is not even 
in standard openSUSE's repositories).

So I propose to add option -i --inplace AND/OR add -o --output option that 
handles in place editing correctly. 

I have submitted pyp to Fedora 
(https://bugzilla.redhat.com/show_bug.cgi?id=805332) but there is no copyright 
notice or license in the source code. This prevents it from being accepted by 
Fedora (and probably most other distros). The website says it is the New BSD 
License, but this would need to be in the source code too to meet the 
requirements of Fedora.

Can a copyright notice and license be put in the source code?

What steps will reproduce the problem?
1. cat file | pyp "len(pp)"         # for a file with 500000 numbers
2.
3.

What is the expected output? What do you see instead?
the expected output is 500000

The problem is the method: def flatten_list(self, iterables)
it is recursive and does too many list manipulations.

I've rewritten it:

    def flatten_list(self, iterables):
        '''
        returns a list of strings from nested lists
        @param iterables: nested list to flatten
        @type iterables: list<str>
        '''
        out = []
        stack=[[iterables,0,len(iterables)]]
        while stack:
            curIter,pos,limit=stack[0]
            if pos==limit:
                stack.pop(0)
                continue
            if type(curIter[pos]) not in [str, PypStr]:
                stack[0][1]+=1
                stack.insert(0,[curIter[pos],0,len(curIter[pos])])
            else:
                out.append(curIter[pos])
                stack[0][1]+=1

        return out

Today is the first day I've used pyp, it's great but I've found this issue,
I think that would be great to make it faster, could anyone test this method
and maybe patch the source?

Thanks,
C

% ls *mp4 | pyp "p.rel('mp.')"       

What is the expected output? What do you see instead?

error: 'PypStr' object has no attribute 'rel' : p.rel('mp.')

What version of the product are you using? On what operating system?

#version 2.11

Please provide any additional information below.

I tried all kinds of variations (ie. relose, rekeep, rkeep, rlose, etc.) and 
the only one that works is p.re(<REGEXP).

Thanks!

It would be useful to have a split/join variable 
<code>t</code>/<code>tab</code> serving as the equivalent of 
<code>p.split('\t')</code> or <code>'\t'.join(p)</code>.

Interesting project which I'm eager to try out.  Looking at the docs I see 
several examples of long-form command-line options which have words separated 
by underscores, such as --macro_save.

This is a bit jarring because all (or almost all) of the tools I have used 
separate such words with dashes instead.  Since dashes are common on the unix 
command-line they are committed to muscle memory, not to mention underscore 
requires the shift key (at least on US keyboards).

It can only help the tool to follow this higher efficiency convention.  For 
example the ls --help command:

  -a, --all 
  -A, --almost-all 
      --block-size=SIZE
  -B, --ignore-backups

...

If you agree, here is an announcement you might upload to Android and Qpython 
forums:

The excellent “pyp” text and command processing utility, runs on Android!

The full manual for “pyp” is at 
https://code.google.com/p/pyp/wiki/pyp_manual

How to get going:       
    Install “Qpython” on your device (it is on Google Play) 

    Download “pyp” to your device.  
    As of this writing, the latest version is “pyp_dev.py” from the link at https://code.google.com/p/pyp/issues/detail?id=25

    Copy it to some directory, renaming it “pyp.py” if you wish.    

    When you start “Qpython”, tap the (very unobtrusive) grey “less than sign” at lower right.  
    This will bring up the link to the console. 
    Change directory (cd) to where “pyp.py” is. 
    Try this    
        echo -e “a \n2\n b \n3” | python pyp.py “p.isdigit()”
    and it should print 
        2 and 3.
    If you wish, you can (probably) shorten the typing by copying “pyp.py” to, say, “y”,    
    adding “#qpy:console” as the first line of “y”, and then doing  
        echo -e “a \n2\n b \n3” | sh y “p.isdigit()”

In the examples involving echo of a newline, such as

   echo "1\n5\n10"

some systems do not interpret '\n' properly (see 
http://stackoverflow.com/questions/8467424/echo-new-line-in-bash-prints-literal-
n ), so recommend you use

   printf "1\n5\n10"

instead of 'echo', or at least use the '-e' flag when needed, like this

   echo -e "1\n5\n10"