Running test 24: spawn.J
Spawn location for J
---- sending commands ----
cJPq
---- awaiting results ----
Test 24 failed:

Program times out and does not return 'actual output' and 'expected output'. This is not the only occurrence, it happens for other tests as well.

Example: implementation quits when g is received and testris is still trying to send the matrix data. You'll get:

Running test 3: io.g
g : given
---- sending commands ----
g
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
m m m m m m m m m m
b b b b b b b b b b
c c c c c c c c c c
g g g g g g g g g g
y y y y y y y y y y
o o o o o o o o o o
r r r r r r r r r r
. . . . . . . . . .
. . . . . . . . . .
Traceback (most recent call last):
  File "testris.py", line 158, in <module>
    main()
  File "testris.py", line 155, in main
    if program_args: run_tests(program_args[0], program_args[1])
  File "testris.py", line 131, in run_tests
    run_test(program, opcodes)
  File "testris.py", line 101, in run_test
    program.stdin.write((cmd + "\n").encode('utf-8'))
IOError: [Errno 22] Invalid argument

There are lots of error conditions testris needs to handle, but there are currently no tests for this error checking. In fact, there are no tests at all. We should fix this.

On windows, running under git bash:

/c/Python33/python testris.py /c/Python33/python learntris.py just times out, even though /c/Python33/python learntris.py runs correctly.

The given learntris.py implementation is:

import sys
sys.stdin.readline()

(Which should pass the first test.) The --shell option makes no difference.

All variations work fine under python 2.7.6:

/c/Python27/python2 testris.py --shell /c/Python33/python learntris.py
/c/Python27/python2 testris.py --shell /c/Python27/python2 learntris.py
/c/Python27/python2 testris.py  /c/Python33/python.exe learntris.py
/c/Python27/python2 testris.py  /c/Python27/python2.exe learntris.py

Right now there is only one test, pertaining to p (print), q (quit), and g (state input). The test suite should cover all working mechanics of Tetris.

Traceback (most recent call last):
File "testris.py", line 165, in main
run_tests(cmdline, use_shell)
File "testris.py", line 135, in run_tests
run_test(program, opcodes)
File "testris.py", line 119, in run_test
program.stdout.read().decode('utf-8').split("\n")]

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe3 in position 18: invalid continuation byte

Oh no! Testris encountered an unexpected problem while
attempting to run your program. Please report the above
traceback in the issue tracker, so that we can help you
with the problem and provide a better error message in
the future.

May be something is wrong with my code .
But it said to report it so here I am.
Here is my code :-https://github.com/ashoknailwal/learntris

why is test2 failing with the following code in "learntris.py"

http://pastebin.com/MH7HYYEC

it produces the said pattern correctly :/

and im using the ./testris.py --shell ./learntris.py command to run it in linux terminal.

There's no need for the run_tests routine to show anything until a test fails.

Probably we could accomplish this by storing the input/output in some kind of TestResult object, and returning that instead of throwing an exception.

This is more in line with how python's own unittest module works, and opens the door to more interesting front ends in the future.

I have executed the following command in the terminal:-
$python testris.py learntris.py
It giving me error:-
Traceback (most recent call last):
File "testris.py", line 158, in
main()
File "testris.py", line 155, in main
if program_args: run_tests(program_args[0], program_args[1])
File "testris.py", line 127, in run_tests
program = spawn(program_args, use_shell)
File "testris.py", line 77, in spawn
stdout=subprocess.PIPE)
File "/usr/lib/python2.7/subprocess.py", line 679, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Currently you need to run extract.py to generate test cases in the tests/ directory.

Now that extract.py generates TestDescription objects, it would make more sense for testris.py to use those objects directly.

Basically, if anything happens other than a successful run of the program (where it finishes and returns error code 0), then we should display a helpful message to the user.

As it stands, a failure to launch causes a traceback to be thrown, which isn't useful for anyone who doesn't happen to be learning python. :) Exceptions should be trapped.

As said in #3, expect is broken and should be replaced by a custom (or existing) Python module that uses subprocess. It should not be too hard to rewrite the expect-using code to use some API with readline() instead.

ccca3d6

Traceback (most recent call last):
  File "testris.py", line 184, in <module>
    main()
  File "testris.py", line 155, in main
    except FileNotFoundError as e:
NameError: global name 'FileNotFoundError' is not defined

Question in the title

This should be one of the first few tests. It would establish the concept that the tests send multiple commands at once to your implementation.

see also #27

Traceback (most recent call last):
  File "testris.py", line 163, in main
    try: run_tests(cmdline, use_shell)
  File "testris.py", line 135, in run_tests
    run_test(program, opcodes)
  File "testris.py", line 119, in run_test
    program.stdout.read().decode('utf-8').split("\n")]
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 32: invalid
 start byte

Oh no! Testris encountered an unexpected problem while
attempting to run your program. Please report the above
traceback in the issue tracker, so that we can help you
with the problem and provide a better error message in
the future.

https://github.com/LelouchVlBritannia/learntris - my code is in there ( its a mess)
MinGW/MSYS -- GCC 4.8.1
Python 3.4

Most likely I am doing something really weird but just reporting because it said to report it.

I read through the RUNNING PROGRAMS OTHER THAN EXECUTABLE BINARIES section and attempted to apply it to Java, but when I try this i keep getting an error that reads:

File Not Found: C:\windows\system32\java.exe

but it's there. I know because if I run the following in CMD, it works as it should (blank output).
C:\windows\system32\java.exe learntris

• The spawn routine should take a list of strings just like subprocess.Popen.
• The program_name variable should just be program and it should be a list.
• The check for whether the program actually exists (in find_learntris needs to be adjusted accordingly. Probably the check for whether the path exists should just be removed, and we should treat a failure to launch the executable like any other exception (i.e., it should stop the test suite and show a test failure...)

When a test fails you get a unified diff output like so:

Test 8 failed: output mismatch:
+ . . . .
+ c c c c
+ . . . .
+ . . . .

It would be very helpful to explain what this means, or switch to a less confusing output (maybe "This line is different" instead of - and +.)

#+name: collide.floor
#+begin_src
> cT vvvv vvvv vvvv vvvv vvvv vvvv Pq
. . . . . . . . . . #  0
. . . . . . . . . . #  1
. . . . . . . . . . #  2
. . . . . . . . . . #  3
. . . . . . . . . . #  4
. . . . . . . . . . #  5
. . . . . . . . . . #  6
. . . . . . . . . . #  7
. . . . . . . . . . #  8
. . . . . . . . . . #  9
. . . . . . . . . . # 10
. . . . . . . . . . # 11
. . . . . . . . . . # 12
. . . . . . . . . . # 13
. . . . . . . . . . # 14
. . . . . . . . . . # 15
. . . . . . . . . . # 16
. . . . . . . . . . # 17
. . . . . . . . . . # 18
. . . . . . . . . . # 19
. . . . M . . . . . # 20
. . . M M M . . . . # 21
= Oh, how the Mighty have Fallen...
: Once upon a time, a T tetramino spawned at the
: top of the well, and used the 'v' command to
: move down by one cell 24 times. Sadly, it was
: unable to burrow into the earth and spent the
: remainder of its life standing on its head at
: the bottom of a well.
#+end_src

It moves down past the end of the screen.

In Tetris, as soon as it hits the bottom of the screen it should be considered a hard drop like V and change to a new tetromino.

Learntris assumes the T is still active even as it goes beyond the scren and is dropped... which makes no sense at all. The tetris implementation should select a new tetromino.

It should also not use P because that implies it's still active.

So it should be something like cT vvvv vvvv vvvv vvvv vvvv pq (move 20 rows down, then print normally.)

cc @tangentstorm

Given the following learntris:

#!/usr/bin/env python
while True: pass

...testris produces the following output:

Running test 1: io.q
q : quit
---- sending commands ----
q
---- awaiting results ----
Test 1 failed: <program timed out>

It's probably for the best that the normal explanatory text for the test is missing, since the student should be working on handling 'q' correctly, or removing an accidental infinite loop.

However, there should be some explanation about exactly what timeouts are (i.e. that their implementation took too long to respond and could be locked up.) and probably a reminder about the q command.

(See discussion and notes from @mattpul in #25)

Running Testris on windows 10, python3.7
Contents of my learntris.py:
import sys
sys.stdin.readline()
the error I'm seeing:

Traceback (most recent call last):
File "testris.py", line 168, in main
run_tests(cmdline, use_shell)
File "testris.py", line 134, in run_tests
program = spawn(program_args, use_shell)
File "testris.py", line 84, in spawn
stdout=subprocess.PIPE)
File "C:\Program Files\Python37\lib\subprocess.py", line 756, in init
restore_signals, start_new_session)
File "C:\Program Files\Python37\lib\subprocess.py", line 1155, in _execute_child
startupinfo)
OSError: [WinError 193] %1 is not a valid Win32 application

Oh no! Testris encountered an unexpected problem while
attempting to run your program. Please report the above
traceback in the issue tracker, so that we can help you
with the problem and provide a better error message in
the future.

https://github.com/LearnProgramming/learntris/issues

This establishes the concept that an implementation should exit when it sees the q command, rather than just exiting by default.

In tetris, the clock usually drives the active tetromino down, and after it moves it clears lines.

I think v/V (move down) should clear lines, and s shouldn't really exist anymore

For one, winpexpect requires pexpect to be installed, so it will always import the latter first without considering the former. This straight away will lead to it not working.

Even with winpexpect, it hangs, presumably while waiting for input or something. I propose the expect-related stuff be switched out for a subprocess-based matcher (See also: #4)

Example: maximally obtuse TDD programmer does exactly what the error messages indicate, and no more:

$ ./testris.py 

Testris is the test runner for Learntris. Learntris is
the part *you* write, in whatever language you prefer.

If you're seeing this message when you tried to run
testris.py, it means testris can't find your code!

Your first step is to write a *console-mode* program
(one that does absolutely nothing!) and tell testris
where to find it:

    ./testris.py [/path/to/learntris] [arguments]

The path should refer to a physical file on disk, so if
you need command line arguments, create a wrapper program.
The default path is "./learntris".

You can pass extra arguments that will be passed to the
guest learntris program.

If you need more help with setting up, try reading:
https://github.com/LearnProgramming/learntris/wiki/Getting-Set-Up

Once testris is able to launch your program, this message
will be replaced with instructions for implementing your
first feature.

File not found: ./learntris
$ touch learntris
$ ./testris.py 

[Errno 13] Permission denied
Couldn't run './learntris' due to a permission error.
Make sure your program is marked as an executable.
$ chmod a+x learntris 
$ ./testris.py 
--------------------------------------------------
Traceback (most recent call last):
  File "./testris.py", line 168, in main
    run_tests(cmdline, use_shell)
  File "./testris.py", line 134, in run_tests
    program = spawn(program_args, use_shell)
  File "./testris.py", line 84, in spawn
    stdout=subprocess.PIPE)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 8] Exec format error
--------------------------------------------------
Oh no! Testris encountered an unexpected problem while
attempting to run your program. Please report the above
traceback in the issue tracker, so that we can help you
with the problem and provide a better error message in
the future.

  https://github.com/LearnProgramming/learntris/issues

There are probably tons of possibilities for OSError, and maybe not all of them can be gracefully handled without risking being misleading. Anyway, food for thought.

Given the following learntris...

#!/usr/bin/env python
print('x')

We get:

Running test 1: io.q
q : quit
---- sending commands ----
q
---- awaiting results ----
: The 'q' command instructs learntris to quit.
:
: Learntris should not produce any output unless
: explicitly instructed to do so.
---- expected results ----

Test 1 failed: output mismatch:
- x

There are several cleanups that should be made:

1. The lines Running test 1: io.q and Test 1 failed: output mismatch: should be modified so that both refer to the test name and neither refers to the number.
2. There should be an explanation of the diff output.

Here is an example of how an improved version might look:

Running test: [io.q]
q : quit
+--- sending commands ----
q
+--- awaiting results ----
: The 'q' command instructs learntris to quit.
:
: Learntris should not produce any output unless
: explicitly instructed to do so.
+--- expected results ----
+--- output mismatch! ----
:
: Test [io.q] failed.
:
: The following diff show the changes required
: to pass the test.
:
:  '-' indicates a line to be removed,
:  '+' indicates a line to be added.
+-------------------------
- x

As it stands, there is nothing showing how to get started with learntris or even really what it does. Ideally there would be a tutorial or just a description of all of the tests that programs should pass.

When I use C:\Users\aaron\learntris>python testris.py
I get the typical message of: "Testris is the test runner for testris... File not found: ./learntris"

Now, this is the message I get whenever I try to run: C:\Users\aaron\learntris> python testris.py learntris.py
I think that this should work (I'm using windows cmd.exe and python is in my path).

Any assistance would be wonderful!

Traceback (most recent call last):
File "testris.py", line 165, in main
run_tests(cmdline, use_shell)
File "testris.py", line 131, in run_tests
program = spawn(program_args, use_shell)
File "testris.py", line 81, in spawn
stdout=subprocess.PIPE)
File "C:\Program Files (x86)\Python35-32\lib\subprocess.py", line 947, in init
restore_signals, start_new_session)
File "C:\Program Files (x86)\Python35-32\lib\subprocess.py", line 1224, in _execute_child
startupinfo)
OSError: [WinError 193] %1 is not a valid Win32 application

Oh no! Testris encountered an unexpected problem while
attempting to run your program. Please report the above
traceback in the issue tracker, so that we can help you
with the problem and provide a better error message in
the future.

https://github.com/LearnProgramming/learntris/issues

The error:

Traceback (most recent call last):
File "testris.py", line 154, in
main()
File "testris.py", line 151, in main
if program_args: run_tests(program_args[0], program_args[1])
File "testris.py", line 121, in run_tests
program = spawn(program_args, use_shell)
File "testris.py", line 77, in spawn
stdout=subprocess.PIPE)
File "C:\Python27\lib\subprocess.py", line 711, in init
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 948, in executechild
startupinfo)
WindowsError: [Error 193] %1 is not a valid Win32 application

Dear community,

I'm new to this so please forgive me if I can't see the wood for the trees. When I compile and run the 'testris.py' file I get the following error relating to a module. The output is pasted below.

Traceback (most recent call last):
  File "F:\Programming\Python\learntris\PythonApplication1\PythonApplication1\PythonApplication1.py", line 159, in main
    run_tests(cmdline, use_shell)
  File "F:\Programming\Python\learntris\PythonApplication1\PythonApplication1\PythonApplication1.py", line 124, in run_tests
    for i, test in enumerate(extract.tests()):
NameError: name 'extract' is not defined

Would someone be so kind as to give me a hint at where I'm going wrong?

I'm using Visual studio 2015 RC to build the testris code and I'll probably continue using python to build my version of 'learntris'

Many thanks

Being able to graphically play your game would be a great motivation.

This would be best done in either standard Python (i.e. with Tkinter), or as a Web app (preferably using the Python standard library again, or at least bottle.py so that we could drop it in.)

A Web app could use WebSockets to communicate, and Canvas or SVG to draw.

If anyone wants to attempt this (feel free!), don't hesitate to drop by #learnprogramming (see the README for a link) and mention it. We wouldn't mind to guiding you.

I was playing around with learntris to learn as well as to contribute to this effort. I have a program which prints out the '.' in the correct format. I verified this from the command line. The correct code is here
http://pastebin.com/jEBVC987
But test2 does not pass