Code Monkey home page Code Monkey logo

voc's Introduction

VOC is currently on hiatus

The BeeWare project has pivoted away from using VOC for Android development. We now use an embedded CPython library to provide Android support.

We still believe there is value in a bytecode compilation approach; however, we're not dedicating any BeeWare resources to VOC development, and we we're not encouraging other to contribute to VOC at this time.

If you're still interested in working on VOC, please get in touch.

VOC

image

image

image

image

image

image

A transpiler that converts Python code into Java bytecode.

This is experimental code. If it breaks, you get to keep all the shiny pieces.

What it does:

  • Provides an API to let you programmatically create Java class files.
  • Compiles Python 3.4 source files into Java class files, enabling you to run Python code on a JVM (including Android's VM).

It isn't a completely compliant Python 3.4 implementation - there are some language features (some builtin functions) that still need to be implemented, and there is only a bare bones standard library implementation. However, it is possible to convert simple Python programs, and even write simple Android applications.

Tutorial

To take VOC for a spin, run through the Getting Started guide, then start with the first tutorial.

If you'd like to contribute to VOC development, we have a guide for first time contributors.

Documentation

Documentation for VOC can be found on Read The Docs.

Why "VOC"?

VOC is a backronym for "Vestigial Output Compiler". Or "Vexing Obtuse Compiler". Or "Valuable Obscure Compiler". Or "Varigated Ocelot Calibrator". It's your choice.

Historically, VOC was the Vereenigde Oostindische Compagnie (VOC), or Dutch East India Company, often considered to be the world's first multinational corporation. It was also the first company to issue shares, and facilitate the trading of those shares. It was granted a 21 year monopoly to carry out trade activities in Asia, primarily the Spice Islands - the Dutch East Indies. They established a major trading port at Batavia - now Jakarta - on the island of Java (now part of Indonesia).

While VOC was a very profitable business for it's shareholders, it operated as a colonial power. Modern historians have expressed criticism of their business practices, which included exploitation of indigenous populations (including use of slave labour), slave trade, use of violence, and environmental destruction (including deforestation).

VOC also played a role in the European discovery of Australia, and there are many VOC shipwrecks on the coast of Western Australia.

The name VOC was chosen by the project because the project founder is from Western Australia, and VOC has a strong non-colonial presence in that state's history. It was also a convenient pun, as VOC was an enterprise Java company.

It should not be considered an endorsement or glorification of the exploitative business practices of the company.

Community

VOC is part of the BeeWare suite. You can talk to the community through:

We foster a welcoming and respectful community as described in our BeeWare Community Code of Conduct.

Contributing

To get started with contributing to VOC, head over to our First Timers Guide.

If you experience problems with VOC, log them on GitHub. If you want to contribute code, please fork the code and submit a pull request.

voc's People

Contributors

ad115 avatar amiryalamov avatar amymok avatar aniket1998 avatar asp1234 avatar bio-boris avatar bpyap avatar bsthorpe742 avatar cflee avatar deep110 avatar eliasdorneles avatar flacode avatar freakboy3742 avatar glasnt avatar goanpeca avatar hwrdch avatar jonkiparsky avatar jvmaia avatar khoobks avatar mcmasts avatar onyb avatar patiences avatar saurabh--kumar avatar soandrew avatar sohit-nayak avatar soummyaah avatar stummjr avatar svmundada avatar uranusjr avatar whydoubt avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

voc's Issues

Implement the Python standard library

Python has a sizeable standard library. It is this standard library the provides a lot of the power of Python as a language.

Much of this standard library is implemented in Python, and could be converted into a Jar file using VOC. VOC's build system should be modified to perform this conversion.

Other parts of the standard library are implemented in C, and will require re-implementation in Java.

Even if there is a pure-python implementation, it may be desirable to re-implement a standard module in Java for efficiency purposes.

This may be dependent on completion of #36 and #38, as you won't be able to use the standard library unless the builtin data types are available.

Improve dev environment setup docs for Windows

I struggled through a few issues to get a dev environment up and running on Windows (e.g., pyenv is not supported on Windows and has little/no interest in being so). I'd like to add some detail in an appropriate place on ways for Windows devs to set things up, just not totally sure where that should go so as not to clutter things unnecessarily for the main path.

Tests failing

I followed instruction on how to set up a dev environment. When I run the tests I get at the end of the procedure a number of errors due to missing files. I am running on an Ubuntu 16.4 VM, Python 3.4.5, javac 1.8.0_91 (openJDK) , ant 1.9.6 . Below a sample of the errors I get:

ERROR: setUpClass (tests.structures.test_java.JavaTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/luca/voc-dev/voc/tests/utils.py", line 276, in setUpClass
    setUpSuite()
  File "/home/luca/voc-dev/voc/tests/utils.py", line 60, in setUpSuite
    out, err = proc.communicate(timeout=15)
  File "/usr/lib/python3.4/subprocess.py", line 959, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.4/subprocess.py", line 1621, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.4/subprocess.py", line 985, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command 'ant java' timed out after 15 seconds

----------------------------------------------------------------------
Ran 31 tests in 4771.214s

FAILED (errors=173)
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/luca/voc-dev/env/lib/python3.4/shutil.py", line 457, in rmtree
    orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/home/luca/voc-dev/voc/tests/tmpbf9imex0'

Exceptions usage fail in unexpected ways

Here are some test cases that don't work as expected (tested creating a file testraise.py, putting these snippets on it then compiling w/ VOC and running):

raise ValueError("test")

results in:

Exception in thread "main" java.lang.NoSuchMethodError: org.python.exceptions.ValueError.<init>(Lorg/python/Object;)V
	at python.testraise.__init__.module$import(testraise.py:1)
	at python.testraise.__init__.main(testraise.py)

However, if you do raise ValueError() (no args), the exception is raised normally.

The inverse happens for KeyError: if you don't give it an argument, it fails with the same error as above, but it works if you give it an argument.

A few other exceptions I tested (TypeError, AttributeError and KeyboardInterrupt) also have the same failure pattern as ValueError, there are probably more of this.

Another thing to note (possibly related) is that this also fails: (EDIT: removed separate error, which was duplicate of #350)

Use a hidden name for python functions

If you define a method in Python, VOC will potentially generate three Java-side methods to represent that method:

  • A binding method, representing the Java interface to the method. This will be typed using the specific types annotated in the Python code
  • A super method, with the same prototype as the binding, but explicitly invoking the super.method()
  • The Implementation, representing the actual Python logic. All arguments to this method will be org.python.Object, as will the return type. All implementation methods are static; the first argument is the org.python.Object representing the instance. This largely mirrors Python's own behavior.

When you invoke a Python method from within Python, the implementation is called directly. However, if you invoke from Java, the binding is invoked; this transforms the typed arguments into Python-compatible objects, and invokes the implementation.

At present, the implementation and binding have the same method name, but different prototypes (pure Java types for the binding; static org.python.Object types for the implementation). However, if a method foobar() needs a super, the super method will be called foobar$super(). This is not a legal name in Java syntax, but it is a legal and accessible symbol name. As a result, we can create a super method without any risk of namespace clashes.

It would be desirable for the implementation method to also have a modifier (foobar$py()) to make it clear that the method should only be invoked from within a Python context.

Review str tests that are passing but shouldn't

I've noticed there are some tests in tests/datatypes/test_str that are passing because they're raising the same error, but aren't doing what they're supposing to be doing.

There are also some tests that can be simplified because they have the same setup code (use the same basic string, but several calls to assertCodeExecution).

I'm adding this issue as a reminder for myself to review and cleanup them (can't do it right in this moment).

I think it would be nice to add an optional parameter expected_output to TranspileTestCase.assertCodeExecution to be checked w/ Python and Java output, to ensure that it's what we expect.

Incorrect behavior when giving wrong arguments

I was testing out the behavior of the method signatures before sending #288, and found some unexpected behaviors.

There is probably a better way of testing this, but I did my tests adding these two test methods to the List object:

    @org.python.Method(
        __doc__ = "",
        default_args = {"arg1", "arg2"}
    )
    public org.python.Object some_method(org.python.Object arg1, org.python.Object arg2) {
        System.out.printf("some_method: arg1: %s\n", arg1);
        System.out.printf("some_method: arg2: %s\n", arg2);
        return org.python.types.NoneType.NONE;
    }

    @org.python.Method(
        __doc__ = "",
        args = {"arg1"},
        varargs = "others"
    )
    public org.python.Object method_varargs(org.python.Object arg1, org.python.types.Tuple others) {
        System.out.printf("method_varargs: arg1: %s\n", arg1);
        System.out.printf("method_varargs: others: %s\n", others);
        return org.python.types.NoneType.NONE;
    }

And then I wrote some different combinations of passing arguments, and these two cases in which I pass unexpected arguments result in unexpected behavior:

li = [1, 2, 3]

print("\nli.some_method(1, 'two', 3)")
li.some_method(1, 'two', 3)

print("\nli.method_varargs()")
li.method_varargs()

This is the output for the above program:

li.some_method(1, 'two', 3)
some_method: arg1: 1
some_method: arg2: two

li.method_varargs()
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: -1
    at java.util.ArrayList.elementData(ArrayList.java:418)
    at java.util.ArrayList.get(ArrayList.java:431)
    at org.python.types.Function.adjustArguments(Function.java:251)
    at org.python.types.Function.invoke(Function.java:320)
    at org.python.types.Method.invoke(Method.java:41)
    at python.argstest.__init__.module$import(argstest.py:7)
    at python.argstest.__init__.main(argstest.py)

In both cases I expected to get a TypeError complaining about passing wrong number of arguments, but in the first case the extra argument was silently ignored and in the second case the missing argument causes a Java runtime level exception.

Use type information to optimize code paths

VOC currently assumes all arguments are instances of objects implementing org.python.Object. This makes the compiler easy to write, but it requires conversions from and to object form of primitive data types.

In some cases, however, type information is available for parameters. This type information could be used to optimize code paths. For example, if we know an argument is an integer, and is only ever used as an integer, there's no need to convert it to org.python.types.Integer and back again just to perform a simple mathematical operation; the integer Java opcodes could be used directly on the argument.

This should result in a significant performance improvement for VOC-compiled code.

Update python version requirement to 3.5+

Please update the python version requirement to 3.5+ instead of 3.4+.
It took me a lot of time to figure this out, since I had Python 3.5.2 installed on my system and still, while installing VOC, I got error message saying:
VOC requires Python 3.4+
This was resolved once I changed this in the setup.py file.

Complete implementation of methods on standard types

The builtin types of Python have a number of methods associated with them - for example, dict objects have update(), items(), keys(), pop() and so on. These methods need to be implemented in Java. In many cases, there will be a simple mapping to an analogous function in the underlying Java type; however, sometimes you'll need to write an implementation using the available Java primitives.

Tests are are also required; they should go in the module corresponding to the data type in the tests/datatypes directory.

Tests are based on comparing the output of some sample Python code when run through CPython to the output when run through VOC. When the two outputs are identical - and I mean byte-identical, down to the text and punctuation of the error message if appropriate - the test passes.

If you want an example to follow, look at the implementation of clear() on dict:

  • The @org.python.Method annotation is used to expose the clear() method as a symbol that should be available at runtime.
  • All arguments are exposed as org.python.Object, and then cast to the right data type as required.
  • Finally, the method returns a Python None.

Custom exception raising.

It is now possible to raise only exceptions from package org.python.exceptions. So for raise custom python exception there are 2 option:

  1. Rebuild java-python-support.jar with your exception.
  2. Write on java exception proxy-class.

Impossible to install via pip - no links on PyPI page

When trying to install Voc as suggested in the documentation:

pip install voc

an error is raised:

Collecting voc
  Could not find a version that satisfies the requirement voc (from versions: )
  No matching distribution found for voc

The version of pip is not really the problem (it doesn't work with pip 6.1.1 nor with pip 7.1.2).
Trying to install it with the verbose option leads to the probable root cause:

(voc)p@pc:~/voc$ pip install -v voc
Collecting voc
  Getting page https://pypi.python.org/simple/voc/
  Starting new HTTPS connection (1): pypi.python.org
  "GET /simple/voc/ HTTP/1.1" 200 111
  1 location(s) to search for versions of voc:
  * https://pypi.python.org/simple/voc/
  Getting page https://pypi.python.org/simple/voc/
  Analyzing links from page https://pypi.python.org/simple/voc/
  Could not find a version that satisfies the requirement voc (from versions: )
Cleaning up...
No matching distribution found for voc
Exception information:
Traceback (most recent call last):
  File "/home/p/.virtualenvs/voc/local/lib/python2.7/site-packages/pip/basecommand.py", line 211, in main
    status = self.run(options, args)
  File "/home/p/.virtualenvs/voc/local/lib/python2.7/site-packages/pip/commands/install.py", line 294, in run
    requirement_set.prepare_files(finder)
  File "/home/p/.virtualenvs/voc/local/lib/python2.7/site-packages/pip/req/req_set.py", line 334, in prepare_files
    functools.partial(self._prepare_file, finder))
  File "/home/p/.virtualenvs/voc/local/lib/python2.7/site-packages/pip/req/req_set.py", line 321, in _walk_req_to_install
    more_reqs = handler(req_to_install)
  File "/home/p/.virtualenvs/voc/local/lib/python2.7/site-packages/pip/req/req_set.py", line 461, in _prepare_file
    req_to_install.populate_link(finder, self.upgrade)
  File "/home/p/.virtualenvs/voc/local/lib/python2.7/site-packages/pip/req/req_install.py", line 250, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/home/p/.virtualenvs/voc/local/lib/python2.7/site-packages/pip/index.py", line 571, in find_requirement
    'No matching distribution found for %s' % req
DistributionNotFound: No matching distribution found for voc

The root cause is probably the page https://pypi.python.org/simple/voc/, being it completely blank.

Install issue on Ubuntu 16.04 libpython3.4-testsuite

Hi,
I'm in the process of installing the dev environment for VOC and when I tried to install 3.4 and got the following error

Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package libpython3.4-testsuite
E: Couldn't find any package by glob 'libpython3.4-testsuite'
E: Couldn't find any package by regex 'libpython3.4-testsuite'

I had to run the following command to fix it:

sudo apt-get install libpython3.5-testsuite

Args not defined for inbuilt methods

For most of the methods, args have not been defined, which leads to the following problem.

5*6 is executed succesfully
However, 5.mul(6) throws an error. For every method, the args need to be defined.

The stack trace for:

# test.py
a = 4
print(a.__mul__(5))

Comes out to be:

Exception in thread "main" java.lang.IllegalArgumentException: wrong number of arguments
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.python.types.Function.invoke(Function.java:275)
    at org.python.types.Method.invoke(Method.java:44)
    at python.test.__init__.module$import(test.py:2)
    at python.test.__init__.main(test.py)

The operators are checked, however the underlying methods throw an error.
This arose in a conversion with @cflee
He suggested to bridge this gap by modifying the tests/utils.py to test the math operators.

Complete implementation of operations on standard types

Java has a specific set of allowed operator and operations. Python has a set of operators and operations as well. However, the two don't match exactly. For example, Java will allow any object to be added to a java.lang.String; Python does not. On the other hand, Python allows multiplication of str by int (producing a duplicated string); Java does not.

In order to replicate Python behavior in Java, Python's logic for all the basic operations needs to be implemented in the VOC Java support library.

The test suite contains around 2800 tests in the that are currently marked as "expected failures". These reflect an attempt to do "every operation between every base data type". The task: pick a data type, and write the implementation of one of the dunder operation (e.g., __add__ or __xor__) or comparison (__lt__ or __eq__) methods.

The tests will pass (and become unexpected successes) when the output of Python code doing that operation is the same when run through CPython and VOC - and I mean byte-identical, down to the text and punctuation of the error message if appropriate.

If you want to see an example of what is involved, check out the implementation of add for integers. If you have an int, Python will allows you to add an int or a float to it; all other types raise a TypeError. The list of operations on int that still need to be implemented can be found here; if you add a new operation, delete the line that corresponds to that test, and the test will report as a pass, rather than an expected fail.

Use Array instead of List in Tuple type

Current org.python.types.Tuple uses a List<Object> as its internal implementation; takes a List as its argument constructor and stores a reference to it.

This is sub-optimal, because a Python tuple is immutable. An Array would avoid dragging in all the resize mechanisms.

But further, it is incorrect, because a Python tuple should not change when the value used to construct it changes.

Add Stack Maps to generated Java classfiles

Java 7 added the requirement for a Stack Map to track usage of the stack during operation. The Java 7 JVM validates this Stack Map at startup.

voc doesn't currently generate Stack Maps, and as a result, it is necessary to use -XX:-UseSplitVerifier argument to the JVM.

I believe there is sufficient information in the data model to generate Stack Maps - it's just matter of doing the work.

most tests fail on Windows7

  1. JVM doesn't start:
    File "voc\tests\utils.py", line 500, in runAsJava
    self.jvm.stdin.flush()
    OSError: [Errno 22] Invalid argument
  2. File write failed:
    File "env\lib\encodings\cp1251.py", line 19, in encode
    UnicodeEncodeError: 'charmap' codec can't encode character '\xff' in position 678: character maps to

async/await feature request

python3.5 async/await feature will be a good feature to see split the issue #245 seperate.
clojurescript like csp feature is another way to solve instead of callbacks

Use a single Java session for tests

The current test runner validates Java output by starting a fresh VM for every test. However, JVM warmup is time-expensive operation.

Ideally, we'd be able to use a single JVM instance, and run each test in that instance, rather than restarting the JVM each time.

If necessary, this could be done on a per-TestCase or per test Module level.

Complete implementation of Python builtins

org.Python is a Java implementation of all the Python builtins - methods like print(), input(), list(), abs(), min() and so on.

An implementation of these builtins is required.

for/else is not supported

After some experimenting, I ran into a case where the transpiler gives an error.

for k in []:
    pass
else:
    print("nobreak")

This is as far as I could get to a minimal (non-)working example. Replacing the print with pass makes it get optimised out by Python already, so the transpiler works.

AttributeError: 'module' object has no attribute 'test_codecs'

tansell@tansell-x1c-l:~/foss/pybee/voc$ python3 setup.py test
running test
running egg_info
writing dependency_links to voc.egg-info/dependency_links.txt
writing top-level names to voc.egg-info/top_level.txt
writing entry points to voc.egg-info/entry_points.txt
writing voc.egg-info/PKG-INFO
reading manifest file 'voc.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'Makefile'
warning: no files found matching 'requirements*.txt'
writing manifest file 'voc.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
  File "setup.py", line 44, in <module>
    test_suite='tests'
  File "/usr/lib/python3.4/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.4/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/lib/python3/dist-packages/setuptools/command/test.py", line 135, in run
    self.with_project_on_sys_path(self.run_tests)
  File "/usr/lib/python3/dist-packages/setuptools/command/test.py", line 116, in with_project_on_sys_path
    func()
  File "/usr/lib/python3/dist-packages/setuptools/command/test.py", line 160, in run_tests
    testLoader = cks
  File "/usr/lib/python3.4/unittest/main.py", line 92, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python3.4/unittest/main.py", line 139, in parseArgs
    self.createTests()
  File "/usr/lib/python3.4/unittest/main.py", line 146, in createTests
    self.module)
  File "/usr/lib/python3.4/unittest/loader.py", line 146, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib/python3.4/unittest/loader.py", line 146, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib/python3.4/unittest/loader.py", line 117, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/usr/lib/python3/dist-packages/setuptools/command/test.py", line 36, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/usr/lib/python3.4/unittest/loader.py", line 114, in loadTestsFromName
    parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_codecs'
tansell@tansell-x1c-l:~/foss/pybee/voc$ python3 --version
Python 3.4.3

Improve Java test runner

Currently, the improved Java test runner (refs: issue #13, PR #98) uses a single JVM that runs the python.testdaemon.TestDaemon class, to execute the various transpiled classes in tests/utils.py's runAsJava() and assertJavaExecution(). Class names are fed in via stdin, and the resulting output is retrieved from stdout. For each test, the main Thread's ContextClassLoader is set to a new JoinClassLoader instance that routes org.python.* classes to a URLClassLoader for python-java.jar, and all other classes to a URLClassLoader for the temp/ and java/ directories.

However, there are some issues with this implementation:

  • sys.argv contains the parameters used to invoke the JVM, i.e. the python.testdaemon.TestDaemon class, and not the args that are passed together with the class name on stdin. The current workaround is to fallback to the 'old' Java test runner that starts up a new JVM instance for each test that has non-empty args parameter. This will be increasingly non-ideal if more tests supply args.
  • The runtime parts of voc now have to use java.lang.Thread.currentThread().getContextClassLoader().loadClass() instead of the more common Class.forName(), in order to use the appropriate ClassLoader to load the correct Class. Specifically, it needs to use the current test's instance of JoinClassLoader, instead of the ClassLoader that was used to create the calling class.
    • This is seen in org.python.ImportLib, org.python.types.Type, org.python.java.Type, and org.python.java.Module.
    • There are some subtleties involved in dynamic loading beyond merely classloading. For example, it was observed that a org.python.java.Type object cannot be cast to its supertype org.python.types.Type when the objects/classes are loaded from different ClassLoaders! This is because class identity in the JVM is defined by both the fully qualified class name as well as the class loader.

The easiest way is to entirely switch off the caching that is being done on the classes from python-java.jar, but that makes the test runtime multiply by ~3.5 (250% increase). It would be best if there's some kind of middle ground where there is a small or no increase in runtime, while still being able to switch back to using Class.forName() in the main codebase.

Possible approaches I can think of currently include:

  • Tweaking the current JoinClassLoader approach to adjust what classes are cached and which URLClassLoader is to load what classes
  • Implementing a full ClassLoader to replace URLClassLoader, so that a single ClassLoader can be used in a way that sidesteps the classes-can-only-be-loaded-once behaviour without upsetting the JVM

For more details about the concerns, see this comment on #98 by @freakboy3742.

Type annotations to improve performance

Can type annotations be optionally used to improve performance if enforced at runtime, or the program is strictly analyzed at compile time?

Either for this or any of the other projects.

Fix incorrect handling of __main__

Given this Python code:

print('A')

if __name__ == '__main__':
    print('B')

print('C')

CPython prints ABC, while VOC prints ACB. This causes at least two tests to fail. [1]

This arises from how VOC is transforming the Python if __name__ == "__main__": statement. B is placed into the Java main() method, while A and C are placed into module$import().

At runtime, the JVM starts with the main() method, which first invokes module$import(), then proceeds to B. This is incorrect behaviour.

One possible fix is to move the children of if __name__ == "__main__": into something like module$main(), and change main() to invoke module$import() with a flag main=true. The latter then contains a conditional that invokes module$main() at the spot where is_mainline_def() is found.

[1] tests.structures.test_function.FunctionTests.test_mainline and tests.stdlib.test_sys.SysModuleTests.test_exit

Issues when trying to run tests on Windows

(This is on Windows 8.1; I imagine similar issues appear on other Windows systems)

I've had a try at running the tests under Windows, the main failure that I run into seems to be related to the default encoding under Windows (cp1252 rather than utf-8):

E AssertionError: 'utf-8' codec can't decode byte 0xff in position 10: invalid start byte

This seems to be an issue on both the Java and Python versions, see attached voc_output.txt, which is a dump of the main_code for both runAsPython() and runAsJava()

Another, possibly related issue is that Windows has a "charmap" encoding in it's terminals (cmd and powershell), which doesn't handle some unicode characters. I've also attached
unicode_test.py.txt which blows up with a similar error.

In the process of debugging, I've also found a couple of places where utf-8 encoding seems to have been missed - pull request here: #236

SyntaxError: Non-ASCII character '\xc3' in file /home/tansell/foss/pybee/voc/tests/utils.py on line 628, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details

tansell@tansell-x1c-l:~/foss/pybee/voc$ python setup.py test
running test
running egg_info
creating voc.egg-info
writing voc.egg-info/PKG-INFO
writing top-level names to voc.egg-info/top_level.txt
writing dependency_links to voc.egg-info/dependency_links.txt
writing entry points to voc.egg-info/entry_points.txt
writing manifest file 'voc.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'Makefile'
warning: no files found matching 'requirements*.txt'
writing manifest file 'voc.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
  File "setup.py", line 44, in <module>
    test_suite='tests'
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 135, in run
    self.with_project_on_sys_path(self.run_tests)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 116, in with_project_on_sys_path
    func()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 160, in run_tests
    testLoader = cks
  File "/usr/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python2.7/unittest/main.py", line 149, in parseArgs
    self.createTests()
  File "/usr/lib/python2.7/unittest/main.py", line 158, in createTests
    self.module)
  File "/usr/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib/python2.7/unittest/loader.py", line 103, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 36, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/usr/lib/python2.7/unittest/loader.py", line 103, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 36, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/usr/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName
    module = __import__('.'.join(parts_copy))
  File "/home/tansell/foss/pybee/voc/tests/structures/test_comparisons.py", line 1, in <module>
    from ..utils import TranspileTestCase
  File "/home/tansell/foss/pybee/voc/tests/utils.py", line 628
SyntaxError: Non-ASCII character '\xc3' in file /home/tansell/foss/pybee/voc/tests/utils.py on line 628, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details

AttributeError when using self.__class__

It looks like the common idiom self.__class__ doesn't work with VOC classes.

Steps to reproduce:

Create a file testclass.py with the following code:

class Hello:
    def hello(self):
        print('hello, from class %s' % self.__class__.__name__)

Hello().hello()

Expected behavior:

$ python testclass.py 
hello, from class Hello

However, this is what you get using VOC:

$ voc -v testclass.py
Compiling testclass.py ...
Adding default main method...
Writing ./python/testclass/__init__.class ...
Writing ./python/testclass/Hello.class ...
$ java -cp ../voc/dist/python-java-support.jar:. python.testclass.__init__
Exception in thread "main" AttributeError: 'Hello' object has no attribute '__class__'
	at org.python.types.Object.__getattribute__(Object.java:288)
	at python.testclass.Hello.hello(testclass.py:3)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.python.types.Function.invoke(Function.java:391)
	at org.python.types.Method.invoke(Method.java:48)
	at python.testclass.__init__.module$import(testclass.py:5)
	at python.testclass.__init__.main(testclass.py)

Environment details:

$ javac -version
javac 1.8.0_121
$ python --version
Python 3.4.4

try/except doesn't create variable for named exceptions

try:
    print('success', next(iter([])))
except Exception as e:
    print(e.__class__.__name__)
(env) eatkin@phobos:~/progs/voc/sandbox$ python -m voc -v named_exception_fail.py && java -classpath ../dist/python-java.jar:. python.named_exception_fail.__init__
Compiling named_exception_fail.py ...
Adding default main method...
Writing ./python/named_exception_fail/__init__.class ...
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: org.python.exceptions.NameError: name 'e' is not defined
        at org.python.types.Module.__getattribute__(Module.java:29)
        at python.named_exception_fail.__init__.<clinit>(named_exception_fail.py:4)
(env) eatkin@phobos:~/progs/voc/sandbox$

Integrate Ouroboros into the build

pybee/Ouroboros provides a pure Python implementation of the Python standard library. That Python code can be compiled directly into Java classifies and converted into a standard library that can ship as part of the VOC support libraries, supplemented by native implementations as required (see #74).

Error in test suite when running under Python 3.4.4

Via Kesha Shah:
The CI server runs under Python 3.4.2, and is (usually) in a fully passing state.

However, if you run the test suite under Python 3.4.4, you will get 3 test failures:

======================================================================
FAIL: test_list (tests.builtins.test_int.BuiltinIntFunctionTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/keshashah/VOC/voc-dev/voc/tests/utils.py", line 641, in func
    self.assertBuiltinFunction(x=example, f=function, operation=operation, format=self.format)
  File "/Users/keshashah/VOC/voc-dev/voc/tests/utils.py", line 661, in assertBuiltinFunction
    """ % kwargs, "Error running %(operation)s with f=%(f)s, x=%(x)s" % kwargs)
  File "/Users/keshashah/VOC/voc-dev/voc/tests/utils.py", line 373, in assertCodeExecution
    self.assertEqual(java_out, py_out, context)
AssertionError: "### [52 chars]tring or a number, not 'list'\n    test.py:3\n" != "### [52 chars]tring, a bytes-like object or a number, not 'l[17 chars]:3\n"
  ### EXCEPTION ###
- TypeError: int() argument must be a string or a number, not 'list'
+ TypeError: int() argument must be a string, a bytes-like object or a number, not 'list'
?                                           +++++++++++++++++++++
      test.py:3
 : Global context: Error running f(x) with f=int, x=[]

======================================================================
FAIL: test_none (tests.builtins.test_int.BuiltinIntFunctionTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/keshashah/VOC/voc-dev/voc/tests/utils.py", line 641, in func
    self.assertBuiltinFunction(x=example, f=function, operation=operation, format=self.format)
  File "/Users/keshashah/VOC/voc-dev/voc/tests/utils.py", line 661, in assertBuiltinFunction
    """ % kwargs, "Error running %(operation)s with f=%(f)s, x=%(x)s" % kwargs)
  File "/Users/keshashah/VOC/voc-dev/voc/tests/utils.py", line 373, in assertCodeExecution
    self.assertEqual(java_out, py_out, context)
AssertionError: "### [52 chars]tring or a number, not 'NoneType'\n    test.py:3\n" != "### [52 chars]tring, a bytes-like object or a number, not 'N[21 chars]:3\n"
  ### EXCEPTION ###
- TypeError: int() argument must be a string or a number, not 'NoneType'
+ TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
?                                           +++++++++++++++++++++
      test.py:3
 : Global context: Error running f(x) with f=int, x=None

======================================================================
FAIL: test_tuple (tests.builtins.test_int.BuiltinIntFunctionTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/keshashah/VOC/voc-dev/voc/tests/utils.py", line 641, in func
    self.assertBuiltinFunction(x=example, f=function, operation=operation, format=self.format)
  File "/Users/keshashah/VOC/voc-dev/voc/tests/utils.py", line 661, in assertBuiltinFunction
    """ % kwargs, "Error running %(operation)s with f=%(f)s, x=%(x)s" % kwargs)
  File "/Users/keshashah/VOC/voc-dev/voc/tests/utils.py", line 373, in assertCodeExecution
    self.assertEqual(java_out, py_out, context)
AssertionError: "### [52 chars]tring or a number, not 'tuple'\n    test.py:3\n" != "### [52 chars]tring, a bytes-like object or a number, not 't[18 chars]:3\n"
  ### EXCEPTION ###
- TypeError: int() argument must be a string or a number, not 'tuple'
+ TypeError: int() argument must be a string, a bytes-like object or a number, not 'tuple'
?                                           +++++++++++++++++++++
      test.py:3
 : Global context: Error running f(x) with f=int, x=(1, 2.3456, 'another')

----------------------------------------------------------------------

It appears that Python 3.4.4 changed the error message raised by int() when invalid input is provided. Under 3.4.2, you get:

>>> int(list())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: int() argument must be a string or a number, not ‘list'

However under Python 3.4.4, you get:

>>> int(list())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: int() argument must be a string, bytes-like object or a number, not ‘list’

VOC should standardise on the 3.4.4 message format (since that's the format that will exist into the future); the cleanse_python() method in tests/utils.py should normalize the error string from 3.4.2 format to 3.4.4 format if it is observed in Python test output.

Python exceptions cant be raised.

I have a problem with exception raises.

1.Cant raise exception by class name.

test.py
---------------------
raise ValueError

  File "/home/user/Projects/voc/voc/python/ast.py", line 489, in visit_Raise
    exception = 'org/python/exceptions/%s' % node.exc.func.id
AttributeError: 'Name' object has no attribute 'func'

2.Cant raise instance of exception without arguments

test2.py
---------------------
raise ValueError()

   File "/home/inekipelov/PosterProjects/voc/voc/java/constants.py", line 391, in __init__
    self.descriptor = method_descriptor(descriptor)
  File "/home/inekipelov/PosterProjects/voc/voc/java/signatures.py", line 179, in method_descriptor
    return MethodDescriptor(parameters=matches[:-1], return_type=matches[-1])
IndexError: list index out of range

try/except inside for loop doesn't initialize jump_offset in find_try_except

for x in range(2):
    try:
        print('success', next(iter([])))
    except Exception:
        print('whoops')
(env) eatkin@phobos:~/progs/voc/sandbox$ python -m voc -v try_within_for_fail.py && java -classpath ../dist/python-java.jar:. python.try_within_for_fail.__init__
Compiling try_within_for_fail.py ...
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/eatkin/progs/voc/voc/__main__.py", line 49, in <module>
    main()
  File "/home/eatkin/progs/voc/voc/__main__.py", line 45, in main
    verbosity=args.verbosity
  File "/home/eatkin/progs/voc/voc/transpiler.py", line 18, in transpile
    transpiler.transpile(source_file, prefix)
  File "/home/eatkin/progs/voc/voc/transpiler.py", line 78, in transpile
    self.transpile_code(os.path.abspath(filename)[len(common):], code)
  File "/home/eatkin/progs/voc/voc/transpiler.py", line 91, in transpile_code
    module.extract(code)
  File "/home/eatkin/progs/voc/voc/python/blocks.py", line 77, in extract
    blocks = find_blocks(instructions)
  File "/home/eatkin/progs/voc/voc/python/utils.py", line 785, in find_blocks
    i, block = find_try_except(offset_index, instructions, i)
  File "/home/eatkin/progs/voc/voc/python/utils.py", line 751, in find_try_except
    if jump_offset > end_block_offset:
UnboundLocalError: local variable 'jump_offset' referenced before assignment
(env) eatkin@phobos:~/progs/voc/sandbox$

Links to other documents not working

If we look at the Getting Started document, towards the end we have what looks like a broken link: :doc:start the first tutorial </tutorials/tutorial-0>. I tried to modify it to GitHub relative link format as given here but couldn't get it right. How are these to be fixed?

Adding a linter to travis

A linter needs to be added to travis, one that if possible, reports non-compliance the same way that a test failure reports - "tests passed", but "Python lint failed on file:<>"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.