dmoj / judge-server Goto Github PK

Should be possible to bind a C grader to a JNI interface and provide callbacks. Will require a lot of machine-generated code.

We can set RLIMIT_FSIZE to limit the maximum file size a compiler can possibly create. However, setting this limit in the judge process is not a trivial matter. A clean solution requires a script which sets the limit and execves the compiler, which would require a utility function to spawn compilers as this only works on *nix. The helper might also need to catch SIGXFSZ exit and raise compile error as appropriate.

Needless to say, this has no priority.

Currently, we add some new lines on top of the submission, for optimization, with disastrous results since from __future__ import <something> doesn't work.

We should use an alternative mechanism to make that work.

Would prevent awesome havoc when the system language is set to something like zh_CN. GCC would be more than happy to give translated messages, puzzling everyone.

Note: Herculaneum uses Unicode smart quotes in error messages exactly for this reason.

Don't just silently terminate grading and pretend like things will be fine.

The Java submission:

public class Clazz {
    public static void main(String[] argv) {
    }

    static {
        String x = "x";
        for(int i = 0; i < 1000000; i++) x += x;
    }
}

Will not produce an MLE status code as expected, but rather an IR. The offending line? https://github.com/DMOJ/judge/blob/master/java_executor/src/ca/dmoj/java/SubmissionThread.java#L27

But why? Clearly the OutOfMemoryError is handled. The reason is quite illegit. Let's take a look at the docs for java.lang.reflect.Method#invoke (emphasis mine, cruft removed).

If the underlying method is static, the class that declared the method is initialized if it has not already been initialized.

Throws:

• InvocationTargetException - if the underlying method throws an exception.
• ExceptionInInitializerError - if the initialization provoked by this method fails.

Now, at this time, the class that declared the method is not initialized. So we're not given an InvocationTargetException when the static block runs out of heap space. Easy though, right? Just wrap the invocation to catch ExceptionInInitializerError, and check if the cause is an OutOfMemoryError. No.

In my experience with the Java language and Java documentation, the terms "Exception" and "Error" are used quite interchangeably. InvocationTargetException states that it will be thrown if "if the underlying method throws an exception", but this is not true. It will be thrown if the underlying method throws any Throwable - indeed, the entire Java executor relies on this. It's therefore only logical to assume that ExceptionInInitializerError would act the same. It doesn't.

Unlike what you might expect, it only gets thrown if the initialization throws an Exception. Not a Throwable, as you might believe - an Exception. Hence, "correct" code would wrap the invocation call with a try/catch block for OutOfMemoryError, which, by virtue of being an Error, will not be wrapped in ExceptionInInitializerError.

Thanks, Java.

This rant issue took less time to write than this bug took to identify.

Following the .NET sandbox example, the JVM sandbox should have a similar interface to ease the addition of #37, #38, #39, #41.

When operating on the child's stdin on Windows, and the child ignores all the input, IOError: [Errno 22] Invalid argument is caused when the buffer is flushed. This happens on write or close, and causes exceptions.

Sample tracebacks:

Traceback (most recent call last):
  File "C:\Users\Quantum\Desktop\judge\judge.py", line 281, in _begin_grading
    interactive=('grader' in init_data)):
  File "C:\Users\Quantum\Desktop\judge\judge.py", line 664, in run
    result.proc_output, error = communicate(input_data, outlimit=25165824, errlimit=1048576)
  File "C:\Users\Quantum\Desktop\judge\communicate.py", line 53, in safe_communicate
    proc.stdin.write(input)
IOError: [Errno 22] Invalid argument

Traceback (most recent call last):
  File "C:\Users\Quantum\Desktop\judge\judge.py", line 281, in _begin_grading
    interactive=('grader' in init_data)):
  File "C:\Users\Quantum\Desktop\judge\judge.py", line 664, in run
    result.proc_output, error = communicate(input_data, outlimit=25165824, errlimit=1048576)
  File "C:\Users\Quantum\Desktop\judge\communicate.py", line 63, in safe_communicate
    proc.stdin.close()
IOError: [Errno 22] Invalid argument

Should be possible to export a C grader to Pascal.

Would by highly useful for problemsetters without installing the DMOJ Django site. Could be written as a lightweight Java applet that implements a minimal grading UI.

It would reduce the amount of effort that needs to go into organizing contests, since data would not have to be uploaded for every try. Would also be useful for allowing this judge to be used as a standalone program for private training (there are more stars on this repo than the site repo, which might be indicative that some potential users are interested in private graders).

According to some StackOverflow post, Clang is recommended for students due to "extremely clear diagnostics are definitely easier for beginners to interpret". The idea is rather popular to the point that GCC tries to debunk it as a myth.

It might be worth to add clang as an alternative C/C++ frontend.

1. Add option to only track process creation and exiting PTRACE_O_TRACE*.
2. A SecurePopen-compatible class to log only only these events. Useful for example to track languages with builtin sandbox without performance penalty.
3. Port cptbox to more platforms, most notably linux x32 to use 64-bit powers with only 32-bit pointers.

No reason why it should be fixed into the file, as it makes DMOJ/hephaestus rather painful.

Caused by #22, and the traceback now contains <string>, and python tries to open that as a file and read from it.

A class-based approach is fully modifiable by subclasses instead of making the make_executor function take more and more arguments.

Well you see, normal grader takes string to safe_communicate, checkers also take strings, but these interactive graders, take umm, cStringIO.StringIO???

error: [Errno 10054] An existing connection was forcibly closed by the remote host

Some generators take prohibitively long times to generate testdata, and in a contest scenario it would be beneficial to have a cache of generated data so that subsequent submissions run faster. The cache would likely expire after a few minutes, to free resources.

Generators should automatically stop executing when being aborted, at least stop before spawning the next process to generate the next file. Similar action for compilers would be nice to have.

Same as #19, but for firewall rules.

Importing classes from judge.py for use in interactive graders/custom checkers is nasty and ultimately depends highly on the current judge version, forcing any future changes to the internals to be signature-compatible.

An API could easily resolve this.

Add all the JVM languages!

Is most confusing for newcoming Java users as it is.

When wbox crashes, its generated accounts are never cleaned. They should be cleaned the next time the judge boots.

C:\Users\Tudor>net user

User accounts for \\BERLIN

-------------------------------------------------------------------------------
Administrator            Guest                    Tudor
-----------              wbox0000000000c6afaf     wbox0000000001871be0
wbox00000000018a93b8     wbox000000000191d021     wbox0000000001aa8ccc
wbox0000000001b196a0     wbox000000000ace72e2     wbox000000000b23e734
wbox000000000b296475     wbox000000000b324dc7     wbox000000000ba9c9f5
wbox000000000bb804e8     wbox000000000bbb3434
The command completed successfully.

Would be really nice to be able to measure submission count and average submission judging time for showing on http://status.dmoj.ca/

To speed up IO in Java, the executor uses an ASCII-encoded stream. This has the effect of making Unicode-dependent problems such as http://www.dmoj.ca/problem/denoun2 painful to solve in Java.

The stream encoding should be configurable per-problem.

socket.error: [Errno 10013] An attempt was made to access a socket in a way for idden by its access permissions

After exiting from the issue documented in #5 , Windows judges are not able to connect again.

Because Dart is just so cool.

judge.json is painful because it's JSON according to @Xyene and I don't disagree.

For DMOJ/hephaestus to work, we have to have the option of running all the executors without the sandbox. Ideally this would mean migrating most duplicate executor features into ResourceProxy and renamed BaseExecutor. This BaseExecutor would then be able to expose the basic features as seen fit.

Since the user submission runs in the same process as the judge executor calls to System.exit are blocked such that the user does not exit the grading JVM.

Should be trivial to handle the exit permission request and kill the submission thread instead of throwing an AccessControlException.

The judge servers were written to "chug along", but this has a terrible effect on error messages.

The following traceback is generated when a user misconfigures a problem:

Traceback (most recent call last):

  File "/code/site/judge/bridge/judgehandler.py", line 199, in on_bad_problem
    raise ValueError('\n\n' + packet['message'])

ValueError:

Traceback (most recent call last):
  File "/judge/judge/judge.py", line 185, in _begin_grading
    with open(os.path.join(get_problem_root(problem_id), 'init.json'), 'r') as init_file:
  File "/usr/lib/python2.7/posixpath.py", line 77, in join
    elif path == '' or path.endswith('/'):
AttributeError: 'NoneType' object has no attribute 'endswith'

Intuitive, right? Very helpful to those who are not familiar with the judge's source code.

Hello, I want to try out this repo of judge on Windows ? How to use it ? Any documentation will be helpful. 👍

Since GCC 5.1 released a while ago, we could allow C++14 features to be used on our judges.

Or something among those lines. TCP packets in async node.js are just too painful.

We already have a Java executor.

waterloo first year cs teaches racket exclusively...

Might be best to move the java_executor to its own repository after all these changes.

On powerful computers it may be beneficial to support the running more than one judge without going through the pain of having to copy the judge directory. Simply pass a path to a judge json model as an optional commandline parameter, and default to /data/judge/judge.json for compatibility with judges that are already set up.

A certain online judge doesn't require flushing for interactive problems. A wild guess scientific analysis suggests that they present pty devices for stdin and stdout. This can be arranged on *nix through the use of the pty module's openpty() in place of os.pipe().

For linked list, the checker takes longer than the submission itself to check the data.

Some profile data:

name                                  ncall  tsub      ttot      tavg
..judge/checkers/standard.py:1 check  25     20.60029  39.94269  1.597708

The submission itself takes 7.69s. This is clearly not acceptable. I suggest we start porting checkers to C.

Should this be added in place of Rhino? It ships with Java 8.

@quantum5

The current executor system does not take advantage of inheritance and modularization. This should be changed so that small changes doesn't have to cause all executors to be changed, e.g. #47, #34.

I don't think Haskell "A Plus B" program should take 2.5 seconds...

Also, according to this documentation, -O produced reasonable code at a reasonable speed, while "At the moment, -O2 is unlikely to produce better code than -O."