Code Monkey home page Code Monkey logo

runner's Introduction

codewars/runner

Issue tracker for CodeRunner used on Codewars and Qualified.

Reporting Issues

Please report any language related issues here.
For anything else about Codewars, please use codewars/codewars.com.

Feature Requests

Please open new issues using appropriate issue templates.

Code

CodeRunner itself is not open source, but it's roughly equivalent to the following:

$ WORKDIR=/workspace/
# Create a container
$ C=$(docker container create --rm -w $WORKDIR language-image cmd args)
# Copy files
$ files | preprocess | docker container cp - $C:$WORKDIR
# Run
$ docker container start --attach $C | postprocess
  • WORKDIR is not always /workspace/ and can be a subdirectory of it (planned to be standardized in the future)
  • preprocess is responsible for the file layout and any code modifications necessary for backwards compatibility (e.g., concatenation)
  • postprocess transforms the output when necessary (e.g., Codewars test output from JSON)

Images can be used with local files by changing the step to copy the files:

$ docker container cp ./files/. $C:$WORKDIR
#                            ^^
#                            copy contents and not itself

Container images are available on DockerHub under qualified.

Contributing

Contributions are welcomed!

For now, please look at the following places:

Feel free to open issues to ask us if you'd like to contribute in other ways.

Supported Languages

Stable

Beta

Requested

See issues with language request label.

runner's People

Contributors

kacarott avatar kazk avatar pabloandermann avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

runner's Issues

Missing Documentation for C environment?

I was just planning to author a C translation for one of my Kata involving the Preloaded section and received an error stating that a struct type I defined in the Preloaded section could not be recognized. I figured that it was probably because I had to import the filename associated with the Preloaded section to use it (in the Solution and Test Cases) so I searched documentation/environments to look for how the runner for the C language is structured on CW. To my surprise, I looked at every single Markdown file and could not find one corresponding to C. Perhaps it was accidentally left out?

Add Scheme

Add Scheme support. Need more information like which implementation and test framework.

Good day, Codewars!

It's been a while since I've been on here, and I am very happy to see new features and languages being actively added here on Codewars. Thank you for all your immense hard work!

If I may, I would like to request support for the Scheme programming language. Scheme belongs to the family of LISP languages such as, for example, Clojure and Common LISP, but possesses its own unique identity.

Being mostly "barebones" in terms of "built-ins", with a specification of not more than 90 pages, yet powerful, Scheme can potentially become a perfect language for katas from 8 kyu to 1, and all sorts of programming challenges.

Kind Regards,
lilsweetcaligula

https://www.codewars.com/topics/5235fedb397cbfd6aa0000ea#59da4ed02a7acc5c660000be


  • Installing
    • Choose which implementation to support
  • Executing
  • Testing
  • Editor. CodeMirror scheme mode
  • Icon

Please enable SQL placeholders

Looking at sequel it seems it's possible to run something like

DB['select * from items where name = ?', name]

so I think you could modify your run_sql in order to accept placeholders.

Benefits:

  • you would be able to send different parameters for testing
  • you can easily translate practically all existing function Kata's to SQL without using functions i.e. you'll have ANSI SQL solutions and you could try the same Kata for all SQL vendors supported

Sample cases:

Echo function:

DB['select ? as echo', any]

Hello name function:

DB['select 'Hello ' || ? || '!' as greeting', name]

Add Vala

It would be great if you will add programming language Vala.
Thank you!

Define language support levels

As the list of languages grow, we should officially define a tiered system of support for languages. Once formalized these languages should be documented so that its clear to all where things stand.

Tier 1 languages, such as JS, Python, Ruby are well supported by the core team and support things like frameworks, packages, multiple versions, multiple testing frameworks, project mode, etc.

2nd tier languages are those that are well supported but are missing some features, that we would like to add so that they can eventually become tier 1. These include Java and C#.

3rd tier languages are those languages that Qualified/Codewars supports as a baseline but are not likely to invest much of its own resources into maintaining.

4th tier languages are basically those that are supported solely through community contributions, with Codewars only supporting in terms of the most minimal effort needed to integrate into the site.

Add Haskell GHC 8.8.x (LTS 16.x) with `-O1`

Name: Haskell
Version: Haskell 8.2.2 (LTS 10.4) -> Haskell 8.6.3 (LTS 13.4)
Release Notes

IMO, the biggest change is in the base package in version 4.11, which includes the enhanced Semigroup and Monoid along with their exposure to Prelude and improved instances of Maybe. Plus, albeit not documented, liftA2 is now in Prelude.

While I don't think updating the GHC and base will break anything, switching the Stackage LTS version might involve a few big changes in other bundled packages.

Add JavaFX

Related to codewars/codewars-runner-cli#595.

You can install it trough OpenJFX If you are on a Debian based Linux distro you can run something like

sudo apt-get install openjfx

It works like a charm with OpenJDK.

It has a huge fanbase and even if elder Java programmers don't like it it's very powerful to do reactive programming due to its observables and thread handling.

R: non-ASCII characters in test name are not printed properly

Related: #67

test_that("two_oldest_ages ยต", {
  expect_equal(two_oldest_ages(c(1,5,87,45,8,8)), c(45,87))
  expect_equal(two_oldest_ages(c(6,5,83,5,3,18)), c(18,83))
})

gives

image

Note: I'm not sure if the issue comes from CW formatter or the test framework itself.

Handle early return in JS

The current setup for JS runner is:

function () {
  // preloaded
  ;
  // user code
  ;(function() {
    var Test = global.Test, describe = global.describe, it = global.it, before = global.before, after = global.after;
    // test code
    ;
  })();
}

A top-level return will cause everything after it to stop executing. It is because the code is not actually run at top level.

Thinking of code as running in top-level is not a good practice anyway, since modules in Node.js are not considered at top level, and even top-level awaits are currently at proposal stage 2. So the entire test code should be conceived as this:

(async () => {
  // test fixture
})().catch((err) => {
  // handle unhandled errors thrown from test fixture
  console.log(err);
});

Then the obvious way to ensure that text fixture does not early return (aka run to its end) is to do this:

var nonceKey = 'nonceVal';
(async () => {
  await (async function() {
    // start of user code
    function multiply(a,b) {
      return a*b;
    }
    // end of user code
    (function tests() {
      // start of test code
      Test.assertEquals(multiply(3,5),15);
      // end of test code
    })();
  })();
  return nonceKey;
})().then((val) => {
  if(val !== nonceKey) throw new Error('text fixture did not run to end');
  // maybe also do other stuff
}).catch((err) => {
  // handle unhandled errors thrown from test fixture
  console.log(err);
  throw err;
});

(Test fixture is wrapped in another async function so arguments.callee.toString() will not reveal the nonce.)

Obviously, nonceKey and nonceVal should be randomly generated every time so it's impossible to crack it.

Add Octave support

I know a lot of engineers who only know these languages, so they might be happy to be able to at least do kumite in them. Eventually, we'll have to have test frameworks.


๐Ÿ‘ reaction might help.

Add libraries for Data Driven Testing

Feature

Add libraries for data driven testing e. g. ddt in Python.

Rationale

  1. More diversified testing environment supporting multiple types/styles of testing.
  2. IMO: Simplifying kata craeation and translation: One can create/convert a data structure without writing/copying a lot of boilerplate code.

OCaml compiler error has incorrect line numbers due to concatenation

It's pretty much in the title. The line number reported was off by one when compared to the sample code in the editor window. This makes it very hard to debug.

Also, the ocaml compiler on its own will show you the offending line with errors underlined, which is not available in the web interface either. That would be a very helpful improvement.

Finally, providing automatic indentation of OCaml code using the ocp-indent tool would help a lot.

Add library for checking imported/hidden modules for PureScript

From codewars/codewars-runner-cli#691 :

Idk if this is a bit of a stretch but would it be reasonable for me to request something similar to Test.Codewars.BlackList in PureScript? Currently, quite a few Haskell kata aren't suitable for translation into PureScript (e.g. "The Five Fundamental Monads", "foldMap all the things") only because they require certain built-ins to be hidden and AFAIK there is no module/library in PureScript for writing such tests. Or are such Kata (requiring certain built-ins to be hidden) highly discouraged and only kept as legacy content?

Replace C++ Igloo with a better testing framework

I'm looking at Igloo and there are lots of problems with it:

  • Very outdated (last commit is 3 years ago) and definitely out of support
  • Very poor documentation: no docs at all except an example at official repo, the official domain for the project is expired, what's remaining is an (official?) wordpress site, but that doesn't show much at all either
  • Outdated in terms of C++ version: CW is using C++14, and Igloo isn't even aiming for C++11 (so no lambdas and such). In fact the recommendations for Igloo I've seen are... mostly back in 2010
  • Section names are forced to be a valid token in the language. Ideally section names should be arbitrarily, like a string

I'm looking at a wide range of C++ test framework and it seems that Catch2 is a nice replacement. Header-only, light-weight, does the same thing, supports strings as section names, support predicates and such. RapidCheck, a C++ QuickCheck clone, also uses Catch within itself.

googletest and boost.org test are two that are actively maintained and fairly popular too. But I really want want arbitrary section names, so I personally likes Catch2 more.

C# custom TestFixture/Test description does not handle surrogate pairs properly

A simple test fixture:

using NUnit.Framework;
[TestFixture(Description = "๐Ÿ‘๐Ÿ‘๐Ÿ‘๐Ÿ‘๐ ˆ“๐ ˆ“๐ ˆ“๐ ˆ“")]
public class SolutionTest
{
  [Test, Description("\U0001F01C\U0001F01C\U0001F01C\U0001F01C\U0001F604\U0001F604\U0001F604\U0001F604")]
  public void MyTest()
  {
    Assert.AreEqual("expected", "actual");
  }
}

results in this:

image

Surrogate pairs are shown incorrectly.

Note: I'm not sure if the issue comes from CW formatter or the test framework itself.

Add pytest as a testing framework for Python

Feature

pytest test framework support for Python 2 and 3.

Rationale

Issues with the current framework(s)

The currently available test frameworks involve a lot of boilerplate. This is true first and foremost about the most widely used test.assert_equals.

The code using this test framework has very high noise to signal ratio and it's hard to parse visually as test.assert_equals isn't syntax-highlighted and there's no significantly visible separation (it's a single comma) between actual and expected value.* Also, this framework is custom and can't be used outside of CW.

I'll use the following example (taken from this discussion) in which 37.5% is boilerplate, to support my point

# Example:

test.assert_equals(changed((0, 0, 1, 1)), 2)
test.assert_equals(changed((10, 10, 10, 10, 99, 99)), 4)
test.assert_equals(changed(([], [], [], [], (), ())), 4)
test.assert_equals(changed(("this", "that")), 1)
test.assert_equals(changed(()), None)
test.assert_equals(changed((1,)), None)
test.assert_equals(changed((1, 1, 1, 1, 1)), None)

# Analysis

# test.assert_equals(changed((1, 1, 1, 1, 1)), None)
# \----------------/
#         noise

code_length = len(of_your_code_including_line_breaks) == 336
test_clutter = len(all_of_test_assert_equals_parens_excluded) == 126
test_clutter/code_length == 0.375 # (i.e. 37.5% is noise) 

A comparison with the same tests written in Groovy using junit demonstrate how cluttered this Python code is. The two main differences I want to point out:

  1. The code isn't that much shorter but it is easier to parse visually as assert is highlighted and there's a highlighted separation (==) between actual and expected values (the highlighting on CW is even better in this respect).
  2. The same code can be used outside CW.
@Test
void "Example Tests"() {
    assert changed([0, 0, 1, 1]) == 2
    assert changed([10, 10, 10, 10, 99, 99]) == 4
    assert changed([[], [], [], [], (), ()]) == 4
    assert changed(["this", "that"]) == 1
    assert changed([]) == null
    assert changed([1,]) == null
    assert changed([1, 1, 1, 1, 1]) == null
}

Proposed Solution: pytest

pytest test framework offers user experience similar to Groovy with little clutter and detailed info about the failing tests. As a demonstration: The Groovy code above can be used in pytest with minimal changes and the code (as it is) can be used outside CW too.

def test_examples():
    assert changed([0, 0, 1, 1]) == 2
    assert changed([10, 10, 10, 10, 99, 99]) == 4
    assert changed([[], [], [], [], (), ()]) == 4
    assert changed(["this", "that"]) == 1
    assert changed([]) == None
    assert changed([1,]) == None
    assert changed([1, 1, 1, 1, 1]) == None

Notes

* IMHO the use of this framework can be by no means to be considered as best practice.

Flush stdout

Recently I was playing with some dangerous stuff with Python, and I discovered that if I segfault at the very end of the code, everything I've printed out from the start of the code is all lost, and nothing is shown on the output. So it means stdout is not flushed at all. Then I flush the stdout before segfaulting the interpreter, and the output are shown properly.

If I recall properly, only some languages flush stdout during some point at program execution, and some of them don't even flush till the end of the program. Also timing out doesn't flush stdout, it just completely terminates the program, which means everything printed during the process is lost too.

So there are basically 2 problems:

  1. stdout is not flushed by default. Typical setting for stdout flushing should be by every line I think?, and it's easy to set that as default in most programming languages.
  2. additional flushing should be done at the beginning and end of a test block (both describe and it), and every test assertion. (I think this is already done for some languages.)

Test assertion for *passing* a test if it reaches a certain time limit

There are currently Test.assertNotEquals, Test.assertNotContains, Test.expectNoError... but there are no controls for the timeout. We can only make tests that fail on timeout, not succeed on timeout.

Use case: When there are 2 versions of a kata and one is the performance version, we'd like to block solutions that can pass the performance version from the easy version, so others can't just look for these solutions in the solution page and solve the performance version for free. @myjinxin2015's Coding 5min/Shortest Code series used to have this feature: if your code is short enough for the challenge version, submitting this code on the non-challenge version will cause tests to fail with the error message telling you to go submit your code in the challenge version instead.

Speaking of which, this also needs codewars/python-test-framework#4.

Include documentation on how to prevent monkey patching for a given language

This is for Ruby but I'm guessing it also applies to Python/Javascript. I've noticed a user overwriting comparison methods(and methods related to the return type of the method) to bypass test cases.
I tested this myself and was able to pass the few katas I tried simply by monkey patching Object or NilClass methods. For some katas it's as simple as

class Object
  def ==(o)
    true
  end
end

or

class Object
  def !=(o)
    false
  end
end

See cw-2.rb#L112, related kumite
Maybe core classes should be frozen like the Test class is, though that could interfere with katas that rely on the user monkey patching. A better solution might be to use untouched/unmodified comparison methods.

Add tensorflow

I was trying to run some tensorflow code on CW, which then I found that tensorflow is somehow not installed anymore. It had always been there before (and documentation says so too), so I believe it's due to the new runner.

Add support for pydash for python runner

JS runner has lodash, I believe it's unfair.

Lo-/pydash lets you write more functional code and spend less time writing boilerplate code which is really useful when you're doing high level katas

Add WebAssembly (WebAssembly text format)

Adding wasm language shouldn't be very hard since Node can call wasm binaries trivally, which means the majority of Node environment can be reused.

As far as coding goes wat (WebAssembly text format) is the standard so we can code in wat, and then let it be converted to wasm to be run by Node environment.

Steps:

  1. Install WebAssembly toolkit https://github.com/webassembly/wabt (or https://github.com/mafintosh/webassembly-binary-toolkit for npm)
  2. Setup pipeline to pass solution.wat to wat2wasm to get wasm file
  3. Setup Node script to load compiled wasm file: https://webassembly.github.io/wabt/demo/wat2wasm/

And I think it's done?

Add MongoDB

eg. MongoDB

Currently we have SQL kata's but all databases are relational, learning some non-relational databases might be a great plus.

pgSQL: Non-query syntax errors are dropped sliently

If you play with other pgSQL sandboxes online (e.g SQL fiddle or Rextester), if you declare a function with syntax errors in it they'll be reported properly. However, in CW it's dropped silently.

I can't express how many hours I've post trying to do SQL katas that involves creating a function when what I only get is PG::UndefinedFunction: ERROR: function my_function() does not exist at the select statement when my function declaration is not correct. And because I don't have any error messages about the function itself, I can only take stabs in the dark to try to fix it. This is horrible.

Edit: It seems that all query-related errors are caught properly, but all function-related are dropped silently. Besides function declarations, EXECUTE is also exhibiting this behaviour.

Improve STDERR display

STDERR doesn't have formatting support and this can lead to a huge wall of text that's very difficult to read.

Add Common Lisp

The code runner already supports running SBCL 1.3. Need to update and figure out the testing.

Test Frameworks:



๐Ÿ‘ reaction might help.

BF: interpreter EOF format is not specified

https://esolangs.org/wiki/Brainfuck#Implementation_issues

EOF is a controversial issue. Many implementations return 0, some return -1, and several notable brainfuck programmers suggest that on EOF the memory cell should be left as-is, without modification. In the original distribution the compiler left the cell unchanged, while the interpreter used the EOF from C which, strictly speaking, could be any negative integer, but which usually is -1 (which, in the case of byte-sized cells, ends up as 255).

The current interpreter used for the BF language version uses 0xFF as EOF, which poses a trap for both kata makers and kata solvers:

  • input needs to be zero-terminated manually in the input, or they really aren't terminated at all
  • we cannot assume that reading pass the end of the input will zero a cell, so we need to do [-] instead of , to zero a cell.

At least this should be documented somewhere.

Add Smalltalk


  • Installing (apt get install -y gnu-smalltalk)
  • Executing. Command to compile and run (gst file.st).
  • Testing
    • Research test framework choices (commonly used ones preferred)
    • Implement Codewars output
  • Editor. CodeMirror smalltalk mode
  • Icon

๐Ÿ‘ reaction might help.

Add Perl

Please complete the following information about the language:

  • Name: Perl
  • Website: Perl
  • Language Version: Perl 5.28.x or later

The following are optional, but will help us add the language:


Progress:

  • Run
  • Test
  • Output

๐Ÿ‘ reaction might help.

Cassandra

I understand that a critical piece of big data infrastructure, Cassandra, is in incredibly high demand right now amongst Fortune 500 companies.

We may wish to support this in the future, since support kata for this may be valuable enough to go out of the way to support it.

Add arc4random for Swift with modulemap

Currently working in translation in Swift Kumite Translation
for some reason arc4random and arc4ramdon_uniform are causing unresolved identifier errors when testing the Kata. They seemed to work fine without doing any imports on local machine.

Is Darwin.C.stdlib missing?

Add preloaded code support for NASM

When trying to create a NASM translation, I noticed that none of the code in the Preloaded section is used by the Runner. Even if I type in complete rubbish, it is totally ignored, no compile or link error whatsoever.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.