ragnargrootkoerkamp / bapctools Goto Github PK
View Code? Open in Web Editor NEWTools for developing ICPC-style programming contest problems.
License: GNU General Public License v3.0
Tools for developing ICPC-style programming contest problems.
License: GNU General Public License v3.0
It's not always immediately clear whether a program finished by itself or was killed. Would be nice if we could show this disctinction.
We could find gaps in testdata by mutation testing the accepted c++
submissions.
Any (most?) mutated solutions should fail.
If a line wrap happens the overwriting system doesn't work anymore and will only overwrite the wrapped part, not the first line. We should avoid going longer than the terminal width if we can.
Verifyproblem doesn't like it when submissions are close to each other in time; maybe we can just skip some submissions or at least mark them as unimportant for timing?
We don't have to rebuild all sources when only tests are updated.
Useful to not spoil results on hidden testcases
Currently the BAPCtools repo must be a git submodule of the contest repo in order to make the validation.h
symlinks work. It would be nice if we could circumvent this requirement.
Possible solutions:
validation.h
a copy. This is problematic if bugs are found.headers
directory to the paths the compiler searches for header files. Small drawback is that it won't be possible to compile the IO validators without the BAPCtools, and the validators will need to be included when exporting to DJ/kattis.Some solutions can fail in multiple ways. DomJudge allows adding a list of expected outcomes. We should support this (with the same syntax) as well.
See also Kattis/problemtools#48.
Teams should be able to run this is contest mode:
test
generators/gen.yaml
See #50.
bt problem
should only copy relevant things.
Project wide settings can opt out of:
We should run generators twice to see if they produce the same result.
Would be nice if at least run
works on windows.
Generate should work fine when there are no validators
Hi, my favorite input format validator is VIVA which for many problems allows very concise input format validators that meet Kattis/problemtools standards. In many cases, it entirely eliminates the need for additional input validation before the problem can be approved.
In Kattis problemtools, it recognizes the presence of .viva files in the input_format_validator directory, e.g. here
Consider using it/adding support for it.
Docs should be updated now that there are some new features.
Also put things in order of how often they're used, instead of usage order
Files in data/bad
can be used to test the validators.
The plan is to implement the following:
A single data/bad/test.in
should fail at least one input validator.
A triple data/bad/test.{in,ans,out}
should pass all input validators and should fail at least one output validator, where the .ans
is OK, but .out
is bad. If only one of .ans
and .out
is given, it will be assumed that a .ans
is not needed to validate a .out
and the single file will be used both as .ans
and .out
.
This can easily be changed when this is standardised in the problemarchive format. See also Kattis/problemtools#51 and Kattis/problemtools#86
We should support a contest-local or git-repo-local languages.yaml like problemtools' global config
Especially validator.{ctd,viva,cpp}.template
And also submissions/accepted/ragnar.cpp.template
The new-problem
subcommand should (interactively) ask for required configuration values:
Furthermore, the source
and source_url
fields should be extracted from a contest-level configuration file.
We should limit memory usage so that we can test such RTE solutions nicely without freezing the system.
We should support interactive problems.
By default only the contest itself should be build. Adding -a/--all
can build it for all problems as well.
Currently the /skel/problem/generators/input_validator
symlink is copied as a directory.
but keep it for validators.
Instead of just having an infinite default timeout.
It's nicer to use a well defined library than using manual string replace commands in latex template files.
Would be nice to fall back to python[3]
when pypy[3]
is not found.
When an AC solution gives WA/TLE, y/n prompt to move it to that directory.
Also allow --copy to copy AC solutions to the right directory. This allows fixing them in the AC dir, while keeping track of non-working version.
Proposal:
Add {input,output}_visualizer/
directory in the root of the problem directory (next to generators/
).
Different types of visualizers:
<visualizer> testcase.in
or <visualizer> < testcase.in
<visualizer> testcase.in testcase.ans
or <visualizer> <testcase.in <testcase.ans
.There is at most one visualizer of each type, and the entire input_visualizer
is treated as a single program (similar to a validator or generator program).
The latter form of piping both the input and output into stdin is useful for Asymptote.
Using sys.platform
we can do platform dependent stuff, so porting the windows
changes to master
should be possible.
In addition we could add generator --move-manual <dir>
to also move all inline manual cases to the specified directory (e.g. generators/manual
)
Generated input/answers should be validated.
We could improve the stability of our output validators by fuzz testing them.
Rewrite the current zip --kattis command to just zip the kattis directory that is created by the kattis subcommand.
Given a custom generator for testcases, it would be cool if we could automatically could create a lot of testcases with random parameters within some bounds, and then only keep the ones that make AC solutions disagree/many WA solutions fail.
To speed up evals, we could cache (hash of submission, hash of testcase)
pairs and do a lookup there instead of rerunning the code.
stats
count input files, but we always require answer files, even for problems with custom validation that don't strictly need them.
Unmodified code examples (input/output validator, example submission) should not be counted.
This is actually rather annoying because it can only be obtained via os.wait4(pid, 0)
, which doesn't handle timeouts.
The subprocess
module doesn't seem to have this functionality.
Also, rusage
isn't supported on Windows.
When using tools.py test, the stderr output of the solution under test is not displayed, which is really anoying for testing
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.