gatoreducator / gatorgrade Goto Github PK
View Code? Open in Web Editor NEW:heavy_check_mark: Generate or Run a Suite of GatorGrader Checks
License: GNU General Public License v3.0
:heavy_check_mark: Generate or Run a Suite of GatorGrader Checks
License: GNU General Public License v3.0
As an instructor, I want to be able to automatically generate a gatorgrade.yml
file in a solution repository, so that I can quickly start defining GatorGrader checks.
As a student, if all GatorGrader checks passed, I want to see the percentage in green, so that I can easily tell when I have completed all of the requirements.
As an instructor, I want to see documentation that explains the format of a gatorgrade.yml
file, so that I know how to define the GatorGrader checks I want.
Here is what a gatorgrade.yml
file should look like. GatorGrade should be able to parse this file and run all of the GatorGrader/command checks defined in the file. Additionally, the gatorgrade generate
command should generate a file that conforms to the same format.
setup: |
./script.sh
poetry install
echo "Setup Complete!"
---
- path/to:
- file.py:
- description: Complete all TODOs
check: MatchFileFragment
options:
fragment: TODO
count: 0
exact: true
- description: Use an if statement
check: MatchFileRegex
options:
regex: "if .*?:"
count: 1
exact: false
- file2.py:
- description: Complete all TODOs
check: MatchFileFragment
options:
fragment: TODO
count: 0
exact: true
- description: Pass MDL
command: mdl .
- description: Pass HTMLHint
command: htmlhint
- description: Have a total of 8 commits, 5 of which were created by you
check: CountCommits
options:
count: 8
As an instructor, I want confirmation that a gatorgrade.yml
file was generated successfully, so that I can be sure that the gatorgrade.yml
file will work for the solution repository before I begin to add configuration to it.
As an instructor, I want to be able to specify GatorGrader checks that do not correspond to file paths, so that I can verify behavior that is not file-specific.
We have started adding type hints with PR #91. For maintainability, we should add type hints (at least for all function inputs and outputs) throughout the entire system. Because the system is relatively small, this would be a great onboarding task for a new maintainer as it will help them develop an understanding of the system's architecture.
Create a white list and the generate tool will put the file names only if they are under the specific paths.
Research gatorgradle documentation, consult the user, consult professional aides
As an instructor, I want to be able to specify GatorGrader checks that correspond to file paths, so that I can verify requirements specific to a file.
As an instructor, I want to be able to specify a list of shell commands to run before the GatorGrader checks, so that I can set up the environment.
Is it possible to get a JSON output? I know that will be helpful for integration with other tools? I would take a stab at it but I'd like to hear others thoughts on it before I go about implementing it, or if anyone else wants to do it?
Probably needs a flag in typer to enable that output and then if so just output to terminal in JSON format? Maybe we can also add a file output?
As an instructor, I want to know if something went wrong with the generation of the gatorgrade.yml
file, so that I can try to debug the issue.
When attempting this command inside a repo with a gatorgrade.yml file; "gatorgrade --config config/gatorgrade.yml" , there was the following error:
Traceback (most recent call last):
File "C:\Users\me\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\me\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\me\AppData\Local\Programs\Python\Python310\Scripts\gatorgrade.exe\__main__.py", line 4, in <module>
File "C:\Users\me\AppData\Local\Programs\Python\Python310\lib\site-packages\gatorgrade\main.py", line 36, in <module>
root: Path = typer.Argument(
TypeError: Argument() got an unexpected keyword argument 'help'
Uninstalling and reinstalling gatorgrade through pipx didn't fix it. Trying to shorten the command by going inside the file with the gatorgrade.yml file yielded the same results. On Windows 11 Pro.
As a GatorGrade user, I want to be able to install GatorGrade from PyPI, so that I can quickly start using GatorGrade.
The name of this project should be capitalized in README.md
.
The workflow should run one specific (preferably the most recent at this time) version of poetry so that it's consistent for the development cycle. That way if there is an update for poetry and we choose to update, we can handle those errors at that time.
As a student, I want to see the description of each GatorGrader check, so that I can easily understand the requirements.
The generate file should be module orientated. Functions may do a good job in this case.
This is more possible future work: there should probably be somewhere more central to get things like the assignment name, so that it is possible to configure them if desired in the future.
Originally posted by @Michionlion in #91 (comment)
Currently, the assignment name (i.e. the name that is printed in the final message, Passed x/x (xx%) of checks for <ASSIGNMENT-NAME>!
) is automatically generated as the current directory. However, it might be useful to be able to configure this assignment name in the front matter of the gatorgrade.yml
.
As an instructor, I want it to be difficult for students to accidentally generate a gatorgrade.yml
file, so that students do not accidentally overwrite the gatorgrade.yml
file I provide for an assignment.
As an instructor, I want the gatorgrade.yml
file to list all of the files in the solution repository, so that I can define GatorGrader checks for any file.
In the future, we may want to add a warning if we get a check in the
checks
array that we cannot run.
Originally posted by @Michionlion in #91 (comment)
Currently, in gatorgrade/input/command_line_generator.py
, we read in the data assuming that the user has provided a correctly formatted gatorgrade.yml
file. If checks are improperly formatted (e.g. a shell check does not contain a command
value), we allow them to fail at the "run" phase, which occurs in gatorgrade/output/output.py
. We may want to add validation of checks at the "read" phase (i.e. in command_line_generator.py
) that ensures that the gatorgrade.yml
file contains all necessary data for each check and provides information to the user if it does not.
From Pull Request #54
Lines 21 and 33:
To resolve the issue with sending None
to this function explicitly, here's what you could do: move the path
parameter to the end of the parameter list, and then give it a default value of None
. Something like:
Suggested change (Line 21):
- def add_checks_to_list(path, data_list, reformatted_data):
+ def add_checks_to_list(data_list, reformatted_data, path=None):
Then, you can call add_checks_to_list
with no path
parameter to specify starting at the root (this would replace where you're currently manually specifying None), or give it a path if you're starting an inner recursive call (where you're calling it inside this function).
(Note: Would also need to change line 33 to put the path
at the end of the function call)
Lines 26-28 and 33-35:
To make this code look better after running black (i.e. not have the line splitting on the parameter list), you should place the comment above the if statement, instead of after it. This will result in cleaner-looking code!
Line 38:
Instead of modifying reformatted_data and passing it by reference to ensure the values are returned, you should directly create and return a new list in this function, rather than creating it in reformat_yaml_data. When processing the result of a recursive call, just use reformatted_data.extend to add on the recursively-generated checks to the current data.
In the future, we may want to handle these exceptions more explicitly, similarly to how the GatorGrader command line interface handles them. This feedback should probably be an issue for future work.
Originally posted by @Michionlion in #91 (comment)
Currently, _run_gg_check
in gatorgrade/output/output.py
only reports GatorGrade exception type to the user in the form of a diagnostic with the content "{command_exception.__class__}" thrown by GatorGrader
. These diagnostics could be improved by catching more specific exceptions and handling each case differently, as well as referencing some included information in the exception, such as args
-- the GatorGrader check arguments for some exceptions. Visit the link in the quote above for more information on how this is handled in GatorGrader itself, which should be similar to what is needed for GatorGrade.
Add a list of contributors to the readme in honor of all of our hard work!
As a student, I want to see documentation that describes how I can install and run GatorGrade, so that I can use GatorGrade for my assignments.
When I tried to run gatorgrade
locally on my Windows machine for an assignment, I had a few problems with what was being executed in the setup
block in the configuration file. First, it couldn't run the commands because it was treated as one whole command per line, which is invalid. @gkapfham and I suspected that it could be a problem with how the separator of different commands was recognized in different operating systems.
Original gatorgrade.yml
Error from the original:
Running set up commands...
The system cannot find the path specified.
The set up command "cd project/clang; make" failed. Exiting GatorGrade.
I then changed the setup block in that configuration file so it looked like the following:
cd project/clang && make bounded-cpu
cd project/clang && make unbounded-cpu
cd project/golang/ && make bounded-cpu
cd project/golang/ && make unbounded-cpu
gatorgrader
successfully ran, but there were some failures on my local checks that passed in GitHub Actions.
-~- FAILURES -~-
✘ Use go fmt to confirm that bounded-cpu.go is formatted correctly
✘ Use go fmt to confirm that unbounded-cpu.go is formatted correctly
✘ Run the bin/bounded-cpu program in the project/clang
✘ Run the bin/bounded-cpu program in the project/clang and confirm two non-blank output lines
✘ Run the bin/bounded-cpu program in the project/clang and confirm fragment in output
✘ Run the bin/bounded-cpu program in the project/golang
✘ Run the bin/bounded-cpu program in the project/golang and confirm two non-blank output lines
✘ Run the bin/bounded-cpu program in the project/golang and confirm fragment in output
Settings for Reproduction:
Python 3.10.7
gatorgrade
0.3.0
Windows 11 Pro
Powershell 7.2.6
As a student, I want to see the output of the GatorGrader checks that are configured for an assignment, so that I know which requirements I have completed and which requirements I still need to complete.
The generate tool should filter the paths and files according to both the white list and the black list
As a student, I want to see green checkmarks next to passing GatorGrader checks and red crossmarks next to failing GatorGrader checks, so that I can easily tell which GatorGrader checks passed and which failed.
As an instructor, I want to be able to specify a shell command check instead of a GatorGrader check, so that I can run programs like mdl
and pylint
.
From Pull Request #69
Lines 21-24:
# Create an empty list to store dictionaries
for file_path in path_names:
# Iterate through items in path_names
file_path_fixed = file_path.replace("./", "")
This segment of code could possibly work better if it were put in its' own function.
When an assignment grades in GitHub Actions, even if there exists failing checks, the assignment still receives a passing build. An example of this: test assignment.
TODO
)reflection.md
)Finish writing test cases for the generate program to have confidence that the program works as expected.
As a GatorGrade maintainer, I want an automatically generated template for my pull request descriptions, so that I know I am providing all of the necessary information about my proposed changes.
As a student, I want to see the percentage of GatorGrader checks that passed, so that I can gauge my progress.
As a GatorGrade maintainer, I want to have continuous integration that lints Markdown, checks spelling, and runs tests, so that I can confidently make changes.
As an instructor, I want to be able to specify file paths either as individual keys or nested keys, so that I can avoid writing duplicate paths for files in the same directory.
The project needs a README.md
file. Some things you should probably include are:
As per @Michionlion feedback:
"I don't think this PR should fix this, but I think you'll probably want two separate jobs for MkDocs -- one for actually building and publishing documentation on merges to main, and another for linting the configuration and files (attempting a build, essentially) that runs on pull requests."
As an instructor, I want to be able to define GatorGrader checks in a gatorgrade.yml
file, so that I can distribute a custom GatorGrade configuration with each assignment.
As a GatorGrade user, I want an easy-to-use command-line interface, so that I can easily use GatorGrade.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.