gatoreducator / gatorgrade Goto Github PK

This is more possible future work: there should probably be somewhere more central to get things like the assignment name, so that it is possible to configure them if desired in the future.

Originally posted by @Michionlion in #91 (comment)

Currently, the assignment name (i.e. the name that is printed in the final message, Passed x/x (xx%) of checks for <ASSIGNMENT-NAME>!) is automatically generated as the current directory. However, it might be useful to be able to configure this assignment name in the front matter of the gatorgrade.yml.

As a GatorGrade maintainer, I want to have continuous integration that lints Markdown, checks spelling, and runs tests, so that I can confidently make changes.

As a GatorGrade maintainer, I want an automatically generated template for my pull request descriptions, so that I know I am providing all of the necessary information about my proposed changes.

As an instructor, I want to see documentation that explains the format of a gatorgrade.yml file, so that I know how to define the GatorGrader checks I want.

The generate file should be module orientated. Functions may do a good job in this case.

When I tried to run gatorgrade locally on my Windows machine for an assignment, I had a few problems with what was being executed in the setup block in the configuration file. First, it couldn't run the commands because it was treated as one whole command per line, which is invalid. @gkapfham and I suspected that it could be a problem with how the separator of different commands was recognized in different operating systems.

Original gatorgrade.yml
Error from the original:

Running set up commands...
The system cannot find the path specified.
The set up command "cd project/clang; make" failed.                Exiting GatorGrade.

I then changed the setup block in that configuration file so it looked like the following:

cd project/clang && make bounded-cpu
cd project/clang && make unbounded-cpu
cd project/golang/ && make bounded-cpu
cd project/golang/ && make unbounded-cpu

gatorgrader successfully ran, but there were some failures on my local checks that passed in GitHub Actions.

-~-  FAILURES  -~-

✘  Use go fmt to confirm that bounded-cpu.go is formatted correctly
✘  Use go fmt to confirm that unbounded-cpu.go is formatted correctly
✘  Run the bin/bounded-cpu program in the project/clang
✘  Run the bin/bounded-cpu program in the project/clang and confirm two non-blank output lines
✘  Run the bin/bounded-cpu program in the project/clang and confirm fragment in output
✘  Run the bin/bounded-cpu program in the project/golang
✘  Run the bin/bounded-cpu program in the project/golang and confirm two non-blank output lines
✘  Run the bin/bounded-cpu program in the project/golang and confirm fragment in output

Settings for Reproduction:

Python 3.10.7

gatorgrade 0.3.0

Windows 11 Pro

Powershell 7.2.6

In the future, we may want to handle these exceptions more explicitly, similarly to how the GatorGrader command line interface handles them. This feedback should probably be an issue for future work.

Originally posted by @Michionlion in #91 (comment)

Currently, _run_gg_check in gatorgrade/output/output.py only reports GatorGrade exception type to the user in the form of a diagnostic with the content "{command_exception.__class__}" thrown by GatorGrader. These diagnostics could be improved by catching more specific exceptions and handling each case differently, as well as referencing some included information in the exception, such as args -- the GatorGrader check arguments for some exceptions. Visit the link in the quote above for more information on how this is handled in GatorGrader itself, which should be similar to what is needed for GatorGrade.

As an instructor, I want it to be difficult for students to accidentally generate a gatorgrade.yml file, so that students do not accidentally overwrite the gatorgrade.yml file I provide for an assignment.

As per @Michionlion feedback:

"I don't think this PR should fix this, but I think you'll probably want two separate jobs for MkDocs -- one for actually building and publishing documentation on merges to main, and another for linting the configuration and files (attempting a build, essentially) that runs on pull requests."

When attempting this command inside a repo with a gatorgrade.yml file; "gatorgrade --config config/gatorgrade.yml" , there was the following error:

Traceback (most recent call last):
  File "C:\Users\me\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\me\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\me\AppData\Local\Programs\Python\Python310\Scripts\gatorgrade.exe\__main__.py", line 4, in <module>
  File "C:\Users\me\AppData\Local\Programs\Python\Python310\lib\site-packages\gatorgrade\main.py", line 36, in <module>
    root: Path = typer.Argument(
TypeError: Argument() got an unexpected keyword argument 'help'

Uninstalling and reinstalling gatorgrade through pipx didn't fix it. Trying to shorten the command by going inside the file with the gatorgrade.yml file yielded the same results. On Windows 11 Pro.

Issue

When an assignment grades in GitHub Actions, even if there exists failing checks, the assignment still receives a passing build. An example of this: test assignment.

Steps taken to reproduce

• Changed an assignment parameter to fail it (added a TODO)
• Changed additional assignment specs to be sure that this wasn't an "off by one" issue (removed words from reflection.md)
• Tried on an additional independent assignment to isolate any build configuration issues

As an instructor, I want to know if something went wrong with the generation of the gatorgrade.yml file, so that I can try to debug the issue.

As a GatorGrade user, I want to be able to install GatorGrade from PyPI, so that I can quickly start using GatorGrade.

As a student, I want to see documentation that describes how I can install and run GatorGrade, so that I can use GatorGrade for my assignments.

From Pull Request #69

Lines 21-24:

# Create an empty list to store dictionaries
    for file_path in path_names:
        # Iterate through items in path_names
        file_path_fixed = file_path.replace("./", "")

This segment of code could possibly work better if it were put in its' own function.

As an instructor, I want to be able to define GatorGrader checks in a gatorgrade.yml file, so that I can distribute a custom GatorGrade configuration with each assignment.

As a student, I want to see the description of each GatorGrader check, so that I can easily understand the requirements.

As a student, I want to see the percentage of GatorGrader checks that passed, so that I can gauge my progress.

We have started adding type hints with PR #91. For maintainability, we should add type hints (at least for all function inputs and outputs) throughout the entire system. Because the system is relatively small, this would be a great onboarding task for a new maintainer as it will help them develop an understanding of the system's architecture.

As an instructor, I want to be able to specify file paths either as individual keys or nested keys, so that I can avoid writing duplicate paths for files in the same directory.

As an instructor, I want to be able to automatically generate a gatorgrade.yml file in a solution repository, so that I can quickly start defining GatorGrader checks.

Create a white list and the generate tool will put the file names only if they are under the specific paths.

As an instructor, I want confirmation that a gatorgrade.yml file was generated successfully, so that I can be sure that the gatorgrade.yml file will work for the solution repository before I begin to add configuration to it.

The name of this project should be capitalized in README.md.

As an instructor, I want to be able to specify GatorGrader checks that do not correspond to file paths, so that I can verify behavior that is not file-specific.

From Pull Request #54

Lines 21 and 33:
To resolve the issue with sending None to this function explicitly, here's what you could do: move the path parameter to the end of the parameter list, and then give it a default value of None. Something like:

Suggested change (Line 21):

- def add_checks_to_list(path, data_list, reformatted_data):
+ def add_checks_to_list(data_list, reformatted_data, path=None):

Then, you can call add_checks_to_list with no path parameter to specify starting at the root (this would replace where you're currently manually specifying None), or give it a path if you're starting an inner recursive call (where you're calling it inside this function).

(Note: Would also need to change line 33 to put the path at the end of the function call)

Lines 26-28 and 33-35:
To make this code look better after running black (i.e. not have the line splitting on the parameter list), you should place the comment above the if statement, instead of after it. This will result in cleaner-looking code!

Line 38:
Instead of modifying reformatted_data and passing it by reference to ensure the values are returned, you should directly create and return a new list in this function, rather than creating it in reformat_yaml_data. When processing the result of a recursive call, just use reformatted_data.extend to add on the recursively-generated checks to the current data.

Here is what a gatorgrade.yml file should look like. GatorGrade should be able to parse this file and run all of the GatorGrader/command checks defined in the file. Additionally, the gatorgrade generate command should generate a file that conforms to the same format.

setup: |
  ./script.sh
  poetry install
  echo "Setup Complete!"
---
- path/to:
  - file.py:
    - description: Complete all TODOs
      check: MatchFileFragment
      options:
        fragment: TODO
        count: 0
        exact: true
    - description: Use an if statement
      check: MatchFileRegex
      options:
        regex: "if .*?:"
        count: 1
        exact: false
  - file2.py:
    - description: Complete all TODOs
      check: MatchFileFragment
      options:
        fragment: TODO
        count: 0
        exact: true
- description: Pass MDL
  command: mdl .
- description: Pass HTMLHint
  command: htmlhint
- description: Have a total of 8 commits, 5 of which were created by you
  check: CountCommits
  options:
    count: 8

As an instructor, I want to be able to specify a shell command check instead of a GatorGrader check, so that I can run programs like mdl and pylint.

As an instructor, I want to be able to specify a list of shell commands to run before the GatorGrader checks, so that I can set up the environment.

The workflow should run one specific (preferably the most recent at this time) version of poetry so that it's consistent for the development cycle. That way if there is an update for poetry and we choose to update, we can handle those errors at that time.

As a student, I want to see green checkmarks next to passing GatorGrader checks and red crossmarks next to failing GatorGrader checks, so that I can easily tell which GatorGrader checks passed and which failed.

The project needs a README.md file. Some things you should probably include are:

• Repo Analytics Banner
• Short description of what the project is
• How to use the project (Might just want to make a section for it and leave it blank for now)
• How to contribute (Specifically, install instructions for the dev environment)
• A reference to the contribution instructions file
• A list of contributors / special thanks

Finish writing test cases for the generate program to have confidence that the program works as expected.

As a student, if all GatorGrader checks passed, I want to see the percentage in green, so that I can easily tell when I have completed all of the requirements.

As a GatorGrade user, I want an easy-to-use command-line interface, so that I can easily use GatorGrade.

As an instructor, I want to be able to specify GatorGrader checks that correspond to file paths, so that I can verify requirements specific to a file.

Research gatorgradle documentation, consult the user, consult professional aides

The generate tool should filter the paths and files according to both the white list and the black list

In the future, we may want to add a warning if we get a check in the checks array that we cannot run.

Originally posted by @Michionlion in #91 (comment)

Currently, in gatorgrade/input/command_line_generator.py, we read in the data assuming that the user has provided a correctly formatted gatorgrade.yml file. If checks are improperly formatted (e.g. a shell check does not contain a command value), we allow them to fail at the "run" phase, which occurs in gatorgrade/output/output.py. We may want to add validation of checks at the "read" phase (i.e. in command_line_generator.py) that ensures that the gatorgrade.yml file contains all necessary data for each check and provides information to the user if it does not.

As a student, I want to see the output of the GatorGrader checks that are configured for an assignment, so that I know which requirements I have completed and which requirements I still need to complete.

Is it possible to get a JSON output? I know that will be helpful for integration with other tools? I would take a stab at it but I'd like to hear others thoughts on it before I go about implementing it, or if anyone else wants to do it?

Probably needs a flag in typer to enable that output and then if so just output to terminal in JSON format? Maybe we can also add a file output?

As an instructor, I want the gatorgrade.yml file to list all of the files in the solution repository, so that I can define GatorGrader checks for any file.

Add a list of contributors to the readme in honor of all of our hard work!