Let's try to capture some steps of the human evaluation process in a tool so that we can "pre-check" the contents of a workflow repository:

• Is there a README?
• Is there a license?
• Does the README mention the figures and tables from the paper (clear connection between outputs and code files)?
• Is there an environment documentation or file (cf. repo2docker REES spec)
• Do all links (and DOIs) resolve?
• Are execution times mentioned in the README?
• If HPC is mentioned, is a sample dataset also mentioned?
• Are any full paths in documentation or code files (/home/.., C:\Users\.., etc.)
• ...

Tools that show how the interaction could work:

• https://github.com/MangoTheCat/goodpractice/
• https://docs.ropensci.org/checkers/

• Add paper authors as "Contributors" (role "other"?)
• Allow to pre-reserve a DOI without filling in all the metadata
• Make filling out the metadata in Zenodo online (which might be more accessible to users) and then retrieving these changes before making any updates works well
• Some metadata, suchs as the paper reference, might not be available the first time - should we not publish the report until then?

I couldn't get much joy from zen4R today. It didn't upload metadata, and was complaining about the licence being unrecognised.

> codecheck:::set_zenodo_metadata(zenodo, record, metadata)
[zen4R][ERROR] ZenodoRecord - License with id 'other-open' doesn't exist in Zenodo 
Error in draft$setLicense(licenseId = "other-open") : 
  License with id 'other-open' doesn't exist in Zenodo

changing the code to use 'MIT' as licence did not work.... removing the line to setLicense silenced the error, but no metadata were uploaded....

shrug...

I'd love to replace the kind of crufy solution I have now for including manifest files in the certificate, e.g. I have :

{r, echo=FALSE, fig.cap=manifest_df[1:4,"comment"]}
knitr::include_graphics(manifest_df[1, "dest"])
cat('\n\n')
knitr::include_graphics(manifest_df[2, "dest"])
cat('\n\n')
knitr::include_graphics(manifest_df[3, "dest"])
cat('\n\n')
knitr::include_graphics(manifest_df[4, "dest"])
cat('\n\n')

whereas what I'd like is a simple chunk that by default will include all the manifest files intelligently into the Rmd. e.g. if given a figure (png/pdf/jpg), it calls include_graphics, else for csv it shows a tiblle summary of the data frame... just need some R magic to get it working.

A package for streamlined creation of YAML headers for documents. Could be useful at some point: https://cran.r-project.org/web/packages/ymlthis/index.html

see https://ropensci.org/technotes/2020/11/19/moving-away-travis/

Its currently a bitmap, whereas at the least we can make it a nice vector format.

and see also discussion re: making the badges dynamic.

codecheckers/discussion#24 (comment)

Besided the validation of the codecheck.yml, a register update can also make sure

• the repository with the check files has the tag codecheck: https://github.com/search?q=topic%3Acodecheck+fork%3Atrue+org%3Acodecheckers&type=Repositories
• the repository has a license
• ...

Currently citation() generates a citation of the current codecheck certificate. the function name clashes though with base::citation() which should be left alone. (incidentally, citation("codecheck") is already half-usable.)

should we simply not export the function, and rely on codecheck:::citation() or rename the function to something like codecheck_cite_cert() or cite_certificate()? My preference would be option 3.

We can write a function that generates (or fills in a template) a DOCX report to be filled out by codecheckers. We can try to make the coverpage look very similar to the one coming out of the Rmd template.

Note to self: This is where I created docx in R before using the awesome officer package: https://gitlab.com/nuest/forumcs-2019-osf-proceedings/-/blob/master/proceedings.Rmd#L231

It could give much nicer error messages if integrated with the used test package.

https://jacob.wujciak.de/octolog/index.html

The ttdo package provides nice diffs for tests from the tinytest package, which could be used for validation of codecheck.yml files:

http://dirk.eddelbuettel.com/blog/2020/05/09/#ttdo_0.0.5

Also useful for tests in general!

It would be great if the codecheck package can also be helpful for reviews conducted by non-R users. For this, we could try a Shiny-based UI. The shiny app could either run in the cloud (free with RStudio) or as a Binder.

The Shiny app would focus on editing the report and pushing metadata to Zenodo.

https://ropensci.org/blog/2022/02/01/pkgcheck-action/

Can query Crossref (using rcrossref and warn if author names or title differ.

See https://stackoverflow.com/a/64799580/261210

Currently I see no place in codecheck.yml for where we should list the URL of the original code repository that we are checking.

how about a new field source: to point to a URL of where the software came from? Or that might need to be an open string, as sometimes we will need to describe several URLs, one for the code, one for the data, ....

@sje30 I suggest to move the functions from the check_register script in https://github.com/codecheckers/register/ into this package. Fine with you?

I'll also happily do the work :-)

this used to work on our latex templates...

If an author has an ORCID, it doesn't currently get rendered.

The repository on GitHub/GitLab should be "archived" and thereby marked as read-only.

Only existing ORCIDs should be used.

rorcid::orcid_id(orcid = "0000-0002-9341-7985")
# should not be empty

Also, we can point to rorcid package in the documentaiton, which might streamline lookup of ORCIDs for names. Note when I tried this in a container, the callback for the authentication did not work :-/.

Also add rorcid to Suggests in the DESCRIPTION file.

First, manually deposit the register once on Zenodo to preserve it. Then add a function to update that deposit with one function call.

Eventually, when the register rendering is moved to CI, the function could be extended with a check of how recent the deposit is and then update regularly, e.g. once every quarter.

The register validation function should check that

• the deposit title contains the check number
• the deposit metadata contains a reference to the published article
• the deposit metadata contains a reference to the GitHub repository
• the file listing contains a file codecheck.pdf
• the deposit is part of the codecheck community
• ...

Related to #14 and #19

codecheck:::set_zenodo_certificate(zenodo, record, "codecheck.pdf")

didn't work for me. I got this error:

Error in record$id : $ operator is invalid for atomic vectors

it doesn't automatically load 'zen4R'or 'yaml' when the package is
loaded. yet, those packages are listed on the Imports: part of
DESCRIPTION. What am I missing?

> setwd('/Users/stephen/proj/2020/codecheckers/rmr-solver/codecheck')
> library("codecheck")
Loading required package: gh
Loading required package: R.cache
R.cache v0.16.0 (2022-07-21 16:20:02 UTC) successfully loaded. See ?R.cache for help.
Loading required package: parsedate
Loading required package: git2r
> metadata = read_yaml( "../codecheck.yml")
Error in read_yaml("../codecheck.yml") : 
  could not find function "read_yaml"

I've used testthat in the past, but I'd like to give tinytest with ttdo a try:

• https://cran.r-project.org/package=tinytest
• https://cran.r-project.org/package=ttdo

Details: https://docs.github.com/en/organizations/collaborating-with-groups-in-organizations/customizing-your-organizations-profile#pinned-repositories

This tweet https://twitter.com/daranzolin/status/1254486875303075841/photo/1

got me thinking: that function is surely useful for the validation of codecheck.yml, should we make this package Depends: R (>= 4.0.0) ?

Note: In the register I already introduced assertthat for checks, so this might be a moot point.

The template codecheck.Rmd is useful, but doesn't work out of the box ("make" will fail.) I think this is bad for us, as we can't even test that the template works.

I propose a simple change to create_codecheck_files() that will generate two pngs, called "codecheck-deleteme1.png" and "codecheck-deleteme2.png" in the root of the project. They can be included in the template manifest so that the template works. The manifest will then have comments to indicate the pngs should be deleted once the codecheck has started.

There is an oddity with the zenodo DOIs.

e.g.
report: https://doi.org/10.5281/zenodo.10213244

the id, 10213244, is the latest DOI. But there is also a DOI 10213243, which is the 'master DOI' and will automatically link to the latest version of the report. It might be a good idea to explicitly document this. End users might prefer refering to the 'master DOI' rather than the first version.

A question and answer style, similar to devtools::release(), built on cli: https://cran.r-project.org/web/packages/cli/index.html

Should definitely use crayon for nice coloured output.

whereami could also be helpful.

And ask & praise.