As an interviewer, when I load the Dashboard/index using Safari, the "pencil-and-paper" links for interview assignments do not render.

How it looks in Safari

(links missing)

How it looks in Chrome

(links render properly)

As an application reviewer, I would like to see a running "leaderboard" of how many reviews reviewers have submitted, for motivational purposes.

As a reviewer, I might like to see (links to) my recent reviews, assigned interviews, etc.

As an admin, I might like to see useful review statistics and links:

• number of apps
• number of reviews
• trends of the above
• ...

Appy primarily prioritizes applications for review according to their (lack of) reviews. It should secondarily prioritize applications so as to resolve ambiguity between the reviews they have received.

For example:

1. first list applications with 0 or 1 reviews (existing ordering)
2. then list applications whose difference between total application approvals and non-approvals and/or …???
3. … (random ordering)

Or, really, whatever yields:

1. …
2. applications with 1 yes and 1 "not yes" (either "no" or "unknown")
3. …with 2 unknowns
4. …with 2 yes, 2 no
5. …

As a reviewer, I would like to be able to search for an applicant's application, so that I can review someone I know.

I need not be able to list all applications, (and perhaps the search term should have a minimum length or perform an exact case-insensitive match against last, and optionally first, name).

copy/move applicant name and email display from second to first part of application listing

As an admin, I would like to see a listing of reviewers, with various views, listing/sorting/grouping them by "interesting" features:

• many reviews
• many reviews with high average score
• few reviews

🙌 was mistaken by a reviewer for ❓ – he marked applications "exceptional," which he'd meant to mark "cannot determine."

Perhaps this symbol could be more clear.

As an admin, I would like to see a listing of applications, ordered by number of reviews (descending), each linking to an application page, listing the application's reviews, and linking to reviewers.

appy-reviews was initiated as a private repository out of an excess of caution.

It should be open-source; but, we'll want to ensure that it contains nothing sensitive or "secret" (passwords, etc.) – either in its current code (HEAD) or in any of its history.

As a Round 2 Interviewer, I would like to see the Round 1 Interviewer's review and comments on the interview review page.

This work was largely done for Round 1 Interviews in #19, but a listing of interview reviews was omitted.

The review count message was designed with the assumption that reviewers would only submit a few reviews – it doesn't take many before it adds an exclamation point for every review submitted!!!!!!!!!!!!!!!!!!

The message should be recalibrated with the expectation that the reviewer will submit tens of reviews.

Investigate confusing situation(s) wherein a user tries to sign in via one method (say social auth) but their account has only been provisioned for another method (say username+password).

During unavoidable downtime, (primarily during ETL reloads), switch DNS to S3 bucket serving a temporary maintenance page.

As an interviewing reviewer, I can access a new application detail page, through which I can submit a review of my interview with an applicant who was assigned to me.

GET /review/interview/<key>/ returns a page displaying:

• the application and references
• a listing of reviews the application received
• a listing of any (other) interview reviews
• an (interview) review form (like the previous, application review, one)

POST /review/interview/<key>/ stores the new interview review and redirects me to the interview listing.

Automatically collapse review instructions when reviewer has reviewed at least one application.

As a reviewer, I would like to be able to optionally include advice to the applicant in my application review.

For example, I may feel that the applicant is a really good fit, but just needs to "level up" a particular skill and reapply.

This advice could be sent directly to the applicant, and could make it less discouraging for some of the people who don’t get interviews. Plus, at least 2 people are seeing every application, so in many cases the personalized feedback exists, and it would be great to capture it.

As an interviewer, I would like to be able to review all of an application's review comments/suggestions at once, in full.

As a reviewer, my reviews should be deprecable, such that they are discounted or devalued when prioritizing applications in need of review according to the number of reviews that they have received.

hi, I got issues trying to test the repository locally. After successful steps 1 and 2 of the installation instructions from the README I run manage build on the terminal with no extra configuration and it creates two docker images dsapp/appy-reviews/web and 093198349272.dkr.ecr.us-west-2.amazonaws.com/dsapp/appy-reviews/web which are identical. Then I run:

docker run -itd -p 8000:8000 dsapp/appy-reviews/web

And I got a Bad Request (400) message when I access http://localhost:8000 and no response from the IP of the container and port 8000.

Is there something I'm missing during the installation? I'm using macOS 10.14.3 and I had to use pyenv virtualenv 3.7.1 appy because 3.7.2 wasn't available. Thank you.

As a reviewer, I would like to be able to review an applicant's social science aptitude. This form element should be listed right under "data handling and manipulation." It can be called it “social science.”

As a trusted, a.k.a. super, user, I should be able to quickly submit my review, without filling out the (entire) review form.

As a person accessing the database, I find it difficult to query (against) the schema's applications which are available to reviewers to review.

As this logic is not entirely straight-forward, and it is otherwise "baked" into appy, it might be added to a database view/function, which appy provisions (via database migration).

(Obviously it should be somehow ensured or suggested to those looking at and handling appy's code that changes to this logic should be propagated to this function, through further database migration.)

It is likely that this must be a database function, which accepts the single argument of the current program year (though this could alternatively default to the year of the current date).

As a reviewer, I would like to be able to flag an application as exceptional, and deserving an expedited review process.

For example, such applicants might be interviewed even when they only have two reviews, or pushed to the front of the review queue.

This might take the form of an additional form field, e.g. a checkbox.

Appy (appears to) make deprecated GitHub API requests, which will cease to work in 2020.

This is ostensibly during the GitHub OAuth flow.

Updating the library in use for this process may be sufficient.

https://developer.github.com/changes/2020-02-10-deprecating-auth-through-query-param/

appy email address [email protected] should probably forward mail it receives to somewhere someone checks.

Applicants may be imported with varying, mixed-case email addresses, such that their resources aren't properly linked.

For example, these two applications likely should have been one: (1475, 1084).

Appy should ensure that applicant emails are inserted lowercased, and that references and applications, etc. are properly linked.

ETL should be run periodically by an automated process to (re)load data into the database.

• Elastic Beanstalk supports configuration of cron on (one of its) servers.
• Additionally, Elastic Beanstalk supports "worker" servers.
• Appy is a Docker-deployed application, which could be run as an AWS ECS Task.

In 2019, ETL was run during the application stage from the management CLI, with variations of the following:

manage etl --all --stage=application

…and in the review stage:

manage etl --all --stage=review

Note, however, that Appy does not ship with the repo's manage.py. The above CLI commands wrap and document the applicable underlying commands, (run through docker run --rm --user webapp -eDATABASE_URL -eWUFOO_API_KEY appyreviews_web):

./manage.py loadwufoo -v 1 --stage application -

./manage.py loadapps execute

and:

./manage.py loadwufoo -v 1 --stage review -

./manage.py loadapps --closed execute

Appy looks decent in Chrome; but, it looks bad in Firefox. (And who knows about the others….)

Appy should identify the range of possible values in pick-one fields from the application/survey, particularly in references' evaluations, so that as a reviewer I understand the significance of the evaluation.

Reproduce the exact, original language of the questions that the applicant saw, (rather than, as currently, needlessly abbreviating some field labels).

• interview-review page should display peers' interview reviews for all interview assignments preceding the current one
  • on round one, only round-zero reviews should be displayed (i.e. no interview reviews)
  • on round two, round-one reviews should be displayed
  • on any round three, round-one and round-two reviews should be displayed
• an (optional) interview3_decision should be added to application
• 2018 round-two interview assignments should be marked round-three, as appropriate, (and the remainder's applications' decisions updated)

As a reviewer, I would like to be able to view a listing of the reviews I have submitted, view them (with the same page UX as I originally saw), and update them.

As an admin, I would like to (optionally) see all reviews.

As a reviewer, I would prefer to review applicants who share my academic/professional background.

As a reviewer, I would like to be able to use Appy on any size screen … even on that of my mobile phone!

As a reviewer, I should be assigned some arbitrary set of applications to review, which I will be asked to review first, before being asked to review applications according to the existing, general prioritization rules.

(This need not be configurable via any Web page, for now.)

As an interviewing reviewer, I see a listing of (links to) applications of those applicants assigned to me to interview.

GET /review/interview/

The page at /accounts/password/reset/done/ is not styled.