We want people to be able to deploy their own inference endpoints and give us an API key.

We also want to be able to run deployed full-time endpoints and perform fine-tuning on react native eg.

Then users will just pay us.

Description:
Want to test backend lambda + API gateway functionality dependent of #75

Validation:
Video showing call using postman and returning a review

tech steps:
add route to the apigateway. This route will be called /demoReview and will be a POST.
route needs no authentication.
this route needs rate limiting turned on in API gateway!!!!
add lambda integration to the demoReview lambda.

Error calling openai completions

Possible causes

• Request is too long for the model token length
• Mallformed request.

'code-review-gpt test'

Should test the prompt end to end against a pre defined set of criteria.

Add it to ci

Description: need to be able to use users API keys for review. This ticket involves creating the lambda to add the API key to the db

Validation:
trigger lambda in console with test API key and see it in dynamo.

Tech steps:
create a new empty lambda function updateUser in core service.
follow the instructions on the site https://www.dynamodbtoolbox.com/docs/introduction/quick-start
create an entity
User:
PK: USERID#.... string
SK: ROOT string
apiKey: .... string

in the lambda we are going to take the APIkey and the userid from the request and add it to dynamo.

We want to support gitlab

Description:
When a user first signs up to the platform. We want to save their email address and user sub to dynamo. Cognito latency is super slow and expensive.

Description:
Create a dynamodb adapter for nextauth in the webapp. dependant on #61 and #64

Validation;
Screenshot of user data in auth db

Tech steps:
Create a dynamodb table in the sst stack.

Follow this tutorial
https://authjs.dev/reference/adapter/dynamodb

Problem we're trying to solve is clear

Secure the endpoints against DDOS attacks and DoW attacks

Success Measures

• protected against DDOS attacks to the best of our ability

Scope of feature

• Cloudfront
• Waf rules

Edge Cases

Perfect user stories

❌

Risks/Dependencies identified

Dependent on endpoints implemented

Technical Strategy is known

✅

Uncertainties are known

not sure on exact Waf rules or pricing model

Dependencies are unblocked

Testing strategy is known

• artillery?

Need a frontend with a big box which allows you to write in code.

Review button.

Problem we're trying to solve is clear

Allow people to use the tool with installing or downloading anything

Success Measures

• A website at the oriontools/demo url

Scope of feature

• A box that is pasted in text and get code reviewed

Edge Cases

✅

Perfect user stories

❌

Risks/Dependencies identified

✅

Technical Strategy is known

• Make a front end it NextJS

Uncertainties are known

• Specific styling, no UI to work with

Dependencies are unblocked

✅

Testing strategy is known

• Unit test front end
• NO END TO END TESTS

When in debug mode, print the prompt and responses of the model when a call is made

The idea here is that we want a hosted and free demo accessible on the public internet for people to play with.

Beause we will be paying for this the API gateway endpoint needs rate limiting to prevent DoW attacks and also the lambda needs request limiting to prevent overuse of GPT4 $$$

Problem we're trying to solve is clear

Allow people to use the tool with installing or downloading anything

Success Measures

• A backend endpoint which can be hit with postman

Scope of feature

• An endpoint that is hit with text and get code reviewed

Edge Cases

✅

Perfect user stories

❌

Risks/Dependencies identified

• Have to add in rate limiting due to it being an open end point

Technical Strategy is known

• Make a lambda which will handle the compute
• Connect together with API Gateway
• Add cloudfront
• Add WAF rules

Uncertainties are known

• Specific WAF rules

Dependencies are unblocked

✅

Testing strategy is known

• Integration test the backend
• NO END TO END TESTS

Make a landing page on GitHub pages. Super minimal. Leave a space for a logo

Needs a link to the GitHub and the code to get started

'npm i code-review-gpt && npx code-review-gpt configure'

Problem we're trying to solve is clear

We want user to be able to add the github app to the repo.

Success Measures

Use can add github app to repo and it comment on their PR

Scope of feature

The GitHub app will receive webhooks to

1. setup the repo
2. on pr open and on commit

Edge Cases

Perfect user stories

❌

Risks/Dependencies identified

• API gateway needs lambda auth to verify Github is sending request

Technical Strategy is known

✅

Uncertainties are known

✅

Dependencies are unblocked

✅

Testing strategy is known

• integration test the endpoints

Description:
We want to repackage the review functionality of our app into a lambda which will eventually be called by a Github API.
The npm package should still be functioning after this ticket!!!

Validation:
Test in the console and show a review is returned when you give it some code. Screenshot

Tech step:
in root/services -> create cdk app (core)
in functions/ create review lambda -> config (construct), index (handler)
stacks/ in the core stack -> call the review lambda construct
test in the console.
weird thing with tsconfig

Tech steps

• Create a git folder in src/common
• Create functions for git commands (now in src/review/prompt/gitCommand.ts)
• Create a function that uses the github env variables and returns a list of objects of type {fileName:string, fileContent:string, changedLines: string}
• (Write tests for these functions ?)
• In global index.ts, in case "review", call the git function first, and pass in the list of objects in the review function

Description:
Currently we have an open /postReview endpoint (dependent on #70 ) We need to validate requests to this endpoint to check they are actually coming from Github and importantly that the repo they are coming from is requested to authenticated user on our platform.

Validation:
Screenshot

Tech steps:

• Add GITHUB_SIGNATURE_HEADER_KEY to the lambda environment, and populate it in the paramstore.
• In the review lambda, create a authenticate function
• Use the auth function as a gaurd before doing anything else.
• Use the github example as our auth function - Use this https://docs.github.com/en/webhooks-and-events/webhooks/securing-your-webhooks#validating-payloads-from-github

Description:
Some formatting with the comment to make it look nicer.

No one knows what LOGAF means. Use Risk and rate 1 low -> 5 high.
Also add Review to the sign off

Debug log should show the comand used, flags, model, prompt, and response. This should be enabled by Github actions running in debug mode.

Use tslog to define a minLevel on the logger that is

• 4 when running on the ci (only warn, error and fatal logs)
• 3 when running locally (info, warn, error, and fatal)
• 2 when running with debug (debug, info, warn, error, and fatal)

We want to make sure that the npm package we release remains up to date.

Description:
First ticket for backend saas. The idea is to have a deployed dynamodb table to AWS.

Validiation:
Screenshot of dynamo table in console.

Tech steps:
in services. cdk init --typescript core
in a resources folder add a dynamo table: PK: pk, SK:sk.
standard setup, check in a previous project. RemovalPolicy to be a function of the stage... billing mode to pay per request.
helper functions

Store all changes in a local vector store using chroma db

Description:

Working CI pipeline so we can be sure of our changes with automatic tests and automatic push to prod (npm/aws deploy).

Add templates for issues.

Description:
Use next auth with Github as a provider. Register a new app with github. Orion Tools Auth. Add next auth to the site and it will add a login screen.

Validation;
Video of sign in flow with Github

Tech:
https://authjs.dev/getting-started/introduction
Follow the tutorial in the above link.
for stuff with github app as @mattzcarey for any details

currently we are doing a bunch of git commands around the code base they should be in one place.

Description:
Make a demo review lambda which takes code and reviews it. Use apiKey and Langchain Smith env variables from parameter store.

This is different from the Github Review lambda

Validation:
screenshot testing the lambda in console.

Tech steps:
arguments: {
code: .....
}

return: {
reviewedCode: ...
riskLevel: ....
}

Functionality:

1. get the openai apikey and langsmith API key from SSM, add to the env variables of lambda
2. check the length of the code is not outside the context window.. If not return error.
3. append the prompt to the code
4. return a result to the frontend

We need to have backend endpoints to get user details and updated user details.

On frontend we need an admin dashboard thing which allows a user to see the repo connected to the account and update the API key we have on file.

Tech Steps:

• In src/common/model create a file called createMemoryStore.ts
• Create a function called createMemoryStore which takes a single argument, initialFiles and returns a VectorStore
• Write test to make sure it works as expected

Description:
Need to be able to sign out dependent of #64

Validation:
Video recording

Tech steps:
https://next-auth.js.org/getting-started/client#signout

Make a button on the user page which appears when signed in.
use the sign-out method from next-auth.
redirect to home page.
Display users email once signed in, to show when someone is signed in and who is signed in.

Description:
We want users to be able to update their openai API key that we store on file as part of their user in dynamo.

Validation:
Screenshot in postman sending post request to the updateUser endpoint with field apiKey. Recieve a 200 response.

Tech steps:
create API stack
create API gateway
add a post method to the updateUser endpoint
add a lambda integration to trigger the lambda.

• in args add an option line-by-line commenting
• in review/index.ts make a const isLinebyLine using the boolean from argv
• in constants.ts add to the main prompt a fileLine attribute to the returned json
• log that and test locally
• in comment on PR add the line by line commenting behind the isLinebyLine command (see dans draft pr for info)

This should all be in one place. Currently they are in test and review.

We are currently assuming that the model response is a valid markdown with the correct template. We should make sure that the response is valid and if not regenerate the response to prevent malformed result

Rank feedbacks and only return first 5 instead (or less) of spamming with comments

• Create a prompt to rank the feedbacks
• Function to call the model and rank the feedbacks
• Filtering logic to limit to the first 5 if they exist

ERROR Error in callModelJSON
SyntaxError Unterminated string in JSON at position 7947
error stack:
•

• AIModel.ts maxTry
/src/common/model/AIModel.ts:35
• task_queues processTicksAndRejections
internal/process/task_queues:95

Move all logs to use Tslog.

Description:
Need to return: userId to the frontend

Problem we're trying to solve is clear

User needs to be able to manage their installed repos (add new repo, remove repo, change API key)
User needs to be able to see how many requests have occurred for each repo and any errors.

Success Measures

• page accessible only to logged in users with the above functionality

Scope of feature

• page deployed to oriontools.ai/user

Edge Cases

Perfect user stories

❌

Risks/Dependencies identified

• no UI design

Technical Strategy is known

✅

Uncertainties are known

✅

Dependencies are unblocked

✅

Testing strategy is known

• integration test the endpoints

make folder services perform create-next-app.

Follow instructions on SST website to add sst in dropin mode host.

Site should have a title only on the main page. Tailwind, app router. all the good stuff.

Page to direct users to.

This should contain a logo, title, and installation instructions.

Also contain contact info to aleios and the repo

Tech stack nextjs app with sst hosting.

Problem we're trying to solve is clear

Creating a landing page which has installations instructions, an about the tool and a demo video, and a call action to try demo

Success Measures

• a landing page deployed to oriontools.ai

Scope of feature

• a pretty page with the info above shown. link to the demo and the repo

Edge Cases

Perfect user stories

❌

Risks/Dependencies identified

• no UI design

Technical Strategy is known

✅

Uncertainties are known

✅

Dependencies are unblocked

✅

Testing strategy is known

• no testing needed for a static page

Auth service to allow users to login to the platform online.

BE:

Problem we're trying to solve is clear

Auth service to allow users to login to the platform online.

Success Measures

• be able to sign up, login, and logout via postman

Scope of feature

• auth
• database to store auth details

Edge Cases

Perfect user stories

❌

Risks/Dependencies identified

• never used Next auth personally but used internally before

Technical Strategy is known

✅

Uncertainties are known

✅

Dependencies are unblocked

✅

Testing strategy is known

• integration test the endpoints

Description:
We need a webhook url to supply to the github app in github. This URL should be apigateway /postReview and forward on the request to the review Lambda.

Validation:
install the app in an example repo. Make a PR and see that the review lambda is triggered (going into it's own ticket)

• Screenshot of postman 200 response

Tech steps:

• create API gateway in API stack
• make post method postReview
• lambda integration to the review lambda.
• deploy and and get the URL.
• test in postman
• then give URL to matt to add to github app.

Hello, cool project! Any plans to support gitlab?

For v3 we want to support people just paying for us directly. We handle the AI API setup.

Use 3rd party with Stripe to handle the payment.

We currently ask for GPT to select the most important comments according to the model. The issue with that is that if we have several files, we often end up with the same comment several time

We need to save user data.

We need a dynamo db auth table. Make this in the same stack as the web-app service with sst

Validation:
Screenshots of a deployed dynamodb table

When using this tool locally, it would be nice to be able to specify the commit SHAs to be reviewed instead of only working on staged files. I've very briefly looked at some of the source code and I think the least effort way to do this would be use most of the current CI logic but without involving github and the github token. Does this seem viable?