faros-ai / airbyte-connectors Goto Github PK

Pull same data as Okta connector (org_*)

#160

Pull cicd objects

https://docs.microsoft.com/en-us/rest/api/azure/devops/pipelines/?view=azure-devops-rest-6.0

A good default is to do the same as our GitHub connector: create a project and a board for each repository. Tasks (issues) should then link to those.
https://github.com/faros-ai/airbyte-connectors/blob/main/destinations/airbyte-faros-destination/src/converters/github/common.ts#L92

On Faros CE, Buildkite connections cannot write cicd_BuildCommitAssociation entries because its converters need to properly set the uid, somewhat similar to this PR:
146cb16#diff-78e89bf564fd6dbf0d497de07baf220468bbee05a9bc73996c698f136970691aL47

This shall be fixed for all converters writing that entity.

2022-04-27 22:59:27 destination > Error processing input: Failed to write cicd_BuildCommitAssociation record: [{"extensions":{"path":"$.selectionSet.insert_cicd_BuildCommitAssociation_one.args.object[0].vcs_Commit.data.vcs_Repository.data","code":"constraint-violation"},"message":"Not-NULL violation. null value in column \"uid\" of relation \"vcs_Repository\" violates not-null constraint"}]

Pull vcs objects from Azure
https://docs.microsoft.com/en-us/rest/api/azure/devops/git/?view=azure-devops-rest-6.0

Please see if this draft is useful to work off of #122

Add a Harness source and converter based on Harness feed

Write a source and Faros converter based on the Changeset feed

Probably a bug in pagination?

We would like to build a connecter and converters for OpsGenie. To start, this should be a similar connector to that of PagerDuty as we are interested in incident data.

Data to retrieve:

• Incidents
• Users

API Docs

Note: Be aware of rate-limiting when using node.js API client as it does not appear to handle 429 responses

Auth can be tricky, first iteration can just have basic auth

https://developer.nulab.com/docs/backlog/#

Create Victorops source and converter based on the Victorops feed.

It's a bit convoluted, to work around API quirks, apologies in advance.

https://octopus.com/docs/octopus-rest-api

https://appexchange.salesforce.com/listingDetail?listingId=a0N30000000ps3jEAA

Should be similar to Circle CI (Will to confirm)

We need to write cicd_BuildCommitAssociation objects from jenkins when available. See feed as reference on how this is done.

Currently its hardcoded to 20000, but some issues have descriptions that cause Squadcast API responses to go past that length. Make this a configurable source option.

Hi @huongtn!

As it was discussed with you, please add the start_date argument to the following connectors (where applicable):

Merged:

1. Agile Accelerator
2. Backlog
3. BitBucket
4. Customer – io
5. Datalog
6. Docker
7. FireHydrant
8. Google Calendar
9. Harness
10. Jenkins
11. Okta
12. PagerDuty
13. Phabricator
14. Shortcut
15. SquadCast
16. StatusPage
17. VictorOps

Not Merged:

18. Azure Active Director
19. Azure Git
20. Azure Pipeline
21. Bamboo
22. BuildKite

"start_date": {
        "type": "string",
        "title": "Start Date",
        "description": "The date from which you'd like to replicate data for GitHub in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated",
        "examples": ["2021-03-01T00:00:00Z"],
        "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$"
 }

Example of this argument in GitHub -
https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/spec.json#L65-L71

Also, attaching here the initial description of the request from Matthew:
https://devcube-group.slack.com/archives/C01U644P2P4/p1647962158678039

A good default is to do the same as our GitHub connector: create a project and a board for each repository. Tasks (issues) should then link to those.
https://github.com/faros-ai/airbyte-connectors/blob/main/destinations/airbyte-faros-destination/src/converters/github/common.ts#L92

https://www.perforce.com/

https://docs.microsoft.com/en-us/graph/api/user-list?view=graph-rest-1.0&tabs=http

In the original Bitbucket feed we write out repository information as follows - https://github.com/faros-ai/feeds/blob/master/feeds/vcs/bitbucket-feed/src/feed.ts#L143-L154

We need to add the same object into Butbucket source as well.

Please also go through the sources and add descriptions for all the fields in the spec.json files that don’t currently have descriptions.

cc: @cjwooo

Running build from the root of the project fails with tsc

> [email protected] build /airbyte-connectors/faros-airbyte-cdk
> tsc -p src

src/destinations/destination-runner.ts:87:19 - error TS2339: Property 'message' does not exist on type 'unknown'.

87                 e.message ? e.message : e
                     ~~~~~~~

Add converters for Jenkins streams in Faros Destination

Applies to Faros Community Edition

How to reproduce:

• on a GitHub connection, sync with the canned streams
• delete reviews and review_comments streams, trigger the reset

the reset fails with the following error
2022-04-06 17:04:22 ERROR i.a.c.i.LineGobbler(voidCall):82 - Failed to reset vcs_PullRequest data for origin mygithubsrc: [{"extensions":{"path":"$","code":"constraint-violation"},"message":"Foreign key violation. update or delete on table \"vcs_PullRequest\" violates foreign key constraint \"vcs_PullRequestComment_pullRequest_fkey\" on table \"vcs_PullRequestComment\""}]
which is likely because the reset does not delete the data from the removed streams.

Workaround once in that situation: update source schema, reset, delete unwanted streams.

Resolve locations using Faros /geocoding/lookup API - https://dev.api.faros.ai/docs.html#Geocoding.

Example from org import feed - https://github.com/faros-ai/feeds/blob/master/feeds/org/org-import-feed/src/feed.ts#L346