data-connector-server's People
Forkers
snyatadata-connector-server's Issues
[New Connector] Google Portability API (Search history)
Add support for Google's Data Portability API.
Start by implementing support for saving a user's search history.
Notes:
- Need to request the scope
https://www.googleapis.com/auth/dataportability.myactivity.search
- No other scopes can be requested at the same time as the data portability API, so it requires it's own connection / provider
- Documentation is here https://developers.google.com/data-portability
![Screen Shot 2024-07-22 at 6 47 09 pm](https://private-user-images.githubusercontent.com/164973/350909152-3d994806-9454-4cef-8c64-86149ce6b5ba.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM2NDI3MzksIm5iZiI6MTcyMzY0MjQzOSwicGF0aCI6Ii8xNjQ5NzMvMzUwOTA5MTUyLTNkOTk0ODA2LTk0NTQtNGNlZi04YzY0LTg2MTQ5Y2U2YjViYS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODE0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgxNFQxMzMzNTlaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1hNTllZjNjMjM5Njc3ZDM0ZTlkNGNiMTNiNmJjMjA3MzY2ODQ1NGExYzMyYzEwNTE2MjhkNzJiZmRhZDMzMTQyJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.aR0AJevfrPLMNwxNusGDy4dwlGtlgm6ebrl1ABv1UrI)
Refactor to use a single sync() endpoint
We are changing the security model to assume the data connector server is operating within a secure enclave.
As such, this server can be modified to accept a single sync()
command that accepts a user's DID and their Verida: Vault
private key.
Add youtube handler inside google provider
Create basic dashboard
Requirements:
-
/dashboard/connections
Connect a provider, Sync a provider, View sync status -
/dashboard/data
View data for a schema, List common schemas
Support signing API responses when running within a secure enclave
Fetch the file at /app/id.sec
which is an ed25519
key and use that to sign all API responses.
If that file doesn't exist, don't sign responses.
Handle sync errors and timeouts
Need to handle the following scenarios:
- Sync has an error, retry X times, then fail permanently
- Sync has lost credentials, user needs to re-authenticate
- Sync timed out and needs to be reset after X time
Obsidian connector
Implement Youtube connector
Implement Youtube handlers in the Google provider.
Phase 1:
User the facebook/following.ts
and the google/gmail.ts
files as examples to following when building the Youtube handler.
- Create a new
youtube-following.ts
handler within the existingGoogle
provider - Map Youtube subscriptions data to the Verida
following
schema (https://common.schemas.verida.io/social/following/v0.1.0/schema.json) Seefacebook/following.ts
for example. - Add the necessary youtube scopes to the existing Google provider
getScopes()
method. - Add a
getYoutube()
method into theyoutube-following.ts
library to instantiate a connection to the Youtube library.
Then, let's have another call to work on next steps.
Refactor for efficiency
The current architecture has the server create a new database for every sync request.
It then provides the Wallet access to that database, so the wallet can sync this new data. This is inefficient from a storage perspective, creates timing issues and makes it difficult to make the sync data in a way to only fetch the new items.
Now that we support accessTokens
, we can provide time limited access for the server to read and write directly to user's vault storage collections. This does increase the security risk surface area a little, but the performance and user experience trade-offs are worth it.
It will work like this:
- User authenticates via the data connector server (no change)
- Verida Vault saves the user's
accessToken
andrefreshToken
for the connection into it's own private database of connections (no change) - Verida Vault periodically calls a
sync()
method on the data connector server - Data connector server handles the
sync()
request, pulls the latest data from the connector API (ie: facebook) and saves the new data directly into the user's Verida Vault database
In order for this to work, the data api connector sync()
method will need to accept the following parameters:
veridaDatabaseEndpointUri
The endpoint of the Verida database to connect toveridaDatabaseAccessToken
The Verida database access token that has 20 minute expiryveridaDatabaseEncryptionKey
The Verida database encryption key that is used to encrypt / decrypt data from the Verida databaseaccessToken
The access token for the connection API (ie: facebook)refreshToken
The refresh token for the connection API (ie: facebook)
Where veridaDatabase
is an encrypted database on the Verida network, owned and controlled by the Verida Vault application.
[New handler] Google Drive Documents
Add GSuite drive files into a new "documents" datastore.
Request the scope https://www.googleapis.com/auth/drive.readonly
Support the following document types:
- Docs (Google documents, Word documents, PDFs)
- Slides
- Spreadsheets
We don't want to store the actual document, but do want to store the text of the document and a link to the actual document.
Note: PDF files can be converted to text with this code:
Session data isn't being saved
The connect()
endpoint sets session data which is then accessed in the callback()
endpoint.
However, as you can see when running the code, the session information isn't retained.
This is a problem as the redirect
URL is specified in connect()
, but then can't be used in callback()
to redirect the user back to the relevant application that initiated the connect request.
Is there an issue with session
config / initialization?
Steps to reproduce:
- Run the server (
yarn run start
) - Hit the
/connect
endpoint (ie:http://localhost:5021/connect/discord?did=did:vda:mainnet:0xCDEdd96AfA6956f0299580225C2d9a52aca8487A&key=abc82b917d4f44708f35a618247e70d5243b1c66c0b60a939bca0bc67eadddef&redirect=http://localhost:3001/connections
) - Hit the
/callback
endpoint (ie:http://localhost:5021/callback/discord?code=abc123
) - See the console output on the server and you'll see the session data from
/callback
endpoint doesn't match/connect
endpoint
[New Handler] Google Calendar
Add support for syncing Google Calendar data.
Required scopes:
- https://www.googleapis.com/auth/calendar.readonly
- https://www.googleapis.com/auth/calendar.events.readonly
API docs:
https://developers.google.com/calendar/api/guides/overview
Will require creation of new schemas:
- calendar
- event
Update to latest protocol v2.1.3
[New connector] Implement Telegram connector
Investigate implementing a Telegram connector.
- Pull down all the user's chatrooms
- Pull down all the user's chats
Will need to design a schema for each of those data sets.
Telegram supports the tdlib
library that exposes full access to Telegram functionality, enough to support creating new Telegram clients.
There are node.js wrappers around this library:
- https://www.npmjs.com/package/telegram
- https://github.com/alik0211/mtproto-core (types: https://www.npmjs.com/package/telegram-api-schema-methods)
- https://www.npmjs.com/package/tdlib-native
- https://www.npmjs.com/package/tdl
- https://www.npmjs.com/package/@telepilotco/tdl
- https://www.npmjs.com/package/prebuilt-tdlib
Telegram doesn't support OAuth and has rather complex auth options, so we will need to implement a specific telegram page in the data connector server that handles auth.
Twitter app breaks auth
Platform: iOS
If a user has the official Twitter app installed and attempts to authorize the data connector, the Twitter app hijacks the authorization request, opens the Twitter app and shows the consent screen in an embedded browser in the Twitter app.
That is fine, however, when the user clicks authorize they are redirected back to the data connector server -- but the query params appear to have been stripped so the data connector server can't obtain the tokens.
Memory leak?
Production server seems to have a memory leak that spirals out of control.
Make existing HTTP link to "Complete Connection" a redirect
Currently we make the user click a link to "complete connection" Instead we can either do a HTTP 302 redirect to that link or a Javascript redirect to https://vault.verida.io with appropriate parameters.
https://stackoverflow.com/a/30001343/280795 has an example of the Javascript redirect (in that case to a custom scheme, but should work the same for HTTP)
This can be replaced by a custom Vault scheme in #9
Switch to using veridavault URL scheme
Once https://github.com/verida/vault-mobile/issues/599 is complete in the Vault, switch to using that instead of the HTTP link vault.verida.io
Notion connector
Otter.ai connector
Support multiple connections for the same provider
A verida identity can currently only have a single connection per provider, however we need to support multiple.
For example; A user may want to connect multiple Google accounts.
Support API versions
Improve sync handling
- Handle errors
- Handle sync timeout
- Handle sync not completing
Support generating profile VCs
Support syncing all user data for facebook and twitter
We currently limit requests to a single page (20 results).
Need to implement pagination via the underlying API's to support syncing much larger data sets (ie: The full history of a user's posts).
Still have a maximum of 3,000 entries for now.
Ideally implement in a way that allows for the pagination pattern to be generic code that can be reused across other API's.
Support minting SBTs
Improve facebook implementation
Add support for pulling user posts
Support access token instead of private keys
Currently the /connect
and /sync
endpoints expects the private key (or seed phrase) of an identity.
This was just for the PoC and is poor security practice. This needs to be upgraded to accept either one of:
- Recent
access
token for theVerida: Vault
context (Permits time limited read / write access) - Keychain entropy for the
Verida: Vault
context (Permits permanent read / write access)
Fireflies connector
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.