Comments (26)
Here is a sketch of how to use webhooks with Grist. The procedure isn't as smooth as I'd like, which is why I'm writing it in a github comment rather than our support site:
-
If using hosted Grist, check that the webhook you need is for a domain listed in #76 (comment) - if it is not, ask Grist staff to add the domain. If self-hosting, follow the instructions in that comment for setting
ALLOWED_WEBHOOK_DOMAINS
. -
Make a practice Grist document that has a table called
Stuff
, a text column calledName
, and a toggle column calledReady
. Make a few rows. Make sure all Readycells
are unset/blank/off - it will represent whether a row is available for sending to the webhook. You can read about readiness columns here: https://support.getgrist.com/integrators/#readiness-column. They are optional, but make testing simpler. All table and column names are arbitrary and have no special significance. -
Get comfortable using the Grist API, documented at https://support.getgrist.com/rest-api/, e.g. do some exercises to add rows to that table (be sure to replace
XXXX
with your api key andDDDD
with the document identifier in the document’s URL, and the http://docs.getgrist.com part if self-hosting):
curl -H "Authorization: Bearer XXXX" -H "Content-Type: application/json" \
https://docs.getgrist.com/api/docs/DDDD/tables/Stuff/records \
-XPOST \
-d '{"records": [{"fields": {"Name": "Test"}}]}'
- Practice adding and removing a dummy webhook via the API. A handy place to get one is webhook.site, you can visit there and click “copy to clipboard” beside “Your unique URL”.
- Here’s how to add a webhook, using the unofficial
_subscribe
endpoint. Be sure to replace my example webhook.site url with your own one:
curl -H "Authorization: Bearer XXXX" -H "Content-Type: application/json" \
https://docs.getgrist.com/api/docs/DDDD/tables/Stuff/_subscribe \
-XPOST \
-d '{"eventTypes": ["add", "update"], "url": "https://webhook.site/f02fdeea-6b67-4a03-932a-1a13f8d9e020", "isReadyColumn": "Ready"}'
- The
eventTypes
list is something you can tweak - includeadd
to get new rows (once they are marked ready), andupdate
to get changed rows, and both to get both kinds of event. - You should get back a result with some important keys, like this:
{"unsubscribeKey":"60fe7777-d7f8-4521-8e77-b6a9f93b0588","triggerId":1,"webhookId":"f717d5ce-5c26-4d3b-ac4e-49b5dbd93761"}
- These are important to keep track of, because you need them to remove a webhook. There’s no way currently to remove a webhook without them.
- Now, toggle a cell in the
Ready
column to ON for one of the rows in the document. You should hopefully see something arrive on your webhook.site page! Something like:
[
{
"id": 6,
"manualSort": 6,
"Ready": true,
"Name": "Miguel de Cervantes"
}
]
- It takes a long time for information sent to a bad webhook to be abandoned, if anything goes wrong, you can reset the whole process and clear any faulty requests that might be queued for a document by calling:
curl -H "Authorization: Bearer XXXX" -XDELETE https://docs.getgrist.com/api/docs/DDDD/webhooks/queue
- Go ahead and practice removing the webhook by sending the unofficial
_unsubscribe
endpoint exactly what_subscribe
returned earlier:
curl -H "Authorization: Bearer XXXX" -H "Content-Type: application/json" \
https://docs.getgrist.com/api/docs/DDDD/tables/Stuff/_unsubscribe \
-XPOST \
-d '{"unsubscribeKey":"60fe7777-d7f8-4521-8e77-b6a9f93b0588","triggerId":1,"webhookId":"f717d5ce-5c26-4d3b-ac4e-49b5dbd93761"}'
- You should get a result with
"success": true
in it back. Okay! After that practice, we’re ready to try the real thing. - Log in to Pabbly Connect, click
Create Workflow
, name the workflow, then click theWebhook
trigger. - Click
Copy
to get the Webhook URL in your clipboard. It should look like:
https://connect.pabbly.com/workflow/sendwebhookdata/XXXXXXXXXX (https://connect.pabbly.com/workflow/sendwebhookdata/XXXXXXX_3D)
- Make sure to set the
Simple Response
toggle to OFF! Grist will send the data with some nested structure that could otherwise confuse Pabbly. - Now, repeat the
_subscribe
call we did earlier, but with the URL replaced with the real thing (https://connect.pabbly.com/workflow/sendwebhookdata/XXXXXXX_3D). - Keep the API result you get in case you want to later call
_unsubscribe
. - Now, toggle a cell in the
Ready
column to ON for one of the rows in the document. - Go back to Pabbly, and you should see that information arrived! You can go ahead and map it using the usual Pabbly process.
@HermesRatgers I'd be interested to hear how this goes for you, and if you hit any blockers or points of confusion.
from grist-core.
@paulfitz Added it and it works, I also added this thread to Pabbly so they can add the webhook aswell. Btw the integration can be managed by you aswell, simply request this by email them your pabbly emailadres and request editor permission.
https://pabbly.hellonext.co/b/Update-Existing-Application/p/webhook-triggers-for-getgrist
from grist-core.
Just to add some extra clarification, a column of type Date will always return a number like that no matter what Date Format you set in the column options, but a formula column of type Text or returning a Python string as in the suggestions will store the same string and return it in the API so it'll match what you see in the UI.
from grist-core.
Give me /events, not webhooks is interesting and may be easier to implement and support.
from grist-core.
@simplynail do you mean a UI for creating webhooks? No, there's no ETA for that, sorry. We do want to get it done when we have capacity, I know that isn't very helpful for you.
from grist-core.
Update: webhooks are available from the UI now https://support.getgrist.com/newsletters/2023-05/#webhooks
There is no longer any restriction on the domains our hosted service will deliver to. Self-hosted Grist can be configured similarly by setting ALLOWED_WEBHOOK_DOMAINS
to *
but, depending on your trust model, you may either want to:
- Stick with explicitly allowing certain domains, or
- Configure a secure proxy (and tell Grist to use it with
GRIST_HTTPS_PROXY
) to deliver webhook payloads.
A threat to watch out for is accidentally allowing untrusted users to poke around at your internal endpoints.
from grist-core.
This is getting closer. If anyone would be interested in being a beta tester of webhooks (and particularly Zapier instant triggers), we'd like to hear from you.
from grist-core.
This is getting closer. If anyone would be interested in being a beta tester of webhooks (and particularly Zapier instant triggers), we'd like to hear from you.
I am interested, but mainly for Pabbly Connect.
from grist-core.
Great, thanks @HermesRatgers. We'll add Pabbly Connect to the domains allowed for webhooks, and prepare instructions for how to get this set up. Are you comfortable using Grist's API (https://support.getgrist.com/rest-api/)? One of the steps will involve making a manual API call, since we don't have a way to edit webhooks visually in the app yet.
from grist-core.
I know how to make api calls (I even know how to create integrations from within Pabbly)
from grist-core.
@paulfitz Can the date be changed? In webhook I receive a date as Birthday | 494726400 in stead of 1985-09-05
from grist-core.
@HermesRatgers if something isn't formatted as you like, one option could be to add another formula column that reformats it. For example, if you have a column called Date with dates in it, you could make another column called (for example) DateString that is just =str($Date)
. Would that work for you? There's some more information on formatting dates on the support site https://support.getgrist.com/dates/#additional-resources
from grist-core.
@paulfitz No, thats not it. The date is Grist is formatted and displayed correct as 1985-09-05. But in the webhook its formatted as 494726400.
from grist-core.
Grist stores dates as seconds since the Unix epoch (https://en.wikipedia.org/wiki/Unix_time), and then formats and displays them as the user wishes in the UI. With webhooks, you are currently getting the data as Grist stores it. There's no way to control that right now, so I'm wondering if a workaround might be to add a second column that calculates a value that has the formatting you want applied to it?
We're looking at giving more control over how data is rendered across our APIs, and for webhooks.
from grist-core.
Hi, is there any ETA for the webhooks feature coming in?
from grist-core.
Hi, yes, I meant UI actually. I just was wondering if it's still on your radar and perhaps coming soon.
Thanks
from grist-core.
I'd like to propose (hope this is the right place) two new domains to be added to the ALLOWED_WEBHOOK_DOMAINS list:
- amazonaws.com
- on.aws
this will allow to configure webhooks that can reach AWS API Gateway and Lambda Function URLs endpoints, allowing developers (possibly many of them) to easily test custom integrations.
For reference:
- Lambda Function URL structure is: https://<url-id>.lambda-url.<region>.on.aws
- API GW URS structure is: https://<api_id>.execute-api.<region>.amazonaws.com
See here, here and here for the official AWS documentation.
Thanks
from grist-core.
Hi @fabiomoratti, this is the right place to ask. Normally we can can add domains to an allow list pretty quickly. But access to AWS endpoints in particular may need to wait for a little refactor before we're comfortable that it doesn't create a new kind of attack surface when Grist itself is hosted on AWS infrastructure. Sorry for the delay.
from grist-core.
@paulfitz , thank you for your reply.
In a way, amazonaws.com
and on.aws
may be considered "general-purpose" domains useful for all the users that deploy their webhooks on AWS, as an alternative it should be possible, on my side, to configure a custom domain for the AWS API GW / Lambda and add it, on your side, to the ALLOWED_WEBHOOK_DOMAINS; of course this solution does not scale because it works for one customer (me, in this case) and on the long run it's not reasonable for you to maintain a long list of allowed domains. Given the above, what is your sentiment about addirng a custom domain while the refactoring is being implemented?
As for alternatives I understand that the way to go is polling: do you have any suggestion for a reasonable polling interval that does not disturb Grist's infrastructure? API calls are limited to 5000 per day per document so one call per minute is well below the limit, but it's worth checking.
Thank you in advance for your support.
from grist-core.
Hello, thanks very much for making Grist. It's been very easy to work with and I just found out about the support for outgoing webhooks. This is a really nice feature.
I was wondering if you could add netlify (api.netlify.com
) to the list of supported domains?
Thanks.
from grist-core.
@acidturtle I added support for netlify.com
to the queue, should take effect on November 28, 2022.
@fabiomoratti we could add your custom domain, as a stop-gap (write [email protected] and mention my name if you don't want to post it here).
from grist-core.
@paulfitz: thank you for the quick response and for adding netlify.com
, much appreciated.
from grist-core.
- Prepare to do a practice run on a fresh, disposable Grist document, rather than the document you ultimately intend to use. This is because it can take a long time for information sent to a bad webhook to be abandoned, and there’s no easy way yet to reset that process. So it is better to practice on something low-stakes.
There is now a new API endpoint (still unofficial) that can reset the whole process and clear any queued requests for a document.
curl -H "Authorization: Bearer XXXX" -XDELETE https://docs.getgrist.com/api/docs/DDDD/webhooks/queue
from grist-core.
We added a new endpoint /webhooks
. It returns some basic configuration and monitoring information about all webhooks registered in a document. It is available currently only for document owners
. You can query it using:
curl -H "Authorization: Bearer XXXX" -XGET https://docs.getgrist.com/api/docs/DDDD/webhooks
It returns a JSON with a following schema:
{
webhooks: [
{
id: string, // webhook id
fields: {
url: string,
unsubscribeKey: string,
eventTypes: string[],
isReadyColumn?: string|null,
enabled: boolean,
tableId: string,
},
usage: {
lastEventBatch?: { // last attempt information
status: 'success'|'failure'|'rejected',
httpStatus: number,
errorMessage: string | null,
size: number, // number of requests in the batch
attempts: number, // number of attempts
},
lastSuccessTime?: number, // last time the webhook request succeeded
lastFailureTime?: number,
lastErrorMessage?: string | null, // last error message and HTTP status
lastHttpStatus?: number|null,
updatedTime?: number|null,
numWaiting: int, // how many requests are queued currently
status: 'idle'|'sending'|'retrying'|'postponed'|'error'
}
}
]
}
Webhook status description:
idle
- there are no active requestssending
- there is an active requestretrying
- there is an active request, but the last attempt failedpostponed
- all attempts have failed, webhook was added to the end of the queueerror
- all attempts have failed, but the queue is too long, so some requests were dropped.lastEventBatch
will haverejected
status.
We also modified the /_unsubscribe
endpoint. Now, it only requires knowledge of the webhook id (if it is called by the owner) or the webhook id and unsubscribeKey otherwise. Owners can query both these keys using the new /webhooks
endpoint, so there is no need to memorize them anymore.
from grist-core.
@fabiomoratti we could add your custom domain, as a stop-gap (write [email protected] and mention my name if you don't want to post it here).
@paulfitz: thank you for your support; I've been playing both with Grist APIs and webhooks and I'm confident I can make it work so I'll write to the mail you mentioned to have my custom domain added waiting for the refactor.
Additionally if you feel it may be useful/interesting for other people I'm willing share the code (a PoC-like version) I use to implement the AWS Lambda webhook; I was going for a public GitHub repo but if you have a better place just let me know.
from grist-core.
@fabiomoratti we could add your custom domain, as a stop-gap (write [email protected] and mention my name if you don't want to post it here).
@paulfitz: thank you for your support; I've been playing both with Grist APIs and webhooks and I'm confident I can make it work so I'll write to the mail you mentioned to have my custom domain added waiting for the refactor. Additionally if you feel it may be useful/interesting for other people I'm willing share the code (a PoC-like version) I use to implement the AWS Lambda webhook; I was going for a public GitHub repo but if you have a better place just let me know.
Your custom domain will be supported from Monday. Sharing your PoC in a public github repo sounds great, thanks @fabiomoratti !
from grist-core.
Related Issues (20)
- decimal number entry broken with locale "Denmark (English)" HOT 6
- UI for 'Terms of use' modale HOT 4
- Mandatory field for widgets
- Test
- self-hosted: hide snapshot UI if not enabled
- FR: search all users registered on closed self-hosted instance
- Work out a design for two-way references HOT 1
- Design UI for the first version of notifications HOT 3
- Access Rules for Pages HOT 2
- test
- Setting GRIST_ANON_PLAYGROUND=false should auto-redirect to login
- SAML extra fields
- Only radio buttons available for Reference List columns in Form Widget
- Form Widget does not follow column dropdown filters
- Make a better UI for charts
- Allow configuring start of the week to either Sunday or Monday HOT 2
- Forms do not respect table access rules
- row height expands even if word-wrap is unselected HOT 2
- Force-reload may raise a 429 (TOO_MANY_REQUESTS) error HOT 3
- EACCES: permission denied (container restart cyclically after update from 15.07.2024 for latest tag) HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from grist-core.