`datasette publish vercel` having issues with exporting JSON about datasette-publish-vercel HOT 18 CLOSED

Well this is a fascinating bug! I've never seen it before, and I can't think of anything obvious that would cause it. I will dig in and see if I can replicate it.

from datasette-publish-vercel.

Only current workaround I can suggest is trying Heroku or Cloud Run instead, but that's deeply unsatisfying.

from datasette-publish-vercel.

CSV is affected by this too: https://thesession.vercel.app/thesession.csv?sql=select%20*%20from%20tunes%20where%20name%20like%20%22%wise%20maid%%22

from datasette-publish-vercel.

... but the .json and .csv links on the table page at https://thesession.vercel.app/thesession/tunes behave correctly - so this bug only affects arbitrary SQL queries.

from datasette-publish-vercel.

It's the % characters.

https://thesession.vercel.app/thesession.json?sql=select+*+from+tunes+where+name+like+%3Asearch%0D%0A&search=Wise+Maid%2C+The returns results correctly

So does https://thesession.vercel.app/thesession.json?sql=select%20*%20from%20tunes%20where%20name%20like%20:search&search=%Wise%20Maid,%20The%

This does NOT work: https://thesession.vercel.app/thesession.json?sql=select%20*%20from%20tunes%20where%20name%20like%20:search&search=%wise%20maid%

But this does: https://thesession.vercel.app/thesession.json?sql=select%20*%20from%20tunes%20where%20name%20like%20:search&search=%25wise%20maid%25

So in some circumstances a % that is not correctly URL-encoded works - but in other circumstances it does not and must be encoded as a %25.

from datasette-publish-vercel.

By Jove, I think you've cracked it!

from datasette-publish-vercel.

The JSON from https://thesession.vercel.app/thesession.json?sql=select%20*%20from%20tunes%20where%20name%20like%20:search&search=%wise%20maid% has another clue:

{
  "ok": true,
  "query": {
    "sql": "select * from tunes where name like :search",
    "params": {
      "search": "wise maid"
    }
  },
  "error": null,
  "private": false,
  "allow_execute_sql": true,
  "query_ms": 18.020902999978716
}

Note that the "search" parameter is shown as "wise maid" without those % wildcards. In https://thesession.vercel.app/thesession.json?sql=select%20*%20from%20tunes%20where%20name%20like%20:search&search=%25wise%20maid%25 (the same thing but with % in the URL encoded as %25 it comes back as "%wise maid%".

I think there may be two bugs here.

Firstly, I suspect Vercel have a weird bug where under certain circumstances raw % in a query string fail to be passed through to the underlying application.

But... Datasette itself shouldn't be generating links to pages where the links have a % rather than a %25 in them.

from datasette-publish-vercel.

Confirmed: https://thesession.vercel.app/thesession?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A is a page where the URL correctly encoded % as %25 - but then in the HTML on that page that links to the CSV and JSON versions we get this:

<p class="export-links">This data as
  <a href="/thesession.json?sql=select * from tunes where name like &#34;%wise maid%&#34;
">json</a>,
  <a href="/thesession.csv?sql=select * from tunes where name like &#34;%wise maid%&#34;
&amp;_size=max">CSV</a>
</p>

Those CSV and JSON links are incorrect.

from datasette-publish-vercel.

Here's evidence of some Vercel weirdness - take a look at the 'query_string': output in these examples:

• https://til.simonwillison.net/-/asgi-scope?q=% - 'query_string': b'q=' - the % is ignored)
• https://til.simonwillison.net/-/asgi-scope?q=%1 - 'query_string': b'q=' - the %1 is ignored)
• https://til.simonwillison.net/-/asgi-scope?q=%12 - 'query_string': b'q=%12' - the %12 makes it through
• https://til.simonwillison.net/-/asgi-scope?q=%25 - 'query_string': b'q=%25' - the %25 makes it through

from datasette-publish-vercel.

Thanks so much for working on this!

from datasette-publish-vercel.

@adactio I am finding this impossible to replicate on my own deployments to Vercel. Could you do me a favour and add --install datasette-debug-asgi to your datasette publish vercel command?

Doing so will add this page: https://til.simonwillison.net/-/asgi-scope to your thesession.vercel.app deployment which could help me figure out what's going on.

The big mystery: https://thesession.vercel.app/thesession?sql=select+*+from+tunes+where+name+like+%22%25a%25%22%0D%0A exhibits the bug (your Vercel deployment) - but the equivalent query https://til.simonwillison.net/fixtures?sql=select+*+from+searchable+where+text1+like+%22%25a%25%22%0D%0A against my Vercel deployment does NOT - if you view source on that page the .csv and .json links are correctly URL-escaped.

from datasette-publish-vercel.

Could you do me a favour and add --install datasette-debug-asgi to your datasette publish vercel command?

Done!

https://thesession.vercel.app/-/asgi-scope

from datasette-publish-vercel.

@now/python is deprecated, please use @vercel/python instead

If you're using Zero Config then this will happen automatically (rename source to /api/my-function.py, no config needed)

from datasette-publish-vercel.

We're using @vercel/python already:

from datasette-publish-vercel.

Well this is VERY weird. We have two instances, both hosted on Vercel.

https://til.simonwillison.net/-/asgi-scope/thesession?sql=select+*+from+tunes+where+name+like+%22%25a%25%22%0D%0A

 'http_version': '1.1',
 'method': 'GET',
 'path': '/-/asgi-scope/thesession',
 'query_string': b'sql=select%20*%20from%20tunes%20where%20name%20like%20%22%25'
                 b'a%25%22%0D%0A',
 'raw_path': b'/-/asgi-scope/thesession',
 'root_path': '',
 'scheme': 'https',
 'server': ('til.simonwillison.net', 80),

But... this one with the exact same path and query string returns very different results:

https://thesession.vercel.app/-/asgi-scope/thesession?sql=select+*+from+tunes+where+name+like+%22%25a%25%22%0D%0A

 'http_version': '1.1',
 'method': 'GET',
 'path': '/-/asgi-scope/thesession',
 'query_string': b'sql=select * from tunes where name like "%a%"\r\n',
 'raw_path': b'/-/asgi-scope/thesession',
 'root_path': '',
 'scheme': 'https',
 'server': ('thesession.vercel.app', 80),

The one on thesession.vercel.app has had the query_string decoded. The one on til.simonwillison.net has not.

from datasette-publish-vercel.

Another difference between the two (though not relevant to the bug we are investigating here):

https://til.simonwillison.net/-/asgi-scope/foo%2Fbar returns this:

 'path': '/-/asgi-scope/foo%2Fbar',
 'query_string': b'',
 'raw_path': b'/-/asgi-scope/foo%2Fbar',

https://thesession.vercel.app/-/asgi-scope/foo%2Fbar returns this:

 'path': '/-/asgi-scope/foo/bar',
 'query_string': b'',
 'raw_path': b'/-/asgi-scope/foo/bar',

According to the ASGI spec neither of those are correct - the expected return for that should be:

 'path': '/-/asgi-scope/foo/bar',
 'query_string': b'',
 'raw_path': b'/-/asgi-scope/foo%2Fbar',

from datasette-publish-vercel.

The most recent commit I can find to vercel/python that modified how path and raw_path were handled is this one: vercel/vercel@d3e6c2d#diff-349c8f0014a53eaa71bbc2f1a4c88f57eaedabb76ddfb65338d93f0b51122665

from datasette-publish-vercel.

https://til.simonwillison.net is using @vercel/python

https://thesession.vercel.app is using @now/python

You're seeing differences since @now/python hasn't received updates in 9 months

from datasette-publish-vercel.