Is your feature request related to a problem? Please describe.
with the new onedrive uri support: adobe/helix-onedrive-support#68
it is benefitial to pass the onedrive uri form to data-embed in order to avoid double sharelink lookups.

@trieloff it's hard to document on the already existing files. it would be nice if you could reset the repository and commit the initial work as pull request.

Description
during a recent outage, non of the invocations to content proxy was able to deliver content, if it was fetching it from another runtime action. the error was:

Unhandled error Error [ERR_HTTP2_INVALID_SESSION]: The session has been destroyed at ClientHttp2Session.request (internal/http2/core.js:1558:13)

The proxy requests to github still worked.

During the same time, Adobe I/O Runtime changed some routing of their traffic.

After forcing HTTP1 and redeploying the action, everything worked again normally. Interestingly, deploying another version with http2 still enabled also worked. also, requesting older versions e.g. 2.0.7 also works now.

also, before the deployment with http2 disabled, I never tried to invoke the content proxy directly, the errors were only observed in coralogix. so it might well be, that it worked before invoking the action directly with curl (and not via fastly)

also, looking at the activations, there were many cold starts, which I assume rules out any problems in caches inside the nodejs process. My wildest guess is, that some network information (dns, route, cert) somehow changed but wasn't updated in the cache of the container environment and caused problems establishing the http2 session....

Unfortunately, it's not possible to reproduce the issue. there was a similar behaviour in helix-static and the typekit fonts: adobe/helix-static#351

/cc @stefan-guggisberg @tysonnorris

Description
Leading // in the path parameter results in status 500 or 502, depending on the specific path.

To Reproduce
E.g. try path=//foo.json (-> 500) or path=//foo.md (-> 502).

Expected behavior
paths with leading // should result in a 400 (bad request).

Description
A timeout error throws an exception which is not caught inside the main function and leads to an error reported in epsagon. so far, all other actions handle their exceptions internally.

2020-06-14T07:33:01.787Z       stderr: [ERROR] error { params: { path: '/en/drafts/migrated/2013/07/30/7-main-benefits-closedloop-marketing.md', __ow_method: 'get', ref: 'd43fd8683f7fd8fcf04755ae3cf23f46cc02c509', __ow_headers: { accept: 'application/json, text/*;q=0.9, */*;q=0.8', 'accept-encoding': 'br, gzip;q=0.8, deflate;q=0.5', connection: 'close', 'epsagon-trace-id': '8cb3499e26574249b0b5851c81e1fee7:be3b5057894b6d00:8a248b4a1c372e3f:1', host: 'controller-a', 'perf-br-req-in': '1592119960.014', 'user-agent': 'helix-fetch', 'x-cdn-request-id': '70bf3936-2ac7-57f9-ab20-d80672b19b41', 'x-cdn-url': 'https://theblog-adobe.project-helix.page/en/drafts/migrated/2013/07/30/7-main-benefits-closedloop-marketing.html', 'x-forwarded-for': '52.30.165.11, 10.250.206.51', 'x-forwarded-host': 'adobeioruntime.net', 'x-forwarded-port': '443', 'x-forwarded-proto': 'https', 'x-real-ip': '10.250.206.51', 'x-request-id': '70bf3936-2ac7-57f9-ab20-d80672b19b41' }, __ow_path: '', owner: 'adobe', repo: 'theblog' }, error: Error: GET https://adobeioruntime.net/api/v1/web/helix/helix-services/word2md@v1?path=%2Fen%2Fdrafts%2Fmigrated%2F2013%2F07%2F30%2F7-main-benefits-closedloop-marketing.docx&shareLink=https%3A%2F%2Fadobe.sharepoint.com%2Fsites%2FTheBlog%2FShared%2520Documents%2Ftheblog&rid=70bf3936-2ac7-57f9-ab20-d80672b19b41&src=adobe%2Ftheblog%2Fd43fd8683f7fd8fcf04755ae3cf23f46cc02c509 timed out after 20000 ms
       at timeoutError (/nodejsAction/HmeHxYJZ/main.js:58212:16)
       at Timeout.setTimeout [as _onTimeout] (/nodejsAction/HmeHxYJZ/main.js:58235:28)
       at ontimeout (timers.js:436:11)
       at tryOnTimeout (timers.js:300:5)
       at listOnTimeout (timers.js:263:5)
       at Timer.processTimers (timers.js:223:10) }

The timeout error is thrown from within helix-fetch.

See AdobeDocs/hlxdemo#1

Depends on adobe/helix-shared#410

Is your feature request related to a problem? Please describe.
one the content-proxy supports fallbacks to github, it will become a problem for local development. where we still want to serve github content from the local repository.

suggestion

add parameter: ignore or reject that can specify the mount point type to ignore.

example ?ignore=github.

alternative

for local development, it might be ok to load the local content first.

Is your feature request related to a problem? Please describe.
once adobe/helix-data-embed#136 is fixed, the new v2 version will have a different response body which should be taken care of.

downstream scripts can already prepare by anticipating the change:

const json = await response.json();
const data = Array.isArray(json) ? json : json.data;

Description
The lookup URL fom the _taxonomy.xlsx is missing the _ prefix and therefore giving a 404.

https://adobeioruntime.net/api/v1/web/helix/helix-services/content-proxy@v2?owner=adobe&repo=theblog&ref=master&path=%2F&lookup=https%3A%2F%2Fadobe.sharepoint.com%2F%3Ax%3A%2Fr%2Fsites%2FTheBlog%2F_layouts%2F15%2FDoc.aspx%3Fsourcedoc%3D%257B565E00D2-2D27-44B2-9DDA-35D182F0F698%257D%26file%3D_taxonomy.xlsx%26action%3Ddefault%26mobileredirect%3Dtrue

To Reproduce
Steps to reproduce the behavior:

1. Go to https://adobeioruntime.net/api/v1/web/helix/helix-services/content-proxy@v2?owner=adobe&repo=theblog&ref=master&path=%2F&lookup=https%3A%2F%2Fadobe.sharepoint.com%2F%3Ax%3A%2Fr%2Fsites%2FTheBlog%2F_layouts%2F15%2FDoc.aspx%3Fsourcedoc%3D%257B565E00D2-2D27-44B2-9DDA-35D182F0F698%257D%26file%3D_taxonomy.xlsx%26action%3Ddefault%26mobileredirect%3Dtrue
2. Observe URL: https://theblog--adobe.hlx.page/en/topics/taxonomy.json (404)

Expected behavior
URL should be https://theblog--adobe.hlx.page/en/topics/_taxonomy.json

Description
When using the edit lookup on a page based on an md file in sharepoint, it goes straight to download the md file rather than showing the text editor.

To Reproduce
Steps to reproduce the behavior:

1. Go to https://adobeioruntime.net/api/v1/web/helix/helix-services/content-proxy@v2?owner=adobe&repo=theblog&ref=staging&path=%2F&edit=https%3A%2F%2Fstaging--theblog--adobe.hlx.page%2Fen%2Fpublish%2F2018%2F01%2F26%2Feverything-need-know-creating-smooth-login-experiences.html
2. See downloaded source file

Expected behavior
A redirect to the sharepoint text editor with the file.

Version:
v2

Description
if a resolve of a google doc results in a 404, the error is not cought in the action but bubbles up to epsagon:

Is your feature request related to a problem? Please describe.
It would be cool to be able to mount directly a document, eg:

mountpoints:
  /index.md: https://adobe-my.sharepoint.com/personal/tripod_adobe_com/Documents/helix-content/sub/welcome.docx
  /en: https://adobe-my.sharepoint.com/personal/tripod_adobe_com/Documents/helix-content
  /data.json: https://adobe-my.sharepoint.com/personal/tripod_adobe_com/Documents/helix-content/cars.xlsx

this would allow to un-claim the / subtree and have proper fallbacks for other toplevel elements.
it could even be simplified to:

mountpoints:
 /: https://adobe-my.sharepoint.com/personal/tripod_adobe_com/Documents/helix-content/sub/welcome.docx

which would internally be expanded to the /index.md form. this also has the advantage, that a new helix-pages site only needs 1 document to start with, using a document with an arbitrary name (eg. homepage.docx).

/cc @davidnuescheler @trieloff

Related to adobe/helix-publish#452: if the helix-bot has access to the repository, then we can treat all resources that are served with a branch name instead of a proper ref as indefinitely shared cacheable (not browser cacheable).

@tripodsan how hard would it be to check the install status of the bot for each repository?

Check this activation id: c978ed8cecea428db8ed8cecea928d36 (https://experience.adobe.com/#/@helix/custom-apps/325368-TraceDebug/#c978ed8cecea428db8ed8cecea928d36)

Why does it return a 202 ?

Description
once https://github.com/adobe/helix-word2md/pull/419 is merged, content proxy's resolve should use the same transformation to create the external urls.

it looks like a google changed their format slightly omitting the /u/0 in the URLs, not sure if that trips anything up... but a ?sharing at the end of the URL also seems to trip up the folder id extraction.

should be a lot more robust as this is usually one of the first steps in a helix journey.

While observing our logs in Coralogix, the large majority of the warn I can see are coming from this project from lines like:

log[utils.logLevelForStatusCode(response.status)](`Unable to fetch...`);

which in 99% of the cases I am observing at the moment are for "valid" 404 requests which are part of our "concurrent check for resources" which for sure tries combos to files which do not exist. Those do not deserve a warn but a debug (eventually a info).

The overflow of those warnings hides the "real" warnings we should be worried about.

@trieloff @tripodsan WDYT ?

This one works together with adobe/helix-pipeline#908
In this normal flow of requests https://helix.coralogix.com/#/query/logs?id=2AtbscscPSC, 103 warnings, 1 is a "real" warning.

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: config
Error type: Invalid allowedVersions
Message: The following allowedVersions does not parse as a valid version or range: "<persister-fs"

Google and Word currently have a time-based cache, but we can treat at least the Shared-Cache surrogate-control as immutable, because they will be invalidated on the fastly level by the two bots.

Is your feature request related to a problem? Please describe.
in order to be able to issue a purge request based on the surrogate-key, the observation event only knows the driveid/itemid of the modified workbook. the x-source-location reported by the content-proxy is therefor not usable:

 curl -sD- https://helix-pages-test--tripodsan.project-helix.page/office/sub/countries.json
...
x-source-location: https://adobe-my.sharepoint.com/personal/tripod_adobe_com/_layouts/15/Doc.aspx?sourcedoc=%7BE9A0979D-75A1-47A8-B9DB-9911C1EA7F65%7D&file=countries.xlsx&action=default&mobileredirect=true

See adobe/helix-home#121 (comment)

Fallbacks will be implemented in helix-content-proxy and executed in parallel

This closes #20 and resolves it: no defat fallbacks, especially not to GitHub

Description
requesting a specific sheet or table via a sheet query parameter doesn't work.
the problem is that the content-proxy doesn't forward those parameters to data-embed.

see adobe/helix-data-embed#118

Description
the cached fstab can yield to unexpected behaviour during development, when users try out different configurations.
it's currently not possible to flush the cache - other than redeploy the action.

things get even more complicated, when different versions of an fstab are cached in different containers / regions.

To Reproduce

1. make a change to fstab.yaml
2. try out and see that it had no effect

Expected behavior
Changes to fstab.yaml shoud immediately by picked up by content-proxy

Additional context
in order to reduce the amount of github traffic, it could be acceptable to cache for 1-2 minutes.
and/or have a reliable way to clear the cache.

That makes partial invalidation impossible.

Description
the source location of a google sheet is: https://docs.google.com/spreadsheets/d/${sheetId}/view which is sub-optimal since the share url could change. it would be better to just use the sheetId.

For valid page: https://blog.adobe.com/en/2020/12/01/what-does-the-cmo50-tell-us-about-modern-marketing.html#gs.re0ega

Edit URL: https://adobeioruntime.net/api/v1/web/helix/helix-services/content-proxy@v2?owner=adobe&repo=theblog&ref=master&path=%2F&edit=https%3A%2F%2Fblog.adobe.com%2Fen%2F2020%2F12%2F01%2Fwhat-does-the-cmo50-tell-us-about-modern-marketing.html%23gs.re0ega

returns Internal Server Error.

Activation logs:

wsk activation logs fc8055048cc24fc38055048cc22fc32c
2021-01-28T10:17:37.385Z       stdout: instrumenting epsagon.
2021-01-28T10:17:37.386Z       stdout: action-status-begin {"container":{"uuid":"1329520efe5a0d3b619c117839e97724","numInvocations":6,"begin":{"mem":174407680,"age":174227,"concurrency":1}}}
2021-01-28T10:17:37.956Z       stderr: [WARN] StatusCodeError
    at Function.fromError (/nodejsAction/VNX8IJnR/index.js:5135:17)
    at OneDrive.listChildren (/nodejsAction/VNX8IJnR/index.js:4387:37)
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  statusCode: 404,
  details: { code: 'itemNotFound', message: 'The resource could not be found.', innerError: { date: '2021-01-28T10:17:37', 'request-id': 'd697e254-021b-480f-9da2-230f67002e79', 'client-request-id': 'd697e254-021b-480f-9da2-230f67002e79' } }
}
2021-01-28T10:17:37.958Z       stderr: [ERROR] error {
  params: {},
  error: StatusCodeError
      at Function.fromError (/nodejsAction/VNX8IJnR/index.js:5135:17)
      at OneDrive.listChildren (/nodejsAction/VNX8IJnR/index.js:4387:37)
      at processTicksAndRejections (internal/process/task_queues.js:97:5) {
    statusCode: 404,
    details: { code: 'itemNotFound', message: 'The resource could not be found.', innerError: { date: '2021-01-28T10:17:37', 'request-id': 'd697e254-021b-480f-9da2-230f67002e79', 'client-request-id': 'd697e254-021b-480f-9da2-230f67002e79' } }
  }
}
2021-01-28T10:17:37.959Z       stderr: error while invoking function TypeError: Cannot read property 'get' of undefined
    at ensureUTF8Charset (/nodejsAction/VNX8IJnR/index.js:579:29)
    at openwhisk (/nodejsAction/VNX8IJnR/index.js:231:5)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async /nodejsAction/VNX8IJnR/index.js:716:14
    at async NodeActionRunner.userScriptMain (/nodejsAction/VNX8IJnR/index.js:853:15)
2021-01-28T10:17:37.959Z       stdout: action-status-end {"container":{"uuid":"1329520efe5a0d3b619c117839e97724","numInvocations":6,"begin":{"mem":174407680,"age":174227,"concurrency":1},"end":{"mem":175591424,"age":174800,"concurrency":1},"delta":{"mem":1183744,"age":573}}}

Ideally, helix-content-proxy should work like a drop-in-replacement to https://raw.githubusercontent.com/.

Description
The urls that resolve to spreadsheets are do not support author friendly urls.

To Reproduce
requesting https://theblog--adobe.hlx.page/en/topics/taxonomy.json results in a 404, since the real document's name is _taxonomy.xslx.

Expected behavior
The data is returned.

Overview

we currently seem to have an ad-hoc set of caches used in content proxy, which makes it very hard to get an expected result

Details

there is a range of caches from the path to id resolution, helix-fetch, fstab, etc. that lead to many situations where content gets stuck in content proxy and changes in google drive / one drive are not reflected immediately

Proposed Actions

for now i would recommend to err on the side of consistency vs. performance, meaning probably to suppress caches at this stage unless we can guarantee flushes on edit or we feel that there are very few changes

For Google and Word. no-cache.

In helix-pipeline, requesting a non-existing item in Word/GDocs will result in another request to GitHub.

In helix-content-proxy, you just get a 404.

Additionally, there is a suggestion for helix-pipeline to only served Word/GDocs if the file does not exist in GitHub. adobe/helix-pipeline#726

Between the three options:

1. GitHub should override Word
2. Word should override GitHub
3. There are no overrides at all

One is clearly the best. I just don't know which one.

I created a word2md test version ci2009 which prepends a # testing string to the result md.

tier 1: test the inner cdn

requesting the normal md:

$ curl https://theblog--adobe.hlx.page/en/drafts/alex/test.md
...
# **A Semi-Surreal 3D Tutorial: Leaves on Water with Mue Studio**

requesting with the test version:

$ curl -sD- https://theblog--adobe.hlx.page/en/drafts/alex/test.md -H 'x-ow-version-lock: word2md=word2md@ci2009'
...
# testing

# **A Semi-Surreal 3D Tutorial: Leaves on Water with Mue Studio**

playing around with those 2 requests, sometimes show the test response delivered with the normal request and vice versa.

tier 2: test content-proxy

the normal version:

curl -sD - 'https://adobeioruntime.net/api/v1/web/helix-pages/helix-services/[email protected]?owner=adobe&path=%2Fen%2Fdrafts%2Falex%2Ftest.md&ref=a4ad47406912c2327273ce63f462376a20d9e11f&repo=theblog'
...
# **A Semi-Surreal 3D Tutorial: Leaves on Water with Mue Studio**

the testing version:

curl -sD - 'https://adobeioruntime.net/api/v1/web/helix-pages/helix-services/[email protected]?owner=adobe&path=%2Fen%2Fdrafts%2Falex%2Ftest.md&ref=a4ad47406912c2327273ce63f462376a20d9e11f&repo=theblog' -H 'x-ow-version-lock: word2md=word2md@ci2009'
...
# testing

# **A Semi-Surreal 3D Tutorial: Leaves on Water with Mue Studio**

again, playing with those 2 requests alternatively show the wrong results.

note, that the X-GW-Cache: HIT header of the runtime action is HIT, when the wrong content is returned.

Description
the word2md and gdocs2md are invoked on the helix namespace, which is sub-optimal as they activation should be on the customers namespace:

suggestion

the the same namespace as the content-proxy action is using.

i.e. use process.env.__OW_NAMESPACE instead.

Description
having a double slash in the path is valid, but the content proxy returns a 400.
it would be better to use a 404, in order to avoid upstream 502.

404!

i tried to develop with the .json representation of an excel spreadsheet locally with the helix-cli and due to #56 i wasn't able to, so i quickly copied the output of

https://excel-index--theblog--adobe.hlx.page/en/query-index.json

into a query-index.json in the root of the repo and changed the path in my fetch from /en/query-index.json to /query-index.json which worked beautifully locally... however when i pushed it into my branch i am getting a 404 for that resource...

https://excel-index--theblog--adobe.hlx.page/query-index.json

while i can work around this for this instance, i am worried that this may be related to the content-proxy hijacking all .json requests, which obviously would break any helix pages site that is using json files in github.

Description
https://adobeioruntime.net/api/v1/web/helix/helix-services/content-proxy@v2?owner=adobe&repo=theblog&ref=master&path=%2F&lookup=https%3A%2F%2Fadobe.sharepoint.com%2Fsites%2FTheBlog%2F_layouts%2F15%2FDoc.aspx%3Fsourcedoc%3D%257BBFD9A19C-4A68-4DBF-8641-DA2F1283C895%257D%26file%3Dindex.docx%26action%3Ddefault%26mobileredirect%3Dtrue%26cid%3Db6e62d68-f53a-471f-a686-90e69458fdd7

Description
it looks like the URLs are still inclusive of uppercase and special characters...
it would be great if we could change that to lowercase and dashed notation...

using helix-data-embed

Description
Required for adobe/helix-pages#531, helix pages appears to still default to the master branch by default when using hlx.page.

To Reproduce
Steps to reproduce the behavior:

1. Create a new repo
2. Add index.md to new repo
3. Visit mynewrepo-myghuser.hlx.page
4. You will receive a 404.
5. Add a master branch.
6. Visit mynewrepo-myghuser.hlx.page\
7. You will see your content.

Expected behavior
Github's default branch should be used, or worse: the main branch as a constant.

Additional context
I'm not sure if GH has an API to get the default branch, or if that's worth the overhead instead of just using a constant.

I do think the main README here should be updated to illustrate how to get to different branches with the -- convention.

Requests with a ref=master should be rejected, as this resolution should be done beforehand.

currently there is a "reverse" lookup that triggers a redirect #126 for a given document id to a preview (inner CDN) URL
it would be great if the same functionality existed the other way around with a redirect to the authoring experience, this would allow us to place an "edit" button into the the sidekick for preview links.

Required for adobe/helix-pages#531: ref might be undetermined when requesting the content-proxy (default branch must be used). The content-proxy must then:

• allow ref parameter to be omitted
• if this is the case, determine the ref (default branch) - should be responsibility of adobe/helix-resolve-git-ref. See adobe/helix-resolve-git-ref#287.

cc @trieloff @tripodsan

For status code 200: cache forever
For status code 404: cache forever, too (we are using immutable URLs, so this won't change)
For all other status codes: don't cache, try again

in helix-fetch, the timeout signal timer is started when the signal is created:

• https://github.com/adobe/helix-fetch/blob/a14a4035f739c03531a20e5d7220c0cd46e361c9/src/index.js#L169

but in content-proxy, the options are created before the requests are issued. so all the operations that happen before already count against the external request timeout.

IMO, the timeout should only be used for the actual external request.

Is your feature request related to a problem? Please describe.
see adobe/helix-home#141

see adobe/project-helix#508

• migrate code to use helix-deploy adapter
• update CI to use helix-deploy