citgm-smoker-nobuild seems broken about citgm HOT 23 OPEN

I suspect the existing script is relying on line breaks in the html listings, but the new page has everything on a single line.

FWIW https://ci.nodejs.org/job/node-test-node-addon-api-new/ does something similar and also failing.

from citgm.

I'm confused -- I thought https://nodejs.org/download/release/ was still supposed to be served from the DO/Joyent servers and not R2 so shouldn't have changed?

Only /dist is still served from DO/Joyent. Other routes are served from R2 since Nov 8.

from citgm.

Working on a fix on the worker side of things here nodejs/release-cloudflare-worker#65

from citgm.

FWIW the job is running:

cd $WORKSPACE && rm -rf *
if [ -n "$DOWNLOAD_LOCATION" ]; then
	DOWNLOAD_DIR="https://nodejs.org/download/$DOWNLOAD_LOCATION/"
else
	DOWNLOAD_DIR="https://nodejs.org/download/release/"
fi
LINK=`curl $DOWNLOAD_DIR | grep $NODE_VERSION | sort -t. -k 1,1n -k 2,2n -k 3,3n | tail -1 | cut -d\" -f 2 | tr -d /` 
case $nodes in
  *-ppcle|*-ppc64le) OS=linux; ARCH=ppc64le; EXT=tar.gz;;
  ubuntu*|debian*|fedora*|centos*|rhel*-x64) OS=linux; ARCH=x64; EXT=tar.gz;;
  osx*) OS=darwin; ARCH=x64; EXT=tar.gz;;
  *-s390x) OS=linux; ARCH=s390x; EXT=tar.gz;;
  aix*) OS=aix; ARCH=ppc64; EXT=tar.gz;;
  win*) OS=win; ARCH=x64; EXT=zip;;
esac
curl -O "$DOWNLOAD_DIR$LINK/node-$LINK-$OS-$ARCH.$EXT"
case $nodes in
  win*) unzip node-$LINK-$OS-$ARCH.$EXT ;;
  *) gzip -cd node-$LINK-$OS-$ARCH.$EXT | tar xf - ;;
esac
mv node-$LINK-$OS-$ARCH node

from citgm.

/cc @ovflowd @flakey5

from citgm.

Quick stupid question, but are really making a GET request with a huge html snippet in the URL?

from citgm.

Or what exactly is the issue here?

from citgm.

Quick stupid question, but are really making a GET request with a huge html snippet in the URL?

+1 to that and also is this relying on parsing whatever html is returned?

from citgm.

(Just trying to understand whay the script is trying to do and what is failing here)

from citgm.

Quick stupid question, but are really making a GET request with a huge html snippet in the URL?

+1 to that and also is this relying on parsing whatever html is returned?

The html looks like the index of that path, maybe something is wrong and accidentally it is trying to send the html, I have no idea. If someone can explain what this script is doing and what it is supposed to do

from citgm.

The script does this:

curl https://nodejs.org/download/release/ | grep v20 | sort -t. -k 1,1n -k 2,2n -k 3,3n | tail -1 | cut -d\" -f 2 | tr -d /

from citgm.

You can compare with direct:

curl https://direct.nodejs.org/download/release/ | grep v20 | sort -t. -k 1,1n -k 2,2n -k 3,3n | tail -1 | cut -d\" -f 2 | tr -d /

from citgm.

The script does this:

curl https://nodejs.org/download/release/ | grep v20 | sort -t. -k 1,1n -k 2,2n -k 3,3n | tail -1 | cut -d\" -f 2 | tr -d /

The script might need to be adjusted to accommodate the new html structure of our directory listing pages.

Heh, @flakey5 I did say somewhere we were relying on the format of the directory listing k

from citgm.

It would be unfair (and a lot of time lost in total) to expect everyone who was parsing nginx output to change their scripts if there's a way to fix that in the new infra.

from citgm.

I'm confused -- I thought https://nodejs.org/download/release/ was still supposed to be served from the DO/Joyent servers and not R2 so shouldn't have changed?

from citgm.

It would be unfair (and a lot of time lost in total) to expect everyone who was parsing nginx output to change their scripts if there's a way to fix that in the new infra.

Oh yeah, definitely agree with you here.

from citgm.

I suspect the existing script is relying on line breaks in the html listings, but the new page has everything on a single line.

+1 I think this is what it's relying on. Formatted the html response and got it to return this

When using the test command @targos pointed out it returns just v20.9.0

from citgm.

I suspect the existing script is relying on line breaks in the html listings, but the new page has everything on a single line.

+1 I think this is what it's relying on. Formatted the html response and got it to return this

When using the test command @targos pointed out it returns just v20.9.0

Do we have any updates here? Or issue to keep track of this work on the worker side of things?

from citgm.

@flakey5 good to hear you are working on it. Let me know when the fix is in place so that I can confirm it fixes the Node-api tests which were also broken (https://ci.nodejs.org/view/x%20-%20Abi%20stable%20module%20API/job/node-test-node-addon-api-new/).

from citgm.

The jobs should be fixed now. We moved the worker off of /download until nodejs/release-cloudflare-worker#74 is resolved due to multiple issues that we've been seeing

from citgm.

Well, can we manually test that they will not break once we move them back? We just merged the new directory listing can we check that?

from citgm.

Once nodejs/release-cloudflare-worker#76 lands this issue should be fixed on the worker's end

from citgm.

Ran the command against the worker again and it works

from citgm.