(moved to here from (phetsims/chipper#627)

@pixelzoom recently thought that I had published an RC of a sim for which he is the responsible dev due to file permissions. The following dialog resulted. Bottom line: We should use something other than my account info for transferring files from the build server to the dev server.

Slack thread:

@pixelzoom [5:14 PM]
I see that you published function-builder 1.0.9-rc.1 on 11/17, so it has entered an RC phase. What’s going on?

@jbphet [8:54 AM]
I think you meant to ping JO on this (kinda freaked me out, since I didn't recall it at all).

@pixelzoom [8:55 AM]
drwxrwsr-x 2 jblanco physphet 4096 Nov 17 18:51 1.0.9-rc.1/

@jbphet [8:55 AM]
https://github.com/phetsims/function-builder/commits/1.0

@pixelzoom [8:56 AM]
hmmm… how did the files on spot end up owned by you?

@jbphet [8:57 AM]
Not sure. I know rosetta uses my account information for some file transfer operations, maybe this is done in other places in the build tools too. If so, probably best to modify that to use a machine account of some kind.

As I think about it, but build server is likely using my account info. I'll put that on my list to investigate today.

• .htaccess should be copied to /SIM/VERSION/protected, it is currently copied to /SIM/VERSION/
• rc deploys are not completing the task and are causing subsequent builds to queue indefinitely
• Initial production deploys are consistently failing due to repos failing to pull master. Subsequent deploys complete successfully.

Changes are tracked in the phet-io-changes branch.

Note: Since this is a public repo, I'm going to avoid using specific group ID and user ID names in this ticket.

While checking the permissions and ownership of the build for the 1.0.0-rc.2 version of Neuron, I found that the files are owned by a different group from other repositories that I've deployed in the past. It looks like this is tied to the way in which the original directory was created. We developers had a discussion about it on Skype, but weren't clear on exactly what if anything needs to be done. At the end of the discussion, @pixelzoom said:

Looks like it's exclusively a group id problem with directories created by @samreid .

and

If it's causing problems, see 'chgrp -R'.

We will discuss in developer meeting a figure out next steps.

the paths are concatenated like DIR+SUBDIR but this requires slashes in DIR, which is in the build-config.json file. It seems risky to require slashes there since they cannot be properly commented in json. Let's get rid of the slashes and add them in code. After all, // is much better than no slash at all.

As seen in https://github.com/phetsims/phet-io/issues/726#issuecomment-269708276 @jonathanolson and I identified a few inconsistencies on the build server. First, we saw files created by @zepumph that looked like they had incorrect permissions. Second, it looked like files on /data/share/phet/phet-repos were being used for something other than builds--we wanted to make sure everyone is aware that those repos are for use by the build-server only.

@jonathanolson can you please provide more information about how @zepumph can make sure his permissions are correct?

@zepumph can you please comment on whether these files are being used for other features?

We just ran into an issue where version 1.0.8 of Function Builder was essentially overwritten by a deployment of 1.0.7. Our best guess as to the cause of this is that the link for the build request got pasted into Skype and something crawled the chat and hit the link. The build server code should probably be modified to prevent deployment over top of any existing previous deployment, and the failure should cause an email notification to be sent.

In the past I've had trouble testing development in a git branch because build-server runs git checkout master on the perennial directory. This doesn't change the current running process, but when there is a failure (as expected during development) the build-server automatically tries to restart itself, and then reverts to running from master. It would be nice if it did not do this so that we can reliably work in a future branch on the dev machine OR work from a previous stable sha in production if we prefer to do development in master.

I'm currently trying to track down the cause of several build failures related to translation. By going through the log, I can see that the command grunt build-for-server is failing, but I can't see much about why. Here is an excerpt:

Oct 31 03:19:31 phet-server.int.colorado.edu build-server[83644]: Oct 31 2017 03:19:31 MDT - info: running command: grunt build-for-server --brand=phet --locales=eu
Oct 31 03:19:54 phet-server.int.colorado.edu build-server[83644]: Oct 31 2017 03:19:54 MDT - error: error running command: grunt build-for-server --brand=phet --locales=eu in ../circuit-construction-kit-dc-virtual-lab, err: Error: Command failed: /bin/sh -c grunt build-for-server --brand=phet --locales=eu
Oct 31 03:19:54 phet-server.int.colorado.edu build-server[83644]: , stdout: Running "lint-all" task
Oct 31 03:19:54 phet-server.int.colorado.edu build-server[83644]: Running "clean" task
Oct 31 03:19:54 phet-server.int.colorado.edu build-server[83644]: Running "requirejs-build" task
Oct 31 03:19:54 phet-server.int.colorado.edu build-server[83644]: Running "after-requirejs-build" task
Oct 31 03:19:54 phet-server.int.colorado.edu build-server[83644]: [135B blob data]
Oct 31 03:19:54 phet-server.int.colorado.edu build-server[83644]: , build aborted.

From this, I can see that the failure must have occurred in the "after-requirejs-build", but then it just says [135B blob data], which seems odd and is entirely unhelpful. If this could instead include whatever was printed to the screen, bit would be most helpful in situations like this.

Adding helpful scripts and documentation for the maintenance release process (testing with phetsims/chipper#429).

The build server sent out a failure message early yesterday morn. The first portion looked like this:

Build failed with error: error running command grunt build-for-server --brand=phet --locales=sk: Error: Command failed: /bin/sh -c grunt build-for-server --brand=phet --locales=sk
WARN: Output exceeds 32000 characters
WARN: Output exceeds 32000 characters
 Sim = function-builder-basics Version = 1.0.1 Locales = sk Shas = 
...

I remember a few people having this issue for local builds a week or two ago and asked @jessegreenberg about it. He referred me to phetsims/chipper#605.

On Monday, January 11, 2016, @pixelzoom attempted to deploy a maintenance release for Beer's Law Lab, but the existing translations did not get properly rebuild and redeployed. Below is the Skype dialog that chronicles the discussion. This is here mainly for record keeping, and I'll try to summarize the problem and solution(s) in subsequent comments.

[1/11/2016 4:30:04 PM] @pixelzoom Need someone's immediate help. I just published a maintenance release of Beer's Law Lab, and all of the translated html files now appear to be gone!! I did not get an email indicating a successful or failed build. Last line of the figaro:/data/share/phet/phet-repos/perennial/build-server.log is:
[1/11/2016 4:30:06 PM] @pixelzoom 2016-01-11T23:19:17.575Z - info: build for beers-law-lab finished successfully
[1/11/2016 4:30:47 PM] @pixelzoom But all links to translated versions are now 404 Not Found.
[1/11/2016 4:32:02 PM] @pixelzoom The English version appear to be OK and has the expected version number, 1.3.1.
[1/11/2016 4:36:03 PM] @pixelzoom Looks like build-server ran with --locales=en. Isn't it supposed to rebuild all locales?
[1/11/2016 4:40:12 PM] @jbphet I'm taking a look. I have to catch a bus shortly, but can take a look at it from home in a bit too.
[1/11/2016 4:40:47 PM | Edited 4:41:02 PM] @pixelzoom I've tried to deploy twice, and it's only building English, and not sending email.
[1/11/2016 4:41:15 PM] @pixelzoom I'm going to try another deploy with --locales=* to see if that does anything.
[1/11/2016 4:42:52 PM] @pixelzoom --locales=* is not passed to build-server. From log:
[1/11/2016 4:42:53 PM] @pixelzoom 2016-01-11T23:41:23.783Z - info: running command: grunt build-for-server --brand=phet --locales=en
[1/11/2016 4:43:55 PM] @pixelzoom malleyc@figaro[64]: pwd
/beers-law-lab
[65]: ls 1.3.0
beers-law-lab-128.png beers-law-lab_ko.html
beers-law-lab-600.png beers-law-lab_pl.html
beers-law-lab_ar.html beers-law-lab_pt_BR.html
beers-law-lab_da.html beers-law-lab_te.html
beers-law-lab_en.html beers-law-lab_vi.html
beers-law-lab_en-iframe.html beers-law-lab.xml
beers-law-lab_es_PE.html beers-law-lab_zh_TW.html
beers-law-lab_fa.html dependencies.json
beers-law-lab_it.html
[66]: ls 1.3.1
beers-law-lab-128.png beers-law-lab_en.html beers-law-lab.xml
beers-law-lab-600.png beers-law-lab_en-iframe.html dependencies.json
[1/11/2016 4:44:14 PM] @pixelzoom Definitely not building translated versions.
[1/11/2016 4:44:37 PM] @jbphet I'm seeing the same thing in the build server log - it's just building the English version.
[1/11/2016 4:44:48 PM] @pixelzoom Is there a config file for build-server on figaro?
[1/11/2016 4:45:16 PM] @pixelzoom build-server.js doc says:
[1/11/2016 4:45:17 PM] @pixelzoom * - locales - a comma-separated list of locales to build [optional, defaults to all locales in babel]
[1/11/2016 4:45:58 PM] @jbphet Yes, it is supposed to do all locales in babel by default. Aaron and I worked on that a bit just before he left.
[1/11/2016 4:46:36 PM] @pixelzoom This is the first time I've deployed a maintenance release of a sim that has translation support. Lucky me.
[1/11/2016 4:47:05 PM] @samreid: I'm not sure what's wrong...
[1/11/2016 4:47:14 PM | Edited 4:47:25 PM] @jbphet I did it before, ran into issues, worked with Aaron to resolve. I thought some had been done since then, but possibly not.
[1/11/2016 4:48:05 PM] @samreid: Workaround brainstorm: iterate through the translations and approve them one by one...
[1/11/2016 4:48:25 PM] @pixelzoom How can I do that? I can't even get them to build?
[1/11/2016 4:48:43 PM] @jbphet Let's not go into workaround mode yet. It's not worth the possible issues.
[1/11/2016 4:49:07 PM] @jbphet I need to run to catch the bus, and will check in once I'm home.
[1/11/2016 4:49:16 PM] @pixelzoom ok thanks.
[1/11/2016 4:49:21 PM] @samreid: Sounds good
[1/11/2016 4:51:43 PM] @jonathanolson: available now, looking into it
[1/11/2016 4:52:25 PM] @jonathanolson: until it is fixed, should I revert the deploy and bring back the previous files?
[1/11/2016 4:52:39 PM] @pixelzoom i guess so
[1/11/2016 4:52:39 PM] @jonathanolson: or I guess changing the symbolic link will do that
[1/11/2016 4:52:58 PM] @pixelzoom is it a symlink or htaccess?
[1/11/2016 4:53:15 PM] @jonathanolson: sorry, htaccess
[1/11/2016 4:53:29 PM] @jonathanolson: 1.3.0 is good, 1.3.1 is bad?
[1/11/2016 4:53:33 PM] @pixelzoom 1.3.0 good
[1/11/2016 4:53:44 PM] @pixelzoom /data/web/htdocs/phetsims/sims/html/beers-law-lab/.htaccess
[1/11/2016 4:53:51 PM] @jonathanolson: changed .htaccess
[1/11/2016 4:53:55 PM] @pixelzoom thanks
[1/11/2016 4:54:25 PM] @pixelzoom verified that we're back online
[1/11/2016 4:55:19 PM] @jonathanolson: revered version on the website, so it doesn't show out-of-date notification
[1/11/2016 4:55:30 PM] @jonathanolson: OK, now for the deploy bit
[1/11/2016 4:55:41 PM] @pixelzoom what is "revered version" ?
[1/11/2016 4:55:57 PM] @pixelzoom oh... reverted?
[1/11/2016 4:56:11 PM] @jonathanolson: reverted, sorry
[1/11/2016 4:56:22 PM] @pixelzoom I thought maybe we were getting religious :)
[1/11/2016 4:56:43 PM] @jonathanolson: My "holy logic" was missing a case!
[1/11/2016 4:57:11 PM] @pixelzoom I have the feeling we're dealing with holey logic, not holy.
[1/11/2016 4:57:17 PM] @jonathanolson: babel on figaro shows many BLL translations
[1/11/2016 4:57:25 PM] @pixelzoom where does that live on figaro
[1/11/2016 4:57:26 PM] @pixelzoom ?
[1/11/2016 4:57:32 PM] @jonathanolson: /data/share/phet/phet-repos/babel
[1/11/2016 4:57:37 PM] @pixelzoom thanks.
[1/11/2016 4:58:14 PM] @pixelzoom build-server.js line 424 says:
[1/11/2016 4:58:15 PM] @jonathanolson: I found out on Friday that the build-server and rosetta seem to be using the same checked-out repos (I think?), so there may be a whole range of bugs available
[1/11/2016 4:58:15 PM] @pixelzoom Get all of the deployed locales from the latest version before publishing the next version,
[1/11/2016 4:58:29 PM] @pixelzoom Is it possibly looking at 1.2.0 instead of 1.3.0?
[1/11/2016 4:59:10 PM] @pixelzoom I don't understand why it would be looking a the latest version instead of babel. But perhaps its definition of "latest" version is faulty.
[1/11/2016 4:59:46 PM] @pixelzoom That doc is for a function named getLocales.
[1/11/2016 5:00:27 PM] @pixelzoom Hate code like this:
[1/11/2016 5:00:29 PM] @pixelzoom // from rosetta
if ( locales && locales !== '*' ) {
callbackLocales = locales;
}

// from grunt deploy-production
else {

[1/11/2016 5:00:39 PM] @pixelzoom Where there's a big chunk of space between if-else.
[1/11/2016 5:02:03 PM] @jonathanolson: yeah
[1/11/2016 5:02:12 PM] @pixelzoom fixed and pushed
[1/11/2016 5:02:42 PM] @jonathanolson: also now line 463, and possibly elsewhere :(
[1/11/2016 5:03:00 PM] @pixelzoom ok, i'll fix that too and look for others
[1/11/2016 5:06:36 PM] @jonathanolson: Looks like the logic to inspect "current" translations was for https://github.com/phetsims/rosetta/issues/101
[1/11/2016 5:07:25 PM] @jonathanolson: which was all done after my code-review of build-server, which is why I didn't recall that code
[1/11/2016 5:08:57 PM] @pixelzoom ah... s*t. you know what's going on? I'm doing a maintenance release, and therefore have an older version of chipper.
[1/11/2016 5:08:58 PM] @jonathanolson: https://github.com/phetsims/rosetta/issues/101#issuecomment-155251142
[1/11/2016 5:09:07 PM] @jonathanolson: > however maintenance releases of sims that don't have the latest chipper will need to use grunt deploy-production --locales=
[1/11/2016 5:09:14 PM] @jonathanolson: yup
[1/11/2016 5:09:20 PM] @pixelzoom I tried that and it's not passed through.
[1/11/2016 5:09:36 PM] @pixelzoom grunt deploy-production --noDev --locales=*
[1/11/2016 5:10:15 PM] @jonathanolson: we can probably manually put together the build-server call that should build it
[1/11/2016 5:10:32 PM] @jonathanolson: or just build locally and SCP it up
[1/11/2016 5:12:30 PM] @jonathanolson: or patch the old deployProduction.js equivalent to pass locales=*
[1/11/2016 5:12:39 PM] @pixelzoom or build with chipper from the maintenance branch, then check out chipper master to do the deploy-production?
[1/11/2016 5:12:52 PM] @jonathanolson: no
[1/11/2016 5:12:55 PM] @pixelzoom ?
[1/11/2016 5:13:01 PM] @jonathanolson: it will try to build with that chipper SHA then
[1/11/2016 5:13:06 PM] @jonathanolson: on the build-server
[1/11/2016 5:13:49 PM] @pixelzoom ah right, deploy-production ships over the shas.
[1/11/2016 5:14:29 PM] @jonathanolson: I don't understand this:
However, we don't want to build locales=* because that will build everything that has strings in babel.
[1/11/2016 5:14:38 PM] @jonathanolson: (comment in https://github.com/phetsims/rosetta/issues/101#issuecomment-155251142)
[1/11/2016 5:15:00 PM] @pixelzoom yeah... build-server has a lot of odd stuff in it that makes little sense to me.
[1/11/2016 5:15:02 PM] @jonathanolson: I thought babel kept strings "not yet ready" in its DB
[1/11/2016 5:15:41 PM] @pixelzoom maybe that's why getLocales function looks at "latest version" and builds only locales that were in that version.
[1/11/2016 5:16:01 PM] @jonathanolson: yes, that's rosetta #101, which I don't think should have been fixed like that
[1/11/2016 5:16:12 PM] @jonathanolson: It seems like defaulting to * is exactly what is desired
[1/11/2016 5:17:26 PM] @pixelzoom well... shall I attempt to patch deployProduction.js ?
[1/11/2016 5:17:56 PM] @jonathanolson: Probably the easiest, yes
[1/11/2016 5:18:08 PM] @pixelzoom feels like it might be a rathole.
[1/11/2016 5:18:09 PM] @jonathanolson: https://github.com/phetsims/perennial/commit/98c820aa08c5da92df01a7b0ef33c06f14fb657c#diff-becbf176979403e2eaca3527f9aea5c5 is the commit that moved things over to looking at "latest version"
[1/11/2016 5:18:17 PM] @jonathanolson: or just build it on your computer and SCP it
[1/11/2016 5:18:28 PM] @jonathanolson: although the server XML is wrong
[1/11/2016 5:18:30 PM] @jonathanolson: so don't do that
[1/11/2016 5:18:41 PM] @samreid: CM ask JO if my brainstorm workaround would be better
[1/11/2016 5:19:14 PM] @pixelzoom SR proposed that we should open each translation in Rosetta, pretend you are a translator. and then press "submit".
[1/11/2016 5:19:30 PM] @pixelzoom ... presumably to trigger a build of each translation,
[1/11/2016 5:19:40 PM] @samreid: But I haven't been able to follow the other proposed workarounds, so I am not sure what is better
[1/11/2016 5:19:45 PM] @jonathanolson: presumably easier to patch deployProduction
[1/11/2016 5:19:49 PM] @jonathanolson: I can assist with that
[1/11/2016 5:19:56 PM] @samreid: OK I trust your judgment
[1/11/2016 5:19:57 PM] @jonathanolson: that would trigger the developer as a translator
[1/11/2016 5:20:14 PM] @jonathanolson: and could cause the same translation-credits website-side failures due to lack of lastname
[1/11/2016 5:20:20 PM] @pixelzoom i don't want to figure out how to use rosetta, and possibly find more problems there.
[1/11/2016 5:20:55 PM] @pixelzoom if I patch deployProduction.js, do I need to commit it?
[1/11/2016 5:21:04 PM] @jonathanolson: presumably not
[1/11/2016 5:21:14 PM] @pixelzoom meant "edit locally" vs "patch".
[1/11/2016 5:21:31 PM] @jonathanolson: understood, presumably no need to commit
[1/11/2016 5:22:38 PM] @samreid: Afk probably working later tonight
[1/11/2016 5:23:01 PM] @pixelzoom I'm going to try applying this commit: https://github.com/phetsims/chipper/commit/dfa45c1613a09c16d71afc9a896e8a246adc41da
[1/11/2016 5:23:25 PM] @jonathanolson: ok
[1/11/2016 5:24:05 PM] @pixelzoom skeptical, because my deployProduction.js looks very different.
[1/11/2016 5:24:39 PM | Edited 5:25:02 PM] @pixelzoom bet a default of locales=\ doesn't get passed through, since explicit locales=\ didn't.
[1/11/2016 5:25:17 PM] @jonathanolson: http://phet.colorado.edu/deploy-html-simulation, with all the query parameters filled in
[1/11/2016 5:25:37 PM] @jonathanolson: though encoding the stringified array is a bit more complicated
[1/11/2016 5:26:32 PM] @pixelzoom I'm not even going to try this patched deployProduction.js. I can tell it's not going to work.
[1/11/2016 5:27:27 PM] @jonathanolson: Recommendation: checkout master chipper, replace request( url, ... ) with console.log( url ). Replace chipper SHA in that with the chipper SHA you need, and load the URL in a browser
[1/11/2016 5:28:06 PM] @pixelzoom what might go wrong if I build translated html files locally and scp them to figaro?
[1/11/2016 5:28:33 PM] @jonathanolson: The beers-law-lab.xml that the website uses to determine titles/translation list wouldn't exist, and would need to be copied (fixable)
[1/11/2016 5:28:55 PM] @jonathanolson: make sure to copy dependencies.json
[1/11/2016 5:29:07 PM] @pixelzoom The only thing missing from beers-law-lab/1.3.1/ on figaro is the translated html files.
[1/11/2016 5:29:26 PM] @pixelzoom dependencies.json is already there, as is the English version.
[1/11/2016 5:29:37 PM] @jonathanolson: MISSING, yes, but 1.3.1/beers-law-lab.xml indicates English is the only translation
[1/11/2016 5:29:49 PM] @pixelzoom so I should be able to scp the translated html files, then fix beers-law-lab.xml.
[1/11/2016 5:29:55 PM] @jonathanolson: 1.3.0/beers-law-lab.xml includes all translations
[1/11/2016 5:30:10 PM] @jonathanolson: yes, then notify me and/or "synchronize" the project
[1/11/2016 5:30:13 PM] @pixelzoom cp 1.3.0/beers-law-lab.xml 1.3.1/
[1/11/2016 5:30:19 PM] @jonathanolson: yes
[1/11/2016 5:30:27 PM] @pixelzoom ok, i'm going to give that a try
[1/11/2016 5:30:33 PM] @jonathanolson: ok
[1/11/2016 5:30:41 PM] @pixelzoom all this for a 30-second maintenance fix
[1/11/2016 5:31:09 PM] @jonathanolson: :(
[1/11/2016 5:32:07 PM] @pixelzoom i suppose 'grunt --locales=*' will build everything in babel, and only the translations that were deployed in 1.3.0 should be scp'ed to 1.3.1
[1/11/2016 5:32:25 PM] @jonathanolson: It is my understanding that if it's IN babel, it's a live translation
[1/11/2016 5:32:53 PM] @jonathanolson: but I've run across enough stuff recently that I don't like to make assumptions
[1/11/2016 5:32:59 PM] @pixelzoom Not the case. There are translations in babel that were ported from Java and have not been officially submitted by a translator.
[1/11/2016 5:34:14 PM] @pixelzoom For BLL, 33 translations were just built from babel. 11 are deployed in 1.3.0.
[1/11/2016 5:37:18 PM] @jbphet I'm home. I'll be making dinner but can help in parallel.
[1/11/2016 5:38:31 PM] @jbphet I don't think it's true that everything in babel is a live translation. Java strings were moved in, and are used to help translators, but if no translator has touched it, than it is not yet live.
[1/11/2016 5:39:08 PM] @jbphet Sorry, just realized that I reiterated CM's last post. Didn't quite scroll to the end of the dialog so far.
[1/11/2016 5:45:11 PM] @pixelzoom now i can't copy the translated html files to beers-law-lab/1.3.1 due to permissions.
[1/11/2016 5:45:24 PM] @pixelzoom they are in
[1/11/2016 5:45:46 PM] @jonathanolson: ahh, the phet-admin account probably
[1/11/2016 5:46:08 PM] @pixelzoom i'm about to be kicked out of the coffee shop/
[1/11/2016 5:46:18 PM] @jonathanolson: ok, I can complete it
[1/11/2016 5:46:24 PM] @jonathanolson: FYI, phet-admin credentials are in https://phet.unfuddle.com/a#/projects/9404/notebooks/17996/pages/105766/latest
[1/11/2016 5:46:46 PM] @pixelzoom thanks, that would be appreciated. it's my 30th wedding anniversary and I'm already 1.5 hours late because of this mess.
[1/11/2016 5:47:26 PM] @jonathanolson: Congratulations, and sorry! I'll definitely tackle from here.
[1/11/2016 5:47:30 PM] @jonathanolson: ls
[1/11/2016 5:47:38 PM] @pixelzoom thanks. afk
[1/11/2016 5:52:29 PM] @jonathanolson: ok, should be up and fixed now
[1/11/2016 6:04:14 PM] @jbphet Thanks JO. I just spot checked a few of the translations and they look good.
[1/11/2016 6:07:04 PM] @samreid: working
[1/11/2016 6:08:04 PM] @samreid: it would be good to get a TLDR of the above scenario—it’s a bit difficult to follow and I’m not sure what the next steps or takeaway message area
[1/11/2016 6:09:13 PM] @jbphet Summary is that a workaround was done for this deployment, a long term solution should be implemented.
[1/11/2016 6:11:00 PM] @samreid: long term solution = the ability to deploy maintenance releases of translated sims?
[1/11/2016 6:13:48 PM] @jbphet Yes, for releases with an older version of Chipper. Some things were changed about the interaction between Chipper and the build server, and some fixes were made (see https://github.com/phetsims/rosetta/issues/101) but something about the fix isn't working.
[1/11/2016 6:14:25 PM] @jbphet It's now on agenda for dev meeting tomorrow.

Most likely caused by running git pull after modifying the simInfoArray file.

error: Your local changes to the following files would be overwritten by merge:
        data/simInfoArray.json
Please, commit your changes or stash them before you can merge.

There is a method in build-server.js called getLocales that is supposed to find the most recent directory, then find the file within that directory that contains the set of translated locales. The code that attempts to identify the most recent directory is inadequate. It looks like this:

        var files = fs.readdirSync( simDirectory );
        var latest = files.sort()[ files.length - 1 ];

This suffers from several issues, not the least of which is that it will identify 1.1.9 as being the latest once 1.1.10 is published. It also identifies non-directories as being directories. This issue recently caused an issue with maintenance deployments.

This problem already cropped up in Rosetta (the translation utility), see phetsims/rosetta#137, and we could use a similar solution. This will result in some duplicated code across repositories, but at this point that is probably acceptable.

When a fix needs to be pushed out (e.g. recently https://github.com/phetsims/phet-android-app/issues/16, phetsims/phetcommon#37, phetsims/vibe#30, https://github.com/phetsims/special-ops/issues/68), I've patched master and every release branch for every live simulation.

This currently misses simulations that have a different branch (or are not on the website yet) for RCs, e.g. phetsims/graphing-slope-intercept#17.

What's the best way to handle identifying all of the extra branches that I'd need to patch (new sims with an RC branch, existing sims with a newer RC branch like BASE, etc.)?

As of this writing, the build server copies the results of its build to the dev server (currently spot) when doing a release candidate (RC), but copies the local version (i.e. from the developer's machine) to the dev server when doing a production release. I'm not sure why it was done this way, but we discussed it in a recent dev meeting and decided that the version built by the build-server should be copied to the dev server in both cases.

Here's a few lines of code in build-server.js that will need to change, there may well be other changes required too:

  if ( option === 'rc' ) {
    spotScp( afterDeploy );
  }
  else {
    // ...

See phetsims/chipper#445 for details on why this is needed.

Will want "npm prune" before every "npm install".

The changes to inspect the .xml to see what translations were previously deployed also seems to have broken this particular case. The file in question (/data/web/htdocs/phetsims/sims/html/molecule-shapes-basics/1.0.0-dev.5/molecule-shapes-basics.xml) does NOT exist, as it was deployed a long time ago.

2016-01-14T22:08:56.730Z - info: detecting version number: 1.1.0-rc.2
2016-01-14T22:08:56.730Z - info: building sim molecule-shapes-basics
2016-01-14T22:08:56.731Z - info: wrote file ./js/build-server/tmp/dependencies.json
2016-01-14T22:08:56.731Z - info: running command: git pull
2016-01-14T22:08:57.009Z - info: git pull ran successfully in directory: .
2016-01-14T22:08:57.009Z - info: running command: npm install
2016-01-14T22:08:58.920Z - info: npm install ran successfully in directory: .
2016-01-14T22:08:58.920Z - info: running command: ./chipper/bin/clone-missing-repos.sh
2016-01-14T22:08:58.945Z - info: ./chipper/bin/clone-missing-repos.sh ran successfully in directory: ..
2016-01-14T22:08:58.946Z - info: pulling from assert
2016-01-14T22:08:58.954Z - info: pulling from axon
2016-01-14T22:08:58.961Z - info: pulling from brand
2016-01-14T22:08:58.966Z - info: pulling from chipper
2016-01-14T22:08:58.969Z - info: pulling from dot
2016-01-14T22:08:58.973Z - info: pulling from joist
2016-01-14T22:08:58.978Z - info: pulling from kite
2016-01-14T22:08:58.984Z - info: pulling from molecule-shapes
2016-01-14T22:08:58.989Z - info: pulling from molecule-shapes-basics
2016-01-14T22:08:58.996Z - info: pulling from nitroglycerin
2016-01-14T22:08:59.001Z - info: pulling from phet-core
2016-01-14T22:08:59.006Z - info: pulling from phetcommon
2016-01-14T22:08:59.019Z - info: pulling from scenery
2016-01-14T22:08:59.024Z - info: pulling from scenery-phet
2016-01-14T22:08:59.028Z - info: pulling from sherpa
2016-01-14T22:08:59.031Z - info: pulling from sun
2016-01-14T22:08:59.034Z - info: pulling from tandem
2016-01-14T22:08:59.560Z - info: running command: grunt checkout-shas --buildServer=true --repo=molecule-shapes-basics
2016-01-14T22:09:00.179Z - info: grunt checkout-shas --buildServer=true --repo=molecule-shapes-basics ran successfully in directory: .
2016-01-14T22:09:00.179Z - info: running command: git checkout 199cd6ef219cb9776231e28ab7a812a6984029db
2016-01-14T22:09:00.190Z - info: git checkout 199cd6ef219cb9776231e28ab7a812a6984029db ran successfully in directory: ../molecule-shapes-basics
2016-01-14T22:09:00.190Z - info: running command: npm install
2016-01-14T22:09:01.456Z - info: npm install ran successfully in directory: ../chipper
2016-01-14T22:09:01.456Z - info: running command: npm install
2016-01-14T22:09:15.931Z - info: npm install ran successfully in directory: ../molecule-shapes-basics

fs.js:438
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^
Error: ENOENT, no such file or directory '/data/web/htdocs/phetsims/sims/html/molecule-shapes-basics/1.0.0-dev.5/molecule-shapes-basics.xml'
    at Object.fs.openSync (fs.js:438:18)
    at Object.fs.readFileSync (fs.js:289:15)
    at getLocales (/data/share/phet/phet-repos/perennial/js/build-server/build-server.js:451:28)
    at /data/share/phet/phet-repos/perennial/js/build-server/build-server.js:818:25
    at /data/share/phet/phet-repos/perennial/js/build-server/build-server.js:288:27
    at ChildProcess.exithandler (child_process.js:646:7)
    at ChildProcess.emit (events.js:98:17)
    at maybeClose (child_process.js:756:16)
    at Socket.<anonymous> (child_process.js:969:11)
    at Socket.emit (events.js:95:17)

Encountered while attempting to publish a maintenance release in phetsims/unit-rates#203 (comment).

'status.sh' (and any other piece of production code, for that matter) should stop when it encounters fatal errors.

Example:

% status.sh
....
chipperfatal: ambiguous argument 'origin/phetioAPIDocs...HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
                                   2d1b574e80bcf70cbc1ae7ccda6289a51a98875b/Users/cmalley/PhET/GitHub/chipper/bin/status.sh: line 69: [: : integer expression expected
/Users/cmalley/PhET/GitHub/chipper/bin/status.sh: line 73: [: : integer expression expected

...

scenery-phetfatal: ambiguous argument 'origin/dag-movable-drag-handler': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
                                   6213dc27544de739f31722d59f1a756e20ca03fb/Users/cmalley/PhET/GitHub/chipper/bin/status.sh: line 69: [: : integer expression expected
/Users/cmalley/PhET/GitHub/chipper/bin/status.sh: line 73: [: : integer expression expected
...
%

These "fatal" error were buried way back in the output, so they took me awhile to notice.

Assigning to @ariel-phet to prioritize and assign. Since this is in perennial, it is unrelated to "chipper 2.0". @jonathanolson is the author of status.sh.

Investigate moving away from callback pyramid of doom. Suggestions for improvement were Async await (@samreid), Promises(@jbphet), and fibers/future(@mattpen).

On April 1 at 3:25 PM, a bulk rebuild of a bunch of sims was initiated. During this process the build server copied translated files for some, but not all, of the simulations to spot (our development server). I'm not at all sure why this would occur, and have dug into it a little bit and recorded my findings below.

For an example where the translated sims were copied over, see http://www.colorado.edu/physics/phet/dev/html/graphing-lines/1.1.4-rc.1/. For an example where they weren't, see http://www.colorado.edu/physics/phet/dev/html/faradays-law/1.1.4-rc.1/.

In the build server log, the messages associated with the bulk rebuild start at time stamp Apr 01 2016 15:25:28 MDT.

There are several problems with this:

• all of these translated sim files on spot will chew up disk space
• if the translations are all getting rebuilt, this process will take longer
• all the transfers to spot cause an occasional failure of the scp command, which in turn results in a "BUILD FAILED" email getting sent out.

I'd like to clarify the desired behavior with the other devs. My assumption is that translations should not be rebuilt when an RC build is performed. For production builds, translations should get rebuilt, but they should not be copied to spot. @pixelzoom, @samreid, and @jonathanolson - can you chime in and let me know if you concur with this? Thanks.

As of this writing, the build server log contains many instances of the following (with different time stamps):

 Apr 01 2016 15:41:47 MDT - info: sent email [object Object]

I believe that the intent was to print out the contents of the message in the log, but the code currently uses message.toString() to print what I believe is a JavaScript object. This should probably be changed to something like JSON.stringify( message).

It looks like addTranslator() was added in 5ac71c3 as part of phetsims/rosetta#83 after I code-reviewed the build server. It apparently now makes a direct DB connection to the production website server.

This can:

1. Cause random failures to insert translators.
2. Cause random failures on the website side with things that deal with translators and sim translations, and has the potential to cause website stability issues.
3. Cause errors similar to what @samreid and I experienced when we were trying to remove an HTML sim translation, as the translation appeared in some places, but didn't appear in the website's administration interface (totally explains the weirdness I experienced earlier).

@ariel-phet, I'm tagging this as high priority due to the above, and the fact that I see this has errored out 149 times in the build server log. Feel free to reprioritize (assigned to you).

Reasons:

1. The website uses an in-memory cache in front of the DB. If you query the DB for information, it can be out of date, and more seriously, if you insert information into the DB, it can (a) conflict with the in-memory representation and possibly trigger errors the website tries to flush the cache to the DB, and (b) the website won't realize for a certain amount of time (possibly indefinitely?) that the information has been updated.
2. Due to this, the website essentially "owns" the database. Modifications to it should go through Hibernate and its cache. This should only be bypassed by manual fixes by a website developer (in the past, I've had to flush the cache to the DB, make the change, and clear the cache, but build-server isn't doing that, and shouldn't). Removing the caching almost certainly not a good solution due to performance reasons.
3. E.g. the following error in build-server's logs (since things got out-of-sync):

Feb 04 2016 07:41:34 MST - info: running SQL command: INSERT INTO user_localized_simulation_mapping SELECT localized_simulation.id, phet_user.id from localized_simulation, simulation, project, phet_user WHERE simulation.name =
 'ph-scale-basics' AND locale = 'vi' AND phet_user.id = 137535 AND simulation = simulation.id AND simulation.project = project.id AND project.type = 2
Feb 04 2016 07:41:34 MST - error: adding to user_localized_simulation_mapping, translator probably already exists for this sim and locale
Feb 04 2016 07:41:34 MST - error:  error: duplicate key value violates unique constraint "user_localized_simulation_mapping_pkey"
    at Connection.parseE (/data/share/phet/phet-repos/perennial/node_modules/pg/lib/connection.js:539:11)
    at Connection.parseMessage (/data/share/phet/phet-repos/perennial/node_modules/pg/lib/connection.js:366:17)
    at Socket.<anonymous> (/data/share/phet/phet-repos/perennial/node_modules/pg/lib/connection.js:105:22)
    at Socket.emit (events.js:95:17)
    at Socket.<anonymous> (_stream_readable.js:764:14)
    at Socket.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:426:10)
    at emitReadable (_stream_readable.js:422:5)
    at readableAddChunk (_stream_readable.js:165:9)
    at Socket.Readable.push (_stream_readable.js:127:10)

I'd like to coordinate with @jbphet (for the build-server/rosetta parts) and @mattpen (for the website) so we can set up a way where the build-server/rosetta can send an (authenticated) request to the website that will cause a translation entry to be created.

It also looks like Rosetta is requiring pg-query and setting it up so it can make queries, but it never actually does so. This should also be removed.

As I was reviewing the api reference for express, I noticed that the version we are using (3) is now deprecated. The current supported version is 4 and 5 is in alpha. After reviewing the migration guide it looks like the update should be relatively straightforward as we are only using one function that was removed.

If we upgrade build-server, it probably makes sense to upgrade rosetta as well.
@jonathanolson @jbphet - Does this sound ok with you? Any concerns?

From 9/29/15 dev meeting...

Take a closer look at what should be moved over from chipper.

checkoutShas.js was copied from chipper, then modified. One of these should be deleted or renamed.

In phetsims/chipper#565 we stopped using "grunt-eslint" and removed it from all package.json (including perennial's). However, I'm unsure how perennial works and whether that grunt-eslint entry may be necessary for building maintenance versions. I suspect not, because the sim will have the grunt-eslint entry but I'm just not 100% sure. Assigned to some people that I saw on recent commits in perennial.

I received this PUSH FAILED email on the aforementioned date:

Error: Repo createContents error

Error: Repo createContents error. Error committing to file acid-base-solutions/acid-base-solutions-strings_te.json

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Assigning @jbphet since it is related to build-server. Tagging @pixelzoom since it relates to acid-base-solutions.

EDIT: Formatting

@samreid noticed that the dependencies.json file is not being copied to spot for RC deployments, possibly production deployments too. I took a look through the code and the history, and I think that this commit may be implicated:

da982bd

It was an attempt at a solution to #20.

This change was probably not quite right, and needs to also make sure that the dependencies files are moved. Assigning to @mattpen.

Was doing RC deployments for maintenance releases (fairly rapid-fire) and saw an email alert:

***** Nagios *****

Notification Type: PROBLEM

Service: nodejs build-server process
Host: figaro.colorado.edu:
Address: figaro.colorado.edu
State: CRITICAL

Date/Time: Sat Mar 5 02:25:22 MST 2016

Additional Info:

PROCS CRITICAL: 0 processes with args /usr/local/nodejs/bin/node /data/share/phet/phet-repos/perennial/js/build-server/build-server.js

copied build-server log out, restarted build-server (via /etc/init.d/build-server start).

Notable lines:

/data/share/phet/phet-repos/perennial/node_modules/async/lib/async.js:30
            if (called) throw new Error("Callback was already called.");
                              ^
Error: Callback was already called.
    at /data/share/phet/phet-repos/perennial/node_modules/async/lib/async.js:30:31
    at /data/share/phet/phet-repos/perennial/js/build-server/build-server.js:308:13
    at /data/share/phet/phet-repos/perennial/js/build-server/build-server.js:295:27
    at ChildProcess.exithandler (child_process.js:646:7)
    at ChildProcess.emit (events.js:98:17)
    at maybeClose (child_process.js:756:16)
    at Socket.<anonymous> (child_process.js:969:11)
    at Socket.emit (events.js:95:17)
    at Pipe.close (net.js:465:12)

Looks like a potential async-related bug where our taskCallback is getting called twice.

Additionally, the fact that I had to semi-manually restart build-server worries me. We may want it to auto-restart if it dies.

@jbphet, can we discuss this on Monday?

This branch was created to separate unstable changes for the chipper:2.0 project from master. Currently the build-server switches perennial to the master branch as part of the deployment process, so we should not put anything that is not ready for production in the master branch.

The log currently logs events using UTC, and this makes them a bit tricky to interpret. It would be easier to work with these logs if they were in Mountain Time instead. This will need to be done for Rosetta too, see phetsims/rosetta#107.

Looks like most (all?) perennial issues should have the "chipper:2.0" label. Rather than label them all, I'm creating this "blanket" issue.

See phetsims/phet-info#39 (comment)

perennial and chipper have some identical grunt tasks, specifically checkout-shas and checkout-master. For perennial, those tasks require a --repo option. Which I habitually forget because I'm used to the chipper tasks. And I get this downstream error message:

% cd perennial
% grunt checkout-shas
Running "checkout-shas" task
Warning: Unable to read "../undefined/dependencies.json" file (Error code: ENOENT). Use --force to continue.

Aborted due to warnings.
%

Yet another example of not testing that something required is indeed present.

The request received by the build server is not currently shown in the log. This makes it tricky when trying to debug issues, because different versions of chipper can end up sending somewhat different requests. The full URL itself should appear in the log.

There is an issue where npm install doesn't work on Windows due to some oddness about invoking it from scripts. As of this writing, the problematic code looks like this:

this.execute( 'npm', [ 'install' ], '../' + repo, function() {
        callback();
      } );

@jonathanolson put a workaround for a similar issue into test-server.js in chipper, and it looked like this:

  var npmInstall = spawn( 'node', [ 'C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js', 'install' ], {
    cwd: rootDir + simName
  } );

This is, however, not a portable solution. We need to figure out something that will work on Windows as well as Unix-ish machines (i.e. OSX and Linux).

According to its README.md, perennial is "Maintenance tools that won't change with different versions of chipper checked out". It consists of 5 .js files.

(1) Why does it have 17 devDependencies in package.json?

(2) Does it really need to use lodash? (See phetsims/sherpa#58 (comment).)

Discussed over Skype that it would be useful to be notified of some build fails but not all.

CM: 'd like to be notified of the build status of my own builds, but not particularly interested in other builds. would it be possible to add an optional mailTo address to build-local.json, that gets passed through to build-server?

I have found that this folder name is misleading. It is named as though anything on the server could access the repo needed to access its content. Recently I was told that in fact this is not the case, because the build-server uses these repos to checkout shas and make production builds. Maybe it would be more clear to rename this to 'phet-repos-for-build-server' or 'phet-repos-no-touching' or something to dictate how it's off limits. Then we could have another folder called 'come-one-come-all-for-the-phet-repos-that-you-need'. This directory could server all of our other phet-repo needs on phet-server. Here are the cases I have run into so far:

• phet-io-website ( to deploy to the static dir)
• phet-office-mix (for the chron job that updates the store.html)
• installer-builder (perhaps deprecated because it is running on bayes now)

All of these currently have a top level directories in /data/share/phet/. This seems like a slippery slope to me. Marking for developer meeting to discuss some organization of phet-server:/data/share/phet/

I noticed a few places where there were lint errors, I don't think they should be there.

Ongoing work for #12. I'd like to have it fully cleaned up and documented before it is moved to master.

We are running into some issues with the buildup of historical versions of the simulations and their translations, see https://github.com/phetsims/installer-builder/issues/10. During the 3/17/2016 developer meeting it was decided that we don't need to keep these old versions around, since we will have them on spot or its replacement anyway. There is an issue to remove them from the web server itself, see https://github.com/phetsims/website/issues/408, but the build server will also need to be modified to remove old versions each time new ones are deployed.

The file build-server.js has a require statement for pg-query, and the it appears to configure a query but never use it. The actual code looks like this:

var query = require( 'pg-query' ); // eslint-disable-line
   .
   .
   .
// configure postgres connection
if ( BUILD_SERVER_CONFIG.pgConnectionString ) {
  query.connectionParameters = BUILD_SERVER_CONFIG.pgConnectionString;
}
else {
  query.connectionParameters = 'postgresql://localhost/rosetta';
}

I've looked through the history of the code and the query code was originally added on 10/27/2015 (commit 5ac71c3) in order to put translation credits into the database. However, the current version of the code adds translation credits by using a URL to our server. It looks like the code that makes the query was removed, but the code that includes it and sets it up was not.

I'm going to remove it, then work with @mattpen to get this inspected and tested.

We are occasionally seeing errors with the build server when it is transferring a lot of files from the server on which it's running (figaro as of this writing) to the dev server (spot as of this writing). The errors in the build server log look like this:

Apr 01 2016 16:16:06 MDT - error: error running command: scp wave-on-a-string_nl.html [email protected]:/htdocs/physics/phet/dev/html/wave-on-a-string/1.1.2-rc.1 in ../wave-o n-a-string/build. build aborted.

@jonathanolson says that he thinks that this might be due to limitations imposed by CU's IT infrastructure that limit the number of sequential SSH sessions that can be set up, and each one of these SCP requests may be creating a new SSH session. We may be able to fix this using something where a connection is established and multiple files are transferred.

During phetsims/fraction-matcher#92

I did:

$ grunt deploy-rc
Running "deploy-rc" task

Then after a minute or two it printed out:

Warning: Build request failed with status code 503. Use --force to continue.

I'm unfamiliar with perennial and the build-server so I'm not sure what's wrong.

Currently the URL outputted from grunt deploy-production/rc is all you need to start the build. This is because it is a GET request, and the authentication is housed in the URL itself. This is quite unsafe. We realized this when a url was pasted into the skype chat, and 12 hours later a Skype bot had started a build.

Two ways to secure the build requests:

• Make the request a POST.
• Put authentication into the headers, basic authentication is probably all we need.

build-server.js current has what is often referred to as a 'pyramid of doom' (see https://en.wikipedia.org/wiki/Pyramid_of_doom_(programming) caused by a long sequence of asynchronous system calls. Here is a screenshot of the code:

This is hard to read and maintain, so I started investigating a JavaScript construct called "promises" as an alternative way to handle this. However, during this investigation, I found that there may be a simpler alternative: child_process.execSync. The code currently uses child_process.exec, which is asynchronous and leads to the pyramid, but it may be possible to fix this by simply using a synchronous version of the same call.

I've used up my budgeted time for this sort of investigation for this week, but will pick it up again in a week or two.

The API for the build server, which is essentially a URL, should be modified to include a version number. This will make it easier to make changes to the build server's behavior without introducing incompatibilities with previous chipper versions, which are used to make maintenance releases of simulations. For background on the motivation for this, see #8 and phetsims/rosetta#101.

Branch involved at determining what the latest published sim will be on the phet-io website. Discussion in https://github.com/phetsims/phet-io-wrappers/issues/30#issuecomment-333233629.

Phethelp received an email this morning requesting to readd the ar translations for concentration and molecules-and-light to the website. Inspecting the archived built sim files, it was discovered that the built translations for these two sims are missing from the versions after June 22. The last version with all of the translations is 1.2.5 for concentration 1.1.5 for molecules-and-light.

Currently, if a user submits a translation, then makes some mods and resubmits, the build server will log an error in the log file. This error indicates that there is already an entry for this user-sim-locale combination. Since this is a normal condition, it would be good to keep the logs clean and either check the DB first or catch the error.

The error looks like this:

12437 2015-11-20T14:08:51.540Z - info: running SQL command: INSERT INTO user_localized_simulation_mapping SELECT localized_simulation.id, phet_user.id from localized_simulation, simulation, project, phet_      user WHERE simulation.name = 'ph-scale' AND locale = 'da' AND phet_user.id = 130982 AND simulation = simulation.id AND simulation.project = project.id AND project.type = 2
12438 2015-11-20T14:08:51.557Z - error: adding to user_localized_simulation_mapping, translator probably already exists for this sim and locale
12439 2015-11-20T14:08:51.557Z - error:  error: duplicate key value violates unique constraint "user_localized_simulation_mapping_pkey"
12440     at Connection.parseE (/data/share/phet/phet-repos/perennial/node_modules/pg/lib/connection.js:539:11)
12441     at Connection.parseMessage (/data/share/phet/phet-repos/perennial/node_modules/pg/lib/connection.js:366:17)
12442     at Socket.<anonymous> (/data/share/phet/phet-repos/perennial/node_modules/pg/lib/connection.js:105:22)
12443     at Socket.emit (events.js:95:17)
12444     at Socket.<anonymous> (_stream_readable.js:764:14)
12445     at Socket.emit (events.js:92:17)
12446     at emitReadable_ (_stream_readable.js:426:10)
12447     at emitReadable (_stream_readable.js:422:5)
12448     at readableAddChunk (_stream_readable.js:165:9)
12449     at Socket.Readable.push (_stream_readable.js:127:10)