I destroyed my stack and tried to recreate it, getting this error:

TASK: [authdb | configure fxa-auth-db-server] ********************************* 
fatal: [ec2-54-189-182-138.us-west-2.compute.amazonaws.com] => {'msg': "One or more undefined variables: 'dict object' has no attribute 'stack'", 'failed': True}
fatal: [ec2-54-189-182-138.us-west-2.compute.amazonaws.com] => {'msg': "One or more undefined variables: 'dict object' has no attribute 'stack'", 'failed': True}

FATAL: all hosts have already failed -- aborting

Should fxa-oauth-console be part of fxa-dev?

https://github.com/mozilla/fxa-oauth-console

@dannycoates @ckarlof any concerns about this?
Should I file a PR to add it to fxa-dev?
i.e https://oauth-console-latest.dev.lcip.org/

See mozilla/fxa-content-server#1596

Per mozilla/fxa-content-server#1596 (comment) it seems like @vladikoff had to SSH in to the latest.dev box and manually update the bower deps in order to force the latest legal-docs content.

Seems like it is stuck on 0.30

{"version":"0.30.0","commit":"f31671fd602ace652b65f8b7721869f67185f4fb","l10n":"42c7e65027","tosPp":"29ec46747a"}

https://oauth-latest.dev.lcip.org/console/ gives an error.

I think we can get @pdehaan on this one:

https://github.com/dannycoates/fxa-dev/blob/f5f80c54d553b60b61c010d284ac21676d001e56/roles/content/tasks/main.yml#L12

To get this working I had specify an explicit inventory containing:

[localhost]
127.0.0.1 ansible_python_interpreter=/usr/local/bin/python

and then in the aws dir, I did:

ansible-playbook dev.yml -i <path to inventory file>

I installed ansible and boto with sudo pip install

WIll file a PR shortly, just filing this and some other to keep track while trying a scratch fxa-dev build.

Anything less than [email protected] is 1. flaky, and 2. unsupported. So upgrade from the version that comes with nodejs 0.10.x

Dev and prod ones here:

https://bugzilla.mozilla.org/show_bug.cgi?id=1052883#c9

For some reason I'm getting:

Both got stuck on: TASK: [create basic stack (this will take a while)]

The configs I tried were vlad2 and vlad:

---
region: us-west-2
subdomain: vlad2.dev
hosted_zone: lcip.org
ssl_certificate_name: wildcard.dev.lcip.org
rds_password: Q&}PzHU79J8Ex}3,
cron_time:
  minute: 0
  hour: 0
  month: 1
  day: 1

This is a new stack, did I forget some step? I just need a deploy with oauth_git_version: register for testing.

The marketplace stage config (https://marketplace.allizom.org) returns to https://marketplace.stage.mozaws.net/fxa/authorize but we'll need it to mirror the dev config. We may as well set up production also if that isn't already.

Stage: https://marketplace.allizom.org/fxa-authorize
Production: https://marketplace.firefox.com/fxa-authorize

Marketplace dev config: https://github.com/washort/fxa-dev/blob/44929fb433cd26ee2fce6b1d26c75e47926fff32/roles/oauth/templates/config.json.j2#L36

I'm not sure what needs to change for this but I'm happy to make the change.

In cloud services show and tell, it seems to be all the rage with the kids.

@st3fan says he has already converted the FxA infrastructure to Docker. Thoughts?

The page you are looking for is temporarily unavailable. Please try again later. ⚠️

Per @ckarlof's #44 (comment)

Can we please move this to mozilla ownership @rfk?

Earlier, it looks like the latest/nightly boxes were stuck on older versions of fxa-content-server repo (specifically mozilla/fxa-content-server@4b1c510 — @ckarlof manually kicked off some server magic to get it refreshed).

Not sure what reporting we have on the fxa-dev boxes or if they're sending out emails on failures. It'd be nice to add the cloud services QA alias added to notifications, and whatever devs also want to be notified.

@jrgm noted in IRC:

So, somehow /data/fxa-content-server/server/templates/pages/dist is owned by root, and the app user is not able to rebuild over that with grunt build.

The API docs mention this URL but it seems to not be deployed properly.

After #12 is ready, it would be nice to replace our existing dev Oauth integration infrastructure with a "dannybox". The existing one uses a memory DB for the Oauth DB and that's inconvenient for them.

This stack should use the production auth API server (api.accounts.firefox.com), like our existing awsbox.

Let's keep the one for Marketplace one for now, but in addition let's have a generic one that's sandboxed (i.e., not connected to prod accounts, but uses the included auth server) and uses persistent storage. Let's also make sure it has more generic DNS name (i.e., doesn't include the word "marketplace").

How do we update these boxes if we didn't originally create them?

get email working for the basic stack on aws

http://docs.aws.amazon.com/ses/latest/DeveloperGuide/postfix.html?tag=vig-20

Not sure why this is happening yet

My fix in #127 is just to make stuff work for now (latest was down).
But that doesn't feel like the right fix (for one thing, it has redundant strings of 'latest.dev' and 'lcip.org').

Other boxes will need a fix soon.

Hey @dannycoates, weird issue. Seems like I can only get one verification email out. no matter how many times I press "Resend email" it never works.

• Go to https://vlad.dev.lcip.org/signup, sign up
• click resend email a few times.
• Only 1 email will be inbox.

This breaks the OAuth functional test because it tests unverified OAuth sign in.

https://vlad.dev.lcip.org:4443/
{"version":"0.13.0","commit":"dfea74224e4106fd4c85eb8ebc19d18f3385cc8c"}

Sometimes (maybe always) this keypair is not being generated (or not in the right place).

fs.js:439
return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
^
Error: ENOENT, no such file or directory '/data/fxa-rp/secret-key.json'
at Object.fs.openSync (fs.js:439:18)
at Object.fs.readFileSync (fs.js:290:15)
at Object.KeyPair (/data/fxa-rp/node_modules/

The latest.dev.lcip.org server may suddenly update while the content server functional tests are running. This happens when multiple PRs get merged quickly. This leads to page load timeouts and test failures.

@dannycoates do you have any ideas how to fix this up?

Here are the build Triggers that our TeamCity server offers:

Currently, it exposes the different services using different ports, which reduces readability and IMO is less intuitive. It would be nice if the different services could be exposed as, e.g., accounts.dev.licp.org and api-accounts.dev.lcip.org, like the current setup.

This is odd. Just built a box from scratch and it kept dying with

TASK: [common | start supervisord] ******************************************** 
failed: [ec2-52-10-10-127.us-west-2.compute.amazonaws.com] => {"failed": true}
msg: start: Job failed to start

After looking, it appears that maybe pip is now installing into /usr/local/bin, so supervisord isnt' where initctl is looking for it. Weird.

502 Bad Gateway

Hilariously failing on "install nonsense". Not sure we need phantomjs in fxa-dev, but hey, I just work here.

TASK: [content | install nonsense] ********************************************
changed: [ec2-54-214-150-49.us-west-2.compute.amazonaws.com] => (item=bower)
changed: [ec2-54-214-150-49.us-west-2.compute.amazonaws.com] => (item=grunt-cli)
failed: [ec2-54-214-150-49.us-west-2.compute.amazonaws.com] => (item=phantomjs) => {"cmd": "/usr/bin/npm install --global phantomjs", "failed": true, "item": "phantomjs", "rc": 1}
stderr: Error requesting archive.
Status: 503
Request options: {
  "uri": "https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.7-linux-x86_64.tar.bz2",
  "encoding": null,
  "followRedirect": true,
  "headers": {},
  "strictSSL": true
}
Response headers: {
  "server": "nginx/1.5.10",
  "date": "Thu, 02 Oct 2014 00:24:03 GMT",
  "content-type": "text/html",
  "transfer-encoding": "chunked",
  "connection": "keep-alive",
  "cache-control": "no-cache"
}
Make sure your network and proxy settings are correct.

If you continue to have issues, please report this full log at https://github.com/Medium/phantomjs

npm ERR! [email protected] install: `node install.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the phantomjs package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node install.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls phantomjs
npm ERR! There is likely additional logging output above.
npm ERR! System Linux 3.10.35-43.137.amzn1.x86_64
npm ERR! command "/usr/bin/node" "/usr/bin/npm" "install" "--global" "phantomjs"
npm ERR! cwd /home/ec2-user
npm ERR! node -v v0.10.32
npm ERR! npm -v 1.4.28
npm ERR! code ELIFECYCLE
npm ERR! not ok code 0

stdout: /usr/bin/phantomjs -> /usr/lib/node_modules/phantomjs/bin/phantomjs

> [email protected] install /usr/lib/node_modules/phantomjs
> node install.js

Looks like an `npm install -g`; unable to check for already installed version.
Downloading https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.7-linux-x86_64.tar.bz2
Saving to /usr/lib/node_modules/phantomjs/phantomjs/phantomjs-1.9.7-linux-x86_64.tar.bz2
Receiving...


msg: Error requesting archive.
Status: 503
Request options: {
  "uri": "https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.7-linux-x86_64.tar.bz2",
  "encoding": null,
  "followRedirect": true,
  "headers": {},
  "strictSSL": true
}
Response headers: {
  "server": "nginx/1.5.10",
  "date": "Thu, 02 Oct 2014 00:24:03 GMT",
  "content-type": "text/html",
  "transfer-encoding": "chunked",
  "connection": "keep-alive",
  "cache-control": "no-cache"
}
Make sure your network and proxy settings are correct.

If you continue to have issues, please report this full log at https://github.com/Medium/phantomjs

npm ERR! [email protected] install: `node install.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the phantomjs package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node install.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls phantomjs
npm ERR! There is likely additional logging output above.
npm ERR! System Linux 3.10.35-43.137.amzn1.x86_64
npm ERR! command "/usr/bin/node" "/usr/bin/npm" "install" "--global" "phantomjs"
npm ERR! cwd /home/ec2-user
npm ERR! node -v v0.10.32
npm ERR! npm -v 1.4.28
npm ERR! code ELIFECYCLE
npm ERR! not ok code 0

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/Users/Chris/dev.retry

E.g., for the Marketplace stack, can we preload the Marketplace devs' SSH keys?

@seanmonstar @jrgm I'm trying to debug the /v1/developer/activate endpoint.

The oauth-server log is reporting it as a 404.

{"Timestamp":1430319186915000000,"Logger":"fxa-oauth-server","Type":"summary.summary","Severity":6,"Pid":28961,"EnvVersion":"2.0","Fields":{"code":200,"errno":0,"method":"post","path":"/v1/verify","t":6,"auth":"null","payload":"[\"token\"]"}}
{"Timestamp":1430319186938000000,"Logger":"fxa-oauth-server","Type":"summary.summary","Severity":6,"Pid":28961,"EnvVersion":"2.0","Fields":{"code":404,"errno":999,"method":"post","path":"/v1/developer/activate","t":0,"auth":"null","payload":"[]"}}

Local curl seems to be getting to it:

[ec2-user@ip-172-31-12-251 ~]$ curl -XPOST 'http://127.0.0.1:9111/v1/developer/activate'
{"code":401,"errno":111,"error":"Unauthorized","message":"Unauthorized for route","info":"https://github.com/mozilla/fxa-oauth-server/blob/master/docs/api.md#errors","detail":"Bearer token not provided"}[ec2-user@ip-172-31-12-251 ~]$ 

We did add a new nginx config in this PR: #138
I have reloaded and restarted nginx on the box, but still getting the same issue.
any ideas?

prevent multiple ansible runs from running at the same time

I sadly haven't paid enough attention to understand this: why do we need to host the oauth stuff on a separate domain like "oauth-stable.dev.lcip.org" rather than a sub-path like "stable.dev.lcip.org/oauth"?

It would be great to simplify the hosting here to avoid confusion for new RPs, who we regularly see stumble on knowing that services are on which hostname.

Can we either reuse awsbox.json or designate a new config file that this environment is configured to load (requires setting CONFIG_FILES=<path/to/config.json>)? Then we could control certain build properties of this deployment from our repo without messing with defaults.

How to resolve

Either

• Link to possible vagrantbox.es
• Provide link to customized image

OR

• Install steps ends up with a fully working independent deployment of FxA
• (optional) do not require initial database

Building the VM:

Related notes

• Making work fxa-dev Vagrant VM from a vanilla fedora20 VM without database. that were done after the steps in this issue.

Initial procedure

Note Progress is described in the comment thread.

This procedure assumes one doesn’t have access to dcoates/fedora20 config.vm.box Vagrant box.

Environment:

• MacOS X as host
• Vagrant v1.6.5
• VirtualBox v4.3.6
• Installed Ansible through brew on Mac OS X host

Steps

1. Install Ansible

   brew install ansible
   

2. Adjust Vagrantfile with:

   config.vm.box = "fedora20"
   config.vm.box_url = "https://dl.dropboxusercontent.com/u/15733306/vagrant/fedora-20-netinst-2014_01_05-minimal-puppet-guestadditions.box"
   

3. Boot the VM for the first time

   First time it fails, its OK.

     make
   

4. Fun in the VM the following:

   vagrant ssh
   

   Inside the VM

     sudo -s
     yum update
     yum install -y npm git ansible
   

5. Halt the VM, restart the process

   exit
   

   Back on the host;

     vagrant halt
     make 
   

6. Add what’s missing inside the VM

   Inside the VM, as root;

     adduser app
     mkdir /data
     chown -R app:app /data/
     easy_install supervisor
     npm install -g grunt-cli
   

7. Get the code

   On the host;

     make update-code
   

   Wait.

   It breaks at fxa-auth-server, see npm-debug.log

8. Breaks halfway

   Here is how /data looks like

     drwxr-xr-x. 10 app app 4096 Dec  3 16:14 fxa-auth-db-server
     drwxr-xr-x.  6 app app 4096 Dec  3 16:11 fxa-auth-mailer
     drwxr-xr-x. 14 app app 4096 Dec  3 16:45 fxa-auth-server
     drwxr-xr-x. 10 app app 4096 Dec  3 16:17 fxa-content-server
     drwxr-xr-x. 10 app app 4096 Dec  3 16:22 fxa-customs-server
   

9. Delete an try again

   After reading the npm-debug.log. It mentionned to delete and try again.

   Inside the VM;
   rm -rf /root/tmp/npm-

   Trying again from the host;

     make update-code
   

   Got a different error message. It talked about missing gmp.h.

   

     yum install -y gmp.x86_64 gmp-devel.x86_64
   

10. Another make update-code attempt

    It worked! /o/

1. ... and a few make update

If you forget to specify https:// when connecting to the server, you'll get timeouts/connectivity errors. We should consider doing a redirect from http to https to avoid confusion.

Marketplace team will be happy.

[2014-08-27 21:58:59.598] fxa.CRITICAL: Error: PUBLIC_DIR does not exist: /data/fxa-profile-server/var/public
    at Object.<anonymous> (/data/fxa-profile-server/lib/img/local.js:16:9)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/data/fxa-profile-server/lib/img/index.js:12:22)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
[2014-08-27 21:59:04.647] fxa.CRITICAL: Error: PUBLIC_DIR does not exist: /data/fxa-profile-server/var/public
    at Object.<anonymous> (/data/fxa-profile-server/lib/img/local.js:16:9)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/data/fxa-profile-server/lib/img/index.js:12:22)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
[2014-08-27 21:59:09.068] fxa.CRITICAL: Error: PUBLIC_DIR does not exist: /data/fxa-profile-server/var/public

We had to make var/public manually.

123done now has a configuration option config_uri. This is used to allow 123done to embed FxA in an iframe. See mozilla/fxa-content-server#1386 and mozilla/123done#125.

Needs config changes for the fxa-auth-server and 123done made for all the environments.

It's low-hanging fruit to get more reliable delivery to test accounts, and avoid bounces.

To reproduce in fxa-content-server run:
npm run test-latest grep="Firefox Desktop Sync sign_in - sign in verified"

I think Jenkins is already running the functional tests against latest, but we should maybe add the remote auth server tests, too, and make sure the proper people are notified.

The whole spin-up-cloudformation-stack task takes a good long while and doesn't seem to print any progress updates. It might be fun to e.g. print out cloudformation events as they happen during creation of the stack for a bit of realtime feedback.

Found in https://latest.dev.lcip.org/ver.json (commit 276123d278dd096846d29223ce27d8e74a3566e2)

{
  "version": "0.13.0",
  "commit": "276123d278dd096846d29223ce27d8e74a3566e2",
  "l10n": "17102b27e7"
}

Steps to reproduce:

1. Open your developer tools (Network tab)
2. Open https://latest.dev.lcip.org/signup
3. Open a new tab.
4. Open https://accounts-latest.dev.lcip.org/signup

Actual results:

1. https://latest.dev.lcip.org/signup
   76 requests, 785.15 KB, 2.31s
2. https://accounts-latest.dev.lcip.org/signup
   9 requests, 273.78 KB, 0.28s

Expected results:

Not sure if latest.dev should be serving minimized, similar to accounts-latest.