Code Monkey home page Code Monkey logo

fxa-dev's Introduction

AWS Ansible-based docker development environment for Firefox Accounts

Prerequisites

macOS

Use: brew install ansible && pip install boto3

Usage

To run on AWS change directory to aws

cd aws
  1. Set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables
  2. create a environments/foo.yml file ('foo' can be anything) a) see environments/EXAMPLE.yml for a base reference b) it is recommended that you set values for owner and reaper_spare_me
  3. run make foo

After the cloudformation stacks has been created, cloud-init will run an initial ansible playbook to set up the box. A cronjob run every 10 minutes will pick up changes as needed. The logs for the initial playbook run are in /var/log/cloud-init-output.log. If the cloudformation was created OK, but the services do not come up, check that log output for why.

SSH

You can ssh into the EC2 instance with ssh ec2-user@meta-{{ whatever you configured in foo.yml }}.

Custom Docker tags

By default, the latest tag will be used. This can be adjusted to use other image tags by setting any of {auth_docker_tag, authdb_docker_tag, content_docker_tag, customs_docker_tag, oauth_docker_tag, profile_docker_tag, rp_docker_tag} in your environments/foo.yml configuration file.

NOTE: you must commit and push changes to that file to affect an existing EC2 instance.

Docker stopped|started:

By default, all docker containers are 'started'. If you want to selectively keep a service 'stopped', you can set any of {auth_docker_state, authdb_docker_state, basket_docker_state, content_docker_state, customs_docker_state, oauth_docker_state, profile_docker_state, rp_docker_state} in your environments/foo.yml configuration file.

NOTE: you must commit and push changes to that file to affect an existing EC2 instance.

Custom fxa-dev branch

You can control the branch of fxa-dev for each environment by changing the {fxadev_git_version} value in the environment configuration file.

Layout Notes

  • fxa sources are in /data/fxa-dev.
  • node processes are run by docker
    • config is setup by ansible docker_container module (e.g., roles/auth/tasks/main.yml)
    • run docker ps; docker images for info
  • ansible will do a docker pull, and restart the container if the image, or configuration, has changed.
  • nginx is the web frontend
    • config in /etc/nginx/conf.d
  • node process logs are available with, e.g., docker logs auth-server.

Example urls

About using docker_container and quoting of environment values

docker_container (>=2.8) now insists that environment values be quoted. However, when evaluating "{{ foo }}", those quotes are removed. So use the to_json jinja2 filter to ensure that the value is quoted. Note: I use to_json instead of quote because quote will not quote Boolean values true and false.

If not quoted, the error will look like "Non-string value found for env option. Ambiguous env options must be wrapped in quotes to avoid them being interpreted. Key: ENV_VAR_NAME". If you see this error, add a to_json in your templates and try again.

fxa-dev's People

Contributors

bbangert avatar chilts avatar ckarlof avatar ckolos avatar dannycoates avatar jaredhirsch avatar jbuck avatar johngruen avatar jrgm avatar kparlante avatar lmorchard avatar lzoog avatar mozilla-github-standards avatar mstriemer avatar nchapman avatar philbooth avatar rfk avatar seanmonstar avatar udaraweerasinghege avatar vbudhram avatar vladikoff avatar zaach avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fxa-dev's Issues

Build Oauth integration dev server

After #12 is ready, it would be nice to replace our existing dev Oauth integration infrastructure with a "dannybox". The existing one uses a memory DB for the Oauth DB and that's inconvenient for them.

This stack should use the production auth API server (api.accounts.firefox.com), like our existing awsbox.

preload SSH keys?

E.g., for the Marketplace stack, can we preload the Marketplace devs' SSH keys?

/v1/developer/activate is 404ing in logs

@seanmonstar @jrgm I'm trying to debug the /v1/developer/activate endpoint.

The oauth-server log is reporting it as a 404.

{"Timestamp":1430319186915000000,"Logger":"fxa-oauth-server","Type":"summary.summary","Severity":6,"Pid":28961,"EnvVersion":"2.0","Fields":{"code":200,"errno":0,"method":"post","path":"/v1/verify","t":6,"auth":"null","payload":"[\"token\"]"}}
{"Timestamp":1430319186938000000,"Logger":"fxa-oauth-server","Type":"summary.summary","Severity":6,"Pid":28961,"EnvVersion":"2.0","Fields":{"code":404,"errno":999,"method":"post","path":"/v1/developer/activate","t":0,"auth":"null","payload":"[]"}}

Local curl seems to be getting to it:

[ec2-user@ip-172-31-12-251 ~]$ curl -XPOST 'http://127.0.0.1:9111/v1/developer/activate'
{"code":401,"errno":111,"error":"Unauthorized","message":"Unauthorized for route","info":"https://github.com/mozilla/fxa-oauth-server/blob/master/docs/api.md#errors","detail":"Bearer token not provided"}[ec2-user@ip-172-31-12-251 ~]$ 

We did add a new nginx config in this PR: #138
I have reloaded and restarted nginx on the box, but still getting the same issue.
any ideas?

Fresh install fails: "One or more undefined variables: 'dict object' has no attribute 'stack"

I destroyed my stack and tried to recreate it, getting this error:

TASK: [authdb | configure fxa-auth-db-server] ********************************* 
fatal: [ec2-54-189-182-138.us-west-2.compute.amazonaws.com] => {'msg': "One or more undefined variables: 'dict object' has no attribute 'stack'", 'failed': True}
fatal: [ec2-54-189-182-138.us-west-2.compute.amazonaws.com] => {'msg': "One or more undefined variables: 'dict object' has no attribute 'stack'", 'failed': True}

FATAL: all hosts have already failed -- aborting

fix fxa-rp key generation

Sometimes (maybe always) this keypair is not being generated (or not in the right place).

fs.js:439
return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
^
Error: ENOENT, no such file or directory '/data/fxa-rp/secret-key.json'
at Object.fs.openSync (fs.js:439:18)
at Object.fs.readFileSync (fs.js:290:15)
at Object.KeyPair (/data/fxa-rp/node_modules/

Environment specific config, like awsbox.json

Can we either reuse awsbox.json or designate a new config file that this environment is configured to load (requires setting CONFIG_FILES=<path/to/config.json>)? Then we could control certain build properties of this deployment from our repo without messing with defaults.

Remove the need for separate "oauth-stable.*" domain

I sadly haven't paid enough attention to understand this: why do we need to host the oauth stuff on a separate domain like "oauth-stable.dev.lcip.org" rather than a sub-path like "stable.dev.lcip.org/oauth"?

It would be great to simplify the hosting here to avoid confusion for new RPs, who we regularly see stumble on knowing that services are on which hostname.

install npm@2

WIll file a PR shortly, just filing this and some other to keep track while trying a scratch fxa-dev build.

Anything less than [email protected] is 1. flaky, and 2. unsupported. So upgrade from the version that comes with nodejs 0.10.x

https://latest.dev.lcip.org/ is down (503)

$ http https://latest.dev.lcip.org/ver.json
HTTP/1.1 503 Service Unavailable: Back-end server is at capacity
Connection: keep-alive
Content-Length: 0
$ date
Sun Sep  7 19:01:51 EDT 2014

Looking for a better strategy to update latest.dev

The latest.dev.lcip.org server may suddenly update while the content server functional tests are running. This happens when multiple PRs get merged quickly. This leads to page load timeouts and test failures.

@dannycoates do you have any ideas how to fix this up?

Here are the build Triggers that our TeamCity server offers:

image

Having trouble spinning up new stacks

Hilariously failing on "install nonsense". Not sure we need phantomjs in fxa-dev, but hey, I just work here.

TASK: [content | install nonsense] ********************************************
changed: [ec2-54-214-150-49.us-west-2.compute.amazonaws.com] => (item=bower)
changed: [ec2-54-214-150-49.us-west-2.compute.amazonaws.com] => (item=grunt-cli)
failed: [ec2-54-214-150-49.us-west-2.compute.amazonaws.com] => (item=phantomjs) => {"cmd": "/usr/bin/npm install --global phantomjs", "failed": true, "item": "phantomjs", "rc": 1}
stderr: Error requesting archive.
Status: 503
Request options: {
  "uri": "https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.7-linux-x86_64.tar.bz2",
  "encoding": null,
  "followRedirect": true,
  "headers": {},
  "strictSSL": true
}
Response headers: {
  "server": "nginx/1.5.10",
  "date": "Thu, 02 Oct 2014 00:24:03 GMT",
  "content-type": "text/html",
  "transfer-encoding": "chunked",
  "connection": "keep-alive",
  "cache-control": "no-cache"
}
Make sure your network and proxy settings are correct.

If you continue to have issues, please report this full log at https://github.com/Medium/phantomjs

npm ERR! [email protected] install: `node install.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the phantomjs package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node install.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls phantomjs
npm ERR! There is likely additional logging output above.
npm ERR! System Linux 3.10.35-43.137.amzn1.x86_64
npm ERR! command "/usr/bin/node" "/usr/bin/npm" "install" "--global" "phantomjs"
npm ERR! cwd /home/ec2-user
npm ERR! node -v v0.10.32
npm ERR! npm -v 1.4.28
npm ERR! code ELIFECYCLE
npm ERR! not ok code 0

stdout: /usr/bin/phantomjs -> /usr/lib/node_modules/phantomjs/bin/phantomjs

> [email protected] install /usr/lib/node_modules/phantomjs
> node install.js

Looks like an `npm install -g`; unable to check for already installed version.
Downloading https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.7-linux-x86_64.tar.bz2
Saving to /usr/lib/node_modules/phantomjs/phantomjs/phantomjs-1.9.7-linux-x86_64.tar.bz2
Receiving...


msg: Error requesting archive.
Status: 503
Request options: {
  "uri": "https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.7-linux-x86_64.tar.bz2",
  "encoding": null,
  "followRedirect": true,
  "headers": {},
  "strictSSL": true
}
Response headers: {
  "server": "nginx/1.5.10",
  "date": "Thu, 02 Oct 2014 00:24:03 GMT",
  "content-type": "text/html",
  "transfer-encoding": "chunked",
  "connection": "keep-alive",
  "cache-control": "no-cache"
}
Make sure your network and proxy settings are correct.

If you continue to have issues, please report this full log at https://github.com/Medium/phantomjs

npm ERR! [email protected] install: `node install.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the phantomjs package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node install.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls phantomjs
npm ERR! There is likely additional logging output above.
npm ERR! System Linux 3.10.35-43.137.amzn1.x86_64
npm ERR! command "/usr/bin/node" "/usr/bin/npm" "install" "--global" "phantomjs"
npm ERR! cwd /home/ec2-user
npm ERR! node -v v0.10.32
npm ERR! npm -v 1.4.28
npm ERR! code ELIFECYCLE
npm ERR! not ok code 0

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/Users/Chris/dev.retry

pip now installs supervisord into /usr/local/bin, but upstart is configured for /usr/bin

This is odd. Just built a box from scratch and it kept dying with

TASK: [common | start supervisord] ******************************************** 
failed: [ec2-52-10-10-127.us-west-2.compute.amazonaws.com] => {"failed": true}
msg: start: Job failed to start

After looking, it appears that maybe pip is now installing into /usr/local/bin, so supervisord isnt' where initctl is looking for it. Weird.

Docker?

In cloud services show and tell, it seems to be all the rage with the kids.

@st3fan says he has already converted the FxA infrastructure to Docker. Thoughts?

Serve minified files on latest? (similar to accounts-latest)

Found in https://latest.dev.lcip.org/ver.json (commit 276123d278dd096846d29223ce27d8e74a3566e2)

{
  "version": "0.13.0",
  "commit": "276123d278dd096846d29223ce27d8e74a3566e2",
  "l10n": "17102b27e7"
}

Steps to reproduce:

  1. Open your developer tools (Network tab)
  2. Open https://latest.dev.lcip.org/signup
  3. Open a new tab.
  4. Open https://accounts-latest.dev.lcip.org/signup

Actual results:

  1. https://latest.dev.lcip.org/signup
    76 requests, 785.15 KB, 2.31s
  2. https://accounts-latest.dev.lcip.org/signup
    9 requests, 273.78 KB, 0.28s

Expected results:

Not sure if latest.dev should be serving minimized, similar to accounts-latest.

Can't build new stack

NOTIFIED: [authdb | run db patcher] *******************************************
failed: [ec2-54-202-239-52.us-west-2.compute.amazonaws.com] => {"changed": true, "cmd": "NODE_ENV=stage node bin/db_patcher.js ", "delta": "0:00:00.397277", "end": "2014-09-05 22:39:52.756163", "rc": 8, "start": "2014-09-05 22:39:52.358886"}
stderr:
module.js:340
    throw err;
          ^
Error: Cannot find module 'mysql-patcher'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/data/fxa-auth-db-server/bin/db_patcher.js:11:15)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)

FATAL: all hosts have already failed -- aborting

email notifications on fxa-dev build fails

Earlier, it looks like the latest/nightly boxes were stuck on older versions of fxa-content-server repo (specifically mozilla/fxa-content-server@4b1c510@ckarlof manually kicked off some server magic to get it refreshed).

Not sure what reporting we have on the fxa-dev boxes or if they're sending out emails on failures. It'd be nice to add the cloud services QA alias added to notifications, and whatever devs also want to be notified.

@jrgm noted in IRC:

So, somehow /data/fxa-content-server/server/templates/pages/dist is owned by root, and the app user is not able to rebuild over that with grunt build.

Getting it working on Mac OS 10.8.5

To get this working I had specify an explicit inventory containing:

[localhost]
127.0.0.1 ansible_python_interpreter=/usr/local/bin/python

and then in the aws dir, I did:

ansible-playbook dev.yml -i <path to inventory file>

I installed ansible and boto with sudo pip install

Profile server directory issues on latest.

[2014-08-27 21:58:59.598] fxa.CRITICAL: Error: PUBLIC_DIR does not exist: /data/fxa-profile-server/var/public
    at Object.<anonymous> (/data/fxa-profile-server/lib/img/local.js:16:9)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/data/fxa-profile-server/lib/img/index.js:12:22)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
[2014-08-27 21:59:04.647] fxa.CRITICAL: Error: PUBLIC_DIR does not exist: /data/fxa-profile-server/var/public
    at Object.<anonymous> (/data/fxa-profile-server/lib/img/local.js:16:9)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/data/fxa-profile-server/lib/img/index.js:12:22)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
[2014-08-27 21:59:09.068] fxa.CRITICAL: Error: PUBLIC_DIR does not exist: /data/fxa-profile-server/var/public

We had to make var/public manually.

http -> https redirect

If you forget to specify https:// when connecting to the server, you'll get timeouts/connectivity errors. We should consider doing a redirect from http to https to avoid confusion.

Marketplace stage config incorrect

The marketplace stage config (https://marketplace.allizom.org) returns to https://marketplace.stage.mozaws.net/fxa/authorize but we'll need it to mirror the dev config. We may as well set up production also if that isn't already.

Stage: https://marketplace.allizom.org/fxa-authorize
Production: https://marketplace.firefox.com/fxa-authorize

Marketplace dev config: https://github.com/washort/fxa-dev/blob/44929fb433cd26ee2fce6b1d26c75e47926fff32/roles/oauth/templates/config.json.j2#L36

I'm not sure what needs to change for this but I'm happy to make the change.

(wishlist) show progress info in "TASK: [create basic stack]"

The whole spin-up-cloudformation-stack task takes a good long while and doesn't seem to print any progress updates. It might be fun to e.g. print out cloudformation events as they happen during creation of the stack for a bit of realtime feedback.

Cannot deploy, stuck on "create basic stack"

For some reason I'm getting:
image

Both got stuck on: TASK: [create basic stack (this will take a while)]

The configs I tried were vlad2 and vlad:


---
region: us-west-2
subdomain: vlad2.dev
hosted_zone: lcip.org
ssl_certificate_name: wildcard.dev.lcip.org
rds_password: Q&}PzHU79J8Ex}3,
cron_time:
  minute: 0
  hour: 0
  month: 1
  day: 1

This is a new stack, did I forget some step? I just need a deploy with oauth_git_version: register for testing.

Email resend does not work

Hey @dannycoates, weird issue. Seems like I can only get one verification email out. no matter how many times I press "Resend email" it never works.

This breaks the OAuth functional test because it tests unverified OAuth sign in.

https://vlad.dev.lcip.org:4443/
{"version":"0.13.0","commit":"dfea74224e4106fd4c85eb8ebc19d18f3385cc8c"}

Make first vagrant up work for a new contributor.

How to resolve

Either

  • Link to possible vagrantbox.es
  • Provide link to customized image

OR

  • Install steps ends up with a fully working independent deployment of FxA
  • (optional) do not require initial database

Building the VM:

Related notes

Initial procedure

Note Progress is described in the comment thread.

This procedure assumes one doesn’t have access to dcoates/fedora20 config.vm.box Vagrant box.

Environment:

  • MacOS X as host
  • Vagrant v1.6.5
  • VirtualBox v4.3.6
  • Installed Ansible through brew on Mac OS X host

Steps

  1. Install Ansible

    brew install ansible
    
  2. Adjust Vagrantfile with:

    config.vm.box = "fedora20"
    config.vm.box_url = "https://dl.dropboxusercontent.com/u/15733306/vagrant/fedora-20-netinst-2014_01_05-minimal-puppet-guestadditions.box"
    
  3. Boot the VM for the first time

    First time it fails, its OK.

      make
    
  4. Fun in the VM the following:

    vagrant ssh
    

    Inside the VM

      sudo -s
      yum update
      yum install -y npm git ansible
    
  5. Halt the VM, restart the process

    exit
    

    Back on the host;

      vagrant halt
      make 
    
  6. Add what’s missing inside the VM

    Inside the VM, as root;

      adduser app
      mkdir /data
      chown -R app:app /data/
      easy_install supervisor
      npm install -g grunt-cli
    
  7. Get the code

    On the host;

      make update-code
    

    Wait.

    It breaks at fxa-auth-server, see npm-debug.log

  8. Breaks halfway

    Here is how /data looks like

      drwxr-xr-x. 10 app app 4096 Dec  3 16:14 fxa-auth-db-server
      drwxr-xr-x.  6 app app 4096 Dec  3 16:11 fxa-auth-mailer
      drwxr-xr-x. 14 app app 4096 Dec  3 16:45 fxa-auth-server
      drwxr-xr-x. 10 app app 4096 Dec  3 16:17 fxa-content-server
      drwxr-xr-x. 10 app app 4096 Dec  3 16:22 fxa-customs-server
    
  9. Delete an try again

    After reading the npm-debug.log. It mentionned to delete and try again.

    Inside the VM;
    rm -rf /root/tmp/npm-

    Trying again from the host;

      make update-code
    

    Got a different error message. It talked about missing gmp.h.

    screen shot 2014-12-03 at 5 19 53 pm

      yum install -y gmp.x86_64 gmp-devel.x86_64
    
  10. Another make update-code attempt

    It worked! /o/

screen shot 2014-12-03 at 5 20 04 pm

  1. ... and a few make update

Build a sandboxed Oauth integration server

Let's keep the one for Marketplace one for now, but in addition let's have a generic one that's sandboxed (i.e., not connected to prod accounts, but uses the included auth server) and uses persistent storage. Let's also make sure it has more generic DNS name (i.e., doesn't include the word "marketplace").

How do we update these boxes if we didn't originally create them?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.