Code Monkey home page Code Monkey logo

engineyard-serverside's Introduction

engineyard-serverside

Build Status

engineyard-serverside is the serverside component of the Engine Yard Cloud CLI. This gem is invoked either by the Cloud dashboard or by the engineyard gem through helper gem engineyard-serverside-adapter.

ey.yml Configuration

The ey.yml file allows options to be saved for each environment to which an application is deployed.

A typical Rails application will have a config/ey.yml like this:

---
# This is all you need for a typical rails application.
defaults:
  migrate: true
  migration_command: rake db:migrate
  precompile_assets: true

The following ey.yml file shows other things that can be customized. A typical application will not need most of these options.

---
# 'defaults' applies to all environments running this application.
# Only set these options if needed. The defaults are correct for most applications.
defaults:
  bundler: detect                           # By default, bundler is detected via Gemfile. Options: true: always run bundler; false: never run bundler
  composer: detect                          # By default, composer is detected via composer.lock. Options: true: always run composer; false: never run composer
  npm: detect                               # By default, npm is detected via package.json. Options: true: always run npm; false: never run npm
  bundle_without: GROUP1 GROUP2             # exclude groups on bundle install (default: test development)
  bundle_options: --OPTION                  # extra bundle install options (--local, --quiet, etc; does not override bundle_without)
  copy_exclude:                             # don't rsync the following dirs (some people like to skip .git)
  - SOME_LARGE_DIR 
  maintenance_on_restart: false             # show maintenance page during app restart (default: false except for glassfish and mongrel)
  maintenance_on_migrate: true              # show maintenance page during migrations (default: true)
  precompile_assets: true                   # ensure rails assets precompilation (default: assets will be compiled if they are detected)
  precomplie_assets_task: assets:precompile # override the assets:precompile rake task
  precompile_assets_command: rake assets:precompile # override the entire precompile command (ignores precompile_assets_task)
  precompile_unchanged_assets: false        # if true, does not check git for changes before precompiling assets.
  asset_dependencies:                       # a list of relative paths to search for asset changes during each deploy.
  - app/assets                              # default
  - lib/assets                              # default
  - vendor/assets                           # default
  - Gemfile.lock                            # default
  - config/routes.rb                        # default
  - config/application.rb                   # default
  - config/requirejs.yml                    # custom option (be sure to include defaults if you specify this option)
  asset_strategy: shifting                  # choose an alternet asset management strategy. See rails_assets/strategy.rb for more info.
  asset_roles: :all                         # specify on which roles to compile assets (default: [:app, :app_master, :solo])
  ignore_database_adapter_warning: true     # hide database adapter warning if you don't use MySQL or PostgreSQL (default: false)
  ignore_ey_config_warning: true            # hide ey_config warning if you don't use it (default: false)
  ignore_gemfile_lock_warning: true         # hide warning when Gemfile is present but Gemfile.lock is missing. (default: false)
  keep_releases: 3                          # keep more or less releases (default: 3; Recommended: >= 2; MUST BE >= 1, with 1 being experimental)
  keep_failed_releases: 3                   # keep more or less failed releases (default: 3)
  gc: false                                 # if true, run repository garbage collection every deploy. (default: git will run gc as needed)

# Environment specific options apply only to a single environment and override settings in defaults.
environments:
  env_production:
    precompile_unchanged_assets: true       # precompiles assets even if no changes would be detected (does not check for changes at all).
    asset_strategy: shifting                # choose an alternet asset management strategy (shifting, cleaning, private, shared)
    asset_roles: :all                       # specify on which roles to compile assets (default: [:app, :app_master, :solo] - must be an Array)
  env_staging
    asset_strategy: private                 # Use an asset management that always refreshes, so staging enviroments don't get conflicts

These options in ey.yml will only work if the file is committed to your application repository. Make sure to commit this file. Different branches may also have different versions of this file if necessary. The ey.yml file found in the deploying commit will be used for the current deploy.

Rails Assets

If precompile_assets is not set, asset compilation will be detected and failures may be ignored. Set precompile_assets to true or false to ensure proper behavior.

When precopmile_assets is true, git diff will be used to detect changes to the path names specified in asset_dependencies since the revision of the last successful release. If precompile\_assets is true, be sure to git ignore public/assets to avoid conflicts.

When precompile_unchanged_assets is true, assets will always be compiled and the git diff detection will not run.

Strategies

A number of asset persistence strategies are supported by default. Choose one and then customize using deploy hooks if necessary.

private - Precompile assets fresh every time. Shared assets are not symlinked and assets stay with the release that compiled them. The assets of the previous deploy are symlinked as into the current deploy to prevent errors during deploy. When no assets changes are detected, the deploy uses rsync to copy the previous release's assets into the current assets directory.

shared - Basic shared assets. Precompiled assets go into a single shared assets directory. The assets directory is never cleaned, so a deploy hook should be used to clean assets appropriately. When no assets changes are detected, shared directory is only symlinked and precompile task is not run.

cleaning - Precompiled assets are shared across all deploys. Before compiling the active deploying assets, all assets not referenced by the manifest.yml from the previous deploy are removed. After cleaning, the new assets are compiled over the top. The result is an assets dir that contains the last assets and the current assets. When no assets changes are detected, shared directory is only symlinked and cleaning and precompile tasks are not run.

shifting - The default behavior and the one used since the beginning of asset support in engineyard-serverside. Assets are compiled into a fresh shared directory. Previous shared assets are shifted to a last_assets directory to prevent errors during deploy. When no assets changes are detected, the two shared directories are symlinked into the active release without any changes.

Invoking serverside manually

All information that this gem needs to perform a deploy of an application is passed in on the command line. Command line interaction is performed through engineyard-serverside-adapter. Refer to the adapter gem or the source code here for details on the command line options.

The easiest way to run a serverside command manually is to run a normal deploy with the verbose option from the command line, then copy the engineyard-serverside command from the log and paste it into a terminal on the remote machine. Running manually should usually be unnecessary, but could be used to debug problems.

Running the spec suite (ruby 1.8.7+)

Install required gems:

which bundle >/dev/null || gem install bundler
bundle install

Running tests:

bundle exec rake

Bundler will take care of installing and running proper (older versions) of gems.

Running the spec suite (ruby 1.8.6)

Bundler doesn't work on ruby 1.8.6-p287, which is what ey_resin provided to older instances. To test engineyard-serverside under the same ruby, run:

gem build engineyard-serverside.gemspec
gem install --local --development --conservative ./engineyard-serverside-<VERSION>.gem
rake

Due to the nature of --development, that gem install command can take a VERY long time. If you get tired of waiting, you can manually install the indicated versions of rake and rspec. Check the gemspec for current requirements. At the time of this writing, rspec 1.3.0 was the last version known to work on 1.8.6.

Set the 'VERBOSE' environment variable to something to get full output from failing commands.

Releasing

To release the engineyard-serverside gem, use the command below and then follow the instructions it outputs.

bundle exec rake release

This will remove the .pre from the current version, then bump the patch level and add .pre after. A git tag for the version will also be added.

New versions of engineyard-serverside will not be used by Cloud or the engineyard gem until upgraded gems have been pushed. Refer to the engineyard gem release instructions for more details.

Testing manually on a server

The following 2 commands will install the current working copy on an instance:

bundle exec rake install_on[account/envname]
ey deploy --serverside-version VERSION -e envname -a appname -r branch-to-checkout --no-migrate -v

If you don't have an environment running on Engine Yard Cloud, you'll need to do a bit of hacking of both commands. You can look at the Rakefile to see how install_on works. It is simple to adapt it to installing on specified servers.

Next you'll have to run the deploy command manually on the instance. The easiest way to do that is to take an existing serverside command and modify it. An engineyard-serverside command can be extracted from a verbose log for another application's deploy, and then modified. The command looks like this:

bash -lc '/usr/local/ey_resin/ruby/bin/engineyard-serverside _2.1.0.rc1_ deploy --account-name account --app appname --config '\''{"input_ref":"testing","deployed_by":"Märtîn ☃ Èmdé"}'\'' --environment-name envname --framework-env production --instance-names ec2-255-255-255-255.compute-1.amazonaws.com:util --instance-roles ec2-255-255-255-255.compute-1.amazonaws.com:util localhost:solo --instances ec2-255-255-255-255.compute-1.amazonaws.com localhost --no-migrate --ref 5a6de57ca3ce3c51df18cfff0cbea87f2f07872a --repo git://github.com/engineyard/todo.git --stack nginx_passenger3 --verbose'

Run that from any primary web server instance and be sure to use "localhost" for the instance you're running on instead of its public address.

Obligatory Edit

engineyard-serverside's People

Contributors

alenia avatar benburkert avatar calavera avatar conormcd avatar ddiachkov avatar drnic avatar dvalfre avatar ess avatar halorgium avatar indirect avatar jacobo avatar jfuechsl avatar jhsu avatar jlindley avatar lanej avatar larrytheliquid avatar martinemde avatar mkb avatar mpapis avatar mutle avatar outerim avatar rbankston avatar ryansouza avatar shaiguitar avatar slack avatar smerritt avatar thommahoney avatar thorn avatar walski avatar wilson avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

engineyard-serverside's Issues

Use control scripts to restart app servers

Now that the control scripts are deployed, use them to restart the app servers.

This will let us add a new stack and not have to change engineyard-serverside. It will also (potentially) make using it on xcloud easier as they can write their own control scripts to do whatever wacky thing the customer requires, again without modifying engineyard-serverside.

Invoke the control script as an unprivileged user. Do not use sudo.

Bundle --without option in ey.yml isn't loaded for web deploys

Currently the bundle --without option can be set as follows in ey.yml

environments:
  myapp_staging:
    bundle_without: development test my_special_group

The problem is that web deploys are not properly supported without using the eydeploy.rb file as outlined here: http://docs.engineyard.com/customize-your-deployment.html#second

Currently putting the following in eydeploy.rb will work for all deploys, web or cli. This is similar to the previous solution:

class EY::Serverside::Deploy::Configuration
  def bundle_without
    "development test my_special_group"
  end
end

In order to support the first way with ey,yml, the file must be loaded by the serverside deploy system, which it isn't currently. This requires some non-trivial changes to the deploy side.

Handle package deploy failures

This one's going to be a whole Railway all its own.

Basically, if a deploy fails, reverse any performed operations that we reasonably can.

Deploying a jruby app to EY and got exceptions

Hi,

I was trying to deploy a jruby app to EY cloud and got the following exceptions:

~> Deploying revision 61b2bba added thor to trinidad platform
+ 0m 00s ~> Pushing code to all servers
+ 0m 00s ~> Starting full deploy
+ 0m 00s ~> Copying to /data/todo/releases/20120204172654
+ 0m 00s ~> Ensuring proper ownership.
+ 0m 00s ~> Gemfile found.
+ 0m 00s ~> Gemfile.lock found.
+ 0m 00s ~> Bundling gems...
LoadError: library `fiber' could not be loaded: java.lang.ClassNotFoundException: org.jruby.libraries.FiberExtLibrary
     require at org/jruby/RubyKernel.java:1047
     require at /usr/lib/jruby/1.6/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36
      (root) at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/vendor/celluloid/lib/celluloid/fiber.rb:3
     require at org/jruby/RubyKernel.java:1047
     require at /usr/lib/jruby/1.6/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36
      (root) at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/vendor/celluloid/lib/celluloid/fiber.rb:1
     require at org/jruby/RubyKernel.java:1047
     require at /usr/lib/jruby/1.6/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36
      (root) at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/vendor/celluloid/lib/celluloid/core_ext.rb:243
     require at org/jruby/RubyKernel.java:1047
     require at /usr/lib/jruby/1.6/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36
  Serverside at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/vendor/celluloid/lib/celluloid.rb:4
          EY at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/engineyard-serverside/futures/celluloid.rb:2
      (root) at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/engineyard-serverside/futures/celluloid.rb:1
     require at org/jruby/RubyKernel.java:1047
     require at /usr/lib/jruby/1.6/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36
  Serverside at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/engineyard-serverside/futures/celluloid.rb:24
          EY at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/engineyard-serverside/future.rb:2
      (root) at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/engineyard-serverside/future.rb:1
     require at org/jruby/RubyKernel.java:1047
     require at /usr/lib/jruby/1.6/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36
      (root) at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/engineyard-serverside/future.rb:28
     require at org/jruby/RubyKernel.java:1047
     require at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/engineyard-serverside.rb:36
      (root) at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/bin/engineyard-serverside:8
+ 0m 03s ~> Release /data/todo/releases/20120204172654 failed, saving release to /data/todo/releases_failed.
+ 0m 04s ~> [Relax] Your site is still running old code and nothing destructive has occurred.
/usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/engineyard-serverside/task.rb:62:in `run_on_roles': #<EY::Serverside::Future:0xb7cec914 @value=false, @server=#<struct EY::Serverside::Server hostname="localhost", roles=[:solo], name=nil, user="deploy">, @args=[false], @block=#<Proc:0xb7c64618@/usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/engineyard-serverside/task.rb:55>> (EY::Serverside::RemoteFailure)
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/engineyard-serverside/task.rb:48:in `sudo'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/engineyard-serverside/deploy.rb:497:in `check_ruby_bundler'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/engineyard-serverside/deploy.rb:228:in `bundle'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/engineyard-serverside/task.rb:34:in `roles'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/engineyard-serverside/deploy.rb:227:in `bundle'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/engineyard-serverside/deploy.rb:160:in `send'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/engineyard-serverside/deploy.rb:160:in `run_with_callbacks'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/engineyard-serverside/deploy.rb:31:in `cached_deploy'
     ... 12 levels...
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/lib/vendor/thor/lib/thor.rb:124:in `start'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.27/bin/engineyard-serverside:10
    from /usr/local/ey_resin/ruby/bin/engineyard-serverside:19:in `load'
    from /usr/local/ey_resin/ruby/bin/engineyard-serverside:19

Implement a distributor

After generating a package, we need a way to get that package from the build node (app_master) to the other servers in the environment that run the application (app, util).

Options:

  • rsync the file over, much the same as we do with our current deploy scheme
  • Store the package in an s3 bucket so the other servers can grab it.

Don't choke on compound version constraints for bundler.

The following breaks the lockfile parser. We shouldn't even need the lockfile parser anymore because bundler is much more stable and the newest versions will work for the majority of people.

gem 'bundler', '~>1.6', '>= 1.6.2'

Before/After restart hooks

We can use existing platform mechanisms to perform a restart for app instances (including master), but that's not actually the case for utility instances, because the services configured for utility instances are not known to serverside.

To that end, we should provide a mechanism for restart hooks, much like the legacy deploy workflow does.

To make this as simple to implement as we possibly can, the design is something like this:

  • HOOK_PATH=$RELEASE_PATH/.ogun/hooks
  • Hooks have bare names. (before_restart, after_restart)
  • Hooks can be implemented however the customer desires. (ruby, bash, whatever, so long as it's chmod +x)
  • We pass all pertinent information via CLI arguments. For example, $HOOK_PATH/after_restart $RELEASE_NAME $SERVER_ROLE $SERVER_NAME
  • Hook results matter. A hook that exits 0 is considered good. Otherwise, it causes a deploy failure.

Implement a migrations runner

Provided that we've built and distributed a packaged application across the environment, we should provide a mechanism for running database migrations.

Implement a Railway

At its core, deployment is basically a multi-step sequence that can bail out at any point if there is a failure.

That is more or less the essence of Railway-Oriented Programming from FP land.

There is an existing solution for this pattern in Ruby, dry-transaction, but it does not support around 80% of the Ruby versions that we have to support.

So, let's come up with something. Ground rules:

  • This is a pipeline. The output of one successful step should be fed into the next step as its input.
  • If any step returns a failure, subsequent steps should not be run.
  • The output of the final step executed is returned as the result of the entire pipeline.
  • We should provide basic success/failure hooks for post-railway operations.

Choose the deployer based on capability

Basically, add a condition to the CLI workflow helpers that choose the deploy scheme (legacy vs packages) based on whether or not all of the requirements are in place to use the package scheme.

public/system symlink interferes with some applications

At least one application, Pancake, needs to use public/system for its own updates. Deploy overwrites public/system with a symlink after deleting the directory. There is currently no way to avoid this so an option is needed to prevent breaking applications.

Note: When searching for a maintenance page, public/system/maintenance.html.default is searched before using the default engineyard supplied one. We can assume that an app that needs public/system for itself will not store its maintenance page there, so ignoring that search path should be fine.

Support before_deploy and after_deploy hooks

These hooks are placed at the outer edges of the deploy task.

before_deploy will trigger immediately after the active_release ("releases/TIMESTAMP") directory is created on all the servers. This is the earliest it is possible to run the before hook with the expectation that the paths will be correct and usable.

after_deploy will trigger at the very end of the deploy before the script exits. This runs after maintenance pages are removed, and after the git garbage collection task.

Public system symlink check raising when ran in enable/disable maintenance actions

lib/engineyard-serverside/maintenance.rb:112:inrealpath': No such file or directory - /data/kitchen_sink_ruby/releases/20140620220738 (Errno::ENOENT)`

Paths defaults the active_release path to be a new timestamp named directory. Existing maintenance toggling code seems to avoid using any paths that use active_release, but the code added in #98 does. Since this just-generated timestamped path doesn't exist the code raises a no-such-file error.

`assets_strategy` not working as expected

I use shared assets strategy but actually all assets precompile from scratch on every deploy when any asset_dependencies changed. This happens because shared assets directory is moved into last_assets (renamed) and empty new assets directory is created.

Full log — http://pastebin.com/raw.php?i=rkpLaCy5

log cut:

+    24s  ~> Precompiling assets. (precompile_assets: true)
             $ sh -l -c 'rm -rf /data/happy_dog_treats/shared/last_assets && mkdir -p /data/happy_dog_treats/shared/assets /data/happy_dog_treats/shared/last_assets && mv /data/happy_dog_treats/shared/assets /data/happy_dog_treats/shared/last_assets/assets && mkdir -p /data/happy_dog_treats/shared/assets && ln -nfs /data/happy_dog_treats/shared/assets /data/happy_dog_treats/shared/last_assets /data/happy_dog_treats/releases/20131224232025/public'
             $ sh -l -c 'cd /data/happy_dog_treats/releases/20131224232025 && PATH=/data/happy_dog_treats/releases/20131224232025/ey_bundler_binstubs:$PATH RAILS_ENV=staging RACK_ENV=staging NODE_ENV=staging MERB_ENV=staging rake assets:precompile RAILS_GROUPS=assets'

that clearly states:

mv /data/happy_dog_treats/shared/assets /data/happy_dog_treats/shared/last_assets/assets
&&
mkdir -p /data/happy_dog_treats/shared/assets

Maybe I got wrong the description of shared strategy but my point is that instead of full assets compilation we only precompile updated assets when using turbo-sprockets-rails3 gem or similar (I guess even stock assets:precompile can detect if it needs to re-compile asset file).

Implement a source updater

In order to build a new version of the app, we need to be able to update our local copy of the source.

Realistically, this is just creating a Railway that does the same related things as the stuff in the legacy deploy workflow, but in a way that we can use as a step in another Railway.

Implement a deploy failure handler

What happens when a package-based deploy fails?

At a minimum, we should walk back the things that we've done (as possible) and move the failed build to the failed releases location.

Deploy hook symlink commands are outputting verbose messages

I've been using this hook in before_migrate.rb for a while:

run "ln -nfs #{shared_path}/config/redis.yml #{release_path}/config/redis.yml"

We have a dev redis.yml in the Git repo but we want to override it with a custom chef one. This has been working fine for a while but recently when using the CLI to deploy, we've started to see messages like:

ln: creating symbolic link `/data/pas/releases/20101101165521/config/redis.yml': File exists

It seems to be only informative because when I check the current folder the symlink actually exists and is pointing to the one we want. My ey.yml config hasn't changed at all (has no verbose option) and this is when I run a deploy like: ey deploy -e production.

Has anything in the serverside gem changed that displays all stderr messages? Shouldn't the ln -nfs command not output any messages at all and always override the file if it already exists?

Implement the deploy railway

The Deploy railway basically just ties together all of the operations/railways that we've already implemented around the idea of a package-based deploy.

It also sets up the initial input for the rest of the pipeline.

resque wrapper script should allow QUEUES, not just QUEUE

Not sure if serverside is the place to file this, but:

The /engineyard/bin/resque script checks for the presence of a QUEUE setting in config files and exit 1s if not found. This should also allow QUEUES as Resque provides for specifying a priority order.

Maintenance page fails with advanced pages

When deploying with a custom maintenance.html that has quotes. An example would be with images single quoted instead of double quoted. The failures in the log look like this:

<Exception during deploy: #<Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT>

due to the change performed in 249bbb8

[BUG] object allocation during garbage collection phase

There were multipale deploys out put is below. The first deploy was unsuccessful, the second passed, however the third failed again.

Deploy 1

ln: creating symbolic link /data/pmvalor/releases/20110926172307/config/database.yml': File exists ln: creating symbolic link/data/pmvalor/releases/20110926172307/config/newrelic.yml': File exists
~> Migrating: cd /data/pmvalor/releases/20110926172307 && PATH=/data/pmvalor/releases/20110926172307/ey_bundler_binstubs:$PATH RAILS_ENV=production RACK_ENV=production MERB_ENV=production rake db:migrate --trace
~> Symlinking code
/usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.4.1/lib/vendor/open4/lib/open4.rb:39: [BUG] object allocation during garbage collection phase
ruby 1.8.6 (2008-08-11) [i686-linux]

~> [Attention] Maintenance page still up, consider the following before removing:

  • any deploy hooks ran, be careful if they were destructive
  • any migrations ran, be careful if they were destructive
  • your old code is still symlinked as current
    Failed deployment recorded in AppCloud
    Deploy failed

Deploy 2

$ ey deploy -e valor_production
Beginning deploy for 'pmvalor' in 'valor_production' on server...
/usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.4.1/lib/vendor/open4/lib/open4.rb:39: [BUG] Segmentation fault
ruby 1.8.6 (2008-08-11) [i686-linux]

~> Installing engineyard-serverside on ec2-50-16-179-37.compute-1.amazonaws.com
~> Deploying revision 20ad76e Merge branch 'dev-albert' into main-guild

License missing from gemspec

RubyGems.org doesn't report a license for your gem. This is because it is not specified in the gemspec of your last release.

via e.g.

spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']

Including a license in your gemspec is an easy way for rubygems.org and other tools to check how your gem is licensed. As you can image, scanning your repository for a LICENSE file or parsing the README, and then attempting to identify the license or licenses is much more difficult and more error prone. So, even for projects that already specify a license, including a license in your gemspec is a good practice. See, for example, how rubygems.org uses the gemspec to display the rails gem license.

There is even a License Finder gem to help companies/individuals ensure all gems they use meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough issue that even Bundler now generates gems with a default 'MIT' license.

I hope you'll consider specifying a license in your gemspec. If not, please just close the issue with a nice message. In either case, I'll follow up. Thanks for your time!

Appendix:

If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file), GitHub has created a license picker tool. Code without a license specified defaults to 'All rights reserved'-- denying others all rights to use of the code.
Here's a list of the license names I've found and their frequencies

p.s. In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :). See the previous link or my blog post aobut this project for more information.

Implement a package builder

The result of this process should be a tarball that contains everything necessary (compiled assets, configuration, required packages, etc) to run the application.

In a perfect world, this package will be built via a buildpack mechanism.

Rollback assets, if possible, when running a deploy rollback

Ideally, users that decide that rollback is an acceptable deployment strategy for their application will choose to use the asset_strategy: private. The private strategy stores assets in the release directory and does not share or move assets, allowing rollback to function correctly. This is supported already in the current version.

However, if the default shifting asset_strategy is used, we could potentially support one level of rollback.

Deploy Hooks for update_repository_cache task would be useful

I've recently encountered a scenario where I've had an external volume mounted in an app's repository directory structure. Due to the clone and checkout process for Git repositories, the update_repository_cache task would fail if the external volume was still mounted:

/usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.33/lib/engineyard-serverside/task.rb:62:in `run_on_roles': #<EY::Serverside::Future:0xb7366ffc @value=false, @server=#<struct EY::Serverside::Server hostname="ec2-23-21-218-214.compute-1.amazonaws.com", roles=[:app_master], name=nil, user="deploy">, @args=[false], @block=#<Proc:0xb742199c@/usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.33/lib/engineyard-serverside/task.rb:55>> (EY::Serverside::RemoteFailure)
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.33/lib/engineyard-serverside/task.rb:48:in `sudo'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.33/lib/engineyard-serverside/deploy.rb:288:in `copy_repository_cache'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.33/lib/engineyard-serverside/deploy.rb:26:in `cached_deploy'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.33/lib/engineyard-serverside/deploy.rb:17:in `deploy'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.33/lib/engineyard-serverside/cli.rb:61:in `send'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.33/lib/engineyard-serverside/cli.rb:61:in `deploy'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.33/lib/vendor/thor/lib/thor/task.rb:33:in `send'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.33/lib/vendor/thor/lib/thor/task.rb:33:in `run'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.33/lib/vendor/thor/lib/thor/invocation.rb:109:in `invoke'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.33/lib/vendor/thor/lib/thor/invocation.rb:116:in `call'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.33/lib/vendor/thor/lib/thor/invocation.rb:116:in `invoke'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.33/lib/vendor/thor/lib/thor.rb:137:in `start'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.33/lib/vendor/thor/lib/thor/base.rb:378:in `start'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.33/lib/vendor/thor/lib/thor.rb:124:in `start'
    from /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.5.33/bin/engineyard-serverside:10
    from /usr/local/ey_resin/ruby/bin/engineyard-serverside:19:in `load'
    from /usr/local/ey_resin/ruby/bin/engineyard-serverside:19

Needing a way to force an unmount before clone and checkout, I used custom config/eydeploy.rb to override deploy and infer before/after deploy hook callbacks for the update_repository_cache task. Overriding the base deploy method seems more of a workaround, as the documentation for deployment customization doesn't list it as a method that's unlikely to change.

Last_assets fails following a failed deployment

Following the last_assest path for updating and maintaining the assets directory on deployment, if the previous deployment failed, the last_assets directory fails to point to the correct set.

Specifically if deployment fails at the before_symlink hook.

Source should be loaded before Deploy

Separate Source from Deploy so that Integrate is not wacky. If Source was loaded ahead of time in integrate and deploy, then integrate could use a source that had its own behavior. Integrate expects the "source" of deploy to be another existing deploy, so handling that appropriately is better than hacking integrate to get it working.

Implement a release finalizer

This is, effectively, the last step of an otherwise successful deploy.

After the new build is in place and serving requests, we should remove the unarchived version of the previous release at the least, and we should likely also remove all but the most recent N packages.

Error on using cleaning asset_strategy

The comand that removes/cleans the assets is getting an error.
Mostly because the command is too long.

engineyard-serverside-2.6.14/lib/engineyard-serverside/spawner.rb:125:in exec: Argument list too long

I checked the source code and its joining the files to be removed.

    //engineyard-serverside/lib/engineyard-serverside/rails_assets/strategy.rb
     remove_assets = []
            (assets_on_disk - assets_in_manifest).each do |asset|
              remove_assets << "'#{asset}'"
              remove_assets << "'#{asset}.gz'" if all_assets_on_disk.include?("#{asset}.gz")
            end
            run("rm -rf #{remove_assets.join(' ')}")

use ssh deploy key with bundler

I have a project that depends on a gem thats hosted in a private github repository.
I should be able to add my engineyard deploy key as a collaborator in github and then add the dependency to my Gemfile.
Unfortunately it appears that my deploy key is only used for checking out the main project.

Multiple versions aren't tagged in git

On Rubygems I can see these versions:

2.6.14 - December 13, 2016 (6.04 MB)
2.6.13 - December 09, 2016 (6.04 MB)
2.6.12 - December 09, 2016 (6.04 MB)
2.6.11 - November 01, 2016 (6.04 MB)
2.6.10 - August 19, 2016 (6.04 MB)

But the latest one I can see tagged in Git and on Github is v2.6.4

deploy hooks doesn't work with jruby with run or sudo

+    02s  !> FATAL: Exception raised in deploy hook /data/my_app/releases/20130131143500/deploy/before_symlink.rb.
+    02s  !> 
+    02s  !> NoMethodError: undefined method `exitstatus' for [#<Process::Status: pid=30502,exited(0)>, "", ""]:Array
+    02s  !> 
+    02s  !> Please fix this error before retrying.
NoMethodError: undefined method `exitstatus' for [#<Process::Status: pid=30502,exited(0)>, "", ""]:Array
  logged_system at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-2.0.4/lib/engineyard-serverside/shell.rb:64
           sudo at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-2.0.4/lib/engineyard-serverside/deploy_hook.rb:95
         (eval) at (eval):1
  instance_eval at org/jruby/RubyBasicObject.java:1734
      eval_hook at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-2.0.4/lib/engineyard-serverside/deploy_hook.rb:32
           call at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-2.0.4/lib/engineyard-serverside/deploy_hook.rb:25
          chdir at org/jruby/RubyDir.java:466
           call at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-2.0.4/lib/engineyard-serverside/deploy_hook.rb:20
           hook at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-2.0.4/lib/engineyard-serverside/cli.rb:73
       __send__ at org/jruby/RubyBasicObject.java:1704
           send at org/jruby/RubyKernel.java:2101
            run at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-2.0.4/lib/vendor/thor/lib/thor/task.rb:27
    invoke_task at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-2.0.4/lib/vendor/thor/lib/thor/invocation.rb:120
       dispatch at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-2.0.4/lib/vendor/thor/lib/thor.rb:275
          start at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-2.0.4/lib/vendor/thor/lib/thor/base.rb:425
         (root) at /usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-2.0.4/bin/engineyard-serverside:10

Add 'vendor/assets' to the asset_dependencies

The defaults are:

  asset_dependencies:                       # a list of relative paths to search for asset changes during each deploy.
  - app/assets                              # Defaults: app/assets lib/assets vendor/assets Gemfile.lock config/routes.rb config/application.rb
  - lib/assets                              # anything you specify will overwrite the defaults.
  - Gemfile.lock
  - config/application.rb
  - config/requirejs.yml

I was assuming that the `vendor/assets' directory is listed there too.
There are 2 main places to place the assets into (by default), only 2 of them are included by default.

That means if any of 3rd party scripts will be updated, the assets won't be recompiled.

So including the vendor/assets would be a very sensible and consistent default IMO.

Failed Deploy via cli

Output leading to the failed deploy

:: running sh -l -c 'mkdir -p /data/app_name/releases/20101001094637/tmp'
:: running ssh -i /home/deploy/.ssh/internal -o StrictHostKeyChecking=no -o PasswordAuthentication=no [email protected] 'sh -l -c '''mkdir -p /data/app_name/releases/20101001094637/tmp''
:: running ssh -i /home/deploy/.ssh/internal -o StrictHostKeyChecking=no -o PasswordAuthentication=no [email protected] 'sh -l -c '''mkdir -p /data/app_name/releases/20101001094637/tmp''
:: running ssh -i /home/deploy/.ssh/internal -o StrictHostKeyChecking=no -o PasswordAuthentication=no [email protected] 'sh -l -c '''mkdir -p /data/app_name/releases/20101001094637/tmp''
:: running ssh -i /home/deploy/.ssh/internal -o StrictHostKeyChecking=no -o PasswordAuthentication=no [email protected] 'sh -l -c '''mkdir -p /data/app_name/releases/20101001094637/tmp''
/usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/engineyard-serverside-1.2.0/lib/vendor/open4/lib/open4.rb:39: [BUG] object allocation during garbage collection phase

We have ree 2010.02 installed, though I am not sure if the serverside stuff uses that.

I want to be able to run a hook as early as possible in the deploy

This would probably be right after copy_repository_cache, any earlier and the code for the hook wouldn’t exist yet.

An example of when to use this is that if you wanted to run chef recipes on deploy, but one of the things the chef recipes did was to install a unix package, which was then required by a gem in your Gemfile.

We would need to clearly message that your gems are bundled, so running rake tasks probably wouldn’t work.

Moved from http://github.com/engineyard/engineyard/issues#issue/66

Changing 32bit --> 64bit breaks bundled gems

Move the bundled gems off the EBS so that no .so files are carried over when booting a new instance.

Since cloud.ey will tell engineyard-serverside to rebundle anyway, this only means that bringing up new instances will take a little bit longer.

Latest release path calculation should only consider release-shaped locations

Currently, the latest_release path is, effectively, just Dir["/data/appname/releases/*"].last

As we've seen recently, this totally breaks if a non-release item is inadvertently created in the releases directory.

At the least, we should filter for things that match a "probably release directory" regex or something.

Failed git fetch doesn't abort deploy

If you try to clone a private repository to which you don't have permissions, the fetch silently fails, then the checkout complains about /data/$app/shared/cached-copy/.git not being a Git repository. This error message, while true, does not illuminate what went wrong.

When fetching fails, we ought to abort the deploy immediately and have a helpful error message (mentioning deploy keys, at a minimum).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.