civicrm / civicrm-buildkit Goto Github PK

Hello!

I have been experimenting with creating docker containers for CiviCRM which implement the docker principles of composerability, automateability, and so on, and of course the best way to programmatically install CiviCRM is with buildkit. My most recent attempt seems to fit the bill -- it uses separate containers for the web and database servers, which can be swapped out by the administrator at will. I hope there will be some interest in it from the Civi community, since a lot of members seem to be interested in using containers to manage their installations. You can see the results of my attempts here: https://github.com/djcf/civicrm-docker/ I believe the docker design patterns it employs are quite sound, and I hope it might be useful to someone.

In these circumstances, its useful to be able to use the buildkit tools 'civibuild download' and 'civibuild install' to programmatically configure the Civi instance, however it is not desireable to use the buildkit tools to configure the vhost, the web server, or the database server, since in this circumstance we use docker to do that. With the rise of composeable infrastructures, its easy to imagine this scenario might crop up with other tools in the future.

My feature request is simple: a way to use the buildkit install scripts (drupal-clean, backdrop-demo, etc.) without also using amp. My docker container does this at runtime with two simple hacks. Firstly, it removes amp_install() from the install script[1]. Then, because amp sets up environment variables which are now lost, and which, in any case, are managed by the docker daemon (following the third principle of the 12-factor application[2]), it then re-loads those variables back into the build environment on line 728 of src/civibuild.lib.sh after they had been sanitised[3].

Do you think it might be useful for other users to accommodate such a feature in future releases?

You are busy. Thank you for taking the time to read this feature request.

Daniel

1: https://github.com/djcf/civicrm-docker/blob/master/build/install.sh line 48: sed -i 's/amp_install//g' /buildkit/app/config/$SITE_TYPE/install.sh
2: http://12factor.net/config
3: https://github.com/djcf/civicrm-docker/blob/master/build/install.sh line 78: sed -i 's/function drupal_install() {/function drupal_install() {\n source /buildkit/install.conf/g' /buildkit/src/civibuild.lib.sh

When this happens a simple simlink from /usr/local/bin/node to /usr/bin/nodejs works.

the amp test call in the install instructions can never fully succeed because it requires a restart of the web server.

Would be great if an httpd restart/reload command could be configured and invoked when creating vhosts. Maybe with a prompt to confirm a restart?

I am attempting to set up buildkit with vagrant.

Initial run fails as nodejs and npm are missing. After adding the packages to bootstrap.sh, node and npm install fine, but then I get this error:

default: npm http GET https://registry.npmjs.org/bower
default: npm http GET https://registry.npmjs.org/jasmine-core
default: npm http GET https://registry.npmjs.org/jshint
default: npm http GET https://registry.npmjs.org/karma
default: npm http GET https://registry.npmjs.org/karma-chrome-launcher
default: npm http GET https://registry.npmjs.org/karma-jasmine
default: npm http GET https://registry.npmjs.org/karma-junit-reporter
default: npm http GET https://registry.npmjs.org/karma-ng-html2js-preprocessor
default: npm http GET https://registry.npmjs.org/karma-phantomjs-launcher
default: npm ERR! Error: failed to fetch from registry: karma
default: npm ERR! at /usr/share/npm/lib/utils/npm-registry-client/get.js:139:12
default: npm ERR! at cb (/usr/share/npm/lib/utils/npm-registry-client/request.js:31:9)
default: npm ERR! at Request._callback (/usr/share/npm/lib/utils/npm-registry-client/request.js:136:18)
default: npm ERR! at Request.callback (/usr/lib/nodejs/request/main.js:119:22)
default: npm ERR! at Request. (/usr/lib/nodejs/request/main.js:212:58)
default: npm ERR! at Request.emit (events.js:88:20)
default: npm ERR! at ClientRequest. (/usr/lib/nodejs/request/main.js:412:12)
default: npm ERR! at ClientRequest.emit (events.js:67:17)
default: npm ERR! at HTTPParser.onIncoming (http.js:1261:11)
default: npm ERR! at HTTPParser.onHeadersComplete (http.js:102:31)
default: npm ERR! You may report this log at:
default: npm ERR! http://bugs.debian.org/npm
default: npm ERR! or use
default: npm ERR! reportbug --attach /home/vagrant/buildkit/npm-debug.log npm
default: npm ERR!
default: npm ERR! System Linux 3.2.0-23-generic-pae
default: npm ERR! command "node" "/usr/bin/npm" "install"
default: npm ERR! cwd /home/vagrant/buildkit
default: npm ERR! node -v v0.6.12
default: npm ERR! npm -v 1.1.4
default: npm ERR! message failed to fetch from registry: karma
default: npm ERR!
default: npm ERR! Additional logging details can be found in:
default: npm ERR! /home/vagrant/buildkit/npm-debug.log
default: npm not ok

looks like it is failing on 'karma'. On another attempt, it failed on js hint.

Any suggestions on getting this running? My goal was to have my enire team use civibuild, but we are not all on the same platform and a vagrant install would be great. Not to mention, this was the best way to deal with working with devs that use windows machines.

Just upgraded to Ubuntu 15.04 (from 14.10) which includes mysql 5.6

civibuild restore-all fails with:

2015-08-03 23:29:48 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-08-03 23:29:48 0 [Note] mysqld (mysqld 5.6.25-0ubuntu0.15.04.1) starting as process 12081 ...
2015-08-03 23:29:48 12081 [Warning] The option innodb (skip-innodb) is deprecated and will be removed in a future release
2015-08-03 23:29:48 12081 [ERROR] Unknown/unsupported storage engine: InnoDB
2015-08-03 23:29:48 12081 [ERROR] Aborting

on your doc: "New versions of buildkit are likely to include new versions of tools. The new tools will download automatically when you first run "civibuild". If you prefer to download explicitly, then re-run "civi-download-tools"."

but when I run civi-download-tools:

[Composer binary (/usr/src/civicrm-buildkit/bin/composer) already exists. Skipping.]]
[[composer dependencies already installed. Skipping.]]
[[civix binary (/usr/src/civicrm-buildkit/bin/civix) already exists. Skipping.]]
...

Is there a way to force?
ideally, some tools (eg civix are installed using git clone --depth=1) so I can cd extern/civix and git pull

PHP Fatal error: Call to a member function getDriver() on a non-object in /usr/src/civicrm-buildkit/vendor/totten/amp/src/Amp/Database/MySQL.php on line 58

new Datasource(); doesn't return an object
because
/usr/src/civicrm-buildkit/vendor/totten/amp/src/Amp/Database/Datasource.php

Not sure what I missed in the install, I'm following the README

This ticket is not to track progress on improving documentation, but rather simply to form a group decision on a plan for how to improve buildkit doc...

The problem

Currently, buildkit doc is spread out over several partially-overlapping sources:

• dedicated site
• README.md file
• /docs directory
• Dev Guide
• one large SE answer
• CLI help with civibuild --help

Further, many of these documentation sources do not make the presence of the other sources apparent to new users.

At the very least, I would like to see some consolidation and more inter-linking. But I think it's worth stepping back and thinking a little bigger. My hope is that upon closing this issue, a clear plan will exist to help consolidate and improve the buildkit documentation.

I have some ideas, which I'll share in the comments.

If the OS has npm, getting dependencies fails like this. I'll submit a PR in a minute.

symfony/console suggests installing symfony/event-dispatcher ()
phpunit/php-code-coverage suggests installing ext-xdebug (>=2.0.5)
phpunit/phpunit suggests installing phpunit/php-invoker (>=1.1.0,<1.2.0)
pear/console_table suggests installing pear/Console_Color (>=0.0.4)
symfony/templating suggests installing psr/log (For using debug logging in loaders)
symfony/dependency-injection suggests installing symfony/proxy-manager-bridge (Generate service proxies to lazy load them)
wp-cli/wp-cli suggests installing psy/psysh (Enhanced wp shell functionality)
Generating autoload files
npm WARN optional dep failed, continuing [email protected]

[email protected] install /opt/buildkit/node_modules/karma-phantomjs-launcher/node_modules/phantomjs
node install.js

sh: 1: node: not found

On Ubuntu 14.10 (on VirtualBox) buildkit - got "WARNING: Failed to locate command "node". NodeJS (http://nodejs.org/) is required for development of CiviCRM v4.6+." and "WARNING: Failed to locate command "node". NodeJS (http://nodejs.org/) is required for development of CiviCRM v4.6+." - I installed separately following instructions here: https://github.com/joyent/node/wiki/installing-node.js-via-package-manager

Specifically:
-view my contact
-edit my contact
-administer payment processor

It's causing tests to fail, and based on other permissions looks like it should have this.

Just a suggestion - My first attempt at installing buildkit failed because I didn't have it

Hi - love this!

I have the following feedback WRT using vagrant:
had to comment out server setting config.vm.network :private_network, ip: "192.168.168.10"

Need to install the following packages for precise32-standalone (sudo apt-get)
mcrypt
php5-mcrypt
drush

After these need to run:

1. civi-download-tools
2. Then this:
   sudo apt-get purge nodejs npm
   curl -sL https://deb.nodesource.com/setup | sudo bash -
   sudo apt-get install -y nodejs

buildkit doesn't add the drupal entry to the apache conf files
Change apache virtualhost to enable htaccess overrides

Options FollowSymLinks
AllowOverride All

<Directory /opt/buildkit/build/mytestbuild>
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Order allow,deny
allow from all

Then:
CiviCRM does not have permission to write temp files in /opt/buildkit/build/mytestbuild/sites/default/files/civicrm/templates_c/en_US/

I'm not quite sure if this is a docs issue or a civibuild issue.

I installed buildkit and now tried to run civibuild using the command here and it just keeps showing me the -h info again and again.
I'm running: $ civibuild create dmaster --url http://localhost:8001 --admin-pass s3cr3t

ajk@ajk-ThinkPad-R60:~/Documents/civi$ civibuild create dmaster --url http://dmaster.localhost --admin-pass 111
Common options:
  <build-name>        The name of the sub-directory to build
  <ms-id>             The unique id for a site within a multisite build

Syntax: civibuild create    <build-name>[/<ms-id>] [options]
Syntax: civibuild download  <build-name>           [options]
Syntax: civibuild install   <build-name>[/<ms-id>] [options]
Syntax: civibuild reinstall <build-name>[/<ms-id>] [options]
Description: Download and/or install the application
  --type <type>       The name of the download/install scripts. (If omitted, assume <build-name>) [Optional]
  --web-root <path>   The full path to the website root. [Default: /home/ajk/buildkit/build/<build-name>]
  --civi-ver <ver>    The branch or tag of CiviCRM desired (master, 4.4, 4.3, 4.3.0, etc) [Optional]
  --cms-ver <ver>     The release of the CMS desired [Optional]

  --url <url>         The public URL of the site
  --title <title>     The title of the site
  --admin-user        Name of the CMS's administrative user
  --admin-pass        Password for the CMS's administrative user
  --admin-email       Email of the CMS's administrative user
  --demo-user         Name of the CMS's demo user
  --demo-pass         Password for the CMS's demo user
  --demo-email        Email of the CMS's demo user

  --force             If necessary, destroy pre-existing files/directories/DBs
                      (For "reinstall", "--force" is implicit.)

Syntax: civibuild list
Description: Display a list of build-names found in the build directory. 

Syntax: civibuild edit <build-name>[/<ms-id>]
Description: Edit the <build-dir>/<build-name>[.<ms-id>].sh config file

Syntax: civibuild show <build-name>[/<ms-id>] [options]
Description: Show key details about the build
  --html <dir>        A new HTML dir which will report build status in detail
  --new-scan <file>   A new JSON file which summarizes the git structure
  --last-scan <file>  An existing JSON file which summarizes the old git structure
  --full              Show all build options declared

Syntax: civibuild snapshots
Description: List available snapshots

Syntax: civibuild snapshot <build-name>[/<ms-id>] [options]
Syntax: civibuild restore  <build-name>[/<ms-id>] [options]
Description: Create or restore a snapshot of the CMS+CRM DBs
  --snapshot <name>      The name of the snapshot subdirectory. (If omitted, assume <build-name>) [Optional]
  --cms-sql <sql-file>   The path to a SQL backup of the CMS DB [Optional]
  --no-cms               Skip resetting the CMS DB [Optional]
  --civi-sql <sql-file>  The path to a SQL backup of the CiviCRM DB [Optional]
  --no-civi              Skip resetting the CiviCRM DB [Optional]

Syntax: civibuild clone-create <build-name>[/<ms-id>] [options] [snapshot-options]
Syntax: civibuild clone-show <build-name>[/<ms-id>] [options]
Syntax: civibuild clone-destroy <build-name>[/<ms-id>] [options]
Description: Create, export, or destroy a DB clone based on a snapshot
  --clone-id <name>      The name of the specific clone. [Required for clone-create, clone-show]
  --force                If necessary, destroy pre-existing files/directories/DBs

Syntax: civibuild restore-all
Description: Restore *all* HTTP/DB services from their respective snapshots

Syntax: civibuild upgrade-test <build-name>[/<ms-id>] <sql-bz2-files>...
Description: Loads each DB snapshot and runs the current upgrade logic
  <sql-bz2-files>        One or more *.sql.bz2 files.
                         [Default path: /home/ajk/buildkit/vendor/civicrm/upgrade-test/databases/]

Syntax: civibuild destroy <build-name>
Description: Destroy all data about a particular build

I'm getting the below message from terminal after issuing ./bin/civi-download-tools on OS X Sierra

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

I believe that https://civicrm.org/get-buildkit.sh (referred here: https://github.com/civicrm/civicrm-buildkit/blob/master/doc/download-ubuntu.md) was lost when the site was overhauled.

The NodeJS interpreter name (node) has been changed to 'nodejs'. While installing, the following error occurs:

sh: 1: node: not found
npm WARN This failure might be due to the use of legacy binary "node"
npm WARN For further explanations, please read /usr/share/doc/nodejs/README.Debian 

An easy fix that worked for me is installing the nodejs-legacy package. It might be nice to mention it in the documentation.

Further reading: http://stackoverflow.com/questions/21168141/cannot-install-packages-using-node-package-manager-in-ubuntu

sorry @totten - I just rebuilt my docker container and trying to get my build operational

issueing reinstall gets me a working D7 instance but CiviCRM is disable on startup

any clues??

ampuser@e32a1f20b4dd:~$ civibuild reinstall dmaster --url http://dmaster.localhost --admin-pass s3cr>
[[Load saved options from /opt/buildkit/build/dmaster.sh]]
[[Install dmaster/default (type 'drupal-demo' in '/opt/buildkit/build/dmaster')]]
About to re-install "/opt/buildkit/build/dmaster". Are you sure you want destroy existing data? [y/N] y
[[Execute /opt/buildkit/app/config/drupal-demo/uninstall.sh]]
++ source /opt/buildkit/app/config/drupal-demo/uninstall.sh
+++ '[' -f /opt/buildkit/build/dmaster/sites/default/civicrm.settings.php ']'
+++ drupal_uninstall
+++ drupal7_uninstall
+++ cvutil_assertvars drupal7_uninstall CMS_ROOT SITE_ID CMS_URL
+++ _cvutil_assertvars_back=ehxB
+++ set +x
++++ _drupal_multisite_dir http://dmaster.localhost default
++++ '[' default == default ']'
++++ echo default
+++ DRUPAL_SITE_DIR=default
+++ '[' -n default -a -d /opt/buildkit/build/dmaster/sites/default ']'
+++ '[' default == default ']'
+++ pushd /opt/buildkit/build/dmaster
+++ chmod u+w sites/default
+++ '[' -f sites/default/settings.php ']'
+++ chmod u+w sites/default/settings.php
+++ rm -f sites/default/settings.php
+++ '[' -f sites/default/files ']'
+++ popd
+++ '[' -n default -a -d /opt/buildkit/app/private/dmaster/default ']'
+++ amp_uninstall
+++ echo 'WARNING: amp_uninstall: Retaining DB & site config to provide continuity among rebuilds'
WARNING: amp_uninstall: Retaining DB & site config to provide continuity among rebuilds
++ set +ex
[[Execute /opt/buildkit/app/config/drupal-demo/install.sh]]
++ source /opt/buildkit/app/config/drupal-demo/install.sh
+++ amp_install
+++ _amp_install_cms
+++ echo '[[Setup MySQL and HTTP for CMS]]'
[[Setup MySQL and HTTP for CMS]]
+++ cvutil_assertvars _amp_install_cms CMS_ROOT SITE_NAME SITE_ID TMPDIR
+++ _cvutil_assertvars_back=ehxB
+++ set +x
++++ mktemp.php ampvar
+++ local amp_vars_file_path=/tmp/ampvar9vMOpN
+++ local amp_name=cmsdefault
+++ '[' default == default ']'
+++ amp_name=cms
+++ '[' -n http://dmaster.localhost ']'
+++ amp create -f --root=/opt/buildkit/build/dmaster --name=cms --prefix=CMS_ --url=http://dmaster.localhost --output-file=/tmp/ampvar9vMOpN --perm=admin
+++ source /tmp/ampvar9vMOpN
++++ CMS_URL=http://dmaster.localhost
++++ CMS_ROOT=/opt/buildkit/build/dmaster
++++ CMS_DB_DSN='mysql://dmastercms_y21ow:[email protected]:3306/dmastercms_y21ow?new_link=true'
++++ CMS_DB_USER=dmastercms_y21ow
++++ CMS_DB_PASS=6JQyRjooQlcavUOr
++++ CMS_DB_HOST=127.0.0.1
++++ CMS_DB_PORT=3306
++++ CMS_DB_NAME=dmastercms_y21ow
++++ CMS_DB_ARGS='--defaults-file='\''/var/lib/amp/my.cnf.d/my.cnf-e002d0a77fcb578c8a4e4b1ae521eef5'\'' dmastercms_y21ow'
+++ rm -f /tmp/ampvar9vMOpN
+++ _amp_install_civi
+++ echo '[[Setup MySQL for Civi]]'
[[Setup MySQL for Civi]]
+++ cvutil_assertvars _amp_install_civi CMS_ROOT SITE_NAME SITE_ID TMPDIR
+++ _cvutil_assertvars_back=ehxB
+++ set +x
++++ mktemp.php ampvar
+++ local amp_vars_file_path=/tmp/ampvarmeCiXd
+++ local amp_name=cividefault
+++ '[' default == default ']'
+++ amp_name=civi
+++ amp create -f --root=/opt/buildkit/build/dmaster --name=civi --prefix=CIVI_ --skip-url --output-file=/tmp/ampvarmeCiXd --perm=super
+++ source /tmp/ampvarmeCiXd
++++ CIVI_URL=
++++ CIVI_ROOT=/opt/buildkit/build/dmaster
++++ CIVI_DB_DSN='mysql://dmasterciv_htrfu:[email protected]:3306/dmasterciv_htrfu?new_link=true'
++++ CIVI_DB_USER=dmasterciv_htrfu
++++ CIVI_DB_PASS=PuEM9Ds3EuDhRFUO
++++ CIVI_DB_HOST=127.0.0.1
++++ CIVI_DB_PORT=3306
++++ CIVI_DB_NAME=dmasterciv_htrfu
++++ CIVI_DB_ARGS='--defaults-file='\''/var/lib/amp/my.cnf.d/my.cnf-d801f1b090da10f54a24aa165eed070f'\'' dmasterciv_htrfu'
+++ rm -f /tmp/ampvarmeCiXd
+++ _amp_install_test
+++ echo '[[Setup MySQL for Test]]'
[[Setup MySQL for Test]]
+++ cvutil_assertvars _amp_install_test CMS_ROOT SITE_NAME SITE_ID TMPDIR
+++ _cvutil_assertvars_back=ehxB
+++ set +x
++++ mktemp.php ampvar
+++ local amp_vars_file_path=/tmp/ampvart3yFSj
+++ local amp_name=testdefault
+++ '[' default == default ']'
+++ amp_name=test
+++ amp create -f --root=/opt/buildkit/build/dmaster --name=test --prefix=TEST_ --skip-url --output-file=/tmp/ampvart3yFSj --perm=super
+++ source /tmp/ampvart3yFSj
++++ TEST_URL=
++++ TEST_ROOT=/opt/buildkit/build/dmaster
++++ TEST_DB_DSN='mysql://dmastertes_j8kxc:[email protected]:3306/dmastertes_j8kxc?new_link=true'
++++ TEST_DB_USER=dmastertes_j8kxc
++++ TEST_DB_PASS=zWEnl7i7uczByTZk
++++ TEST_DB_HOST=127.0.0.1
++++ TEST_DB_PORT=3306
++++ TEST_DB_NAME=dmastertes_j8kxc
++++ TEST_DB_ARGS='--defaults-file='\''/var/lib/amp/my.cnf.d/my.cnf-0e4f2545763f9f7b31c306153f2c37cb'\'' dmastertes_j8kxc'
+++ rm -f /tmp/ampvart3yFSj
+++ drupal_install
+++ drupal7_install
+++ cvutil_assertvars drupal7_install CMS_ROOT SITE_ID CMS_TITLE CMS_DB_USER CMS_DB_PASS CMS_DB_HOST CMS_DB_NAME ADMIN_USER ADMIN_PASS CMS_URL
+++ _cvutil_assertvars_back=ehxB
+++ set +x
++++ _drupal_multisite_dir http://dmaster.localhost default
++++ '[' default == default ']'
++++ echo default
+++ DRUPAL_SITE_DIR=default
++++ cvutil_build_hostport 127.0.0.1 3306
++++ local host=127.0.0.1
++++ local port=3306
++++ '[' -z 3306 ']'
++++ echo 127.0.0.1:3306
+++ CMS_DB_HOSTPORT=127.0.0.1:3306
+++ pushd /opt/buildkit/build/dmaster
+++ '[' -f sites/default/settings.php ']'
+++ drush site-install -y --db-url=mysql://dmastercms_y21ow:[email protected]:3306/dmastercms_y21ow --account-name=admin --account-pass=s3cr3t [email protected] '--site-name=CiviCRM Sandbox on Drupal' --sites-subdir=default
You are about to create a sites/default/settings.php file and DROP all tables in your 'dmastercms_y21ow' database. Do you want to continue? (y/n): y
No tables to drop.                                                                                [ok]
Starting Drupal installation. This takes a few seconds ...                                        [ok]
sh: 1: /usr/sbin/sendmail: not found
WD mail: Error sending e-mail (from [email protected] to [email protected]).                   [error]
Installation complete.  User name: admin  User password: s3cr3t                                   [ok]
Unable to send e-mail. Contact the site administrator if the problem persists.                 [error]

Description

amp test fails. I ran through amp config with multiple different settings to try and resolve the issue, with no luck.

Environment:

Operating System: CentOS release 6.8 (Final)
Apache: 2.4.23
PHP: 5.6.24
MySQL: 5.5.50
NodeJS: 4.4.7
cPanel: 56.0 (build 28)

Output

root@c3po [/etc/httpd]# amp test
Create test application
Create data directory: /root/.amp/canary/data/tCBxcjTj1Ec4iIrtoANOYJXjkPIFMHp6
AMP_URL='http://localhost:7979'
AMP_ROOT='/root/.amp/canary'
AMP_DB_DSN='mysql://canary_1zgw6:[email protected]:3306/canary_1zgw6?new_link=true'
AMP_DB_USER='canary_1zgw6'
AMP_DB_PASS='8ZDbjqHgNGons7jG'
AMP_DB_HOST='127.0.0.1'
AMP_DB_PORT='3306'
AMP_DB_NAME='canary_1zgw6'
AMP_DB_ARGS='--defaults-file='''/root/.amp/my.cnf.d/my.cnf-0b5b5bf7f16b99daea17a437816adb76''' canary_1zgw6'

Connect to test application
Expect response: "response-code-7juK2zu89U"

Warning: file_get_contents(http://localhost:7979/index.php): failed to open stream: Connection refused in phar:///opt/buildkit/bin/amp/src/Amp/Command/TestCommand.php on line 163
Received incorrect response: ""
Tips for common issues:

• (Re)run "amp config"
• Double-check the httpd_type (apache24) along with any displayed instructions.
• Double-check the httpd_restart_command.
• (Re)run "amp test"
  root@c3po [/etc/httpd]#

I'm following instructions here https://buildkit.civicrm.org/#/tutorials and my first build fails with this error message below. Did i miss a step to grant www-data this permission?

Started by user brian
Building in workspace /var/lib/jenkins/jobs/ScheduledTest-drupal-clean/workspace
[workspace] $ /bin/bash /tmp/hudson5034185699702965011.sh

• export PATH=/opt/buildkit/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
• PATH=/opt/buildkit/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
• BLDNAME=jenkins-1
• BLDURL=http://localhost:8101
• EXITCODE=0
• '[' -d /var/lib/jenkins/jobs/ScheduledTest-drupal-clean/workspace/junit ']'
• '[' -d /var/lib/jenkins/jobs/ScheduledTest-drupal-clean/workspace/civibuild-html ']'
• '[' -d /opt/buildkit/build/jenkins-1 ']'
• mkdir /var/lib/jenkins/jobs/ScheduledTest-drupal-clean/workspace/junit
• mkdir /var/lib/jenkins/jobs/ScheduledTest-drupal-clean/workspace/civibuild-html
• civibuild download jenkins-1 --civi-ver master --type drupal-clean
  PHP Warning: file_put_contents(/opt/buildkit/app/tmp/civi-download-tools.lock): failed to open stream: Permission denied in /opt/buildkit/bin/pidlockfile.php on line 91

Warning: file_put_contents(/opt/buildkit/app/tmp/civi-download-tools.lock): failed to open stream: Permission denied in /opt/buildkit/bin/pidlockfile.php on line 91
touch: cannot touch ‘/opt/buildkit/app/tmp/composer-data.md5’: Permission denied
Build step 'Execute shell' marked build as failure
[htmlpublisher] Archiving HTML reports...

In app/config/wp-demo/install.sh Paths are set hardcoded -- https://github.com/civicrm/civicrm-buildkit/blob/master/app/config/wp-demo/install.sh#L16

Due to WP installer changes the paths for v4.7.alpha4+ should be:
ln 21:

CIVI_SETTINGS="${WEB_ROOT}/wp-content/uploads/civicrm/civicrm.settings.php"
CIVI_FILES="${WEB_ROOT}/wp-content/uploads/civicrm"
CIVI_EXT_DIR="${WEB_ROOT}/wp-content/uploads/civicrm/ext"
CIVI_EXT_URL="${CMS_URL}/wp-content/uploads/civicrm/ext"

Obviously this should be conditional based on the --civi-ver flag.

I can attempt a patch, however not sure when I will get to it, so I am posting here for now.

Well, TBH I just think it's gencode. I'm building CiviCRM tarballs for Fuzion, and without some MySQL configuration, gencode spits out a lot of

Access denied for user ''@'localhost' (using password: NO)

when building the various DAO files, and the resulting tarballs contain invalid / missing SQL.

I looked for a config option to set a DSN, but failed to identify where it's set.

MySQL does not need a connection to any particular DB - it just wants a connection to pass to mysql_real_escape_string() I believe. So a workaround is,

GRANT SELECT ON test.doesnotexist TO ''@localhost;

But I'd rather be able to configure the connection details than do this. How are others managing this in order to successfully build CiviCRM distribution tarballs?

looking at this page: https://github.com/civicrm/civicrm-buildkit/blob/master/doc/path.md

I am just confused because civix isn't in the buildkit/bin

root@6b35adb896d1:~# ls /root/buildkit/bin/
civi-download-tools  find-symlink-tree-orphans  phpcs-civi
civibuild            gitify                     phpunit-each
cividist             givi                       phpunit-xml-cleanup
civihydra            mkpasswd.php               pidlockfile.php
civilint             mktemp.php                 regmv
extract-url          pdepend-civi-csv           releaser
fetch-universe       pdepend-to-csv             timeout.php
find-stale-builds    phpcbf-civi

So i've done something wrong or the example needs changing?

The drupal_install function is mostly generic enough to work with all 3 drupal versions - but there is one line it it which is not d8-compatible. I propose moving it to the specific install scripts

drush vset --yes file_private_path "${PRIVATE_ROOT}/${DRUPAL_SITE_DIR}"

From example in #155, a working implementation.

RM11632

In theory we can switch to the version that does - although it is beta & may be less stable

Sourceforge has been 'temporarily in static offline mode' all morning. They've never been the best hosting service, and I'm worried about their future given the Slashdot layoffs after being sold by Dice.

To get through my earlier error (#121), I re-configured permission to "Loose".

Then, when I ran ./amp test I got:

    civicrm@piglet:/var/www/test/civicrm-buildkit/bin$ ./amp test
    Create test application
    Create data directory: /home/civicrm/.amp/canary/data/xxxxxxxxxxxxxxxxxxxxx
    AMP_URL='http://localhost:7979'
    AMP_ROOT='/home/civicrm/.amp/canary'
    AMP_DB_DSN='mysql://canary_dnil7:[email protected]:3306/canary_dnil7?new_link=true'
    AMP_DB_USER='canary_dnil7'
    AMP_DB_PASS='xxxxxxxxxxxxxxxx'
    AMP_DB_HOST='127.0.0.1'
    AMP_DB_PORT='3306'
    AMP_DB_NAME='canary_dnil7'
    AMP_DB_ARGS='-h 127.0.0.1 -u canary_dnil7 -pxxxxxxxxxxxxxxxx -P 3306 canary_dnil7'

    Connect to test application
    Expect response: "response-code-0scdfw0yPk"
    PHP Warning:  file_get_contents(http://localhost:7979/index.php): failed to open stream: HTTP request failed! HTTP/1.1 500 Internal Server Error
     in /var/www/test/civicrm-buildkit/vendor/totten/amp/src/Amp/Command/TestCommand.php on line 153

    Warning: file_get_contents(http://localhost:7979/index.php): failed to open stream: HTTP request failed! HTTP/1.1 500 Internal Server Error
     in /var/www/test/civicrm-buildkit/vendor/totten/amp/src/Amp/Command/TestCommand.php on line 153
    Received incorrect response: ""
    Tip: Try running "amp config" and/or restarting the webserver.
    civicrm@piglet:/var/www/test/civicrm-buildkit/bin$

And the log file reports:

    [Mon Jan 05 10:03:47 2015] [crit] [client 127.0.0.1] configuration error:  couldn't perform authentication. AuthType not set!: /index.php

Need to either remove the documentation, or implement. It would be useful for the wmff build...

MySQL is MySQL Community 5.5.39 GA from http://dev.mysql.com/downloads/mysql/5.5.html#downloads
MySQL client binary is in PATH at /usr/local/mysql/bin/mysql

Workaround: sudo ln -s /usr/local/mysql/bin/mysql /usr/local/bin/mysql

It seems to be the case that civibuild needs to be run as a user who is not root but who has sudo.

It maybe helpful to say this in the README?

(happy to do a patch if you like)

I'm just looking into setting up the script to generate 1) only drupal 7 & 2) drupal 6 & 7 but not Joomla & WP in 2 different setups.

I know we can pass in things like env GIT_REMOTE=fuzion - are you open to a patch that would allow
env CMS_LIST=drupal6,drupal

I'm also mulling over the fact that we want to untar the tarball & commit to a deployment branch - possibly that would look somewhat similar in terms of using an env to do it - or perhaps it just needs to be in a wrapper script that may or may not be accepted into the buildkit repo

I am trying to create a dev environment of civi + wordpress on my computer.
Note: I was able to create a drupal + civi dev env earlier.

I'm running the command civibuild create wpmaster --url http://wpmaster.localhost --admin-pass <pass>

I seem to hit a snag when civibuild tries to download the wordpress plugin wordpress-importer. It says the plugin could not be found:

...
....
++ wp rewrite flush --hard
Success: Rewrite rules flushed.
++ wp plugin install wordpress-importer --activate
Warning: wordpress-importer: An unexpected error occurred. Something may be wrong with WordPress.org or this server&#8217;s configuration. If you continue to have problems, please try the <a href="https://wordpress.org/support/">support forums</a>.
Activating 'wordpress-importer'...
Warning: The 'wordpress-importer' plugin could not be found.
++ wp import /home/ajk/buildkit/app/config/wp-demo/civicrm-wordpress.xml --authors=create
Error: WordPress Importer needs to be installed. Try 'wp plugin install wordpress-importer --activate'.

I tried going into ~/buildkit/build/wpmaster/wp-content/plugins/civicrm/civicrm and running the command, but it keeps giving the same error. I'm not sure what I can do to debug this (I haven't used wp-cli before)

I keep getting this failure message, below, when running civi-download-tools. I installed phantomjs from source last night but it's not recognized as already installed. I think the issue is that the install script needs to contain nodejs install.js as well as/instead of node install.js .

Also, my installed directory is
~/buildkit/node_modules/karma-chrome-launcher/node_modules/
not
~buildkit/node_modules/karma-phantomjs-launcher/node_modules/phantomjs

so I'll figure out where phantomjs is actually installed.

[email protected] install /home/owner/buildkit/node_modules/karma-phantomjs-launcher/node_modules/phantomjs
node install.js

npm WARN This failure might be due to the use of legacy binary "node"
npm WARN For further explanations, please read
/usr/share/doc/nodejs/README.Debian

npm ERR! [email protected] install: node install.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the phantomjs package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node install.js
npm ERR! You can get their info via:
npm ERR! npm owner ls phantomjs
npm ERR! There is likely additional logging output above.

npm ERR! System Linux 3.16.0-34-generic
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install"
npm ERR! cwd /home/owner/buildkit
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.4.21
npm ERR! code ELIFECYCLE
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /home/owner/buildkit/npm-debug.log
npm ERR! not ok code 0

I don't work with multisite, but this potentially looks like a bug; haven't encountered a problem in the wild, just noting inconsistency.

https://github.com/civicrm/civicrm-buildkit/blob/master/bin/civibuild#L565

should it not use civibuild_save?
Seems like if multi-site params were provided, then they could get saved to the wrong file.

see here in bin/setup.sh

137 ###############################################################################
138 ## Execute tasks
139 set -x
140 
141 if [ -n "$DO_DOWNLOAD" ]; then

Drupal 8 is unable to run

"$PHP5PATH"php bin/cli.php -e System -a flush --triggers 1 --session 1

due to race conditions - esp around symphony components

We can avoid this flag by calling setup.sh from buildkit with all flags except that ie.

if [ -e "xml" -D -g -s -d -e "bin/setup.sh" ];

instead of
if [ -e "xml" -a -e "bin/setup.sh" ]; then

My take on this is the only think possibly lost is calling build menu?

So, options are

1. always call with the above flags rather than -a
2. figure out some d8 conditional
3. fork the install civicrm function in civibuild.lib.sh, much like we have the drupal_install

We might still need to call the system.flush - but in drupal at least we can do using drush.

If I run civibuild create foo --url foo.local --type drupal-demo it fails with the error message "Failed to parse URL: foo.local". Eventually I figured out that I needed to use --url http://foo.local, but it would be nice if civibuild could be smarter about this and accept --url foo.local as well.

Hi,

with various tests and crash during the install, I end up with loads of databases that have the same pattern drupaldemo_{random}. is there a way to

1. have a single database for each site instead of two?
2. control or skip the {random} at the end (rationale: if I have several databases with the same site name, I got something wrong I'd investigate beside of piling stalled databases.

right now, drush civicrm-sql-cli connects to the drupal db, that makes it even more annoying not to be able to see which database is the right one by its name ;)

@totten and I had spoken yesterday about adding tests for buildkit.

So, I began trying to write unit-tests to check the bash functions (as they're the most common in the repo). While doing this I found it very difficult to test basic functions because it's tough to import them. It's not modularized from the script inside the bin folder.

Ideally we should have lib files for each of the bin files which contain all of the functions independently and then source the lib into the bin to use them as required.
Also, some small functions like absdirname() are being using in multiple locations (releaser, phpcs-civi, gitify, civilint, civibuild, and civi-download-tools). We should move them to a common lib file.

I notice that something like this is already there for civibuild - It has a src/civibuild.lib.sh.
Unless it's appropriately modularized, it becomes difficult to add unittests for functions.

Is this something that's wanted ?

If yes, what sort of directory structure should be followed ? I was thinking:

.
|-- bin
|   |-- civibuild
|   |-- cividist
|   `-- etc ...
|-- src
|   |-- common.lib.sh # Used by more than 1 bin
|   |-- civibuild.lib.sh
|   |-- cividist.lib.sh
|   `-- etc ...
`--- etc ...

Note that with this structure, some files would have more than 500 lines of code (I notice that already there are files with 700-800 lines). Is that alright ?

I'm running debian wheezy.

When I ran the command:

./amp config

I chose the linuxACL option and www-data as the web user. No errors.

Then, when I ran:

./amp test I got:

civicrm@piglet:/var/www/test/civicrm-buildkit/bin$ ./amp test
Create test application
Create data directory: /home/civicrm/.amp/canary/data/4cJePYaQav6IYhZLF1MJkVvp2Iq1T38N
PHP Fatal error: Uncaught exception 'RuntimeException' with message 'setfacl: /home/civicrm/.amp/canary/data/4cJePYaQav6IYhZLF1MJkVvp2Iq1T38N: Operation not supported
' in /var/www/test/civicrm-buildkit/vendor/totten/amp/src/Amp/Permission/External.php:43
Stack trace:
#0 /var/www/test/civicrm-buildkit/vendor/totten/amp/src/Amp/Command/DatadirCommand.php(43): Amp\Permission\External->applyDirPermission('write', '/home/civicrm/....')
#1 /var/www/test/civicrm-buildkit/vendor/symfony/console/Symfony/Component/Console/Command/Command.php(241): Amp\Command\DatadirCommand->execute(Object(Symfony\Component\Console\Input\ArrayInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#2 /var/www/test/civicrm-buildkit/vendor/totten/amp/src/Amp/Command/ContainerAwareCommand.php(26): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArrayInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#3 /var/www/test/civicrm-buildkit/vendor/totten/amp/src/Amp/Command/TestCommand.php(117): Amp\Comm in /var/www/test/civicrm-buildkit/vendor/totten/amp/src/Amp/Permission/External.php on line 43

Fatal error: Uncaught exception 'RuntimeException' with message 'setfacl: /home/civicrm/.amp/canary/data/4cJePYaQav6IYhZLF1MJkVvp2Iq1T38N: Operation not supported
' in /var/www/test/civicrm-buildkit/vendor/totten/amp/src/Amp/Permission/External.php:43
Stack trace:
#0 /var/www/test/civicrm-buildkit/vendor/totten/amp/src/Amp/Command/DatadirCommand.php(43): Amp\Permission\External->applyDirPermission('write', '/home/civicrm/....')
#1 /var/www/test/civicrm-buildkit/vendor/symfony/console/Symfony/Component/Console/Command/Command.php(241): Amp\Command\DatadirCommand->execute(Object(Symfony\Component\Console\Input\ArrayInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#2 /var/www/test/civicrm-buildkit/vendor/totten/amp/src/Amp/Command/ContainerAwareCommand.php(26): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArrayInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#3 /var/www/test/civicrm-buildkit/vendor/totten/amp/src/Amp/Command/TestCommand.php(117): Amp\Comm in /var/www/test/civicrm-buildkit/vendor/totten/amp/src/Amp/Permission/External.php on line 43

civicrm@piglet:/var/www/test/civicrm-buildkit/bin$

My user is in the www-group already. Does this need to be run as root?

I'm new to CiViCRM. I want to fetch events which are created in CiViCRM which is hosted on another server. So basically I want to fetch CiViCRM data from one individual site to my site. I don't know that which are the ways to do it and which is the best one. But I think Rest option would be good if I'm not wrong. I'm not able to find any similar example which is near to my task. All I don't understand that how should I use my rest url with code:

1. Option 1:

$api = new civicrm_api3(array ('server' => 'http://localhost/abc',

'api_key'=>'123456789',
'key'=>'123xyzABC'));

2. Option 2:

 $contactxml =       simplexml_load_file("http://localhost/abc/sites/all/modules/civicrm/extern /rest.php?entity=contact&action=get&key=<site    key>&api_key=$apikey&last_name=Koot");
  foreach ($contactxml->children() as $contact) {
    $display = $contact->display_name;
  }

Is there any other options. I want to go for option 1 if that is possible. And I want response in JSON if possible.

Any help wound be appreciated.

Thanks

FYI, re-running install I see buildkit is expecting an old version of nodejs. Might be nice to support current:

Joes-MacBook-Pro:buildkit josephmurray$ ./bin/civi-download-tools
[[Composer binary (/Users/josephmurray/Sites/civicrm/buildkit/bin/composer) already exists. Skipping.]]
[[composer dependencies already installed. Skipping.]]
npm WARN engine [email protected]: wanted: {"node":"~0.8 || ~0.10"} (current: {"node":"0.12.3","npm":"2.9.1"})

Currently civibuild drops out with

Please prepare a 'dist' folder using buildkit and run the command within that folder.

I'd like to better document what needs doing here - there's no obvious civibuild command.

As far as I can tell -

• Create a directory called default (?)
• Create a directory out for tarballs to go to
• Create a directory web for the webroot to point to
• Clone repos in src/ (see https://github.com/civicrm/civicrm-buildkit/blob/master/doc/cividist.md for layout - civicrm-core and then checkouts of civicrm-wordpress, civicrm-drupal, civicrm-packages, civicrm-joomla contained)

.
├── default
├── out
│   ├── config
│   ├── gen
│   ├── tar
│   └── tmp
├── src
│   ├── CRM
│   ├── Civi
│   ├── WordPress
│   ├── ang
│   ├── api
│   ├── bin
│   ├── bower_components
│   ├── css
│   ├── distmaker
│   ├── drupal
│   ├── extern
│   ├── i
│   ├── install
│   ├── joomla
│   ├── js
│   ├── l10n
│   ├── node_modules
│   ├── packages
│   ├── settings
│   ├── sql
│   ├── templates
│   ├── tests
│   ├── tools
│   ├── vendor
│   └── xml
└── web
    ├── by-date
    └── by-hash

I'm working on dockerising civi-download-tools

it contains a series of useful system - Lines 423-444 and 568-590 of this version

It seems to me that this would be best and very useful as a standalone bash script.

What do you think?

Possible with a mode argument? full|essential

civichecksytem full or civichecksytem essential

it would be good to include version checks as well (but that would be another day!)

Installing buildkit on a Mac OS X 10.9.5 yields errors below. It seems it would be good to add installation of nodejs (and its npm dependency) as part of buildkit (or perhaps to change documentation to indicate user should go to nodejs.org and click install before running buildkit install:

~/Sites/civicrm/buildkit ~/Sites/civicrm/buildkit
WARNING: Failed to locate command "node". NodeJS (http://nodejs.org/) is required for development of CiviCRM v4.6+.

It is possible that you have already installed "node" in a non-standard location.
If so, please update the PATH appropriately. Possible matches were found in:

• /usr/local/bin
• /usr/local/bin
  WARNING: Failed to locate command "npm". NodeJS (http://nodejs.org/) is required for development of CiviCRM v4.6+.

It is possible that you have already installed "npm" in a non-standard location.
If so, please update the PATH appropriately. Possible matches were found in:

• /usr/local/bin
• /usr/local/bin

I know it is listed as a req but might be nice to check first. For bonus check if sourceforge is actually up - that took me a while to figure out :)

When spinning up a new drupal install, buildkit creates separate databases with different sql users.
To get views working one must manually copy-paste the db code to settings.php and also grant select permissions to the sql user of one db to the other.