sandstorm-io / vagrant-spk Goto Github PK

When following the packaging tutorial on Ubuntu 14.04.03 LTS up-to-date, using vagrant 1.7.4, Oracle VM VirtualBox Manager 5.0.4, vagrant-spk (commit 7f29d1f), I get the following warning at the end of the process:

==> default: Synchronizing state for nginx.service with sysvinit using update-rc.d...
==> default: Executing /usr/sbin/update-rc.d nginx defaults
==> default: Executing /usr/sbin/update-rc.d nginx disable
==> default: insserv: warning: current start runlevel(s) (empty) of script `nginx' overrides LSB defaults (2 3 4 5).
==> default: insserv: warning: current stop runlevel(s) (0 1 2 3 4 5 6) of script `nginx' overrides LSB defaults (0 1 6).
==> default: Synchronizing state for php5-fpm.service with sysvinit using update-rc.d...
==> default: Executing /usr/sbin/update-rc.d php5-fpm defaults
==> default: Executing /usr/sbin/update-rc.d php5-fpm disable
==> default: insserv: warning: current start runlevel(s) (empty) of script `php5-fpm' overrides LSB defaults (2 3 4 5).
==> default: insserv: warning: current stop runlevel(s) (0 1 2 3 4 5 6) of script `php5-fpm' overrides LSB defaults (0 1 6).
==> default: insserv: warning: current start runlevel(s) (empty) of script `php5-fpm' overrides LSB defaults (2 3 4 5).
==> default: insserv: warning: current stop runlevel(s) (0 1 2 3 4 5 6) of script `php5-fpm' overrides LSB defaults (0 1 6).
==> default: Synchronizing state for mysql.service with sysvinit using update-rc.d...
==> default: Executing /usr/sbin/update-rc.d mysql defaults
==> default: Executing /usr/sbin/update-rc.d mysql disable
==> default: insserv: warning: current start runlevel(s) (empty) of script `mysql' overrides LSB defaults (2 3 4 5).
==> default: insserv: warning: current stop runlevel(s) (0 1 2 3 4 5 6) of script `mysql' overrides LSB defaults (0 1 6).

It's also worth noting that during this process I also see other red text flow past, at least a couple of times it seems to be that the process is expecting input from the tty but not getting it - see these two excerpts:

==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
    default: The guest additions on this VM do not match the installed version of
    default: VirtualBox! In most cases this is fine, but in rare cases it can
    default: prevent things such as shared folders from working properly. If you see
    default: shared folder errors, please make sure the guest additions within the
    default: virtual machine match the version of VirtualBox you have installed on
    default: your host and reload your VM.
    default:
    default: Guest Additions Version: 4.3.18
    default: VirtualBox Version: 5.0
==> default: Mounting shared folders...
    default: /opt/app => /home/datakid/src/work/sandstorm/omeka-2.3.1
    default: /vagrant => /home/datakid/src/work/sandstorm/omeka-2.3.1
    default: /host-dot-sandstorm => /home/datakid/.sandstorm
==> default: Running provisioner: shell...
    default: Running: inline script
==> default: stdin: is not a tty
==> default:   % Total    % Received % Xferd  Average Speed   Time    Time
==> default:      Time  Current
==> default:                            
==> default:   Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 60 53969   60 32482    0     0  28402      0  0:00:01  0:00:01 --:--:-- 28418
100 53969  100 53969    0     0  47118      0  0:00:01  0:00:01 --:--:-- 47134
==> default: Sandstorm requires sysctl kernel.unprivileged_userns_clone to be enabled


==> default: Unpacking mysql-client-5.5 (5.5.44-0+deb8u1) ...
==> default: Selecting previously unselected package mysql-server-core-5.5.
==> default: Preparing to unpack .../mysql-server-core-5.5_5.5.44-0+deb8u1_amd64.deb ...
==> default: Unpacking mysql-server-core-5.5 (5.5.44-0+deb8u1) ...
==> default: Processing triggers for man-db (2.7.0.2-5) ...
==> default: Setting up mysql-common (5.5.44-0+deb8u1) ...
==> default: Selecting previously unselected package mysql-server-5.5.
==> default: (Reading database ...
(Reading database ... 15%abase ... 5%
(Reading database ... 25%abase ... 20%
(Reading database ... 60%abase ... 30%
==> default: (Reading database ... 65%
==> default: (Reading database ... 70%
==> default: (Reading database ... 75%
==> default: (Reading database ... 80%
==> default: (Reading database ... 85%
==> default: (Reading database ... 90%
==> default: (Reading database ... 95%
(Reading database ... 36210 files and directories currently installed.)

Most Windows users probably don't have a Python installation running around, so we should package vagrant-spk into either a single-file vagrant-spk.exe which embeds a python interpreter and can run cleanly, or an installer bundle (.msi?).

I gently prefer the former, since it means there's one less install step for end users.

Possible tools:

• http://www.py2exe.org/ for standalone program
• https://pynsist.readthedocs.org/en/latest/ for installers

@ndarilek writes about this problem here: https://groups.google.com/forum/#!topic/sandstorm-dev/M41u_K8iAgA

To copy & paste:

The last things I see on my vagrant-spk dev console are:

[email protected] node_modules/source-map-support
â""â"?â"? [email protected] ([email protected])
App is now available from Sandstorm server. Ctrl+C to disconnect.
WriteResult({ "nInserted" : 1 })

This is followed by the shutdown message when I terminate the app.

At the top of server/users.ls I have:

console.log("Running")

Meteor.publish "users", ->
  console.log("Subscribing")
  ...

I'd expect to see "Running"/"Subscribing" shortly after I'm told the app
is running.

Thanks.

Steps to reproduce

If you create a new VM with vagrant-spk with the lemp backend, and it happens to be the case that the directory you point nginx to has no index.php, then nginx wants to give you a 403 Forbidden page.

Actual results

With the current setup, what I see is the following gibberish:

�������!�w�ߡ����qB�����O�z�p`�D���=����ؿ_�6�&�NN'�B)�M��j��}��"����
�D2�����6��7���DUQ���E)L�mȉ�c����X���;o�u��$x�Q��}ߌ1PpQ��w����*�������Gؚ�z�s���
@1��':�Ɗ����/���bB:�

(Lots of upside down question marks and one-half signs.)

Expected results

I see the following message:

403 Forbidden

nginx/1.6.2

(or, preferably, a custom message like "This Sandstorm app seems to be serving a 404 at the top path (/). Read $URL to learn more.")

Proposed solution

In our nginx configuration in vagrant-spk, always set

gzip off;

in nginx.conf

I tested that this makes the 403 Forbidden text appear.

This is with a freshly-created vagrant-spk setupvm meteor:

Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'debian/jessie64'...

�[KProgress: 20%
�[KProgress: 30%
�[KProgress: 40%
�[KProgress: 90%
�[K==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'debian/jessie64' is up to date...
==> default: Setting the name of the VM: sandstorm_default_1437148847579_3526
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
default: Adapter 1: nat
==> default: Forwarding ports...
default: 6080 => 6080 (adapter 1)
default: 22 => 2222 (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
default: SSH address: 127.0.0.1:2222
default: SSH username: vagrant
default: SSH auth method: private key
default: Warning: Connection timeout. Retrying...
default:
default: Vagrant insecure key detected. Vagrant will automatically replace
default: this with a newly generated keypair for better security.
default:
default: Inserting generated public key within guest...
default: Removing insecure key from the guest if it's present...
default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
default: The guest additions on this VM do not match the installed version of
default: VirtualBox! In most cases this is fine, but in rare cases it can
default: prevent things such as shared folders from working properly. If you see
default: shared folder errors, please make sure the guest additions within the
default: virtual machine match the version of VirtualBox you have installed on
default: your host and reload your VM.
default:
default: Guest Additions Version: 4.3.18
default: VirtualBox Version: 5.0
==> default: Mounting shared folders...
default: /opt/app => C:/Users/nolan/Projects/Scheduler
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

chown id -u vagrant:id -g vagrant /opt/app

Stdout from the command:

Stderr from the command:

stdin: is not a tty
chown: changing ownership of ‘/opt/app’: Not a directory
Calling 'vagrant' 'up' in c:\users\nolan\Projects\Scheduler.sandstorm
Traceback (most recent call last):
File "c:/users/nolan/appdata/local/bin/vagrant-spk", line 779, in
main()
File "c:/users/nolan/appdata/local/bin/vagrant-spk", line 776, in main
operation(args)
File "c:/users/nolan/appdata/local/bin/vagrant-spk", line 699, in bring_up_vm
call_vagrant_command(sandstorm_dir, "up")
File "c:/users/nolan/appdata/local/bin/vagrant-spk", line 620, in call_vagrant_command
return subprocess.check_call(command, cwd=sandstorm_dir)
File "C:\python2\Lib\subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['vagrant', 'up']' returned non-zero exit status 1

The second vagrant-spk up works, claiming that VirtualBox is already running. Though I don't know what state it's in; perhaps failure to launch the VM should run halt, and up should always attempt a provision? Or an up that fails to create the VM should halt and destroy so Vagrant won't be in an inconsistent state?

Right now it's not very discoverable what stacks exist and what they include. If we want people to use these stacks, we should make them more visible.

On OSX 10.10.5, with Vagrant 1.7.2, if I do vagrant-spk setupvm uwsgi and then vagrant-spk up, I see this, and then the script hangs:

Calling 'vagrant' 'up' in /Users/dwrensha/Desktop/test-vagrant-spk/.sandstorm
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'debian/jessie64'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'debian/jessie64' is up to date...
==> default: A newer version of the box 'debian/jessie64' is available! You currently
==> default: have version '8.1.0'. The latest is version '8.2.0'. Run
==> default: `vagrant box update` to update.
==> default: Setting the name of the VM: test-vagrant-spk_sandstorm_1444573630
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 6080 => 6080 (adapter 1)
    default: 22 => 2222 (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection timeout. Retrying...
    default: 
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default: 
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if its present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Mounting shared folders...
    default: /opt/app => /Users/dwrensha/Desktop/test-vagrant-spk
    default: /vagrant => /Users/dwrensha/Desktop/test-vagrant-spk
    default: /host-dot-sandstorm => /Users/dwrensha/.sandstorm
==> default: Running provisioner: shell...
    default: Running: inline script
==> default: Created symlink from /etc/systemd/system/multi-user.target.wants/sandstorm.service to /etc/systemd/system/sandstorm.service.
==> default: Running provisioner: shell...
    default: Running: inline script

The problem goes away if I revert this commit: 116f943

In this file https://github.com/zarvox/vagrant-spk/blob/master/vagrant-spk

This:

set -eu

cd /opt/app

if [ -f /opt/app/composer.json ] ; then
    if [ ! -f composer.phar ] ; then
        curl -sS https://getcomposer.org/installer | php
    fi
    php composer.phar install
fi

doesn't work. The system is missing the json file because it is in /opt/app/frontend/composer.json at least for my paperwork port. Not sure how it is with other apps.
In my case it must be:

set -eu

cd /opt/app

if [ -f /opt/app/frontend/composer.json ] ; then
    if [ ! -f composer.phar ] ; then
        curl -sS https://getcomposer.org/installer | php
    fi
    cd /opt/app/frontend
    php ../composer.phar install
    php ../composer.phar self-update
fi

Will send a PR, feel free to reject it if this is only about my port.

Right now in the Sandstorm docs https://github.com/sandstorm-io/sandstorm/wiki/Pure-client-apps we tell people how to put together a pure client app.

It'd be nice if there were a vagrant-spk wrapper for this. It could make a very simple platform stack.

For a 0bin package, I need to at least do:

ln -s zerobin/static static

because static assets are configured (in our nginx conf) to come from /opt/app/static.

The 0bin deployment instructions suggest that I add a few other Iines to the nginx configuration, like this:

    location /static/ {
        root  /path/to/zerobin;
        gzip  on;
        gzip_http_version 1.0;
        gzip_vary on;
        gzip_comp_level 6;
        gzip_proxied any;
        gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
        gzip_buffers 16 8k;
        # Disable gzip for certain browsers.
        gzip_disable ~@~\MSIE [1-6].(?!.*SV1)~@~];
        expires modified +90d;
    }

(source: https://0bin.readthedocs.org/en/latest/en/nginx_install.html )

I think doing that is more trouble than it's worth. If I really did want to do that, though, I'd also suggest that the nginx site config live in .sandstorm/. I wonder what @zarvox would think of that, but luckily I'm not suggesting it yet.

I'm OK being the one to own this documentation issue, so I'll self-assign.

https://www.evernote.com/shard/s59/sh/09003436-2e8b-4224-ab09-d6bea7b80024/a006b9a017e7c0aa/deep/0/

argparse wasn't a thing yet in Python 2.5, so Python 2.5 users don't have it.

I think a design goal of vagrant-spk is that it should work on all sorts of people's random computers, so it's wise to remove the argparse dependency (perhaps relying on optparse instead, or perhaps inlining it somehow).

@zarvox curious for your +1 -1 on this. I'm going to write this patch as if you've +1'd it but will be eager to hear what you think.

If VirtualBox isn't installed, vagrant will seemingly hang forever, attempting to configure Hyper-V.

I want to rename the "launcher" script into something called "start-grain", or at least with "grain" in the title.

Rationale:

• I'd like to minimize the number of unique things Sandstorm app packagers have to know, and they're going to have to know the term grain, so we might as well present them with it.
• No end user will see this, so it's not as if we are going down the path of showing this term to users of the apps themselves.

This is not necessarily a show-stopper for anything, but it strikes me as a thing to do sooner rather than later.

I'm curious for @JamborJan 's and/or @zarvox 's +1 or -1 etc.

I've found that vagrant-spk dev doesn't show the icons. This would be helpful for prototyping how the icons look in the actual shell. It does show on the "New" page, but it doesn't show anywhere else. See screenshots:

Right now it seemingly hangs forever and the wrench says it can't find main.js.

@JanJambor discussed with the paperwork team an issue where absolute URLs in Paperwork weren't being generated properly in his Sandstorm package. The problem seemed to be that $_SERVER['https'] needs to be set to on for Paperwork/Laravel to figure things out properly, but that doesn't happen with his nginx etc. setup.

(Full discussion here: paperwork/paperwork#281)

My suggestion is that we should

• 1. adjust vagrant-spk's "LEMP" (Linux ENginx MySQL PHP) configuration to handle this on behalf of frameworks, setting $_SERVER['https'] to on if the X-Forwarded-Proto is https. Having said that, I haven't dug into the sandstorm-http-bridge setup enough to find out if that's going to be feasible.
• 1. Use absolute CSS URLs in https://github.com/paulproteus/php-app-to-package-for-sandstorm app to be able to test that the handling works properly.

Relevant links:

• http://laravel-tricks.com/tricks/force-generation-of-http-https-link-if-behind-caching-server
• http://laravel-tricks.com/tricks/fix-ssl-in-laravel-4-when-server-is-behind-a-load-balancer-or-a-reverse-proxy

This is needed for Ryan Prior's packaging work on NightScout. See http://www.nightscout.info/ for more info on that.

In https://www.evernote.com/shard/s59/sh/85fb8ee1-acf2-434f-b737-34dc39304749/25eea35f0ecefb6c/deep/0/ , @jadeqwang ran into a problem where she had another Vagrant listening on port 6080. In her case, it was the Vagrant from a Sandstorm checkout.

I would love to see vagrant-spk figure out how to resolve this problem, though we might decide it's out of scope. Some things we can do:

• Ask Vagrant/VirtualBox what VMs are running; if some have Sandstorm in the name, we could provide the user a hint that that's where the conflict is.
• It could check if /opt/sandstorm exists and if so, indicate that you might need to sudo sandstorm stop.
• vagrant-spk could pick a random port number; to do this properly, it would need to also modify the sandstorm.conf within the VM. This is tricky but arguably the best answer.

Curious for people's thoughts.

@pwais just did wipe and thought he was going to remove his VM but actually removed his scripts. Apparently it's mostly OK this time, but anyway he would like more --help!

After the "vagrant-spk up" step, when surfing to http://local.sandstorm.io:6080/, if a user account doesn't exist, there is an error message along the lines of "No users exist, run sandstorm admin-token to login".

When run the command returns: "sandstorm: command not found"

Actual action to be taken is:

in top right corner, drop down "Sign In" and choose "dev account"

During tests for the Paperwork package update I realized that it is not possible to upload an grain backup to test the update process.

Desired behavior

It should be possible to run a new version of an app in vagrant-spk devmode and to upload grain backups to test the migration process.

Current behavior

When uploading a backup file the error message Restore failed: 200 App id for uploaded grain not installed: App Id: is shown. The mentioned App Id is exactly the one which is running in dev mode in the background.

Current workaround

A time consuming work around at the moment is:

• Upload or install an older version of the app via upload or market
• Upload the backup of the app
• Uninstall the app again but keep the grain
• Run the app in vagrant-spk dev mode
• Now you can use the backup grain for testing

See https://github.com/quasimotoca/mautic/pull/2 for more.

Symptoms include all sorts of confusing weirdness from bash, such as:

: invalid option name/.sandstorm/global-setup.sh: line 2: set: pipefail

I haven't yet fully confirmed it's due to \r\n vs \n but I have my suspicions.

When using vagrant-spk as described in the README the following will be the result after launching the first instance of the test app:

...** SANDSTORM SUPERVISOR: Starting up grain.
Installing MySQL system tables...
150416  8:21:48 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
OK
Filling help tables...
150416  8:21:49 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:

/usr/bin/mysqladmin -u root password 'new-password'
/usr/bin/mysqladmin -u root -h sandbox password 'new-password'

Alternatively you can run:
/usr/bin/mysql_secure_installation

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the manual for more instructions.

You can start the MySQL daemon with:
cd /usr ; /usr/bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl
cd /usr/mysql-test ; perl mysql-test-run.pl

Please report any problems at http://bugs.mysql.com/

150416  8:21:49 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.

I will fork the repo and try to make it self configurable if possible.

bash-3.1$ vagrant-spk dev
Calling 'vagrant' 'ssh' '-c' '/opt/app/.sandstorm/build.sh && cd /opt/app/.sands
torm && spk dev --pkg-def=/opt/app/.sandstorm/sandstorm-pkgdef.capnp:pkgdef' in
c:\users\nolan\Projects\Scheduler.sandstorm

WARNING: The output directory is under your source tree.

npm WARN package.json [email protected] No description
npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] No README data

[email protected] install /opt/app/.sandstorm/bundle/programs/server/node_modules/f
ibers
node ./build.js

linux-x64-v8-3.14 exists; testing
Binary is fine; exiting
npm ERR! Error: UNKNOWN, symlink '../semver/bin/semver'
npm ERR! If you need help, you may report this entire log,
npm ERR! including the npm and node versions, at:
npm ERR! http://github.com/npm/npm/issues

npm ERR! System Linux 3.16.0-4-amd64
npm ERR! command "/home/vagrant/.meteor/packages/meteor-tool/.1.1.3.4sddkj++os.l
inux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node" "/ho
me/vagrant/.meteor/packages/meteor-tool/.1.1.3.4sddkj++os.linux.x86_64+web.brows
er+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/npm" "install"
npm ERR! cwd /opt/app/.sandstorm/bundle/programs/server
npm ERR! node -v v0.10.36
npm ERR! npm -v 1.4.28
npm ERR! path ../semver/bin/semver
npm ERR! code UNKNOWN
npm ERR! errno -1
npm ERR! not ok code 0
App is now available from Sandstorm server. Ctrl+C to disconnect.
WriteResult({ "nInserted" : 1 })

I don't see a semver directory anywhere in .sandstorm/bundle.

I used to run Meteor on Windows under Vagrant, and remember having to do a bunch of trickery to work around my app running in a shared directory. I wonder if it might be better/easier for the app to build its bundle on a guest filesystem, one that won't fail because of path length limits or casing issues.

https://www.evernote.com/shard/s59/sh/58db80b4-e6b6-44e6-98f5-856b5b05e428/50ac01dc4ea0fe8f/deep/0/

The key here is that with didn't exist in Python 2.5.

Getting the following after:

vagrant-spk setupvm meteor
vagrant-spk up

default: Running: inline script

==> default: stdin: is not a tty
==> default: /opt/app/.sandstorm/global-setup.sh: line 2: set: -
: invalid option
==> default: set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]
==> default: /opt/app/.sandstorm/global-setup.sh: line 5: /host-dot-sandstorm/caches/install.sh
: Protocol error
==> default: /opt/app/.sandstorm/global-setup.sh: line 31: syntax error: unexpected end of file
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.
Calling 'vagrant' 'up' in c:\users\nolan\Projects\Scheduler.sandstorm
Traceback (most recent call last):
File "c:/users/nolan/appdata/local/bin/vagrant-spk", line 779, in
main()
File "c:/users/nolan/appdata/local/bin/vagrant-spk", line 776, in main
operation(args)
File "c:/users/nolan/appdata/local/bin/vagrant-spk", line 699, in bring_up_vm
call_vagrant_command(sandstorm_dir, "up")
File "c:/users/nolan/appdata/local/bin/vagrant-spk", line 620, in call_vagrant_command
return subprocess.check_call(command, cwd=sandstorm_dir)
File "C:\python2\Lib\subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['vagrant', 'up']' returned non-zero exit status 1

Right now, if you want to package something where you want more control over how the VM gets brought up, etc,, you have to pretend you want lemp or something, and then remove things until you discover you are happy.

By contrast, we could give a minimal thing that doesn't work and tell you to customize it. That would be a diy platform stack.

(Are we OK consistently referring to these as "platform stack"s?)

I propose that we set a policy for vagrant-spk that master always works.

Given that, there is a question about what "works" means. I think it means:

If you work through https://docs.sandstorm.io/en/latest/vagrant-spk/packaging-tutorial/ using current master of vagrant-spk, you should always be able to get to the end of the tutorial successfully, or if not, then vagrant-spk should print useful error messages telling you what to do, or it should print a useful error message saying that it can't work at all due to some detail about your system.

• I think one logical conclusion of the above is that if VirtualBox is missing, we should tell the user to install VirtualBox, and move the libvirt driver to some kind of "experimental" status, requiring an "experimental" flag passed in to vagrant-spk as a command line argument, since it doesn't work as readily out-of-the-box nor as consistently across operating systems (even Debian vs. Fedora) as the VirtualBox backend. It's OK with me if we do also tell people they can try libvirt, so long as we make it clear that it's experimental; unless it's equally well-maintained as the VirtualBox backend, I'd like to require people explicitly request the use of it.
• It would also be nice to spend some time cleaning up the output in the case of common error messages. One example is that in the case of e.g. port 6080 being already used, we should make sure useful messages.

I'm OK with owning those tasks. I want to make sure there's consensus for this before I prioritize it.

I think reliability is more important than flexibility in the case of vagrant-spk since it's going to be many Sandstorm app developers' first exposure to packaging, as well as a daily-use tool for Sandstorm package maintainers. My personal concern about this comes from spending a while of my time today running into libvirt-related problems today, and deciding I want to make the package I was trying to make more than I want to make sure vagrant-spk fully supports libvirt (so I took some time to install VirtualBox properly and was back on track).

A different but related thing is that we could commit to a release cycle for vagrant-spk, and then advise people to use releases rather than master. Having version tags is probably a good idea anyway, so that I can say "These docs were written against vagrant-spk 0.93" or something on the docs pages, though I'd like both (a) have releases and (b) keep master constantly usable.

Sorry if this comes off as negative about our libvirt support. I do think it's useful.

As an app developer, it would be nice to be able to get a shell inside a running grain to figure out why something is acting the way it's acting.

The standard recommendation is to run:

vagrant-spk ssh
cd /opt/app/.sandstorm
sudo nsenter --target $(pidof sandstorm-http-bridge) --wd --mount --net --ipc --uts --pid

This issue tracks the creation of a vagrant-spk command (e.g. vagrant-spk entergrain {{grainid}}) to achieve that.

Separately, it would be nice if doing so would not grow the sandstorm-files.list. Since (I believe) that requires cooperation from spk, I also filed sandstorm-io/sandstorm#669

Otherwise people end up commiting their VirtualBox state, which is not something that should live in version control.

It'd be nice to not have to explain-away what local.sandstorm.io is.

TODO: Explain this more; just wanted to file a bug so I don't lose the idea.

What I did

git add .sandstorm

What I expected to happen

My Sandstorm packaging files are added to git, and no system-specific state (like Vagrant state) is added to git.

What happened instead

Vagrant state was added to git.

Proposal

vagrant-spk should create a .sandstorm/.gitignore file with the contents of:

.vagrant

and place it in .sandstorm/.gitignore. Since it lives in .sandstorm/ rather than the top level directory, it will add to (rather than replace) any existing .gitignore rules.

Details

For the fun of it, here is a terminal transcript in which I was surprised.

➜  0bin git:(master) ✗ vagrant-spk init
Calling 'vagrant' 'ssh' '-c' 'spk init -p 8000 --keyring=/host-dot-sandstorm/sandstorm-keyring --output=/opt/app/.sandstorm/sandstorm-pkgdef.capnp  -- /opt/app/.sandstorm/launcher.sh' in /home/paulproteus/projects/0bin-package/0bin/.sandstorm
** WARNING: Keys are being added to:
**   /host-dot-sandstorm/sandstorm-keyring
** Please make a backup of this file and keep it safe. If you lose your keys,
** you won't be able to update your app. If someone steals your keys, they
** will be able to post updates for your app. (Use -q to quiet this warning.)
wrote: /opt/app/.sandstorm/sandstorm-pkgdef.capnp
Connection to 127.0.0.1 closed.
➜  0bin git:(master) ✗ ls
compress.sh  docs  libs  README.rst  screenshot.png  setup.py  tools  zerobin  zerobin.py
➜  0bin git:(master) ✗ nano -w .sandstorm/sandstorm-pkgdef.capnp 
➜  0bin git:(master) ✗ git add .sandstorm/
➜  0bin git:(master) ✗ ls .sandstorm 
build.sh  global-setup.sh  launcher.sh  sandstorm-pkgdef.capnp  setup.sh  stack  Vagrantfile
➜  0bin git:(master) ✗ git add .sandstorm/
➜  0bin git:(master) ✗ git commit -m 'Initial packaging'
[master d64401d] Initial packaging
 13 files changed, 316 insertions(+)
 create mode 100644 .sandstorm/.vagrant/machines/default/virtualbox/action_provision
 create mode 100644 .sandstorm/.vagrant/machines/default/virtualbox/action_set_name
 create mode 100644 .sandstorm/.vagrant/machines/default/virtualbox/id
 create mode 100644 .sandstorm/.vagrant/machines/default/virtualbox/index_uuid
 create mode 100644 .sandstorm/.vagrant/machines/default/virtualbox/private_key
 create mode 100644 .sandstorm/.vagrant/machines/default/virtualbox/synced_folders
 create mode 100644 .sandstorm/Vagrantfile
 create mode 100755 .sandstorm/build.sh
 create mode 100644 .sandstorm/global-setup.sh
 create mode 100755 .sandstorm/launcher.sh
 create mode 100644 .sandstorm/sandstorm-pkgdef.capnp
 create mode 100644 .sandstorm/setup.sh
 create mode 100644 .sandstorm/stack

I realized that we get some pretty big log files in grains. I found that when looking for some information about crashing apps. So I know that log files are useful but should be purged rather quickly.

So for example we have in data/log/nginx the access log which is getting bigger and bigger. I have only in one grain I am using on my own already 2MB (Logs from March 23rd 2015 until today).

Guessing mode:

• I guess this is something which needs to be set in the environment when packaging the app
• So I guess this needs to be done in vagrant-spk rather than sandstorm
• But I guess that I can guess wrong ...

JJ

A user cloning a repo where the developer has already run vagrant-spk setupvm <stack> will jump straight to vagrant-spk up which expects that directory to exist.

Steps to reproduce:

git clone https://github.com/paulproteus/php-app-to-package-for-sandstorm.git
cd php-app-to-package-for-sandstorm
vagrant-spk setupvm lemp
chmod 644 .sandstorm/*
vagrant-spk up  # works fine
vagrant-spk  # fails, see below

(note: I'm doing the chmod as part of the steps to reproduce not because anyone would actually do that chmod, but instead for the reason that on Windows, users are likely to fail to commit the +x bit as part of the git repo, result in packages that fail to build)

Expected behavior:

Grain starts.

Actual behavior:

➜  php-app-to-package-for-sandstorm git:(master) ✗ vagrant-spk dev
Calling 'vagrant' 'ssh' '-c' '/opt/app/.sandstorm/build.sh && cd /opt/app/.sandstorm && spk dev --pkg-def=/opt/app/.sandstorm/sandstorm-pkgdef.capnp:pkgdef' in /tmp/php-app-to-package-for-sandstorm/.sandstorm
spk dev: --pkg-def=/opt/app/.sandstorm/sandstorm-pkgdef.capnp:pkgdef: not found
Try 'spk dev --help' for more information.
Connection to 127.0.0.1 closed.
Traceback (most recent call last):
  File "/usr/local/bin/vagrant-spk", line 367, in <module>
    main()
  File "/usr/local/bin/vagrant-spk", line 364, in main
    operation(args)
  File "/usr/local/bin/vagrant-spk", line 298, in dev
    "spk dev --pkg-def=/opt/app/.sandstorm/sandstorm-pkgdef.capnp:pkgdef"
  File "/usr/local/bin/vagrant-spk", line 177, in call_vagrant_command
    return subprocess.check_call(command, cwd=sandstorm_dir)
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['vagrant', 'ssh', '-c', '/opt/app/.sandstorm/build.sh && cd /opt/app/.sandstorm && spk dev --pkg-def=/opt/app/.sandstorm/sandstorm-pkgdef.capnp:pkgdef']' returned non-zero exit status 1

That way, failure will be more clear.

I, and others, have run into an issue where VirtualBox and Vagrant plus vagrant-spk results in apps within the VM not being able to properly use DNS.

This is my public TODO item to see if there's a way to decrease the breakage there.

Steps to reproduce

• Create a lemp vagrant-spk VM

mkdir demo
echo hello > demo/index.php

• Create a grain
• Surf to http://local.sandstorm.io:6080/grain/{{grainId}}/hello

Expected behavior

nginx redirects the app iframe to /demo/ which then gets a HTTP 200.

Actual behavior

nginx redirects the app iframe to http://sessionrandomthing.local.sanstorm.io:8000/demo/ which raises a CSP exception.

Note that port 8000 is what nginx is listening on, inside the grain.

Current status

I checked that we send a valid Host: header that includes the port. We do.

I tried turning off port_in_redirect but then it redirects to port 80.

I tried turning on server_name_in_redirect but then it redirects to localhost:8000.

What I think I want to be able to do somehow is dynamically set server_name but it seems to always be a literal string, hmm.

So I don't know how to fix this, but I wanted to file it at least. Perhaps a workaround is we can swap out:

try_files $uri $uri/ =404;

with:

try_files $uri =404;

and then we'd 404 on /demo but at least we wouldn't raise this CSP error aka redirect outside the grain.

See sandstorm-io/sandstorm#794 for why.

My goal here is that we should start publishing vagrant-spk "versions", so that people can make sure to get the docs for the version they're using.

@phildini mentioned to me that he would drastically prefer if he never had to see any Cap'n Proto when packaging apps for Sandstorm.

To achieve this, we would need to at least allow the vagrant-spk user to provide the:

• appName, and
• appActionTitle

as parameters when creating the package initially, via vagrant-spk.

I have a new iteration of the paperwork app. Everything works fine with vagrant-spk dev. When running vagrant-spk pack paperwork.spk a surprisingly small file is created. Uploading and using this causes a white screen. Seems that node / gulp stuff which is installed / created during vagrant-spk dev is missing in the package. At least when looking at sandstorm-files.list it seems to be very short.

I tried to find a work around by adding lines to sandstorm-files.list but it only gets more confusing :-)

Can you please take a look and check if vagrant-spk dev creates a proper sandstorm-files.list and vagrant-spk pack creates a usable spk?

You can take my repo as written in the readme and the issue should be 100% reproducible as I completely have cleaned my working directory and started from scratch with git clone also and the result is the same. vagrant-spk dev works vagrant-spk pack not.

@tdfischer claims to have run into this. Looking into it now to see if I can reproduce it.

@jadeqwang noticed this, and I agree it's a problem.

Thanks to @alexose for noticing this during some testing.

I updated to the latest vagrant-spk and this will happen when you want to pack a spk. Seems to have a missing slash between /opt/app and .sandstorm/

Calling 'vagrant' 'ssh' '-c' 'cd /opt/app.sandstorm/ && spk pack --keyring=/host-dot-sandstorm/sandstorm-keyring --pkg-def=/opt/app/.sandstorm/sandstorm-pkgdef.capnp:pkgdef /home/vagrant/sandstorm-package.spk && mv /home/vagrant/sandstorm-package.spk /opt/app/sandstorm-package.spk' in /Volumes/2rockRAID5/Eigene/Development/paperwork/.sandstorm
bash: line 0: cd: /opt/app.sandstorm/: No such file or directory