digitalrebar / provision Goto Github PK

Docs, API bindings, clients and integrations supporting RackN Digital Rebar data center provisioning and workflow system.

Home Page: http://rebar.digital

License: Other

Makefile 0.93% Go 84.82% Shell 11.18% Python 2.79% HTML 0.14% Dockerfile 0.14%

provision's Introduction

END OF LIFE NOTICE FOR VERSION 2.x

Version 2 of Digital Rebar is no longer being maintained. All development has shifted to v3 (Digital Rebar Provision) which is being actively supported. If you need help migrating, please contact RackN for assistance.

Ref: https://github.com/digitalrebar/provision

Welcome to Digital Rebar

The Digital Rebar is a container-ready cloud & hardware provisioning platform that delivers the best of software deployment automation and orchestration without locking you into a single platform or operating system. Our mission is to embrace the heterogenous nature of cloud and data center operations.

While it's been completely rebuilt by DevOps artisans, Digital Rebar history includes years of battle-tested ops learnings by the Crowbar Project founders.

Getting Started & Documentation

Documentation is maintained in Read The Docs and sourced from our doc repository.

Help & Community

Help & Chat
- Gitter: ![Gitter](https://badges.gitter.im/Join Chat.svg)
Documentation
Live Weekly Planning and Design Meetings
Mailing List

Commercial support for Digital Rebar is available from RackN Inc.

Codebase History

Fall 2016: Digital Rebar / DigitalRebar converged into single repo + workloads.
Summer 2015: Digital Rebar / Core (and many repos) restructured into microservice containers
Fall 2013: OpenCrowbar Rearchitecture (v2) was a complete re-write of Crowbar to be composable
Spring 2011: Original Project: Crowbar (still maintained by SUSE) was a Chef wrapper layer

Dangerous and Fun Quick Start command.

We know you'll ignore this advice, but we recommend that you Read The Docs Deployment Install Guide first!

WARNING: This is only for clean systems. If you've already cloned this repo, use the run-in-system script!

From the home directory of a user with sudo capabilities,

curl -fsSL https://raw.githubusercontent.com/digitalrebar/digitalrebar/master/deploy/quickstart.sh | bash

This command will turn the current node into a DigitalRebar admin node in Host mode operating as the IP on the default gateway interface. For cloud instances, this means that it will use the private network and will only safely manage nodes in its private network. UX and API will be available through the public IP of the cloud instance assuming https is allowed through the cloud's network protections.

You may add additional arguments to bash to enable features or change the IP address that ADMIN node will use.

E.g. curl -fsSL https://raw.githubusercontent.com/digitalrebar/digitalrebar/master/deploy/quickstart.sh | bash -s -- --con-provisioner --con-dhcp --admin-ip=1.1.2.3/24

will enable the dhcp server and provisioner for the admin node. You will need to edit the admin-internal network to boot nodes properly. This would also set the admin-ip to 1.1.2.3/24 in the configuration files. This last part is needed if you are using an AWS or google instance and you want to use your admin node for things not directly in your VPC/Network.

NOTE: When enabling the provisioner, you will need about 20GB of disk space. Plan accordingly.

provision's People

Contributors

Stargazers

Watchers

provision's Issues

JSON/YAML marshalling fails on empty file - should error with file doesn't exist

When an operator specifies a non-existent JSON/YAML file for parsing, the system produces the above error. This error leads the operator to believe that content may have triggered the marshalling error. Instead, the actual file does not exist.

EXAMPLE:

ls: cannot access foo.yaml: No such file or directory
# drpcli contents create foo.yaml
Error: Error parsing layer: error unmarshaling JSON: json: cannot unmarshal string into Go value of type models.Content

EXPECTED RESULT:
Something more along the lines of an error message like:
cannot access foo.yaml: no such file or directory

`drpcli catalog create --pkg-version stable|tip` does not produce a valid catalog

The implementation of drpcli catalog create --pkg-version stable|tip in the current tip version of drpcli does not create a valid catalog that can be passed to drpcli catalog updateLocal -c <catalog_just_created>. This is because drpcli catalog updateLocal only downloads content from "real" catalog items and not alias-type entries like -stable and -tip.

The following script has logic that will add the actual version entries if stable and/or tip packages are referenced:

#!/bin/bash

# Get the latest upstream catalog file updated by RackN several times per day
curl --compressed https://repo.rackn.io 2>/dev/null > rackn-catalog.json

# $package_list is just a space-delimited list of content packs and plugins that should be included in the local repository.
# Extract content packs and plugin providers from all the version sets located in content/version_sets/*.yaml
# This will create some bogus entries but they will be ignored
package_list="drp-stable drp-tip drpcli-stable drpcli-tip $(cat content/version_sets/*.yaml | egrep '\- Name:| Version:' | tr -d '^ ' | cut -d: -f2 | sed '$!N;s/\n/-/')"
# Convert to a json list, e.g.: ["bios-stable", "bios-tip"]
package_list_json=$(jq -R -s -c 'split(" ")' < <(echo -n $package_list))

# For every item in $package_list that is stable or tip make sure that we also get its ActualVersion catalog entry because that is what updateLocal uses to download content
# For tip content we use the part of the ActualVersion up to and not including the + sign
package_list_with_actual_versions=$(echo "$package_list $(jq -r --argjson keys "$package_list_json" \
  '.sections.catalog_items[] | select(.Id as $id | $keys | index($id)) | .Meta.Name+"-"+.ActualVersion' rackn-catalog.json)" | cut -d+ -f1 | sort | uniq)
# Convert to a json list, e.g.: ["bios-v4.2.7", "bios-v4.2.8"]
package_list_with_actual_versions_json=$(jq -R -s -c 'split(" ")' < <(echo -n $package_list_with_actual_versions))

# Use the entries in $package_list_with_actual_versions_json to filter the entries in .sections.catalog_items from the upstream catalog
jq --argjson keys "$package_list_with_actual_versions_json" \
  '.meta.Name="catalog" | .meta.Description="Catalog" | .meta.DisplayName="Catalog" |
  { meta: .meta,
    sections: {
      catalog_items: .sections.catalog_items | with_entries( select( .key as $k | $keys | index($k) ) )
    }
  }' rackn-catalog.json

# Cleanup
rm -f rackn-catalog.json

Feature request: Normalize catalog view / installation across UX, CLI, and DRP

Currently the UX catalog view is determined by the catalog_url param with its default value pointing to the upstream catalog. The drpcli cli does not use catalog_url but instead relies on a -c <catalog_file> flag that defaults to the upstream catalog. Finally, when version sets are applied to endpoints the only location from which content can be downloaded is the local catalog as specified by {{ ProvisionerUrl }}/files/rebar-catalog/rackn-catalog.json . The content must exist on the filestore having been previously created via drpcli plugins runaction manager buildCatalog and drpcli catalog updateLocal.

This is a request to normalize catalog views and operations across the various DRP components:

All catalog views should be satisfied by the catalog pointed to by catalog_url
Content is pulled form the catalog pointed to by catalog_url
In drpcli the catalog could be overridden for ad hoc uses via -c <catalog_file>
Version sets download content from whatever is pointed to by catalog_url. If using a local catalog is desired then catalog_url must be explicitly set to the local catalog file. Perhaps fall back to locally installed content so that custom content packs can also be referenced without having to build an entirely local catalog. Or perhaps support two catalogs, one for upstream and one for custom content.
Local/custom catalog creation should be simplified for most-likely use cases, e.g. users probably want whatever is specified in version sets to be in their local catalog.

For prod environments users can simply pin explicit versions and not have to worry about changes even if using the upstream catalog. If they pin to stable or tip then it is expected and correct for content to auto-update whenever an upstream stable or tip is updated if the endpoint is in apply mode.

This is a work in progress as far as what all is desired, but this is a starting point for ideas on simplification of the catalog viewing/management/building process.

Stage parameter RunnerWait cannot be set to false

When creating or updating a Stage, even if you set RunnerWait: false, the returned final object has RunnerWait: true.

This is true for at least versions 4.0.3 and 4.0.13.

drpcli --endpoint parameter doesn't seem to work

As I tried to use drpcli remotely, I tried to use the --endpoint parameter to point to my remote host:

$ drpcli --endpoint https://10.1.60.1:8092 --token <obfuscated> users list
Error: CLIENT_ERROR: Get https://127.0.0.1:8092/api/v3/users: dial tcp 127.0.0.1:8092: connect: connection refused

This is with latest release drpcli v4.0.21 from https://rebar-catalog.s3-us-west-2.amazonaws.com/drp/v4.0.21.zip

Using proxy for downloads with `drpcli -D` has no effect on `uploadiso` command

Steps to reproduce

Install drp into environment where HTTP proxy should be used for internet access.
Try to download items from catalog using drpcli -D:

drpcli -D PROXY contents upload catalog:task-library-stable
drpcli -D PROXY bootenvs uploadiso sledgehammer

Expected result

Every catalog item is downloaded and installed.

Actual result

HTTP proxy specified in -D command line switch has no effect on uploadiso subcommand:

# drpcli -D http://localhost:3131 bootenvs uploadiso sledgehammer
2020/01/30 17:57:04 Unable to connect to https://s3.us-east-2.amazonaws.com/vl-hammer/sledgehammer-9b5276ac5826520829aa73c149fe672fe2363656.arm64.tar: Get https://s3.us-east-2.amazonaws.com/vl-hammer/sledgehammer-9b5276ac5826520829aa73c149fe672fe2363656.arm64.tar: dial tcp 52.219.104.218:443: connect: connection refused: Skipping

Experiment shows that it works when certain environment variables are set (http_proxy, https_proxy) but they aren't seem to be documented.

`drpcli agent` is not documented

Running drpcli agent produces help output of:

  drpcli agent [operation] [flags]

None of the operations are documented. These should be documented so an operator understand the operation arguments that are available.

Sledgehammer doc out-of-date

I was looking at the https://github.com/digitalrebar/provision/blob/v4/doc/arch/sledgehammer.rst / https://provision.readthedocs.io/en/latest/doc/arch/sledgehammer.html doc and it seems like the repository referenced, https://github.com/digitalrebar/sledgehammer , doesn't exist anymore.

Is there any chance someone could update the doc to reflect the new state of things?

drpcli "--endpoint" flag ineffective

$ drpcli version
Version: v4.4.0

$ drpcli --endpoint 'https://hello' bootenvs list
Error: Error creating Session: CLIENT_ERROR: Get https://127.0.0.1:8092/api/v3/users/rocketskates/token: dial tcp 127.0.0.1:8092: connect: connection refused

Enhance old release version handling in install.sh

Currently, the install.sh script exits with an unfriendly error if an older version is specified than is available in the Catalog, like:

EXIT MESSAGE: FATAL: Internal error, get() expects files to get

The Installer should handle this a little more gracefully if possible. Currently at line 731 - we grep through the catalog to find drp-$DRP_VERSION - which may return an empty string, this is passed to the get() call without checking.

Error messaging should be enhanced to catch this.

Profile names can not begin with a number

When a profile name begins with a number a 422 is raised by DRP. The expected behavior is that it would create the profile.

UX feature request: Show button with link to GitHub tag diffs for new plugin versions

Similar to how content packs have a preview and/or diff button for newer versions, plugins could show a button with a link to GitHub to show their code diffs based upon tags.

Install script not downloading dr-provision.zip (Github 404)

Hi!

When running the installer, for any version, the installer silently fails after Downloading file: dr-provision.zip
Adding -v to the curl command shows that:

Installing Version tip of Digital Rebar Provision
>>> Downloading file:  dr-provision.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to github.com port 443 (#0)
*   Trying 140.82.118.4...
* Connected to github.com (140.82.118.4) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=github.com,O="GitHub, Inc.",L=San Francisco,ST=California,C=US,serialNumber=5157550,incorporationState=Delaware,incorporationCountry=US,businessCategory=Private Organization
*       start date: May 08 00:00:00 2018 GMT
*       expire date: Jun 03 12:00:00 2020 GMT
*       common name: github.com
*       issuer: CN=DigiCert SHA2 Extended Validation Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
> GET /digitalrebar//provision/releases/download/tip/dr-provision.zip HTTP/1.1
> User-Agent: curl/7.29.0
> Host: github.com
> Accept: */*
>
* The requested URL returned error: 404 Not Found

Might be a GitHub issue?

`drpcli plugins runaction manager buildCatalog` does not distinguish between stable and tip

When you run drpcli plugins runaction manager buildCatalog the resulting local catalog that gets created and stored at {{ ProvisionerUrl }}/files/rebar-catalog/rackn-catlog.json will contain stable and tip entries for content that both point to the most recent (i.e. tip) actual version of the content.

To reproduce run the following:

drpcli plugins runaction manager buildCatalog
drpcli catalog updateLocal -c rackn-catalog-subset-to-download-locally.json
drpcli plugins runaction manager buildCatalog
jq '.sections.catalog_items."drp-community-content-stable"' /var/drp/drp-data/tftpboot/files/rebar-catalog/rackn-catalog.json

You will see that the stable version of drp-community-content refers to the actual tip version.

The following is the reduced catalog file used for creation as rackn-catalog-subset-to-download-locally.json:

{
  "meta": {
    "Name": "rackn-catalog",
    "Version": "v20.02.16-2",
    "Description": "RackN Catalog",
    "Source": "https://s3-us-west-2.amazonaws.com/rebar-catalog/rackn-catalog.json",
    "Documentation": "",
    "RequiredFeatures": "",
    "Prerequisites": "",
    "Color": "blue",
    "Icon": "book",
    "Author": "RackN",
    "DisplayName": "RackN Catalog",
    "License": "RackN",
    "Copyright": "RackN",
    "CodeSource": "https://s3-us-west-2.amazonaws.com/rebar-catalog/rackn-catalog.json",
    "Order": "2000",
    "Tags": "rackn,catalog,contents,plugins",
    "DocUrl": "https://s3-us-west-2.amazonaws.com/rebar-catalog/rackn-catalog.json",
    "Type": "",
    "Writable": false,
    "Overwritable": false
  },
  "sections": {
    "catalog_items": {
      "drp-community-content-stable": {
        "Validated": false,
        "Available": false,
        "Errors": [],
        "ReadOnly": false,
        "Meta": {
          "Author": "RackN and Digital Rebar Community",
          "CodeSource": "https://github.com/digitalrebar/provision-content",
          "Color": "green",
          "Copyright": "RackN 2019",
          "Description": "Core Provisioning Functionality (Required Libraries)",
          "DisplayName": "Community Core",
          "DocUrl": "https://provision.readthedocs.io/en/latest/doc/content-packages/drp-community-content.html",
          "Icon": "creative commons",
          "License": "APLv2",
          "Name": "drp-community-content",
          "Order": "100",
          "Prerequisites": "BasicStore",
          "Tags": "base,community,linux,APL,quick-start,ssh"
        },
        "Endpoint": "",
        "Id": "drp-community-content-stable",
        "Type": "catalog_items",
        "Name": "drp-community-content",
        "ActualVersion": "v4.2.8",
        "Version": "stable",
        "ContentType": "ContentPackage",
        "Source": "https://s3-us-west-2.amazonaws.com/rebar-catalog/drp-community-content/v4.2.8.json",
        "Shasum256": {
          "any/any": "d1ce505e44f3878dd5879c1e9d876e0a1de71f78b8cff9df894c4b80dd123fa0"
        },
        "NOJQSource": "drp-community-content-stable:::https://s3-us-west-2.amazonaws.com/rebar-catalog/drp-community-content/v4.2.8.json",
        "Tip": false,
        "HotFix": true
      },
      "drp-community-content-tip": {
        "Validated": false,
        "Available": false,
        "Errors": [],
        "ReadOnly": false,
        "Meta": {
          "Author": "RackN and Digital Rebar Community",
          "CodeSource": "https://github.com/digitalrebar/provision-content",
          "Color": "green",
          "Copyright": "RackN 2019",
          "Description": "Core Provisioning Functionality (Required Libraries)",
          "DisplayName": "Community Core",
          "DocUrl": "https://provision.readthedocs.io/en/latest/doc/content-packages/drp-community-content.html",
          "Icon": "creative commons",
          "License": "APLv2",
          "Name": "drp-community-content",
          "Order": "100",
          "Prerequisites": "BasicStore",
          "Tags": "base,community,linux,APL,quick-start,ssh"
        },
        "Endpoint": "",
        "Id": "drp-community-content-tip",
        "Type": "catalog_items",
        "Name": "drp-community-content",
        "ActualVersion": "v4.3.0-alpha2.71+g47bb9dc3f63a6bdd2af246bceac9f720d8985abb",
        "Version": "tip",
        "ContentType": "ContentPackage",
        "Source": "https://s3-us-west-2.amazonaws.com/rebar-catalog/drp-community-content/v4.3.0-alpha2.71%2Bg47bb9dc3f63a6bdd2af246bceac9f720d8985abb.json",
        "Shasum256": {
          "any/any": "ba0746d1237ee2f4222ce8fa79cbcedd35b463554067036e697de2d5ee646e20"
        },
        "NOJQSource": "drp-community-content-tip:::https://s3-us-west-2.amazonaws.com/rebar-catalog/drp-community-content/v4.3.0-alpha2.71%2Bg47bb9dc3f63a6bdd2af246bceac9f720d8985abb.json",
        "Tip": true,
        "HotFix": false
      },
      "drp-community-content-v4.2.8": {
        "Validated": false,
        "Available": false,
        "Errors": [],
        "ReadOnly": false,
        "Meta": {
          "Author": "RackN and Digital Rebar Community",
          "CodeSource": "https://github.com/digitalrebar/provision-content",
          "Color": "green",
          "Copyright": "RackN 2019",
          "Description": "Core Provisioning Functionality (Required Libraries)",
          "DisplayName": "Community Core",
          "DocUrl": "https://provision.readthedocs.io/en/latest/doc/content-packages/drp-community-content.html",
          "Icon": "creative commons",
          "License": "APLv2",
          "Name": "drp-community-content",
          "Order": "100",
          "Prerequisites": "BasicStore",
          "Tags": "base,community,linux,APL,quick-start,ssh"
        },
        "Endpoint": "",
        "Id": "drp-community-content-v4.2.8",
        "Type": "catalog_items",
        "Name": "drp-community-content",
        "ActualVersion": "v4.2.8",
        "Version": "v4.2.8",
        "ContentType": "ContentPackage",
        "Source": "https://s3-us-west-2.amazonaws.com/rebar-catalog/drp-community-content/v4.2.8.json",
        "Shasum256": {
          "any/any": "d1ce505e44f3878dd5879c1e9d876e0a1de71f78b8cff9df894c4b80dd123fa0"
        },
        "NOJQSource": "drp-community-content-v4.2.8:::https://s3-us-west-2.amazonaws.com/rebar-catalog/drp-community-content/v4.2.8.json",
        "Tip": false,
        "HotFix": true
      },
      "drp-community-content-v4.3.0-alpha2.71": {
        "Validated": false,
        "Available": false,
        "Errors": [],
        "ReadOnly": false,
        "Meta": {
          "Author": "RackN and Digital Rebar Community",
          "CodeSource": "https://github.com/digitalrebar/provision-content",
          "Color": "green",
          "Copyright": "RackN 2019",
          "Description": "Core Provisioning Functionality (Required Libraries)",
          "DisplayName": "Community Core",
          "DocUrl": "https://provision.readthedocs.io/en/latest/doc/content-packages/drp-community-content.html",
          "Icon": "creative commons",
          "License": "APLv2",
          "Name": "drp-community-content",
          "Order": "100",
          "Prerequisites": "BasicStore",
          "Tags": "base,community,linux,APL,quick-start,ssh"
        },
        "Endpoint": "",
        "Id": "drp-community-content-v4.3.0-alpha2.71",
        "Type": "catalog_items",
        "Name": "drp-community-content",
        "ActualVersion": "v4.3.0-alpha2.71+g47bb9dc3f63a6bdd2af246bceac9f720d8985abb",
        "Version": "v4.3.0-alpha2.71",
        "ContentType": "ContentPackage",
        "Source": "https://s3-us-west-2.amazonaws.com/rebar-catalog/drp-community-content/v4.3.0-alpha2.71%2Bg47bb9dc3f63a6bdd2af246bceac9f720d8985abb.json",
        "Shasum256": {
          "any/any": "ba0746d1237ee2f4222ce8fa79cbcedd35b463554067036e697de2d5ee646e20"
        },
        "NOJQSource": "drp-community-content-v4.3.0-alpha2.71:::https://s3-us-west-2.amazonaws.com/rebar-catalog/drp-community-content/v4.3.0-alpha2.71%2Bg47bb9dc3f63a6bdd2af246bceac9f720d8985abb.json",
        "Tip": false,
        "HotFix": true
      }
    }
  }
}

UX bug: Navigating file store does not URL encode special characters

When navigating the file store in the UX in stable or tip special characters are not properly URL encoded making it impossible to go further if the item is a directory without manually changing the contents in the URL bar.

For example:

Click on the drpcli tip folder that contains a "+" in it and it will try to go to:

{{ ProvisionerUrl }}/files/rebar-catalog/drpcli/v4.3.0-beta1.5+g3557f4782d5ab750e2a43c2243ba114b418fe90a/

It should encode it thus:

{{ ProvisionerUrl }}/files/rebar-catalog/drpcli/v4.3.0-beta1.5%2Bg3557f4782d5ab750e2a43c2243ba114b418fe90a/

upgrade argument to install script is not working as expected with custom user/pass

Hey guys!

I've been having some difficulty using the upgrade argument to the install script, alongside a custom user/password combination.

To reproduce:

Perform a fresh install with /tmp/provision-4.3.2/tools/install.sh --version=v4.3.2 --systemd install --drp-user=myuser --drp-password=mypass --remove-rocketskates

Output:

<snip>
    "Type": "dynamic",
    "Version": "v4.3.1",
    "Writable": false
  }
}
{
  "Available": true,
  "Bundle": "",
  "Description": "",
  "Endpoint": "",
  "Errors": [],
  "Meta": {},
  "Name": "myuser",
  "ReadOnly": false,
  "Roles": [
    "superuser"
  ],
  "Secret": "XO-Y7mS_xOMwuGTa",
  "Validated": true
}
{
  "Available": true,
  "Bundle": "",
  "Description": "",
  "Endpoint": "",
  "Errors": [],
  "Meta": {},
  "Name": "myuser",
  "ReadOnly": false,
  "Roles": [
    "superuser"
  ],
  "Secret": "jZ0VAZhHWhvWn35X",
  "Validated": true
}
Deleted user rocketskates

# Once dr-provision is started, setup a base discovery configuration
  drpcli bootenvs uploadiso sledgehammer
  drpcli prefs set defaultWorkflow discover-base unknownBootEnv discovery defaultBootEnv sledgehammer defaultStage discover

# Optionally, locally cache the isos for common community operating systems
  drpcli bootenvs uploadiso ubuntu-18.04-install
  drpcli bootenvs uploadiso centos-7-install

RS_KEY=myuser:mypass drpcli info get now works, but, even though I've removed the rocketskates user, RS_KEY=rocketskates:r0cketsk8ts drpcli info get will also work.

Further, if I then stop dr-provision, and run /tmp/provision-4.3.2/tools/install.sh --version=v4.3.2 --systemd upgrade --drp-user=myuser --drp-password=mypass, DRP successfully starts, but the install script errors with:

######### You can start the DigitalRebar Provision service with:
 systemctl daemon-reload &&  systemctl start dr-provision
######### You can enable the DigitalRebar Provision service with:
 systemctl daemon-reload &&  systemctl enable dr-provision
SAVING '/usr/local/bin/drpcli' to backup file (/usr/local/bin/drpcli.drp-installer.backup)
Install script saved to '/usr/local/bin/drp-install.sh'
(run ' /usr/local/bin/drp-install.sh remove' to uninstall DRP - must be root)
RS_KEY is myuser:mypass (this line was debugging I inserted - @funkypenguin)
DRP is not up yet, waiting (0) ...
DRP is not up yet, waiting (1) ...
DRP is not up yet, waiting (2) ...
DRP is not up yet, waiting (3) ...
DRP is not up yet, waiting (4) ...
DRP is not up yet, waiting (5) ...
DRP is not up yet, waiting (6) ...
DRP is not up yet, waiting (7) ...
DRP is not up yet, waiting (8) ...
DRP is not up yet, waiting (9) ...
DRP is not up yet, waiting (10) ...
EXIT MESSAGE: FATAL: DRP Failed to start
(reverse-i-search)`remove': /tmp/provision-4.3.2/tools/install.sh --version=v4.3.2 --systemd upgrade --drp-user=myuser --drp-password=mypass --^Cmove-rocketskates
root@banana:~#

And journalctl -u dr-provision shows me:

Jun 23 11:52:46 banana dr-provision[30821]: [108:22]Failed Authenticated (no user) user myuser from 127.0.0.1

For now I'm falling back to using the install command, rather than upgrade, since that still seems to work.

Cheers!
D

drpcli Support a password prompt

Feature request: Please give an option to supply password via stdin for drpcli. Using command line or environment variables will result in passwords being stored in bash history. Thanks!

Plugin install does not handle arm64 binaries

When installing standard plugins (in my case, IPMI), it appears the wrong architecture binary is installed. This is running on an ARM64 host:

Import plugin failed ipmi: define failed: Skipping /opt/drp/drp-data/plugins/.ipmi.part because fork/exec /opt/drp/drp-data/plugins/.ipmi.part: exec format error:

old bsdtar on Mac causes isos to not explode properly - update to 3.3+ bsdtar (homebrew)

When I boot my nodes, they are failing to find the netboot files from tftpboot.

For both centos-7-install:

And ubuntu-18:

I ran the various commands in the docs to drpcli bootenvs uploadiso ubuntu-18.04-install but all I see are some almost empty install directories The documentation says I should see the ISOs exploded into tftpboot directory:

The uploadiso command will fetch the ISO image as specified in the BootEnv JSON spec, download it, and then “explode” it in to the drp-data/tftpboot/ directory for installation use.

https://provision.readthedocs.io/en/latest/doc/quickstart.html#install-boot-environments-bootenvs

$ sudo tree drp-data/tftpboot/ubuntu-18.04
drp-data/tftpboot/ubuntu-18.04
├── arm64
│   └── install
└── install
$ sudo tree drp-data/tftpboot/centos-7
drp-data/tftpboot/centos-7
├── arm64
│   └── install
└── install

The files that are created into these directories are:

➜  drp-home sudo ls -alR drp-data/tftpboot/centos-7
total 0
drwx------   4 root  staff   128 28 Nov 10:51 .
drwx------  33 root  staff  1056 28 Nov 10:45 ..
drwx------   3 root  staff    96 28 Nov 10:45 arm64
drwx------   3 root  staff    96 28 Nov 10:51 install

drp-data/tftpboot/centos-7/arm64:
total 0
drwx------  3 root  staff   96 28 Nov 10:45 .
drwx------  4 root  staff  128 28 Nov 10:51 ..
drwx------  3 root  staff   96 28 Nov 10:45 install

drp-data/tftpboot/centos-7/arm64/install:
total 8
drwx------  3 root  staff  96 28 Nov 10:45 .
drwx------  3 root  staff  96 28 Nov 10:45 ..
-rw-------  1 root  staff  64 28 Nov 10:45 .centos-7.rebar_canary

drp-data/tftpboot/centos-7/install:
total 8
drwx------  3 root  staff   96 28 Nov 10:51 .
drwx------  4 root  staff  128 28 Nov 10:51 ..
-rw-------  1 root  staff   64 28 Nov 10:51 .centos-7.rebar_canary

$ sudo cat drp-data/tftpboot/centos-7/install/.centos-7.rebar_canary
9a2c47d97b9975452f7d582264e9fc16d108ed8252ac6816239a3b58cef5c53d

I don't see anything in drp.log about an error during drpcli bootenvs uploadiso ubuntu-18.04-install.

Any protips to debug this error?

Unlocking a previously API-locked machine causes inventory/ param type errors

When I unlock a machine I receive param type errors:

Key 'inventory/NICs': invalid val '2': : (root): Invalid type. Expected: integer, given: string
Error:
Key 'inventory/CPUCores': invalid val '10': : (root): Invalid type. Expected: integer, given: string
Error:
Key 'inventory/CPUs': invalid val '40': : (root): Invalid type. Expected: integer, given: string
Error:
Key 'inventory/DIMMs': invalid val '12': : (root): Invalid type. Expected: integer, given: string
Error:
Key 'inventory/RAM': invalid val '192051': : (root): Invalid type. Expected: integer, given: string

join-up.sh unavailable when unknown bootenv differs from discovery-base

Steps to reproduce

drpcli prefs set unknownBootEnv ignore (reasons below)
Try to add existing machine into drp as suggested in https://provision.readthedocs.io/en/latest/doc/content-packages/drp-community-content.html#discovery
curl -fsSL [internal ip]:8091/machines/join-up.sh

Expected result

join-up.sh is fetched.

Actual result

Error 404 which is confusing. Default bootenv requirement for that is unclear for user and doesn't mentioned anywhere in the docs.

Negative filtering on machine parameters removes all parameters

When using ?params=Re(sampleparam-) to filter a given parameter, digital rebar removes all parameters and returns an empty array. This can be worked around by using Re(anything-) first and then appending back desired parameters to the query string.

ExtraClaims in uploaded bundle are not preserved

I'm working on a bundle where a task needs an ExtraClaims.

When uploading the bundle, no errors are declared, but listing the tasks reveals the task without its ExtraClaims statement.

I'm running v4.5.4.

I think this might be specific to updating tasks, or something related to bundles, as I do not have a problem creating a new task with claims, and editing the claims thereafter.

Example Objects

Given this task

---
Description: This task should have some ExtraClaims
ExtraClaims:
  - scope: "*"
    action: "*"
    specific: "*"
Name: new-task-with-claims
Meta:
  color: black
  icon: tasks
  title: hepderp
  feature-flags: sane-exit-codes

A claimless task is returned:

drpcli tasks show new-task-with-claims

  {
  "Available": true,
  "Bundle": "ax-drp-config",
  "Description": "This task should have some ExtraClaims",
  "Documentation": "",
  "Endpoint": "",
  "Errors": [],
  "Meta": {
    "color": "black",
    "feature-flags": "sane-exit-codes",
    "icon": "tasks",
    "title": "hepderp"
  },
  "Name": "new-task-with-claims",
  "OptionalParams": [],
  "Prerequisites": [],
  "ReadOnly": true,
  "RequiredParams": [],
  "Templates": [],
  "Validated": true
}

UEFI PXE Booting Issues When using Proxy Mode on DRP subnet

I have the following subnet:

{
  "ActiveEnd": "192.168.1.254",
  "ActiveLeaseTime": 600,
  "ActiveStart": "192.168.1.6",
  "Available": true,
  "Bundle": "",
  "Description": "",
  "Documentation": "",
  "Enabled": true,
  "Endpoint": "",
  "Errors": [],
  "Meta": {
    "color": "black",
    "icon": "cloud",
    "title": "User added"
  },
  "Name": "ens160",
  "NextServer": "",
  "OnlyReservations": false,
  "Options": [
    {
      "Code": 1,
      "Value": "255.255.255.0"
    },
    {
      "Code": 3,
      "Value": "192.168.1.1"
    },
    {
      "Code": 6,
      "Value": "192.168.1.50"
    },
    {
      "Code": 15,
      "Value": "mrice.internal"
    },
    {
      "Code": 28,
      "Value": "192.168.1.255"
    },
    {
      "Code": 42,
      "Value": "192.168.1.50"
    }
  ],
  "Pickers": [
    "hint",
    "nextFree",
    "mostExpired"
  ],
  "Proxy": true,
  "ReadOnly": false,
  "ReservedLeaseTime": 7200,
  "Strategy": "MAC",
  "Subnet": "192.168.1.93/24",
  "Unmanaged": false,
  "Validated": true
}

When the machine I want to provision is running as a VM on ESXi 6.x or newer when the machine loads its PXE stuff it goes straight into grub. When I am on a subnet where I do not have to use proxy mode then it works as expected.

Recursive entity lookup

While working with the cli I was wondering if it were possible to render all entities recursively in one output.

For example:

If I lookup a workflow I see that it includes stages. Currently I would have to make another call to lookup to see what those stages do. If I want to see what those tasks in those stages do I would have to do another lookup and so on. It would be convenient if I could get an output when looking up any entity A to also get the nested entities B rendered as well and recursively also get the entities of C which B has included and so on.

Say I lookup a workflow, then I would get an output with all the stages as if I would do a lookup for them manually and from those stages their tasks and from those tasks their templates all rendered inline with their contents in one big output.

Provision only binding to ipv6

While installing and trying to access the UI I'm can not. Maybe I'm missing something simple but tried a few things and quickly lost out. Following https://provision.readthedocs.io/en/latest/doc/install.html#

Looks like it's trying to bind to ipv6. I have google fiber so I have ipv6 available but would rather run on ipv4. Am I going to have to disable ipv6 across the server?

OS: ClearOS 7 (CentOS 7 derivative)

cat /etc/redhat-release 
ClearOS release 7.7.1 (Final)

ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp66s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:1b:78:33:c4:98 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.12/16 brd 192.168.255.255 scope global dynamic enp66s0
       valid_lft 68475sec preferred_lft 68475sec
    inet6 <redacted>/64 scope global mngtmpaddr dynamic 
       valid_lft 3599sec preferred_lft 3599sec
    inet6 fe80::21b:78ff:fe33:c498/64 scope link 
       valid_lft forever preferred_lft forever
3: enp68s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:1b:78:33:c4:b0 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/23 brd 10.0.1.255 scope global enp68s0
       valid_lft forever preferred_lft forever
    inet6 fe80::21b:78ff:fe33:c4b0/64 scope link 
       valid_lft forever preferred_lft forever

lsof -Pp 2152
COMMAND    PID USER   FD      TYPE             DEVICE SIZE/OFF      NODE NAME
dr-provis 2152 root  cwd       DIR              253,0      255        64 /
dr-provis 2152 root  rtd       DIR              253,0      255        64 /
dr-provis 2152 root  txt       REG              253,0 61298528 805677470 /usr/local/bin/dr-provision
dr-provis 2152 root    0r      CHR                1,3      0t0      2052 /dev/null
dr-provis 2152 root    1u     unix 0xffff8d7cd5231c00      0t0   1431959 socket
dr-provis 2152 root    2u     unix 0xffff8d7cd5231c00      0t0   1431959 socket
dr-provis 2152 root    3u     IPv6            1433859      0t0       TCP *:8080 (LISTEN)
dr-provis 2152 root    4u  a_inode               0,10        0      7407 [eventpoll]
dr-provis 2152 root    5u     IPv6            1429132      0t0       UDP *:69 
dr-provis 2152 root    6u     IPv6            1429133      0t0       TCP *:8091 (LISTEN)
dr-provis 2152 root    7u      raw                         0t0   1434649 00000000:0001->00000000:0000 st=07
dr-provis 2152 root    8u     IPv4            1434650      0t0       UDP *:67 
dr-provis 2152 root    9u     IPv4            1434651      0t0       UDP *:4011 
dr-provis 2152 root   10u  a_inode               0,10        0      7407 [eventpoll]
dr-provis 2152 root   11r  a_inode               0,10        0      7407 inotify
dr-provis 2152 root   12r     FIFO                0,9      0t0   1429134 pipe
dr-provis 2152 root   13w     FIFO                0,9      0t0   1429134 pipe
dr-provis 2152 root   14u     IPv6            1433860      0t0       TCP *:8092 (LISTEN)

Pruned netstat

netstat -tlpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name             
tcp6       0      0 :::8080                 :::*                    LISTEN      2152/dr-provision      
tcp6       0      0 :::8091                 :::*                    LISTEN      2152/dr-provision   
tcp6       0      0 :::8092                 :::*                    LISTEN      2152/dr-provision

DRP 4.2.0 crash report

dr-provision2020/01/26 21:53:08.507977 Version: v4.2.0

I have had drp crash a few time over the past week or so while doing some non-standard mucking around with machine runners. I cannot say exactly what causes the behavior as I do not have a deterministic recreation, but I have been stopping and starting drpcli machines processjobs a lot lately, running both on host and within docker containers. I am also doing some forced job manipulation (e.g. forcing the current job status to be set to "finished" via drpcli within the task itself) so maybe that could have something to do with it.

Crash logs:

608 dr-provision2020/01/25 20:39:02.844858 [5905:290]:manager [error]: /home/travis/gopath/src/github.com/rackn/provision-plugins/cmds/manager/endpoint-actions.go:462
609 [5905:290]0e:01:00:10:00:02: Queuing Update (endpoint-actions.go:678 - Event (contents.update)):·
610 dr-provision2020/01/25 20:42:29.021363 [46500:291]:frontend [audit]: /home/travis/gopath/src/github.com/rackn/provision-server/frontend/frontend.go:383
611 [46500:291]Authenticated user rocketskates from 172.17.0.1
612 dr-provision2020/01/25 20:42:39.473133 [5972:292]:manager [error]: /home/travis/gopath/src/github.com/rackn/provision-plugins/cmds/manager/endpoint-actions.go:462
613 [5972:292]0e:01:00:10:00:02: Queuing Update (endpoint-actions.go:678 - Event (contents.update)):·
614 dr-provision2020/01/25 20:55:19.443512 [46915:293]:frontend [audit]: /home/travis/gopath/src/github.com/rackn/provision-server/frontend/frontend.go:383
615 [46915:293]Authenticated user rocketskates from 172.17.0.1
616 dr-provision2020/01/25 20:55:29.545223 [6021:294]:manager [error]: /home/travis/gopath/src/github.com/rackn/provision-plugins/cmds/manager/endpoint-actions.go:462
617 [6021:294]0e:01:00:10:00:02: Queuing Update (endpoint-actions.go:678 - Event (contents.update)):·
618 2020/01/25 20:56:11 unblockRunnable did not find anything to unblock!
619 2020/01/25 20:56:11 Running: 0, unblocked: 0, total: 2
620 2020/01/25 20:56:11 Dumping transaction locks:
621 2020/01/25 20:56:11 tx 45087: abort: false, wantWrite: true
622 2020/01/25 20:56:11 owned: map[jobs:0b3350a8-7fa6-4312-9715-50436e214db8:true jobs:5b91efe4-6530-4f0b-8119-d96626763c9c:true machines:251b3805-4598-4f7c-9767-54134f4aa505:fal ↪\se stages:discover:false tasks:enforce-sledgehammer:false]
623 2020/01/25 20:56:11 wanted: [machines:251b3805-4598-4f7c-9767-54134f4aa505]
624 2020/01/25 20:56:11 tx 45089: abort: false, wantWrite: false
625 2020/01/25 20:56:11 owned: map[machines:251b3805-4598-4f7c-9767-54134f4aa505:false]
626 2020/01/25 20:56:11 wanted: [jobs:0b3350a8-7fa6-4312-9715-50436e214db8]
627 panic: unblockRunnable did not unblock anything! [recovered]
628 │▸▸▸panic: unblockRunnable did not unblock anything!
629
630 goroutine 367978 [running]:
631 github.com/rackn/provision-server/v4/backend.(*RequestTracker).do.func1(0xc024f259a0)
632 │▸▸▸/home/travis/gopath/src/github.com/rackn/provision-server/backend/requestTracker.go:394 +0x7b
633 panic(0xec2ee0, 0x11f7680)
634 │▸▸▸/home/travis/.gimme/versions/go1.13.linux.amd64/src/runtime/panic.go:679 +0x1b2
635 github.com/rackn/provision-server/v4/backend.(*txInterlockTracker).unblockRunnable(0x4c508e0)
636 │▸▸▸/home/travis/gopath/src/github.com/rackn/provision-server/backend/tx.go:200 +0x64b
637 github.com/rackn/provision-server/v4/backend.(*txInterlockTracker).release(0x4c508e0, 0xb025)
638 │▸▸▸/home/travis/gopath/src/github.com/rackn/provision-server/backend/tx.go:214 +0x205
639 github.com/rackn/provision-server/v4/backend.(*RequestTracker).unlocker(0xc00b396840, 0xbf8348bac65d8872, 0xa083f2f322dd, 0x4c63d20, 0xc024f25970)
640 │▸▸▸/home/travis/gopath/src/github.com/rackn/provision-server/backend/requestTracker.go:122 +0x576
641 github.com/rackn/provision-server/v4/backend.(*RequestTracker).do(0xc00b396840, 0xc0074dea00, 0x44a300)
642 │▸▸▸/home/travis/gopath/src/github.com/rackn/provision-server/backend/requestTracker.go:419 +0x2e8
643 github.com/rackn/provision-server/v4/backend.(*RequestTracker).Do(0xc00b396840, 0xc024f25a00)
644 │▸▸▸/home/travis/gopath/src/github.com/rackn/provision-server/backend/requestTracker.go:428 +0x54
645 github.com/rackn/provision-server/v4/backend.FindLease(0xc00b396840, 0x107b213, 0x3, 0xc00f2947e0, 0x11, 0xc00311050c, 0x4, 0x11a, 0xc015162b20, 0x1, ...)
646 │▸▸▸/home/travis/gopath/src/github.com/rackn/provision-server/backend/dhcpUtils.go:275 +0x1c6
647 github.com/rackn/provision-server/v4/midlayer.(*DhcpRequest).ServeDHCP(0xc00ecb6780, 0xc0029457d0, 0x16)
648 │▸▸▸/home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:581 +0x1fe0
649 github.com/rackn/provision-server/v4/midlayer.(*DhcpRequest).Process(0xc00ecb6780, 0xc00c61e370, 0xc002952fa8, 0x6fc6eb, 0x4c83238)
650 │▸▸▸/home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:791 +0x8ec
651 github.com/rackn/provision-server/v4/midlayer.(*DhcpRequest).Run(0xc00ecb6780, 0x126)
652 │▸▸▸/home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:816 +0x40
653 created by github.com/rackn/provision-server/v4/midlayer.(*DhcpHandler).Serve
654 │▸▸▸/home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:888 +0x379
NORMAL ~/drp/drp.log unix | utf-8 | log 96% 631:5

UX can't add/edit inline task templates due to incorrect PATCH

It is very difficult to edit content in inline task templates in the UX. It's hard to say the exact process that determines whether or not inline task templates can be edited, but I find that generally trying to edit or add content does not work. I must first add a number of dummy inline template tasks with edit, then conduct some deletions before the primary.

This seems to be an error with the PATCH command that is submitted. When I edit content the PATCH operation is usually a remove when it should probably be a POST for brand new content.

DHCP sometimes handsout expired leases.

Hi,

I´d like to start to say great software, but with software, there are sometimes issues.

I´d like to find a way around this issue.
My problem Is that sometimes my devices do not get an IP from drp.
Mainly this is when connect client pcs and mobiles to the network.
Any Ideas how to troubleshoot?

Br Simon Vallebro

Here are some snippets from the log:

Aug 22 09:54:57 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:54:57.605519 [87378:79822]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/backend/dhcpUtils.go:369
Aug 22 09:54:57 rebar-provision dr-provision[942]: [87378:79822]Subnet ens18: Handing out existing lease for 10.30.0.85 to MAC:1e:7d:f3:8a:8a:22
Aug 22 09:54:57 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:54:57.607773 [87378:79823]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:708
Aug 22 09:54:57 rebar-provision dr-provision[942]: [87378:79823]xid 0xfbd2c7a: Discovery handing out: 10.30.0.85 to 1e:7d:f3:8a:8a:22 via 10.30.0.7
Aug 22 09:54:58 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:54:58.644916 [87379:79824]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/backend/dhcpUtils.go:369
Aug 22 09:54:58 rebar-provision dr-provision[942]: [87379:79824]Subnet ens18: Handing out existing lease for 10.30.0.85 to MAC:1e:7d:f3:8a:8a:22
Aug 22 09:54:58 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:54:58.646920 [87379:79825]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:708
Aug 22 09:54:58 rebar-provision dr-provision[942]: [87379:79825]xid 0xfbd2c7a: Discovery handing out: 10.30.0.85 to 1e:7d:f3:8a:8a:22 via 10.30.0.7
Aug 22 09:55:00 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:55:00.601900 [87380:79826]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/backend/dhcpUtils.go:369
Aug 22 09:55:00 rebar-provision dr-provision[942]: [87380:79826]Subnet ens18: Handing out existing lease for 10.30.0.85 to MAC:1e:7d:f3:8a:8a:22
Aug 22 09:55:00 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:55:00.610138 [87380:79827]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:708
Aug 22 09:55:00 rebar-provision dr-provision[942]: [87380:79827]xid 0xfbd2c7a: Discovery handing out: 10.30.0.85 to 1e:7d:f3:8a:8a:22 via 10.30.0.7
Aug 22 09:55:04 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:55:04.619140 [87381:79828]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/backend/dhcpUtils.go:369
Aug 22 09:55:04 rebar-provision dr-provision[942]: [87381:79828]Subnet ens18: Handing out existing lease for 10.30.0.85 to MAC:1e:7d:f3:8a:8a:22
Aug 22 09:55:04 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:55:04.622147 [87381:79829]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:708
Aug 22 09:55:04 rebar-provision dr-provision[942]: [87381:79829]xid 0xfbd2c7a: Discovery handing out: 10.30.0.85 to 1e:7d:f3:8a:8a:22 via 10.30.0.7
Aug 22 09:55:13 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:55:13.184330 [87382:79830]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/backend/dhcpUtils.go:369
Aug 22 09:55:13 rebar-provision dr-provision[942]: [87382:79830]Subnet ens18: Handing out existing lease for 10.30.0.85 to MAC:1e:7d:f3:8a:8a:22
Aug 22 09:55:13 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:55:13.195856 [87382:79831]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:708
Aug 22 09:55:13 rebar-provision dr-provision[942]: [87382:79831]xid 0xfbd2c7a: Discovery handing out: 10.30.0.85 to 1e:7d:f3:8a:8a:22 via 10.30.0.7
Aug 22 09:55:18 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:55:18.969105 [87383:79832]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/backend/dhcpUtils.go:369
Aug 22 09:55:18 rebar-provision dr-provision[942]: [87383:79832]Subnet ens18: Handing out existing lease for 10.30.0.85 to MAC:1e:7d:f3:8a:8a:22
Aug 22 09:55:18 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:55:18.971494 [87383:79833]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:708
Aug 22 09:55:18 rebar-provision dr-provision[942]: [87383:79833]xid 0xb4f59adc: Discovery handing out: 10.30.0.85 to 1e:7d:f3:8a:8a:22 via 10.30.0.7
Aug 22 09:55:20 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:55:20.003084 [87384:79834]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/backend/dhcpUtils.go:369
Aug 22 09:55:20 rebar-provision dr-provision[942]: [87384:79834]Subnet ens18: Handing out existing lease for 10.30.0.85 to MAC:1e:7d:f3:8a:8a:22
Aug 22 09:55:20 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:55:20.005281 [87384:79835]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:708
Aug 22 09:55:20 rebar-provision dr-provision[942]: [87384:79835]xid 0xb4f59adc: Discovery handing out: 10.30.0.85 to 1e:7d:f3:8a:8a:22 via 10.30.0.7
Aug 22 09:55:22 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:55:22.201623 [87385:79836]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/backend/dhcpUtils.go:369
Aug 22 09:55:22 rebar-provision dr-provision[942]: [87385:79836]Subnet ens18: Handing out existing lease for 10.30.0.85 to MAC:1e:7d:f3:8a:8a:22
Aug 22 09:55:22 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:55:22.204485 [87385:79837]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:708
Aug 22 09:55:22 rebar-provision dr-provision[942]: [87385:79837]xid 0xb4f59adc: Discovery handing out: 10.30.0.85 to 1e:7d:f3:8a:8a:22 via 10.30.0.7
Aug 22 09:55:26 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:55:26.316883 [87386:79838]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/backend/dhcpUtils.go:369
Aug 22 09:55:26 rebar-provision dr-provision[942]: [87386:79838]Subnet ens18: Handing out existing lease for 10.30.0.85 to MAC:1e:7d:f3:8a:8a:22
Aug 22 09:55:26 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:55:26.320739 [87386:79839]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:708
Aug 22 09:55:26 rebar-provision dr-provision[942]: [87386:79839]xid 0xb4f59adc: Discovery handing out: 10.30.0.85 to 1e:7d:f3:8a:8a:22 via 10.30.0.7
Aug 22 09:55:35 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:55:35.101556 [87387:79840]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/backend/dhcpUtils.go:369
Aug 22 09:55:35 rebar-provision dr-provision[942]: [87387:79840]Subnet ens18: Handing out existing lease for 10.30.0.85 to MAC:1e:7d:f3:8a:8a:22
Aug 22 09:55:35 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:55:35.104492 [87387:79841]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:708
Aug 22 09:55:35 rebar-provision dr-provision[942]: [87387:79841]xid 0xb4f59adc: Discovery handing out: 10.30.0.85 to 1e:7d:f3:8a:8a:22 via 10.30.0.7
Aug 22 09:56:53 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:56:53.399234 [87388:79842]cacher:frontend [ info]: /home/travis/gopath/src/github.co

Then the client gives up.

Aug 22 09:58:14 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:58:14.605492 [87467:79845]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/backend/dhcpUtils.go:367
Aug 22 09:58:14 rebar-provision dr-provision[942]: [87467:79845]Subnet ens18: Creating new lease for 10.30.0.85 from MAC:1e:7d:f3:8a:8a:22
Aug 22 09:58:18 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:58:18.346061 [87467:79846]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:708
Aug 22 09:58:18 rebar-provision dr-provision[942]: [87467:79846]xid 0xf11d770: Discovery handing out: 10.30.0.85 to 1e:7d:f3:8a:8a:22 via 10.30.0.7
Aug 22 09:58:21 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:58:21.609931 [87470:79847]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/backend/dhcpUtils.go:369
Aug 22 09:58:21 rebar-provision dr-provision[942]: [87470:79847]Subnet ens18: Handing out existing lease for 10.30.0.85 to MAC:1e:7d:f3:8a:8a:22
Aug 22 09:58:21 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:58:21.611977 [87470:79848]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:708
Aug 22 09:58:21 rebar-provision dr-provision[942]: [87470:79848]xid 0xf11d770: Discovery handing out: 10.30.0.85 to 1e:7d:f3:8a:8a:22 via 10.30.0.7
Aug 22 09:58:21 rebar-provision dr-provision[942]: dr-provision2021/08/22 07:58:21.795202 [87471:79849]dhcp:dhcp [ info]: /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:654
Aug 22 09:58:21 rebar-provision dr-provision[942]: [87471:79849]xid 0xf11d770: Request handing out: 10.30.0.85 to 1e:7d:f3:8a:8a:22 via 10.30.0.7

I'm using DRP in my home network for experimentation and it is the default DHCP server on the network.

Drp is running on address 10.30.0.7

Machine status icon is ! if new content changes workflow stage names

Not sure if this is UX or drp.

If a machine is set to a workflow and the workflow content changes (via content pack upload) with different stage names, then the machine icon will be set to "!" presumably because it thinks it can no longer find a given stage name in the workflow. However, the workflow has been updated and does have the correct stage names, so nothing is really in error.

UX bug: Content packs have a blank entry in the Bundle column

In the default UX view items underneath PROVISION and CONTROL, the Bundle column is blank for content packs. It shows the Bundle if the item in question comes from a Plugin but not regular Content Packs. This seems to be the case for stages, tasks, templates, params, profiles, and bootenvs.

DHCP OnlyReservations flag not respected

When setting Subnets.OnlyReservations: true, expected behavior would be to ignore all DHCP requests unless a Reservation was defined for the MAC. However, the DHCP server is issuing a lease for all requests when this flag is set.

Seen in v4.2.4 and tip

How to remove workflow use drpcli?

drpcli machines stage 1650e1e1-a7ad-46ab-9635-53565de7ddd3 complete

Error: ValidationError: machines/1650e1e1-a7ad-46ab-9635-53565de7ddd3
1650e1e1-a7ad-46ab-9635-53565de7ddd3: Changing machine bootenv not allowed; remove workflow first
1650e1e1-a7ad-46ab-9635-53565de7ddd3: Changing machine stage not allowed; remove workflow first

UX does not show task Prerequisites or allow them to be added

Running

drpcli tasks list | jq -r '.[] | select (.Prerequisites | length > 0) | {Name,Prerequisites}'

shows a number of tasks that have Prerequisites. They are not visible from the tasks view in the UX. You cannot add them in edit mode either.

UX suggestion: Enable forward path (next job) in Job View for the clicked-on Job

It would be nice to have a NEXT JOB link similar to the PREVIOUS JOB link in the Job view. I find that when a job fails I will often fix content, re-upload, then click on the green "Set Runnable" icon to-rerun this job. I then have to navigate back to the Jobs list, refresh, then go into the new job view. Instead of this flow it would be nice to just click "Set Runnable" and then have a NEXT JOB link pop up that can take me to the now-running job view.

UEFI boot not working unless the Rebar VM subnet is defined

UEFI boot not working unless the Rebar subnet is defined

.InstallRepos does not return exactly one repo with the installSource bit set

The docs (https://provision.readthedocs.io/en/latest/doc/arch/data.html#template) state that:

.InstallRepos - Returns exactly one Repo from the list chosen by MachineRepos that has the installSource bit set, and at most one Repo from the MachineRepos that has the securitySource bit set.

.MachineRepos is determined by .Repos which itself is determine by the package-repositories param.

.InstallRepos does not honor any installSource bits set in package-repositories. It does pick at most one repo with securitySource bit set.

unzip archive error (dr-provision.zip)

System info:

$ uname -a
Linux drp.isam.corp 4.18.0-193.14.2.el8_2.x86_64 #1 SMP Sun Jul 26 03:54:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release
CentOS Linux release 8.2.2004 (Core)
$ unzip -v
UnZip 6.00 of 20 April 2009, by Info-ZIP.  Maintained by C. Spieler.  Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

Compiled with gcc 8.2.1 20180905 (Red Hat 8.2.1-3) for Unix (Linux ELF) on Apr  7 2020.

UnZip special compilation options:
        COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
        SET_DIR_ATTRIB
        SYMLINKS (symbolic links supported, if RTL and file system permit)
        TIMESTAMP
        UNIXBACKUP
        USE_EF_UT_TIME
        USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)
        USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)
        UNICODE_SUPPORT [wide-chars, char coding: UTF-8] (handle UTF-8 paths)
        MBCS-support (multibyte character support, MB_CUR_MAX = 6)
        LARGE_FILE_SUPPORT (large files over 2 GiB supported)
        ZIP64_SUPPORT (archives using Zip64 for large files supported)
        USE_BZIP2 (PKZIP 4.6+, using bzip2 lib version 1.0.6, 6-Sept-2010)
        VMS_TEXT_CONV
        [decryption, version 2.11 of 05 Jan 2007]

UnZip and ZipInfo environment options:
           UNZIP:  [none]
        UNZIPOPT:  [none]
         ZIPINFO:  [none]
      ZIPINFOOPT:  [none]

Attempt to unzip:

$ export DRPVERSION="v4.4.1"
$ curl -fsSL https://rebar-catalog.s3-us-west-2.amazonaws.com/drp/$DRPVERSION.zip -o dr-provision.zip
$ curl -fsSL https://rebar-catalog.s3-us-west-2.amazonaws.com/drp/$DRPVERSION.sha256 -o dr-provision.sha256
$ sha256sum -c dr-provision.sha256
dr-provision.zip: OK
$ unzip dr-provision.zip
Archive:  dr-provision.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of dr-provision or
        dr-provision.zip, and cannot find dr-provision.ZIP, period.

Subnet with Proxy and Unmanaged both to "true" should return error; doesn't.

The swagger-ui Subnets information shows an example POST block that has both "unmanaged" and "proxy" set to true. The documentation states that if both set to "true", this should result in an error. However, a 200 response is returned in this case.

Swagger URL example:
https://:8092/swagger-ui/#!/Subnets/createSubnet

  "ReadOnly": true,
  "ReservedLeaseTime": 0,
  "Strategy": "string",
  "Subnet": "string",
  "Unmanaged": true,```

Version: v4.1.1 Panics.

Experiencing this in normal operation. Three times in three days now.

With a total of only 6 hosts booting.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xbcec3c]

goroutine 2115660 [running]:
github.com/rackn/provision-server/v4/backend.(*Machine).validate(0xc00827c2d0, 0xc00a9f42c0)
        /home/travis/gopath/src/github.com/rackn/provision-server/backend/machines.go:609 +0xedc
github.com/rackn/provision-server/v4/backend.(*Machine).beforeSave(0xc00827c2d0, 0xc00a9f42c0, 0xc00827c2d0, 0x7f688a95b270)
        /home/travis/gopath/src/github.com/rackn/provision-server/backend/machines.go:672 +0xb0
github.com/rackn/provision-server/v4/backend.(*RequestTracker).save(0xc00a9f42c0, 0x11a1da0, 0xc00827c2d0, 0x11a1da0, 0xc00827c2d0, 0xc002ff6f01)
        /home/travis/gopath/src/github.com/rackn/provision-server/backend/requestTracker.go:438 +0x1aa
github.com/rackn/provision-server/v4/backend.(*RequestTracker).Save(0xc00a9f42c0, 0x11a1da0, 0xc00827c2d0, 0xc00099ff00, 0x0, 0x0)
        /home/travis/gopath/src/github.com/rackn/provision-server/backend/requestTracker.go:475 +0x42d
github.com/rackn/provision-server/v4/midlayer.(*DhcpRequest).checkMachine.func1(0xc0042bad80)
        /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:314 +0xb5f
github.com/rackn/provision-server/v4/backend.(*RequestTracker).Do(0xc00a9f42c0, 0xc009e1f8b0)
        /home/travis/gopath/src/github.com/rackn/provision-server/backend/requestTracker.go:362 +0x1f4
github.com/rackn/provision-server/v4/midlayer.(*DhcpRequest).checkMachine(0xc0006823c0, 0xc0097e5570)
        /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:241 +0x328
github.com/rackn/provision-server/v4/midlayer.(*DhcpRequest).coalesceOptions(0xc0006823c0, 0xc0097e5570)
        /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:348 +0x498
github.com/rackn/provision-server/v4/midlayer.(*DhcpRequest).buildDhcpOptions(...)
        /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:443
github.com/rackn/provision-server/v4/midlayer.(*DhcpRequest).ServeDHCP(0xc0006823c0, 0xc009e91ce0, 0x4)
        /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:708 +0xdba
github.com/rackn/provision-server/v4/midlayer.(*DhcpRequest).Process(0xc0006823c0, 0x7a8744, 0xc00017a7a8, 0x6e148b, 0x3e98e80)
        /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:791 +0x433
github.com/rackn/provision-server/v4/midlayer.(*DhcpRequest).Run(0xc0006823c0, 0x224)
        /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:816 +0x40
created by github.com/rackn/provision-server/v4/midlayer.(*DhcpHandler).Serve
        /home/travis/gopath/src/github.com/rackn/provision-server/midlayer/dhcp.go:888 +0x340

digitalrebar / provision Goto Github PK

provision's Introduction

END OF LIFE NOTICE FOR VERSION 2.x

Welcome to Digital Rebar

Getting Started & Documentation

Help & Community

Codebase History

Dangerous and Fun Quick Start command.

provision's People

Contributors

Stargazers

Watchers

Forkers

provision's Issues

Steps to reproduce

Expected result

Actual result

Steps to reproduce

Expected result

Actual result

System info:

Attempt to unzip:

Recommend Projects

Recommend Topics

Recommend Org