User Story

in order to return appropriately filtered data to users, each api call exposed in
reporting.proto
must

• make a function call on the incoming ctx to find out which projects they should filter on
• return the data that matches the project filters

An example of the function to be called on the ctx is:

func filterByProjects(ctx context.Context, filters map[string][]string) (map[string][]string, error) {
	projectsFilter, err := auth_context.ProjectsFromIncomingContext(ctx)
	if err != nil {
		return nil, err
	}
	if auth_context.AllProjectsRequested(projectsFilter) {
		return filters, nil
	}

	filters["projects"] = projectsFilter
	return filters, nil
}

An example of adding the project filter to the elasticsearch filters is:

if len(filters["projects"]) > 0 {
		termQuery := elastic.NewTermsQuery("projects", stringArrayToInterfaceArray(filters["projects"])...)
		query = query.Filter(termQuery)
	}

Definition of Done

all apis in the defined proto make a function call on the incoming ctx to find out which projects they should filter by
all apis in the defined proto accept a []string projects filter and return the correct data based on those project filters
[ "project6", "project9" ] -> return all data tagged with project 6 or project 9 or both
[ "(unassigned)" ] -> return all data tagged with no projects tagged
[ "(unassigned)", "project8" ] -> return all data tagged with no projects tagged and return all data tagged with project8
["*" ] -> return all data

Demo Script / Repro Steps

set up a project with ingest rules
send in a client run report, action, and compliance report that match the rules of the created project
make a query for resources in created project
expect to see the resources
make a query for resources in a different project
expect to not see the resources

User Story

the Json response from /reporting/export is not properly formed.

This is an array of json objects but:

1. the array is not surrounded by []
2. the elements of the array are not comma separated.

Definition of Done

when the json is properly formed.

Demo Script / Repro Steps

the basic upgrade path
run chef-automate iam version: expect response of IAM v1.0
create local user
add compliance nodes (chef_load_compliance_scans -N 10). this will create nodes for client runs and compliance.
add some actions (chef_load_actions)
run the upgrade command: chef-automate iam upgrade-to-v2 --beta2.1
logged in as admin:

• navigate to client runs: expect to see the nodes
• navigate to compliance: expect to see the nodes
• navigate to event feed: expect to see the actions

logged in as local user:

• navigate to client runs: expect to see the nodes
• navigate to compliance: expect to see the nodes
• navigate to event feed: expect to see the actions

creating node ingestion rules tags node reports with projects (client runs AND compliance)
< after upgrading to iamv2 >
create some ingestion rules in two different projects. ensure the fields selected will match a subset of the nodes.
request all nodes without projects (via api or ui)
expect to only the nodes that did not match the ingestion rules ("unassigned")
a) request all nodes in one project: ensure data matches
b) request all nodes in both projects: ensure data matches
c) request all nodes in both projects and all nodes that did not match: expect to see all nodes (a + b)
add more nodes.
repeat a, b, and c from above

creating actions ingestion rules tags actions (events) with projects
note: this does not apply to events from scan jobs and profiles. this only refers to "actions"
< after upgrading to iamv2 >
create some ingestion rules in two different projects. ensure the fields selected will match a subset of the actions.
request all actions without projects (via api or ui)
expect to only the actions that did not match the ingestion rules ("unassigned")
a) request all actions in one project: ensure data matches
b) request all actions in both projects: ensure data matches
c) request all actions in both projects and all that did not match: expect to see all actions (a + b)
add more actions.
repeat a, b, and c from above

data access integrity
< after upgrading to iamv2 >
while logged in as admin, request nodes from the earlier referenced projects
navigate to the node details view for one of the nodes in client runs. save the url.
navigate to the node details view for one of the nodes in compliance. save the url.
log in as a different user, that does not have access to the projects
copy and paste the urls; expect 403

original issue: https://chefio.atlassian.net/secure/RapidBoard.jspa?rapidView=261&modal=detail&selectedIssue=UI-166

Overview

Currently, we have two different implementations and styles for our on-page status filter groups. One for compliance related pages and one for client runs related pages. This task is only about updating the styles and sizing for the existing filters on the compliance page, not including updating the implementation for the client run pages.

Background Information

It seems that compliance status filters use chef-phat-radio and chef-option from our Stencil library and that the client runs filters do not use any components from the stencil library. It also seems that the client run filters are keyboard accessible and the compliance filters are not. ❓What would be the best way to proceed so that all pages are using the same implementation?

Design Details

Status Filter Group Clickable Prototype (https://chef.invisionapp.com/share/YWQTP2M4EK5#/351214248_Status_Filter-V5-Total-Selected)

Design Specification for Implementation (https://chef.invisionapp.com/share/YWQTP2M4EK5#/354274680_Status_Filter_Group_Design_Spec)

(The InVision design specification includes all the following content, you can also use the InVision Inspec feature for more details)

Status Filter Group Definition

A status filter group is a linear set of two or more status filters, each of which functions as a mutually exclusive option. Status filters contain text labels, status icons(optional), and numbers. A status selector is used for displaying the roll-up count in each status, and filtering out the content in the same area. A status filter group usually appear on top of a table or a list of cards that displays the filtered out content.

Status Selector Style by States

Currently provide five status: general, critical, warning, success, and unknown. A status filter can and only can be one of the five statuses. The text label should reflect the meaning of the chosen status but does not need to match the exact status name.

Status Filter Sequence

As a rule of thumb, the general status filter (total, all, etc) should always appear on the leftmost. The unknown status filter (unknown, pending) should always appear on the rightmost. In between, the status from left to right should be arranged in the sequence of critical, warning, and success.

Accessibility

Visible focus indicator (browser default) and the ability to select the filter with a keyboard.

Spacing & Sizing

The total with of the status filter group should be dynamic and match with the width of the table or card underneath. However, the horizontal distance between each filter should always be 16px.

Variation

Condensed Status Filter Group

When there is limited space, the status filter group can adjust to a condensed variation. Depending on the total width available to fit the group and the individual filter width, filters can wrap to new lines. The style of the states remains the same.

Spacing & Sizing

The total with of the status filter group should be dynamic and fit into the space available. However, the horizontal distance between each filter should always be 10px.

User Story

Users today have a hard time figuring out what api requests should look like for Automate.
We have some basic swagger docs available at automate-url/api/v0/openapi/ui/#, but these are bare-bones when it comes to instructing the consumer on which fields are required, which fields are for incoming requests, the purpose of the api request, and the expected response codes.

Our swagger docs are auto-generated from our externally exposed api protos, so it all starts with a well documented proto for all externally exposed apis.

An example:

import "protoc-gen-swagger/options/annotations.proto";

message TargetConfig {
	option (grpc.gateway.protoc_gen_swagger.options.openapiv2_schema) = {
		json_schema: {
			title: "Target Config"
			description: "Defines the values needed to connect to the node."
			required: ["host", "port", "backend"]
		}
	};
       string host = 1;
       .....
}

Definition of Done

let's work with the docs team to figure this out

Relevant Resources

docs re: read-only/write-only for swagger: https://swagger.io/docs/specification/data-models/data-types/#readonly-writeonly

swagger editor: http://editor.swagger.io/#/

https://godoc.org/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options#JSONSchema

designs from a past attempt at this project
https://app.zeplin.io/project/5b59e8744f3d5be407a3a101/dashboard
(if you need access to zeplin please connect with @susanev)

Aha! Link: https://chef.aha.io/epics/A2-E-128

User Story

in order to support rbac work, we'll need to add some fields to the inspec reports that are fed through the audit cookbook.
the list of fields we've agreed to support is:
chef-server, organization, environment, role, chef tags, policy group, policy name

that means we'll need to add chef-server, organization, chef tags, policy group, and policy name
to https://github.com/chef-cookbooks/audit/blob/master/libraries/reporters/automate.rb#L12 and get a new release out
follow up work will be to add those fields to the mappings for compliance reports

Definition of Done

new audit cookbook release.
inspec reports coming from audit cookbook include chef-server, organization, environment, role, chef tags, policy group, policy name when applicable

Demo Script / Repro Steps

should be able to test audit cookbook with a2 using https://github.com/chef/a2/tree/master/components/compliance-service/smokin

For a profile with inspec.yml containing these attributes:

attributes:
  - name: bucket_name
    description: Required cloud bucket name
    type: string
    required: true
    default: 'test'

  - name: retries
    description: How many times to retry
    type: numeric
    required: false
    default: 3  

inspec json output returns both the legacy attributes array:

"attributes": [
  {
    "name": "bucket_name",
    "description": "Required cloud bucket name",
    "type": "string",
    "required": true,
    "default": "test"
  },
  {
    "name": "retries",
    "description": "How many times to retry",
    "type": "numeric",
    "required": false,
    "default": 3
  }

and the new inputs array:

"inputs": [
  {
    "name": "bucket_name",
    "options": {
      "description": "Required cloud bucket name",
      "type": "string",
      "required": true,
      "value": "test"
    }
  },
  {
    "name": "retries",
    "options": {
      "description": "How many times to retry",
      "type": "numeric",
      "required": false,
      "value": 3
    }
  }

Observed this on inspec 3.7.11 and 3.9.0.

The automate ingestion logic, elasticsearch mappings and APIs only understand the attributes array at the moment.

Will need to prioritize the backend work required by this attributes -> inputs change in inspec. It requires changes to: elasticsearch mappings, data migration, ingestion logic, API logic, reponses and UI rendering.

User Story

for data durability and in support of rbac
don't really know anything about the size of this work; needs investigation
it could be that we find this work isn't really needed as well.

original issue: https://chefio.atlassian.net/secure/RapidBoard.jspa?rapidView=261&modal=detail&selectedIssue=UI-142

Description

DeleteNotificationDialogComponent (https://github.com/chef/a2/blob/f335f0f841b0776ab2018d59fdb5f287b63712b9/components/automate-ui/src/app/page-components/delete-notification-dialog/delete-notification-dialog.component.ts#L2) & NotificationsComponent (https://github.com/chef/a2/blob/f335f0f841b0776ab2018d59fdb5f287b63712b9/components/automate-ui/src/app/pages/notifications/notifications.component.ts#L5)

in addition, these content updates - should be in line with the delete modal component used in other admin pages:

• Close button (secondary)
• Title: Delete Notification?
• Message: Deleting this notification is permanent and cannot be undone.
• Buttons: Delete Notification (primary), Cancel (tertiary)

User Story

In https://github.com/chef/a2/pull/5067 the client runs ingestion pipeline was modified to use the elasticsearch bulk api with bundled messages. This is a huge improvement for the pipeline, so let's implement it in the compliance ingestion pipeline too.
Please see https://github.com/chef/a2/pull/5067/files for more details.

Definition of Done

compliance ingestion uses es bulk api/bundles msgs

User Story

with the introduction of https://github.com/chef/a2/pull/4483, a significant performance improvement was made to ensure that compliance-service can handle all the reports it is ingesting. as part of that work, we will be rejecting messages if we cannot handle them.

to ensure messages do not get dropped, we need to implement retries in inspec reporting to automate. the same functionality needs to exist in the audit-cookbook.

User Story

inspec version field is returned on api call to compliance/reporting/nodes/id/{id}, which is the node details page

Definition of Done

inspec version in the ui as shown above

Demo Script / Repro Steps

run a scanjob or send in a report. see inspec version on compliance reporting node details page

User Story

In an effort to make things easier for our users, UX has mocked up some designs for creating a credential from within the add node screen.
The credential would need to be saved in the same way it is when using the credential add screen. We would then use the id returned from the credential create to the node create post call.
I am not certain how flawless this flow will be, so let's discuss if there are issues with it.

Default State

Sudo Checked

Available Credentials

Change Credential Type

Add new credential form

User Story

original issue: https://chefio.atlassian.net/browse/A2-248

Problem

https://chefio.atlassian.net/browse/A2-241

Overview

The first iteration of a project filter. Users will be restricted to no more than 6 projects to start.

Video

please see original issue for video. I am unable to upload it to this github issue as it is an mp4

Design Details

https://app.zeplin.io/project/5ba808703eb54372d860c737/dashboard?seid=5c464d2dd363b1bf6d2e0a4e

Menu Functionality

Clicking outside of the menu should close it
If the menu is open and focus is moved outside of the menu, then it should close
Any user changes to the checkboxes are lost after closing the menu, if the user did not Apply Changes
Project filter label and checkboxes functionality
Access to exactly 1 project
Display the project name
Do not display any dropdown functionality
Access to only unassigned resources
Display Unassigned resources
Do not display any dropdown functionality
Access to at least 1 project and the unassigned resources
When nothing is selected display All resources
When a single project is selected display that project's name
When only unassigned resources selected display Unassigned resources
When between 2 – (1 - all) projects selected display Multiple projects with a badge indicating the number of selected items
Note: Selecting unassigned resources in this situation has no effect on the top label
When all projects selected and unassigned resources is not selected display All projects along with a badge that indicates the number selected
When all projects selected and unassigned resources display All resources
Access to more than 1 project and not the unassigned resources
When no selection display All projects along with a badge indicating the number of projects
Note: In this situation the badge is gray, indicating that no selection is made
When exactly 1 project selected display that project's name
When between 2 – (1 - all) projects selected display Multiple projects with a badge indicating the number of selected items
When all projects selected display All projects along with a badge indicating the number of projects
Note: In this situation the badge is blue, indicating that there are selections
Button functionality
Apply Changes button should be disabled until one of the checkboxes changes it state, then it should become enabled
When an action is taken on Apply Changes the menu should close and the view should be refreshed
Accessibility
The project filter needs an aria-label="Projects filter" because there is no on-screen label
It should be possible to tab to the projects filter
When the focus is on the filter it should have a visible focus indicator
space or enter should open/close the menu
esc should close the menu if it is open
Up and down arrow keys should navigate the checkboxes
space on a checkbox should toggle its state
when on the button space or enter should activate it and close the menu

Visual Design

Some details included here, please see Zeplin for more information

Label in top-nav is font-size: 16px; font-weight: 600; color: #212334;
Icon to the right of the label in the top nav is 16px by 16px
14px of space between the PROJECTS label and the top of the menu
Please use text-transform for the uppercase label
Projects header in menu is font-size: 11px; font-weight: 400; color: #777777;
18px of space on the left of the checkboxes and the right of the checkbox labels
Checkbox size is 16px by 16px and should align to the default for chef-checkbox
Checkbox label is font-size: 14px; font-weight: 600; color: #212234 which is different than the default for chef-checkbox
16px of space between checkboxes
24px of space between the last checkbox and the Apply Changes button
16px of space between the Apply Changes button and the bottom of the menu
Other Details
Sharing a link with someone else will not maintain any applied filters

original issue: https://chefio.atlassian.net/secure/RapidBoard.jspa?rapidView=261&modal=detail&selectedIssue=UI-85

User Story

Currently the way that chef-button works when a url is applied to it causes a page refresh when the buttons are used in automate. I am guessing that this is because the nested link uses href where Angular expects routerLink. We should figure out a way to have links that look like buttons so it doesn't cause a page refresh.

User Story

Upon ingestion of an action, evaluate chef-server and org associated with action, and tag the action with the appropriate project.

• ingestion service depends on authz service
• ingestion service calls authz service for rules
• action is tagged with project based on rules it matches
• action is tagged with correct projects after project update event triggered

Definition of Done

All incoming actions are evaluated and tagged on ingestion.

• the only fields applicable for actions are chef-server and organization

End result: projects field on action

Demo Script / Repro Steps

get some mocked ingest mapping rules
send some actions into a2
see action has been tagged with projects

The root check in the workflow-ctl code happens too late. Attempts to run the command as a non-root user results in:

/bin/workflow-ctl: line 2: /hab/svc/automate-workflow-server/config/automate-ctl-config.sh: Permission denied
/bin/workflow-ctl: line 8: bundle: command not found

The workaround is to run the command as root which is what is required anyway, but we should be displaying a nicer message to the user. The issue here is that the configuration file is owned by hab and can get created with rather restrictive permissions.

original issue: #72

User Story

The Automate nodemanager details page designs include a multi-select delete option.

the endpoint for the multi-select delete is: a2-url/api/v0/nodes/delete/ids, which takes a POST msg of form ["123..", "456.."]

Definition of Done

Automate nodemanager details page has multi-select delete for nodes.

The linter noticed this problem and a naive attempt to fix it caused
a number of tests to fail.

If this is intentional we might want to document why and perhaps remove this argument:

https://github.com/chef/automate/blob/master/components/config-mgmt-service/backend/elastic/nodes.go#L64-L66

I ran a compliance scan via the audit cookbook with three profiles:

1. mylinux-success a profile with a few passing controls
2. apache-baseline a profile with controls that skip on the host one by one because the apache service is not installed
3. mywindows a profile that will be skipped entirely by inspec due to platform incompatibility.

• Actual compliance details for the node: the Skipped profiles tooltip showed only an empty list:

• Expected compliance details for the node: the Skipped profiles tooltip shows a list of profiles and reason for skip if it available

User Story

when a user updates rules for a project or creates a new project, we need to re-evaluate all ingested nodes and actions to update project tagging.
in the spike, @lancewf demonstrated this could be done by running a painless script in elastic search that takes a list of project rules and evaluates/re-tags ingested nodes according to the new project rules.

that takes care of the actual re-tagging process.

the other thing that needs to be done is event handling around project rule updates.
@lancewf has a design for this:

so, following the above design + some conversations we've had around message durability, the flow is:

• user creates one or more new projects or updates projects
• auth service triggers rule update event
• ingest service and compliance ingestion are listening for the rule update event
• upon receipt of the rule update event, ingest service and compliance ingestion make a call to elastic to start the painless script. this is an async call that returns a process id for later querying.
• ingest service and compliance ingestion trigger rule update received event
• auth is listening for the rule update received event. if that event is not received within a timely manner (tbd) after sending the rule update event, the auth service should retry
• when ingest service completes the update, it triggers client runs update complete event
• when compliance ingestion completes the update, it triggers compliance update complete event
• auth is listening for the compliance update complete event

Notes: there will be many gotchas in this implementation. how to handle duplicate messages, what should the auth service do if it never receives a complete event from one of the services, etc. we are expecting these issues to crop up and expect to spend a couple weeks adjusting as needed

Definition of Done

a project rule update in auth triggers the event flow to start the painless script

Demo Script / Repro Steps

update a project or create a new one
ensure the expected data is tagged with the correct projects

More Information

there have been concerns about data durability when using the event service. until we can build off the work currently being done in the event service for durability (NATS), we are pushing the data durability problem to the services sending events to the event service. this has been discussed at length with @gpeers and she is on board.

Sub Tasks

• Add project tag update for client runs nodes
• Add project tag update for compliance reports
• Add an authz-service manager to manage the update process with both compliance and client runs.

User Story

original issue: https://chefio.atlassian.net/secure/RapidBoard.jspa?rapidView=261&modal=detail&selectedIssue=UI-161

Note: Due to some design consistency related updates some of the attached screenshots may have the incorrect nav or smaller details, if you have any questions please reach out to susan evans.

Overview

Compliance will become a top-level item with Reports, Scan Jobs, and Profiles appearing in the left-nav when navigating to Compliance. This work should leverage the work done in Settings, where the user is directed to the first page in that list that they have access to, and if they don't have access to any of the compliance pages than Compliance should be hidden from the top-nav.

New Routes

compliance/reports/

compliance/scan-jobs/

compliance/profiles

https://github.com/chef/a2/blob/master/dev-docs/adr/adr-2019-01-23.md

Design Details

Reports

Change the h1 to be Reports

Change the subheading to be Compliance reports describe the status of scanned infrastructure. this was recently changed, so this no longer needs an update

Change the Reports icon to use bar_chart

Scan Jobs

Change the h1 to be Scan Jobs

Change the subheading to be Compliance scan jobs run inspec exec on a set of nodes.

Profiles

Change the h1 to be Profiles

Change the subheading to be Compliance profiles manage security and compliance scans.

Documentation Updates

• Change documentation nav to have Compliance as a top-level item with Reports, Scan Jobs, Profiles, and Nodes API under it.

Reports page

• Rename Compliance page to Reports
• Change the h1 to say Compliance Reports
• Change Compliance Reporting Overview to Overview
• Change The Reporting view under the... to The Reports view under the...
• Change Dates in Compliance Reporting to Dates in Compliance Reports
• Change Compliance Reporting Results and Job ID Filters to Compliance Reports Results and Job ID Filters
• Update https://automate.example.com/compliance/scanner/jobs to be https://automate.example.com/compliance/scan-jobs/jobs
• Change ...by selecting Report, which will redirect to the compliance tab. to by selecting Report, which will redirect to the Reports page.
• Change Use the compliance reporting **Profiles tab..._ to Use the reports **Profiles** tab...
• Update all screenshots

Scan Jobs page

• Rename Chef Automate Scan Jobs page to Scan Jobs
• Change the h1 to Compliance Scan Jobs
• Change About Scan Jobs in Chef Automate to Overview
• Update all screenshots

Profiles page

• Rename Asset Store page to Profiles
• Change the h1 to Compliance Profiles
• Change About the Asset Store to Overview
• Change Find Chef Automate compliance profiles under the Asset Store tab to Navigate to compliance profiles by using the Compliance tab and then selecting the Profiles page
• Any other mention on the page of the Asset Store should be changed to Profiles
• Update all screenshots

NB: Save yourself some work (notes from @msorens):

• See https://github.com/chef/a2/pull/4642 for prior art on Settings tab.

• Wait for about-to-land PR (https://github.com/chef/a2/pull/5039) that genericizes the SettingsLanding component used in that PR, into a handy reusable component.

Designs

When showing the error for unreachable nodes, the UI does an API call to get all of the node's information to show the error message from its last connection attempt.

In #5306 the node list API endpoint was changed to include this information as well so the additional API call is potentially unnecessary.

original issue: https://chefio.atlassian.net/secure/RapidBoard.jspa?rapidView=261&modal=detail&selectedIssue=UI-158

Overview

To unify the user menu with the incoming projects filter this task provides a number of visual changes and updates.

Design Details

Zeplin (https://app.zeplin.io/project/5ba808703eb54372d860c737/dashboard?seid=5c4887fb20f12301729f6199)

All visual design and content are intentional if you have any questions or need more information please connect with susan evans.

Dropdown button

• Remove name label from the top navigation bar
• Remove the circle behind the person icon
• Change the person icon to be 20x20 and #212234, it should be vertically centered in the top navigation bar
• Place an expand_more icon 8 pixels to the right of the person icon, its bottom most point should align with the text labels in the top navigation bar
• Together with the person icon and expand more icon should form a single button that opens a drop down

Dropdown menu

• Add a caret to the top of the dropdown that aligns with the expand more icon

• Please reference the Zeplin designs for text style information, and padding/margins in the menu

• Signed in as and the user's name is not interactive

• Profile takes the user to their profile page

• Version number is not interactive

• About opens the about modal

• License

  should open
  https://www.chef.io/online-master-agreement/
  in a new tab
  - Note: there is currently a modal and we want that to go away

• Release Notes opens the release notes in a new tab

• Sign Out signs the user out and sends them to the sign in page

• Hovering/focusing on any of the interactive items should result in a background color of #dee5fd

User Story

InSpec will soon require users to accept the license by using one of the following methods:

* Interactively agree to license (not practical here, I think)
* run with the option `--chef-license accept`
* create the file `/etc/chef/accepted_licenses/inspec` or `~/.chef/accepted_licenses/inspec` . The file may be empty.

This will be needed for all the inspec commands, such as json, check, detect, and exec.

This change will also be modifying the stdout format. Please see chef/license-acceptance#23 for more information.

These changes will affect: profile upload, profile json-ing, detect jobs, and scan jobs.

Definition of Done

Able to run Automate as always with the new license acceptance functionality.

A gem of the new InSpec functionality is available here: http://artifactory.chef.co:8081/webapp/#/artifacts/browse/tree/General/omnibus-gems-local/gems/inspec-4.0.6.gem

We've requested a hab pkg for testing as well.

User Story

In today's version of Automate, we have these handy-dandy filter buttons on the nodes list and profiles list.

Example:

the filter button is the one just to the left of the scan results button.

With the (soon-arriving) addition of deep filtering, we expect that more users will want to filter by control. Instead of making them type up the control every time, we could add the same filter button to the places in the ui where we list controls.
Those are:

• scan results (accessible from reporting nodes list and reporting profiles list)
• reporting profile details (accessible by clicking on a profile in reporting profiles list)
• reporting node details (accessible by clicking on a node in reporting nodes list)

required: ux designs

Definition of Done

filter buttons on all control lists

Aha! Link: https://chef.aha.io/epics/SH-E-459

User Story

original issue: https://chefio.atlassian.net/secure/RapidBoard.jspa?rapidView=261&modal=detail&selectedIssue=UI-164

Overview

Currently, there is no confirmation modal when removing a node credential; this story adds one.

Design Details

Zeplin

• Leverage the delete confirmation on the other settings pages (Users, Tokens, etc)
• Heading: Remove Node Credential?
• Message: Removing node credential: <credential name> cannot be undone.
• Primary destructive button: Remove Credential
• Tertiary button: Cancel
• Close button: close icon

==> Search for in the code base and you will find 6 examples (at last count) of how this reusable component may be used for this task.

Accessibility Details

Heading should use an h2

Tab Order

1. close button
2. Remove Credential button
3. Cancel button
4. go to 1.

User Story

https://chef.invisionapp.com/share/FWQ0OZMMJR6#/screens/336670667

this view is new to automate
the api call that populates the list of nodes for this view is api/v0/nodes/search -d '{"filters": [{"key": "manager_id", "values": ["e69dc612-7e67-43f2-9b19-256afd385820"]}]}'

• nodemanager metadata (nodemgr name, status) (can get count of nodes from nodes call)
• list of nodes (using nodes api call noted above ^)
• multi-select and single-select removal (depends on backend work for multi select node removal)
• add nodes button that links to current add node page

with existing sorting. do not include last scan/last run columns (that are x'd out)

Overview

Following on from UI-144 and https://github.com/chef/a2/pull/4991 we have a few more design updates to be made to chef-checkbox

Design Details

[Zeplin|https://app.zeplin.io/project/5ba808703eb54372d860c737/screen/5c48f36c01c52637748eb390]

• When a checkbox is disabled its label should appear disabled with an opacity of 0.5
• Checkboxes should be 20 x 20 pixels and have at least 14 px of space between the checkbox and label
• Label is optional
• It should be possible to tab to all enabled checkboxes, and using [space] should check/uncheck the checkbox
• Clicking on the label should check/uncheck the checkbox

In Compliance reporting, on a profile page, for example:

https://automate.example.com/compliance/reporting/profiles/bf5ffac4d08aadd4120cf2316dc1633d90f0f2965b69c2974f925d8bed79c07f

or on the reporting profiles page:

https://automate.example.com/compliance/reporting/profiles

if the Scan Results button is clicked, the sidebar lists the nodes using the default latest_report.end_time ASC sorting. Since only the node name is displayed, the list looks unsorted.

Screenshot from acceptance:

I propose to sort the list of node names alphabetically.

User Story

these are getting an overhaul to provide the user with more details. needs to be done for

• aws ec2
• aws api
• azure vm
• azure api
• gcp api

Designs

From the Node Integration Page after a user presses the add integration button

will take them to the default create screen.

AWS Selected

Detect creds from EC2 checked

If AWS EC2 type is selected.

Microsoft Azure

Detect Creds from Azure Checked

Azure VM

Will add the Authentication to runs Scan Jobs.
The drop down will have 2 values

1. Use Azure's RunCommand
2. Add credentials by tags

The add Credentials by Tags will then show the credentials by tags component.

Google Cloud Platform

Credential by Tag Component

The drop down will use the same component as we use for the Project Picker.

If possible, can we add an extra link within the drop down to add a new credentials? which would then take the user to this page:

after saving, then would return them to the Node Integration page with the Nodes tab displayed.

User Story

The automate-workflow code base was imported as-is from the old Chef Automate 1 code base. This feature is provided for users upgrading from Chef Automate 1 who have already invested in workflow. While we do not expect to do any large-scale feature-work in this code base, it is likely that we have to do ongoing maintenance for this feature until there are no supported users of it.

Towards that end, we should cleanup loose ends that might confuse users and future developers or make maintenance harder.

New Features

• HTTP communications between workflow and authn: Workflow communicates with Authn over HTTP rather than HTTPS. This is currently mitigated by it being a localhost-only communication, but now that the code base is imported it should be easier to support HTTPS.

Cleanup Opportunities

• Remove dangerous or non-functional automate-ctl commands. (In progress: #59)

• Remove unused modules from the workflow-server codebase

  • Insights (#57)
  • Visibility (#57)
  • Notifications (maybe? I haven't investigated whether we can)

• Investigate unused code from the workflow-web codebase. I'm not familiar with this project to know if there is unused code in what we imported.

• Removed unused code from workflow-nginx: #80

Test Integration

On merge we added some very basic tests, but there are more tests in the workflow codebases that haven't been hooked up into CI and/or that could be extended.

• Extend workflow integration test to include creating a pipeline
• Extend workflow integration test to include creating a runner
• Unit tests for automate-workflow-ctl
• Integration tests for automate-workflow-ctl
• Unit tests for automate-workflow-web
• e2e tests for automate-workflow-web
• schema tests for automate-workflow-server
• dialyzer for automate-workflow-server

User Story

original issue: #72

In order to make it easier for our users to navigate to adding nodes to the Automate nodemanager, give them a way to get there by adding a link to the "add nodes" page from the Automate nodemanager details screen.

url for automate details page: a2-url/settings/node-integrations/e69dc612-7e67-43f2-9b19-256afd385820

url for add nodes page: a2-url/compliance/scanner/nodes/add

Definition of Done

Automate nodemanager details page has a link (as shown above) to the add nodes page.

User Story

We don't use scroll for es, so we can't go above 10k nodes (same issue visible in ui if you try to paginate to page 11). But since export doesn't need any pagination, we should apply scroll to that api.
@lancewf did similar work for configmgmt export

Definition of Done

can export 11k nodes

Demo Script / Repro Steps

load 11k+ nodes
download report

Motivation

A goal of Automate 2 is for all customers to trust our automatic management of upgrades and service state. Currently, however, Automate users and Chef Software support engineers cannot answer any of the following questions easily:

• When was the last upgrade of this deployment?
• What happened when this deployment was migrated from Automate 1?
• When was the last time the user changed the deployment configuration?

The ability to answer such questions is essential to debugging problems with Chef Automate installation and upgrades and increasing overall trust in the system.

Feature Description

The Action Log will provide users and support staff with information about the historical and in-progress modifications of their Chef Automate installation via a command-line interface. Information will be provided on both deployment- and service-level events. This would include:

• Logs for deployment-level events such as initial installation, upgrades, configuration requests, and service state remediations.
• Logs for service-level events such as installs, loads, restarts, and reconfigurations

This action log would persist across deployment-service restarts and allow you to query information about the entire history of the Chef Automate installation.

The following is example output of such commands (IDs and Timstamps are reused/inconsistent in the example output for convenience).

Overall Deployment History

> chef-automate history
ID         StartTime                  Description     	Status
17d37a16b  2018-09-01T10:31:22+01:00  Initial Deployment   Success
c6961888c  2018-09-03T09:13:41+01:00  Configuration Set    Success
c17a82a1a  2018-09-04T12:22:13+01:00  Automatic Upgrade    Success
87f4dab7e  2018-09-04T13:30:31+01:00  Service Remediation  FAILURE
e96644f81  2018-09-05T22:12:51+01:00  Service Remediation  In Progress

Or, for an A1 migrate that failed once:

> chef-automate history
ID         StartTime                  Description        Status
17d37a16b  2018-09-01T10:31:22+01:00  Migration from A1  Failure
e96644f81  2018-09-01T10:31:29+01:00  Migration from A1  Success
c6961888c  2018-09-03T09:13:41+01:00  Configuration Set  Success
c17a82a1a  2018-09-04T12:22:13+01:00  Automatic Upgrade  In Progress

Individual Service History

The user can inspect the history of individual services. Every service-level action is tied to some deployment-level action.

> chef-automate service-history ingest-service
ID         StartTime                  Description    Cause                       	Status
056eeacff  2018-09-01T10:31:22+01:00  Install        Initial Deployment(17d37a16b)   Success
3cbaae510  2018-09-01T10:31:22+01:00  Configuration  Initial Deployment(17d37a16b)   Success
cf9978b43  2018-09-01T10:31:22+01:00  Load           Initial Deployment(17d37a16b)   Success
53a6e646e  2018-09-01T10:31:22+01:00  Install        Upgrade(c17a82a1a)          	Success
88fafa013  2018-09-01T10:31:22+01:00  Configuration  Upgrade(c17a82a1a)          	Success
3bf3a594c  2018-09-01T10:31:22+01:00  Load           Upgrade(c17a82a1a)          	Success

Individual Event History

The user can poll the status of individual events. The output here might be event-specific. Since we only have deployment-service logs, we also provide information about how to get logs from the system for this event:

> chef-automate show-event c17a82a1a
  	Event: c17a82a1a
Description: Automate Upgrade
     Status: Success

Deployment Service Log:
time="2018-09-05T10:16:01Z" msg="Starting periodic converge" current_manifest=20180905100418 next_manifest=20180905100418
time="2018-09-05T10:16:01Z" msg="Found hart override" hart="&{chef deployment-service /go/src/github.com/chef/a2/results/chef-deployment-service-0.1.0-20180905100253-x86_64-linux.hart 0 1 0 20180905100253}" name=deployment-service origin=chef

... SNIP (actual log would be longer, with command to control output)...

To read the complete Chef Automate log (including all service
logs) for the duration of this event run:

	journalctl -u chef-automate --after 2018-09-01T10:31:22+01:00 --before 2018-09-01T10:32:33+01:00

Data Retention and Key Events

The log should be persistent for the entire lifetime of an installation. However, since an installation may produce a large number of events, the Action log will have data retention and pruning features to limit data growth while retaining key installation events.

Deployment service actions that resulted in no change (for instance, most periodic converges) will be aggressively pruned from the event history. Users will be able to configure the retention period for these events with a configuration value:

action_log.retention.unchanged_actions = “1h”

Deployment service actions that succeeded will be kept for a small amount of time by default. Users will be able to configure the retention period of these events with a configuration value:

action_log.retention.successful_actions = “2d”

Deployment service actions that failed will be kept for a longer amount of time:

action_log.retention.failed_actions = “20d”

Certain key events such as A1 migration and initial deployment will never be pruned.

Since journalctl will still contain full logs, users who want long-term storage of all cluster events can configure their journal log retention accordingly.

User Story

in order to return appropriately filtered data to users, each api call exposed in
stats.proto
must

• make a function call on the incoming ctx to find out which projects they should filter on
• return the data that matches the project filters

An example of the function to be called on the ctx is:

func filterByProjects(ctx context.Context, filters map[string][]string) (map[string][]string, error) {
	projectsFilter, err := auth_context.ProjectsFromIncomingContext(ctx)
	if err != nil {
		return nil, err
	}
	if auth_context.AllProjectsRequested(projectsFilter) {
		return filters, nil
	}

	filters["projects"] = projectsFilter
	return filters, nil
}

An example of adding the project filter to the elasticsearch filters is:

if len(filters["projects"]) > 0 {
		termQuery := elastic.NewTermsQuery("projects", stringArrayToInterfaceArray(filters["projects"])...)
		query = query.Filter(termQuery)
	}

Definition of Done

all apis in the defined proto make a function call on the incoming ctx to find out which projects they should filter by
all apis in the defined proto accept a []string projects filter and return the correct data based on those project filters
[ "project6", "project9" ] -> return all data tagged with project 6 or project 9 or both
[ "(unassigned)" ] -> return all data tagged with no projects tagged
[ "(unassigned)", "project8" ] -> return all data tagged with no projects tagged and return all data tagged with project8
["*" ] -> return all data

Demo Script / Repro Steps

set up a project with ingest rules
send in a client run report, action, and compliance report that match the rules of the created project
make a query for resources in created project
expect to see the resources
make a query for resources in a different project
expect to not see the resources

My attempts to run this when testing various cleanups always results in this error:

 sudo workflow-ctl migrate-github-project foo
Warning: All cached repos must be cleared out prior to migration.
See goo.gl/Bm7tPc for instructions).
Enter "all repos clear" to confirm and proceed.
all repos clear
Failed to run command:  
Error: ---- Begin output of /hab/pkgs/chef/automate-workflow-server/2.8.61/20190418163121/delivery/bin/github_migration foo ----
STDOUT: escript: exception error: no match of right hand side value 
                 {error,
                     {{already_started,<0.20.0>},
                      {child,undefined,net_sup_dynamic,
                          {erl_distribution,start_link,
                              [['[email protected]',longnames]]},
                          permanent,1000,supervisor,
                          [erl_distribution]}}}
STDERR: 
---- End output of /hab/pkgs/chef/automate-workflow-server/2.8.61/20190418163121/delivery/bin/github_migration foo ----
Ran /hab/pkgs/chef/automate-workflow-server/2.8.61/20190418163121/delivery/bin/github_migration foo returned 127

I have not dug into this issue. Is this command expected to work still?

original issue: https://chefio.atlassian.net/secure/RapidBoard.jspa?rapidView=261&modal=detail&selectedIssue=UI-169

User Story

We recently did a link update to have the colors for links in rest, hover, focus, and active link styles are aligned across a2. We missed the breadcrumb links though because they have some local styles that are overriding that update.

References

https://github.com/chef/a2/pull/5063

User Story

A customer asked me about this today, and when i went to look on the docs page I realized it wasn't documented anywhere, so i had to give him an example in chat.
Let's make sure these "delete manager with nodes" apis are at least minimally documented, and then look if there are other apis we haven't yet documented. These ones are esp important to document b/c we have no ui for them.
The purpose of the delete manager with nodes apis is to give the user an easy way to purge all the nodes associated with a node integration should they decide to delete a node integration.
https://github.com/chef/automate/blob/master/components/automate-gateway/api/nodes/manager/manager.proto#L72-L108

Definition of Done

apis are documented as part of our docs on nodes: https://github.com/chef/automate/blob/master/components/automate-chef-io/content/docs/nodes.md

User Story

the changes introduced here https://github.com/chef/a2/pull/5122 were making it hard to get the gcp tests to work. so i disabled the tests over there. let's turn those tests back on! having a pr only focused on that should make this easier to debug

Definition of Done

gcp tests (compliance-service/api/tests/75_) are running in CI

User Story

from the customer:

My scan jobs controls for the `CIS AWS Foundations Benchmark Level 1` were working, but are now all returning `The security token included in the request is invalid`. Seems to be a problem on the A2 side, as I can run the same profile locally with the same key pair I'm feeding the node integration in A2.

In my initial discovery, I found that this problem does not exist when using the "read my creds from env" option, but is reproducible when using an integration for which an access key/secret was provided.

this was likely always an issue.
inspec/inspec-aws#17 ; we were (incorrectly) swallowing the error in the past and are now exposing it.

need to figure out why access key/secret is not enough, or what the differences are/what's going on

On the /compliance/reporting/nodes page, sorting nodes by the Control Failures triggers an infinite loading loop:

API logs show this 400 response:

[2019-04-10T13:58:38+00:00] automate-gateway.default(O): time="2019-04-10T13:58:38Z" level=error msg="finished unary call with code Unknown" auth.action=search auth.resource="compliance:reporting:nodes" auth.subjects="[team:local:admins user:local:admin]" error="rpc error: code = Unknown desc = elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]" grpc.code=Unknown grpc.method=ListNodes grpc.service=chef.automate.api.compliance.reporting.v1.ReportingService grpc.start_time="2019-04-10T13:58:38Z" grpc.time_ms=14.067 iam.version=iam_v1 peer.address="127.0.0.1:33368" span.kind=server system=grpc

API call body:

{
  "filters": [
    {
      "type": "start_time",
      "values": [
        "2019-03-31T00:00:00Z"
      ]
    },
    {
      "type": "end_time",
      "values": [
        "2019-04-10T23:59:59Z"
      ]
    }
  ],
  "page": 1,
  "per_page": 100,
  "sort": "latest_report.controls.failed.total",
  "order": "ASC"
}

original issue: https://chefio.atlassian.net/browse/UI-143

User Story

In order to remove all angular material components, we need to convert the MatSnackbar found here: https://github.com/chef/a2/blob/f335f0f841b0776ab2018d59fdb5f287b63712b9/components/automate-ui/src/app/pages/notifications/notifications.component.ts#L5
to the chef-snackbar found in our chef-ui-library

Screenshot taken from https://a2-local-inplace-upgrade-dev.cd.chef.co that highlight the bug:

Client Runs page without any filters:

Client Runs page with two platform filters, one exact, one wildcard:

As far as I can tell the culprit for this is that we do a must(AND) between filters and not a should(OR) between filters of the same type here:

https://github.com/chef/a2/blob/1179ba54382958fbdac4373dd4cd85b8cd68ce53/components/config-mgmt-service/backend/elastic/elastic.go#L112

User Story

In the current world, when downloading json/csv reports in compliance, we get a connection to elasticsearch for each report. This can cause some problems, wherein too many connections to es are opened and performance suffers, sometimes blocking the ability to download the report. (https://chefio.slack.com/archives/C07HRQ9AS/p1545303073088900, https://chefio.slack.com/archives/C07HRQ9AS/p1545318578103600)

from rick:

there was a time when we were using a singleton for ES connection
we changed it to one connection per request because, the singleton connection had the potential to become "disconnected", in the case where ES went down and then came back up.
so we moved to one connection/request as this would ensure that if connection could be made, it would
it's entirely possible that the olivere lib has addressed this concern with their latest version.. we can go back to using the singleton and, even if olivere has not improved, we can do some checks.. like if conn is dirty, kill the object and respawn etc..
when we used singleton back when, it behaved itself pretty well.. we just need to test it's ability to recover when connection goes south

this is likely something that needs to be addressed on the infra-automation side of things as well:

User Story

As an Automate User, I want to be able to use the updated multi-region capable inspec-aws resources to be able to run compliance scans against all of my regions in a single compliance run.

Definition of Done

We believe this should already be able to handle a version of InSpec that uses the new inspec-aws resource pack, however it should be tested and we should be allowing Automate customers to make use of the improved multi-region support and ensure the CIS certified AWS Profile can be run correctly within A2

Demo Script / Repro Steps

Run a compliance scan using the updated CIS Profile for Level 2 AWS Foundation Benchmark

User Story

in order to return appropriately filtered data to users, each api call exposed in
event.proto
must

• make a function call on the incoming ctx to find out which projects they should filter on
• return the data that matches the project filters

An example of the function to be called on the ctx is:

func filterByProjects(ctx context.Context, filters map[string][]string) (map[string][]string, error) {
	projectsFilter, err := auth_context.ProjectsFromIncomingContext(ctx)
	if err != nil {
		return nil, err
	}
	if auth_context.AllProjectsRequested(projectsFilter) {
		return filters, nil
	}

	filters["projects"] = projectsFilter
	return filters, nil
}

An example of adding the project filter to the elasticsearch filters is:

if len(filters["projects"]) > 0 {
		termQuery := elastic.NewTermsQuery("projects", stringArrayToInterfaceArray(filters["projects"])...)
		query = query.Filter(termQuery)
	}

Definition of Done

all apis in the defined proto make a function call on the incoming ctx to find out which projects they should filter by
all apis in the defined proto accept a []string projects filter and return the correct data based on those project filters
[ "project6", "project9" ] -> return all data tagged with project 6 or project 9 or both
[ "(unassigned)" ] -> return all data tagged with no projects tagged
[ "(unassigned)", "project8" ] -> return all data tagged with no projects tagged and return all data tagged with project8
["*" ] -> return all data

Demo Script / Repro Steps

set up a project with ingest rules
send in a client run report, action, and compliance report that match the rules of the created project
make a query for resources in created project
expect to see the resources
make a query for resources in a different project
expect to not see the resources

User Story

i.e. the machines referenced here: https://github.com/chef/a2/blob/master/components/automate-gateway/integration/license_usage_nodes_test.go#L81 and here: https://github.com/chef/a2/blob/master/components/automate-gateway/integration/nodes_test.go#L19

those references are sprinkled through the code in a few other places

Definition of Done

use vpn protected instances for testing

some WIP towards making this happen https://github.com/chef/a2/compare/vj/cleanup-cmp

Related to chef/a2#5116

As discussed in the iamv2 meeting yesterday (4/11), for the initial implementation of projects filtering for compliance-service we are ignoring the case where incoming requests have no project data attached. By design all requests should include projects data, so this is guarding at the seams between services that this assumption is being maintained.

Being lenient to non-compliant requests is useful in the short term to allow existing code that has no knowledge of projects to continue working, but eventually we will want to be strict in how that is handled.

User Story

in order to return appropriately filtered data to users, each api call exposed in
cfgmgmt.proto
must

• make a function call on the incoming ctx to find out which projects they should filter on
• return the data that matches the project filters

An example of the function to be called on the ctx is:

func filterByProjects(ctx context.Context, filters map[string][]string) (map[string][]string, error) {
	projectsFilter, err := auth_context.ProjectsFromIncomingContext(ctx)
	if err != nil {
		return nil, err
	}
	if auth_context.AllProjectsRequested(projectsFilter) {
		return filters, nil
	}

	filters["projects"] = projectsFilter
	return filters, nil
}

An example of adding the project filter to the elasticsearch filters is:

if len(filters["projects"]) > 0 {
		termQuery := elastic.NewTermsQuery("projects", stringArrayToInterfaceArray(filters["projects"])...)
		query = query.Filter(termQuery)
	}

Definition of Done

all apis in the defined proto make a function call on the incoming ctx to find out which projects they should filter by
all apis in the defined proto accept a []string projects filter and return the correct data based on those project filters
[ "project6", "project9" ] -> return all data tagged with project 6 or project 9 or both
[ "(unassigned)" ] -> return all data tagged with no projects tagged
[ "(unassigned)", "project8" ] -> return all data tagged with no projects tagged and return all data tagged with project8
["*" ] -> return all data

Demo Script / Repro Steps

set up a project with ingest rules
send in a client run report, action, and compliance report that match the rules of the created project
make a query for resources in created project
expect to see the resources
make a query for resources in a different project
expect to not see the resources