Version I tried:
Pre-built packages on Oct 15 (v7.4.0 and v8.0-SNAPSHOT)

What test I have performed:

As a user with admin privilege,

• Installed all 7 beats by using Pre-built installers on Windows 2012 R2
• Uninstall them by using Windows uninstall program
• Installed/Uninstalled FileBeat and PacketBeat on Windows 7
• Run all the beats as service after installation on both w2k12 and win7
• Use FileBeat as MVP, tried end-to-end use case: install FileBeat->Manually update the yml configure file to integrate with local Kibana/ES->Poulate Index template/Dashboard-> Run filebeat as service
• I also tried keyboard navigation as a simple accessibility testing

Overall, most of the above use cases passed, except for some area which has room for improvement:

1. Publisher/Signing warning, which has been tracked in another ticket: #14

2. User Guide. Comparing with the old way (using zip file), now beats installer will be operated in different folders: Program Files and ProgramData, we need to have a good how-to user guide too. Basically we need to rewrite most windows command, i.e., from Step1 to Step 5 in https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html.

3. It would be great if we can ship with some batch files, for users to populate dashboard and install templates, etc. As we now put configuration files (.yml), Kibana dashboards, security and monitoring modules, etc into %ProgramData%\Elastic{version}\Beats{beat name} directory. I spent quite long time to figure out the format of parameters to use during test. I assume it might be the same for our customers too.

4. On the last page of the installer, it would be nicer that we can have a link pointing to the "what are next steps" document, or maybe just point to the user guide.

5. For Packetbeats, I got a failure when I try start as service. When running from the Command line, the error is about wpcap.dll file missing, which should be installed before running the command &service. It is good to be mentioned in the document too.

6. On accessibility part, hotkeys are available for some buttons, but not for all, for example, in the below screen, there is no hot key for Cancel button

Unfortunately I don't have time to try all the beats with ES/Kibana. But overall, from a QA point of view, I am happy with the quality of these installers. Great job!

Evaluate possibility of migrating existing configuration on upgrade.

Evaluate possibility of using beat-specific MSI window backgrounds and other graphics assets.

Evaluate WixSharp's ManagedProject approach as potential v2. It provides much improved UI customization options, custom action handling then then some.

Functionbeat only has an ELF binary, check docs/consult Beats team to see if we need an msi for it

• Release manager integration #22
• Signing mechanism #14
• CLI shim, #10
• Remove functionbeat, #33
• Remove version from path to data directories #39
• Rename configuration file {beatname}.yml to {beatname}.example.yml to prevent upgrade from deleting the former. #39
• Give user further steps to follow after install finished. E.g: URL to getting started, point to .yaml config file, etc. #40

Optional:

• Talk to InfoSec wrt reuse of existing certificates
• Update Getting Started document to account for new MSI-based workflow.

create the getting started with the MSI content for the *beat's and link in the post-install action to that doc.
Reference to #32

Please document all prerequisites:

• for building
• for running the installer

equal to #5 please compile everything in a markdown file in docs/beats

It is nice UX to expose MSI (and possibly other) logs at the end of the install cycle. This task is to make them available from the UI.

Please document all build parameters of the winlogbeat msi:

• add the name of the parameter
• default value if exists
• required or optional

add these informations as a markdown to a docs/winlogbeat folder in the repo.

TODOs

• Build any beat from cli using packages from artifacts-api.
• Add beat-specific icon to show up in Add/Remove Programs and MSI.
• Support for x86 and x64 msi build workflows.
• Fix quiet install failure (/q, /qn, /q+).
• Fix administrative install failure (/a).
• Add support for all build target - builds all known installers. 4cbc102
• Grab short description of a beat from its README.md file. Dependency removed in 92fa202.
• Cache used ProductId (guid) -> version map in config.yaml. Deferred.
• Evaluate switching to "SDK-style" project format for XXXCompiler projects.
• Create meta-ticket for installer test proces. Connect with Beats team and discuss testing and CI engagement. See Installer Wiki
• Evaluate using elastic/elasticsearch-net-abstractions to talk to artifacts-api. #59
• Add support for latest, signifying latest version of the artifact (container).
  Might not be trivial: https://github.com/elastic/infra/issues/14645, https://github.com/elastic/infra/issues/5193, elastic/elasticsearch-docker#215 (comment) No action
• Figure out when to remove the hard-link so that package can be removed from disk. d518199. No Action: existing cleanup behavior removes links.
• Convert TODO entries to Github issues.

Converted to tickets:

• Implement CLI shim for beats (and ES/Kibana later) to run from any directory using same configuration file as service. #10
• Implement package signing. Add env var support for package signing. #14
• After install (UI sequence) provide means to open/edit config file. #40
• Notify Beats team that installer build process depends on beat's README.md file contents, specifically line 3, containing beat description. Discuss alternatives to make it less brittle. README.md dependency removed in 92fa202, others remain, see #4.
• Configure assembly versions properly for build runner and per-package compilers. #9
• Evaluate possibility of specifying external, pre-written beat-specific .yaml config (filebeat, winlogbeat, etc) file to be picked up by Beats installers. It can be on file system or pulled from GIthub for example. #11
• Evaluate WixSharp's ManagedProject approach as potential v2. It provides much improved UI .customization options, custom action handling then then some. #12
• Evaluate possibility of migrating existing configuration on upgrade. On install over an existing same version or upgrade ask to install fresh configuration or preserve existing. On uninstall, ask to remove data and configuration (default to No). #13
• Evaluate possibility of using beat-specific MSI window backgrounds and other graphics assets. #15
• Evaluate replacing SimpleExec with runner capable of returning exit codes and stdout/stderr (possibly nullean/proc). #16

References

• elastic/beats#163
• https://github.com/elastic/enhancements/issues/7001

Implement CLI shim for beats (and ES/Kibana later) to run from any directory using same configuration file as service.

Some nitpick observations while browsing the code base:

• Move to one .sln file. Solution files suck and are hard to maintain, the less the better
• Move shared folder to a shared library. They are now linked in but over time this becomes harder to maintain and less obvious where everything is used.
• Project names == Default namespace. This is somewhat of a stylistic nit but also has implications when dockerizing.
• The build (ElastiBuild) often refers to Artifacts or Products as Targets, this was a tad confusing on initial reading e.g show Show available build targets
• Should use the editorconfig the rest of the .NET repos use.
• Should use the resharper/rider settings the rest of the .NET repositories use. Stylistically the repos is not far off though at first glance 👍

Contributors would need to check it off

Per #28:

6. On accessibility part, hotkeys are available for some buttons, but not for all, for example, in the below screen, there is no hot key for Cancel button

Similar to #25.

Let's prefix all our service name with our company name for discoverability.

Per @Mpdreamz:

Another nitpick thats wholefully personal is the msi-elastic-stack repository name, stack-windows-installers is potentially stronger and inline with stack-docs

I found these "stack-" like names:

stack-docker
stack-docs
stack-demos
stack-terraform
stack-monitoring-dev

elastic-stack-testing
elastic-stack-monitoring-service

I am leaning towards elastic-stack-installers.

@philkra, what do you think?

Individual products:

• Beats
• Elasticsearch
• Kibana

Sign .msi and possibly executables with a cert.

• Support reading of cert credentials from env vars

Do we want to support elasticmachine efforts to create Github releases and tags like Maxime's APM Server example in https://github.com/elastic/infra/issues/16141 shows?

/cc: @Mpdreamz @philkra @russcam

To unblock RM repo integration (https://github.com/elastic/infra/issues/16141#issuecomment-557271631) add commands to install .net core 3.0 SDK during build.

Evaluate possibility of specifying external, pre-written beat-specific .yaml config (filebeat, winlogbeat, etc) file to be picked up by Beats installers. It can be on file system or pulled from GIthub for example.

Per #28:

User Guide. Comparing with the old way (using zip file), now beats installer will be operated in different folders: Program Files and ProgramData, we need to have a good how-to user guide too. Basically we need to rewrite most windows command, i.e., from Step1 to Step 5 in https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html.

Windows post-install instructions need to be updated.

I was wondering what @Mpdreamz @russcam @urso @philkra think about including {version} component in path? I originally envisioned it to help with isolation and possibility of SxS install .. now I think this might pose a problem with version upgrades.

Current install paths:
Binaries: %ProgramFiles%\Elastic\{version}\Beats\{beat name}
Data: %ProgramData%\Elastic\{version}\Beats\{beat name}

Example:
C:\Program Files\Elastic\8.0.0\Beats\winlogbeat
C:\ProgramData\Elastic\8.0.0\Beats\winlogbeat

Current build process for Windows binaries doesn't add icons to executables. I built icons for all Beats executables from official .svg file. Icons are available here: https://github.com/elastic/msi-elastic-stack/tree/master/src/installer/resources.

@philkra @urso @mikemadden42

One we decide on the name (#27), it would be nice to have updates flow into #microsoft-github Slack channel.

Table of Contents for test-related isues

• Build runner unit tests <issue # HERE>
• Package compiler unit tests <issue # HERE>
• Build configuration persistence unit tests <issue # HERE>

The post-install step should allow the user to:

• define an Elasticsearch output for the winlogbeat, identified as output.elasticsearch.hosts. Please note that this field can take a list of hosts as defined in the documentation.
• allow the user to supply optionally basic authentication params
• checkbox to let the user start the service directly (once installation is completed)
• checkbox for Open winlogbeat.yaml config in my default text editor post installation.

I'd argue we'd want to prefix all our products with Elastic ....

It be good if the product or elastic company logo shows up here as well.

MSI package sets only Read/Execute permissions for mutable data written to %ProgramData%. Permissions need to be adjusted at install time to allow (possibly localized) built-in NT Group "Users" Write/Modify access.

A placeholder for discussing what Logstash would need to do to add a Windows installer.

From a previous discussion with @russcam we would need to:

1. Add a new executable wrapper in addition to the logstash.bat file.
2. Build the installer itself.

@russcam as you can see I'm woefully short on details. What can we provide you to move this forward?

IMHO it would be useful to have a dedicated sub-forum for installer-related questions. Interested parties (/me) could then sub to the forum and respond to questions.

@sophiec20 @philkra @Mpdreamz, what do you think?

EDIT: erroneously pinged {at}sop instead of Sophie

• Assembly versions
• Tag commit SHA MSI package with commit SHA.

Setup of integration tests for *beats. We should run nightly tests after the snapshots have been build to catch breaking changes immediately. Let's use this this ticket for communication of the process/requirements, you name it :)

/cc @Mpdreamz @dolaru @sophiec20

@urso What in your opinion we should do with functionbeat, since it doesn't run on Windows. Even fact that files are installed into PrograFiles is already strange.

refs #32

• Remove version from path to data/config dirs. On upgrade we want to use the data already accumulated from previous version.
• Rename configuration file {beatname}.yml to {beatname}.example.yml to prevent upgrade from deleting the former.

#40 introduced a checkbox on the last wizard page to optionally open data directory once user clicks "Finish". Public property WIXUI_EXITDIALOGOPTIONALCHECKBOX controls the state of the checkbox and whether action is executed or not. When this variable is passed in command line to msiexec:

msiexec /l!*vx "winlogbeat-install.log" /i winlogbeat-7.5.0-windows-x86_64.msi WIXUI_EXITDIALOGOPTIONALCHECKBOX=0

Open directory action is not taken however UI did not update to the new value of the variable, however.

Tell the user about the .example.yml config file, what to do with it and where it is. Present an option to open the data directory.

Add stack_elastic_installers block to build.gradle

Evaluate replacing SimpleExec with runner capable of returning exit codes and stdout/stderr (possibly nullean/proc).

I've seen people running multiple versions of the stack, here's another one (elastic/windows-installers#337).

I have per-version folders for config and data built in, Should we consider allowing installing side-by-side?

@russcam @Mpdreamz @sophiec20 @philkra

Meeting notes below. Notes: beats cross-compiles windows binaries in docker, this will not work for installer.

Short-term goal: have beats installers run the same way windows CI environment currently packages installers.

Long-term goal: add windows box to beats packaging pipeline to build and package windows binaries and installers.

https://docs.google.com/document/d/1zKUwsZU77xjf1hI9L_z-UgbzvxbeCAhIVuwqk2gm5b0/

Repository

• Integrate per-beat installer into elastic/beats

Packaging

• Beat per MSI
• All beats in single MSI (maybe)
• Establish service dependencies (TCP/IP, etc)

Artifacts

• short-term: grab built .zip and build package from that
• long-term: build beats windows binaries and MSI under windows, integrated into general beats pipeline

Bitness

• Provide x86 version. Do we still care about x86/32bit variants?
• Provide x64 version.

UI - Provide per-beat configuration UI to:

• allow the beat to be installed as a service.
• allow the beat to connect to downstream component.
• run Kibana dashboard setup
• configure/enable available modules
• Direct user to the guide page for the beat to continue/modify/extend configuration.

CM / Fleet integration

• ...

Testing / CI

• ...

UPD: restored original text of the issue

Beats installer compiler has dependencies on contents of some artifacts in the Beats repository. This metadata describes each beat and is used in the Installer UI and build process to avoid hard-coded or manual input prone to be out of sync.

• Contents of README.md file, specifically line 3, containing one-liner beat description. (dependency removed in 92fa202)
• Contents of LICENSE.txt file it its entirety. This file is temporarily massaged into .rtf format installer expects.
• Directory names (kibana, module, modules.d, monitors.d) are tagged in config.yaml in this repository as "mutable" and installed into a location with write/modify access by non-elevated users.

Purpose of this notification is to see how Beats team feels about current implementation and solicit feedback. See config.yaml for more sample.

@urso @philkra @sophiec20 @Mpdreamz @russcam

Create docs folder and document:

• env vars
• msiexec public variables matrix #5
• Pre-requisites #6

Related content to look at:
https://www.elastic.co/guide/en/beats/winlogbeat/current/index.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/windows.html

Most modern software allows installation without elevation (slack, chrome, vscode, etc), called per-user install. Contrasted to per-machine install, which puts files into ProgramFiles and writes to HKLM, per-user install writes to %AppData% (Roaming) or %LocalAppData%. I think it is worth considering implementing this option.

refs: https://github.com/elastic/infra/issues/16141

per @Mpdreamz

I think config.yml in the original codebase was a mistake, reckon there should be a way to use v5 > guids off a name to create deterministic guids for product/upgrade codes. A bunch of nuget packages support v5 guids.

Artifact version can have an optional qualifier in the name, see https://github.com/elastic/dev/issues/1021 and https://github.com/elastic/infra/issues/16141

• Add ElastiBuild support
• Add Release Manager command support

Determine if we need to add license header to sources and which one.