elastic / observability-docs Goto Github PK

from this doc:
https://github.com/elastic/beats/blob/master/x-pack/elastic-agent/docs/upgrade-elastic-agent.asciidoc#upgrade-standalone-agents

we need to update the steps to cite:
call the Agent from the directory where it was installed in Windows or as the service executable as in:
elastic-agent upgrade {version}

e.g. C:\Program Files\Elastic\Agent\elastic-agent.exe upgrade 7.10.1
e.g. elastic-agent upgrade 8.0.0-SNAPSHOT

of course, we'll want to test this out to be sure it works.

For the 7.9 release, content for the new Observability overview page is required for the Kibana docs.

Summary

Add a section to each agent's troubleshooting documentation that lists the steps required to disable the APM Agent. This information is available on the agent's configuration pages, but it might not be obvious to our users that this can be used for testing or troubleshooting to avoid disruptions to production applications.

Tasks

• Create troubleshooting documentation for the Go Agent elastic/apm-agent-go#789
• Add missing links to APM Overview troubleshooting elastic/apm-server#4024
• Add troubleshooting documentation explaining how to disable each APM Agent
  • .NET Blocked by elastic/apm-agent-dotnet#122.
  • Go elastic/apm-agent-go#789
  • Java elastic/apm-agent-java#1319
  • Node.js elastic/apm-agent-nodejs#1802
  • Python elastic/apm-agent-python#893
  • Ruby elastic/apm-agent-ruby#837
  • RUM elastic/apm-agent-rum-js#852

elastic/kibana#77690 introduced bulk actions UI in Fleet for taking actions on multiple agents at a time: reassign, unenroll, and (soon to come) upgrade agents.

I don't see docs where we document Fleet UI in granular detail, but just to give a few ideas, we could call out that bulk actions (the ability to easily manage a large amount of agents) is available in:

1. Fleet overview
2. Unenroll agents

Note: Bulk actions is only available with Gold+ license

Add fleet terminology to glossary: https://www.elastic.co/guide/en/elastic-stack-glossary/current/terms.html

Related PRs:
https://github.com/elastic/observability-dev/issues/659
https://github.com/elastic/endpoint-app-team/issues/423

Terms to define:

Agent policy TBD
We need to sort out the inconsistencies across Security and Fleet/Elastic Agent docs. An agent policy contains integration policies.

integration policy an instance of an integration that is configured and added to an agent policy

data stream TBD

dataset TBD

Elastic Agent a single, unified agent that you can deploy to hosts or containers to collect data and protect endpoints.

Fleet TBD

integration a bundle of assets needed to ingest and visualize data or protect endpoints.

namespace a user-defined string used to group data.

standalone

data stream naming scheme https://www.elastic.co/blog/an-introduction-to-the-elastic-data-stream-naming-scheme

Fleet

input

Elastic package registry a service that stores Elastic package definitions in a central location that is hosted by Elastic.

Other terms we might need to define:

As discussed in Slack, we should add a "What's New" in Observability section for 7.10 (and beyond!). Find all of your new Observability goodness in one place!

Kibana has a great header, which Kaarina said we can copy: https://www.elastic.co/guide/en/kibana/current/whats-new.html.

The content for Installing metrics should be rewritten to include step-by-step procedural information. The aim is to provide a better user experience to what is currently offered.

The Get started content should include:

• Getting started with the Elastic Stack
• Installing Metricbeat (from Kibana and the command line)
• Enabling integrations (modules)

Per request from Elastic Security product management, we need to update the Config Settings topic to include some additional security and comms information, as specified in this Google doc.

Copied and pasted as follows:

How are communications secured between endpoint and agent?
Endpoint connects to the agent over loopback TLS on port 6788.  Endpoint validates that the agent has root (Linux and macOS) or SYSTEM (Windows) permissions.  In future versions, the endpoint will validate the digital signature of the agent, and vice versa.

How are communications secured between agent and kibana?
Agent connects to Kibana over TLS and authenticates the certificate presented by Kibana.  Agent then provides an ApiKey as an authentication token, which Kibana validates.

What ES and Kibana ports need to be accessible?
The configuration file generated by the Ingest Manager app already contains the correct Elasticsearch address and port for your setup. If you run everything locally, the address is 127.0.0.1:9200. If you use our hosted Elasticsearch Service on Elastic Cloud, the address corresponds to the Elasticsearch endpoint URL that is listed under Endpoints as described in Work with Elasticsearch.  If not running in the cloud, both the Kibana and the Elasticsearch HTTPS ports must be accessible; by default these are 5601 and 9200 respectively.

We can discuss further during our sync on Thursday.

After the dust settles on renaming and moving Ingest Manager > Fleet, we need to set up website redirects and remove the redirects page that was added in #165.

We should add the Metrics anomaly detection configurations to the Observability Guide as a new topic (inspect-metric-anomalies) within the Analyze Metrics section.

cc @szabosteve @lcawl

The steps are:

1. Copy the attached plist file co.elastic.agent.plist into /Library/LaunchDaemons/
2. Execute sudo chown root:wheel /Library/LaunchDaemons/co.elastic.agent.plist to change plist ownership to root
3. Create the path /Library/Elastic/Agent/
4. Copy over the contents of the elastic agent folder into /Library/Elastic/Agent
   a) This should be the contents of the untarred elastic-agent-X.X.X-darwin-x86_64.tar.gz
5. Execute sudo launchctl load -w /Library/LaunchDaemons/co.elastic.agent.plist

co.elastic.agent.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>co.elastic.agent</string>
    <key>ProgramArguments</key>
    <array>
        <string>/Library/Elastic/Agent/elastic-agent</string>
        <string>run</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>WorkingDirectory</key>
    <string>/Library/Elastic/Agent</string>
    <key>UserName</key>
    <string>root</string>
    <key>KeepAlive</key>
    <true/>
    <key>ThrottleInterval</key>
    <integer>15</integer>
    <key>EnableTransactions</key>
    <false/>
    <key>ExitTimeOut</key>
    <integer>60</integer>
</dict>
</plist>

According to the docs readme, we should now use the titleabbrev attribute to provide abbreviated titles instead of a pass thru block.

So instead of using:

++++
<titleabbrev>XYZ Intro</titleabbrev>
++++

Use

[id=intro_to_xyz,titleabbrev=" XYZ Intro"]
== Intro to XYZ

Hi here are some feedback items, in a general order of urgency, from review over:
#80

1. from the Trouble shooting page, about the Endpoint - we could enhance this to be more accurate now (since its live and we want folks to use it!
   What is the Endpoint integration shown in Ingest Manager?
   Let's change the scripting to something like...
   The Elastic Endpoint Security integration facilitates running the Elastic Endpoint on your Agent controlled host. It provides protection capabilities and is a security-data event shipper that integrates seamless with the Elastic Security Kibana app. Endpoint is managed by Elastic Agent in the same way that other Integrations are, try it out! For more information, please see the relating Elastic Endpoint and Elastic Security solution documentation.

2. from this page:
   https://observability-docs_80.docs-preview.app.elstc.co/guide/en/ingest-management/master/elastic-agent-cmd-options.html

recommend we remove this internal-only intended enrollment option:
--staging
Configures agent to download artifacts from a staging build.

3. I'm uncertain if all of the current 'Limitations' are correct... we should have this content further reviewed.

• for instance, I don't know if we support a stand-alone package registry in 7.9 or not. IF so, we can modify or remove that limitation and add a topic about how to set it up if user has the appropriate license level to do so (I thin its upper level like Gold / Platinum)

4. From the troubleshooting page:
   Where does Elastic Agent store logs after startup?
   When started successfully, Metricbeat logs are stored in data/logs/metricbeat under the folder where Elastic Agent was started. If that log path does not exist, Elastic Agent was unable to start Metricbeat, which is a higher level problem to triage.

I think we should start it with citing the logs for the Agent process itself and then mention Metricbeat. Agent logs are in data/logs location. You can mention Filebeat will have logs in the same location as Mericbeat.

And now that we know more we could cite here that for .deb / .rpm installs, the logs are at:
/var/lib/elastic-agent/logs/elastic-agent-json.log
/var/lib/elastic-agent/logs/default/metricbeat-json.log
/var/lib/elastic-agent/logs/default/filebeat_monitor-json.log

5. from the IM overview: this highlighted blue text could be removed from the bottom, its basically stated above and seems repetitive
   "using SSH, Ansible playbooks, or other configuration methods"

6. from this page:
   https://observability-docs_80.docs-preview.app.elstc.co/guide/en/ingest-management/master/elastic-agent-configuration.html
   we could put a note in that all of these same settings can be modified for the Fleet usage in the Integration and Configuration UIs.

• I'm not sure of this was intended to be 'stand-alone' only content but all has a UI counterpart that we could cite now or in the future if we wanted. Not mandatory

7. do you think we should add a topic for how to delete the Agent and relating files from the host? the dpkg or rpm usage should be easy and clear to most admins. and for Windows there is a .ps1 script to uninstall the service, at which point you can simply stop the agent and delete the folder. Maybe its overkill for now? but we could split it to an enhancement for future doc'ing.

The content for Installing logs should be rewritten to include step-by-step procedural information. The aim is to provide a better user experience to what is currently offered.

The Get started content should include:

• Getting started with the Elastic Stack
• Installing Filebeat (from Kibana and the command line)
• Enabling integrations (modules)

Some of the text strings used in Ingest Manager have changed in 7.9 and the docs need to be updated accordingly.

Docs that need updates (might be others):

• https://www.elastic.co/guide/en/ingest-management/master/ingest-management-getting-started.html
• https://www.elastic.co/guide/en/ingest-management/master/run-elastic-agent.html
• https://www.elastic.co/guide/en/kibana/master/ingest-manager.html (possibly)

Related PRs:
elastic/kibana#71691

Summary

Within the Observability guide, we should add a how-to guide relating to the PCF integration. The content should reflect the different use-cases (personas) listed in this google doc, which will be used as a source for the published content in the Observability guide.

In the proposed outline for the Observability Guide, we have the Visualize data in Kibana section. This issue is to add the Logs monitoring content to this new section. This content will eventually remove the need for a separate Logs monitoring Guide.

Documentation updates:

• Monitor logs - brief overview of Logs monitoring.
  • Stream logs (Stream UI)
  • Inspect anomalies (Anomalies UI)
  • Categorize log entries (Categories) Related blog
  • Configure data sources (Settings UI)
  • Create alerts
  • Tail log files

There are a few loose ends/feedback/cleanup required for 7.10. Creating this issue to track them

• Finalize move to Fleet User Guide:

  • Get all branches of docs building in the Fleet User Guide. elastic/docs#2004 elastic/docs#2004
  • Set up redirects for ingest management > fleet (wildcard). https://github.com/elastic/website-www.elastic.co/issues/7066
  • Remove the ingest management guide

• Test steps (fleet and standalone) on deb/rpm. PRs: #244 and elastic/beats#22562 #248

• Verify upgrade changes here: https://github.com/elastic/beats/pull/22510/files/35ee1d26540a2d1a92875756c571c8d28c33685d..efff156e9abec9439d53c7b00a604d1d50aec467#diff-cb5a5af2dec36df51ec6cca4c8a87876483da9187609b28c0515056fdc6fbe39

• Update links on the download page to the new URLs for Fleet docs. PR: https://github.com/elastic/website-www.elastic.co/issues/7091

• After testing is complete, remove release-state override.

• Make sure redirects are set up for any changed URLs. https://github.com/elastic/website-www.elastic.co/issues/7068

• Add a better screenshot to show Fleet (one that shows multiple agents running).

• Review feedback from @EricDavisX (mostly about upgrading):

FIXED
Agent upgrade points to make sure we have somewhere in the various documentation:
- for 7.10 'install' section we should probably the install with a note about manually stopping and removing any prior Agent deployed
- the 'upgrade' on the edge host for 7.10.0, is still a manual process (but different from what you have doc'ed I'll explain more).  In 7.10 we have introduced a new 'install' option as cited, which facilitates a persistent agent deployment for linux, macOS and Windows.
- Once the 7.10.0 or newer Agent is install, upgrading to subsequent versions can be done manually, like documented in 'install agent' or the user can be do this through the Kibana UI with a simple button click.
- From 7.10 and beyond, if the Kibana version is higher than the version of Agent then the Agent can be upgrade to the version of Kibana deployed.  
...In example, when a 7.10 patch comes out like 7.10.1, to upgrade Elastic Agent, the user can simply click the 'Upgrade agent' actions menu button in the 7.10.0 Fleet UI for the desired Agent.  Or the user may select multiple Agents and use the bulk actions menu to upgrade multiple agents at the same time.
-------------
FIXED there is a typo in the .rpm / .deb steps here: https://www.elastic.co/guide/en/ingest-management/7.10/elastic-agent-installation.html
elastic-agent enroll --kibana-url=KIBANA_URL --enrollment-token=ENROLLMENT_KEY
should be
elastic-agent enroll KIBANA_URL ENROLLMENT_KEY
... just like it was in 7.9 docs https://www.elastic.co/guide/en/ingest-management/7.9/run-elastic-agent.html
-------------
FIXED
- the 'upgrade-elastic-agent' page can be simplified with copy from the blurbs above.
https://www.elastic.co/guide/en/ingest-management/7.10/upgrade-elastic-agent.html
 - we can delete the edge host cli steps and just refer users to the 'install' steps (above) or the new UI based steps.
 - we can note again that 'Upgrade agent' is not enabled in the UI or supported for Agents deployed via .deb and .rpm artifacts.
 - we can NOTE: The 'Upgrade agent' action menu item is only enabled when the Kibana version is higher than the version of the Agent.

-------------

FIXED I have 2 more pages to do quick updates:
https://www.elastic.co/guide/en/ingest-management/7.10/uninstall-elastic-agent.html
this command must be called from the installed running agent… they won’t work if you use the location where you unzipped / untarr’ed the folder (which is not the final place where Agent puts the files, you see).
So we can change the commands for linux and macOS to remove the ./
easy.
and for windows, we can cite the full path to make it explicit:
C:\Program Files\Elastic\Agent\elastic-agent.exe uninstall

--------------
FIXED then reverted
and 2. on this page: https://www.elastic.co/guide/en/ingest-management/7.10/elastic-agent-cmd-options.html
we pulled support for ‘upgrade’ from the Agent command line, so we should take the whole section out.  I see it is still listed in the agent ‘help’ command so I’ll talk to Engineering about removing it. (edited) 
--------------

FIXED # 3) I believe we changed the kibana start up variables… from ‘ingestManager’ to ‘fleet’ now, see one legacy usage noted in: https://www.elastic.co/guide/en/ingest-management/7.10/fleet-troubleshooting.html
we should totally double check that if you aren’t sure, I’m totally not sure.
#4 on same page, I see a citation of ingest_manager in an api reference, that’s changed to ‘fleet’ now

--------------

FIXED - ok… about #2 see this issue.  I guess it should stay - but I never tested it as such, and I recall explicitly when we revoked support… it looks like it went back in but nobody talked about it to the group.  : /  communication is hard.  https://github.com/elastic/beats/issues/22530

Creating this issue for future reference. All issues were raised during the review of the Observability Guide.

• Should we document types of logs users might want to add? Currently only Filebeat is documented.
• Should we document Logstash?
• Should we add a tip to the set-connection-widget that tells users how where to find the cloud ID? (The link sends users away from the main flow.)
• Should the content for installing the APM server be included in the Observability guide?
• Get feedback from Marketing on the introductory content.

Per discussion with Manu he thinks this is something we should document for Docker usage particularly, but it will relate to any Linux system. Any Linux will need some process manager like systemd.

we can post more details here. FYI @blakerouse @ph @mdelapenya thanks.

Now that the Logs/Metrics/Uptime content has moved into the Observability guide, can we deprecate the old guides?

• Permanently set current as the last released version [7.9]
• Move the books into the "Legacy" section of /guide
• Remove the content from the master, 7.x, and 7.10 branches of the observability-docs repo

See https://www.elastic.co/guide/en/infrastructure/guide/current/index.html for an example.

Following the work done in elastic/kibana#74842, we should update the Ingest Manager/Fleet docs to refer to agent policies rather than agent configurations.

We should also update datasets to data streams, but I see that data streams is already being used, so this might not be necessary.

Add a topic to the Fleet docs that describes:

What a policy is
How it's used
How to create a policy
Etc

In the proposed outline for the Observability Guide, we have the Visualize data in Kibana section. This issue is to add the Metrics monitoring content to this new section. This content will eventually remove the need for a separate Metrics monitoring Guide.

Documentation updates:

• Analyze metrics - brief overview of Metrics monitoring.
  • View metrics by topology (Inventory UI)
    • Monitor servers
    • Monitor Kubernetes pods Related blog
    • Monitor Docker containers
    • Monitor AWS services Related blog
  • Group and explore metrics (Metrics Explorer UI) Related blog
  • Configure data sources (Settings UI)
  • Create alerts

Current documentation does not indicate that you change the Elasticsearch host under Settings in Ingest Manager.

As a result, we're seeing issues like this: https://discuss.elastic.co/t/elastic-agent-enrollment-problem/246426/4

Create a topic similar to the Filebeat directory layout topic that describes the directory structure for each platform.

For background, see #90 (comment)

Also add a topic to the FAQ that explains where to find log files.

Eamonn noticed that the following pages need to be updated to included synthetics and UE:

• https://www.elastic.co/guide/en/observability/master/observability-ui.html
• https://www.elastic.co/guide/en/observability/master/observability-introduction.html

Add a topic to the Fleet docs that decsribes:

What an enrollment token is, how it works, how to create a token, how to revoke a token, etc

Page: https://www.elastic.co/guide/en/ingest-management/current/ingest-management-getting-started.html

Our prerequisites section for Ingest Manager mention two separate Kibana fields for encryption keys.

I believe the portion I highlighted should sayxpack.encryptedSavedObjects.encryptionKey instead of xpack.security.encryptionKey.

This came up when I was reviewing this PR elastic/kibana#71686. Apparently xpack.security.encryptionKey works for kibana.dev.yml, but it is not what we want to recommend using.

In the proposed outline for the Observability Guide, we have the Visualize data in Kibana section. This issue is to add the Uptime monitoring content to this new section. The content will eventually remove the need for a separate Uptime monitoring Guide.

Related blog

Documentation updates:

• Monitor uptime - brief overview of Uptime monitoring.
  • View monitors by status (Overview UI) - includes content about certificates.
  • Analyze network endpoints (Monitor UI)
  • Configure Heartbeat indices (Settings UI)
  • Create alerts

@elastic/obs-docs We need to add Observability terms to the Elastic glossary. I've included the terms and definitions that immediately came to mind below. Please feel free to edit and add your own.

Observability

• Elastic Agent (ingest)
• Observability
  Unify your logs, system metrics, uptime data, and application traces to provide granular insights and context into the behavior of services running in your environments.

APM

• APM ^{common term}
• Agent (APM)
  An open-source library, written in the same language as your service, which instruments your code and collects performance data and errors at runtime.
• Server (APM)
  An open-source application that receives data from APM agents and transforms them into Elasticsearch documents.
• Distributed tracing
  The end-to-end collection of performance data throughout your microservices architecture.
• Instrumentation
  Code is considered "instrumented" when it collects and reports performance data to APM.
• Real User Monitoring (RUM)
  Performance monitoring, metrics, and error tracking of web applications.
• Span
  Information about the execution of a specific code path. Spans measure from the start to the end of an activity and can have a parent/child relationship with other spans.
• Trace
  Defines the amount of time an application spends on a request. Traces are made up of a collection of transactions and spans that have a common root.
• Transaction
  A special kind of span that has additional attributes associated with it. Transactions describe an event captured by an Elastic APM agent instrumenting a service.

Logs

• Ingestion
  The process of collecting and sending data from various data sources to Elasticsearch.
• Integration
  Out-of-the-box configurations for common data sources to simplify the collection, parsing, and visualization of logs and metrics. Also known as a module.
• Module
  Out-of-the-box configurations for common data sources to simplify the collection, parsing, and visualization of logs and metrics. Also known as an integration.

Metrics

• Metric ^{common term}

Uptime

• Monitor
  A network endpoint which is monitored to track the performance and availability of applications and services.
• Uptime
  A metric of system reliability used to monitor the status of network endpoints via HTTP/S, TCP, and ICMP.

It looks like I broke the cross-doc includes for https://www.elastic.co/guide/en/observability/7.10/instrument-apps.html when updating the APM Overview quickstart guide. This needs to be fixed for 7.10.

Document User Experience app for 7.10. Not a new app. Just a view/tab in Observability. My notes are below.

Overall goal: Surface key user experience metrics and investigative tools.

Why is it important? One in four people abandons a website if it takes more than four seconds to load. Four in ten Americans give up shopping on a mobile site if it takes longer than three seconds to load. Amazon calculated that a page load slowdown of just one second could cost them $1.6B in sales each year. [source]

80% of what a user perceives as "load" time comes from things on the front-end or in-flight once a response leaves your infrastructure [[need source]]

Testing environments are great, but Elastic UE reflects the real-world user experience of actual users. With just a few lines of code, you can begin to see how users are experiencing your website and take steps to begin improving their experience.

How does it work? After page load, downloads a script, collects data, sends it to APM Server where it is processed, and then onto ES?

As JavaScript gets more and more complex, the need for user experience monitoring and error reporting increases. User experience provides a way to quantify three important aspects of your web application: loading performance, visual stability, and interactivity. You can drill down even further by looking at data by connection type, device type, operating system, browser, and even device variants -- all can have an impact on how your code preforms when an end-user is accessing your site.

High-level overview of your site's performance:

Backend: How much time is being spent on the back-end? How long does it take for your server to respond to the initial request?
Frontend: How much time is spent parsing and painting that content?
Pageviews: Pageviews in [[timeframe]]

High-level is useful for seeing the big picture, but we need to look at more granular metrics to understand performance and be able to fix potential problems.

The following metrics represent the user experience:

• First contentful paint: The timestamp when the browser begins rendering content (images, text, etc.) from the DOM. This is when the user can tell that the page is loading.
• Total blocking time: The total amount of time that the user is blocked from providing input to the page -- no mouse clicks, button presses, etc.
• Number of long tasks: Long tasks are periods of time where the main UI thread is working for 50ms or longer. This could be because of a rerender, event handler, or something else.
• Longest long task duration: The longest of the long tasks?
• Total long tasks duration: Sum duration of all long tasks

KEY CONCEPT --> Developers shouldn't have to become experts at interpreting and acting on metrics like paint timings, long tasks, and blocking times. They should spend their time coding. To help with this, Elastic has embraced...

Core Web Vitals: A recent initiative from Google to introduce a new set of metrics to better categorize good and bad sites by quantifying the real-world user experience. This is done by looking at loading performance, visual stability, and interactivity.

Important because starting in early 2021, Google will start using this as part of their ranking algorithm, and will open up the opportunity for websites to rank in the "top stories" position without needing to leverage AMP.

• Largest contentful paint: Loading performance. Measures the perceived load speed for a user as it measures the time to when the page's main content has likely loaded.
• First input delay: Load responsiveness. How quickly can users interact with your page?
• Cumulative layout shift: Visual stability. Are users experiencing unexpected layout shifts? Is content moving around because of async resource loading or dynamic content additions? These experiences can be frustrating for users.

WHO is on your website? OS/Browser fam/Geographic location --> massive impact on how visitors experience your website. This data can help you prioritize optimizations --
for example, for the most popular browsers visiting your site, or it could be used to gain a general understanding of when users are visiting and where they're visiting from.

• Page load distribution: Page load speeds broken down by percentile.
• Page views: Number of page views per bucket.
  Each can be segmented further by browser, OS, device, and location.

• Average page load duration by region: Average page load by country.
• Visitor breakdown: Website visitors segmented by browser and operating system

Variation in software/hardware makes it nearly impossible to test for every combination. UE removes this blindspot by surfacing JavaScript errors occurring on your website in production.

• JavaScript errors
  • Total Javascript errors
  • Error rate: ??
    Helps minimize noise -- one browser might throw a lot of errors, but if most don't, you won't see it take over.
  • Sorted by the percentage of impacted page loads.

Filter and breakdown data by key groupings like OS/browser. Filter by URL or groups of URLs. Surface meaningful insight from cohorts of visitors instead of the masses. Helps you spot performance outliers.

We should consider moving the the Logs anomaly detection configurations from the Stack docs to the Monitor Logs section in the Observability Guide - preferably within the inspect-log-anomalies topic.

cc @szabosteve @lcawl

Currently the docs about Elastic Agent and Fleet (fka Ingest Manager) are buried under the Ingest Management Guide in the Beats documentation. The docs need to be more prominent. There's not an ideal location on the existing doc landing page, which is siloed by product. This problem will be solved by the site redesign that's in progress, but it might take awhile. In the mean time, should we consider moving the content to a more visible location in https://www.elastic.co/guide/index.html?

We need to think about the security and endpoint use cases, too. Let's discuss the possibilities in this issue.

Assigning this to all writers who are impacted, plus the ingest management PM. Feel free to add participants.

Work with Alexander Reelsen to add the Elastic Observability sample Java app tutorial to the documentation.

• Move steps about running as standalone out of the main "quick start" docs and into its own topic.
• Add a statement to the topic about running standalone to indicate that automatic agent updates are not supported if you're running standalone.
• Update steps to use new install command.

Note that screen captures were updated in a different PR, but they'll need to be confirmed in case UI changes got in after the first BC.

Summary

The following links will break in 7.10 and need to be fixed before the release.

• .Net Agent elastic/apm-agent-dotnet#1004 @bmorelli25

11:31:40 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/apm/agent/dotnet/1.x/log-correlation.html:
11:31:40 INFO:build_docs:   - en/kibana/7.10/xpack-logs.html
11:31:40 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/apm/agent/dotnet/current/log-correlation.html:
11:31:40 INFO:build_docs:   - en/kibana/7.10/xpack-logs.html
11:31:40 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/apm/agent/dotnet/master/log-correlation.html:
11:31:40 INFO:build_docs:   - en/kibana/7.10/xpack-logs.html

• Go Agent elastic/apm-agent-go#839 @bmorelli25

11:31:40 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/apm/agent/go/1.x/log-correlation.html:
11:31:40 INFO:build_docs:   - en/kibana/7.10/xpack-logs.html
11:31:40 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/apm/agent/go/current/log-correlation.html:
11:31:40 INFO:build_docs:   - en/kibana/7.10/xpack-logs.html
11:31:40 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/apm/agent/go/master/log-correlation.html:
11:31:40 INFO:build_docs:   - en/kibana/7.10/xpack-logs.html

• Java tutorial #217 @EamonnTP

11:31:40 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/observability/7.10/monitor-java-app.html:
11:31:40 INFO:build_docs:   - en/kibana/current/lens.html
11:31:40 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/observability/7.x/monitor-java-app.html:
11:31:40 INFO:build_docs:   - en/kibana/current/lens.html
11:31:40 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/observability/current/monitor-java-app.html:
11:31:40 INFO:build_docs:   - en/kibana/current/lens.html
11:31:40 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/observability/master/monitor-java-app.html:
11:31:40 INFO:build_docs:   - en/kibana/current/lens.html

• ECS elastic/ecs#1073 / elastic/ecs#1074 @EamonnTP

14:44:53 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/ecs/1.3/ecs-products-solutions.html:
14:44:53 INFO:build_docs:   - en/logs/guide/7.10/logs-app-overview.html
14:44:53 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/ecs/1.4/ecs-products-solutions.html:
14:44:53 INFO:build_docs:   - en/logs/guide/7.10/logs-app-overview.html
14:44:53 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/ecs/1.5/ecs-getting-started.html:
14:44:53 INFO:build_docs:   - en/logs/guide/7.10/logs-fields-reference.html
14:44:53 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/ecs/1.5/ecs-products-solutions.html:
14:44:53 INFO:build_docs:   - en/logs/guide/7.10/logs-app-overview.html
14:44:53 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/ecs/1.6/ecs-getting-started.html:
14:44:53 INFO:build_docs:   - en/logs/guide/7.10/logs-fields-reference.html
14:44:53 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/ecs/1.6/ecs-products-solutions.html:
14:44:53 INFO:build_docs:   - en/logs/guide/7.10/logs-app-overview.html
14:44:53 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/ecs/1.7/ecs-getting-started.html:
14:44:53 INFO:build_docs:   - en/logs/guide/7.10/logs-fields-reference.html
14:44:53 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/ecs/1.7/ecs-products-solutions.html:
14:44:53 INFO:build_docs:   - en/logs/guide/7.10/logs-app-overview.html
14:44:53 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/ecs/1.x/ecs-getting-started.html:
14:44:53 INFO:build_docs:   - en/logs/guide/7.10/logs-fields-reference.html
14:44:53 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/ecs/1.x/ecs-products-solutions.html:
14:44:53 INFO:build_docs:   - en/logs/guide/7.10/logs-app-overview.html
14:44:53 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/ecs/current/ecs-getting-started.html:
14:44:53 INFO:build_docs:   - en/logs/guide/7.10/logs-fields-reference.html
14:44:53 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/ecs/current/ecs-products-solutions.html:
14:44:53 INFO:build_docs:   - en/logs/guide/7.10/logs-app-overview.html
14:44:53 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/ecs/master/ecs-getting-started.html:
14:44:53 INFO:build_docs:   - en/logs/guide/7.10/logs-fields-reference.html
14:44:53 INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/ecs/master/ecs-products-solutions.html:
14:44:53 INFO:build_docs:   - en/logs/guide/7.10/logs-app-overview.html

Related

For elastic/docs#1994.
Full console: https://elasticsearch-ci.elastic.co/job/elastic+docs+pull-request+build/288/console.

The APM stack docs content needs to be replaced with Observability:
https://www.elastic.co/guide/en/elastic-stack/7.9/elastic-stack-highlights.html

APM's What's new section needs to be deprecated and linked to the new Obs section:
https://www.elastic.co/guide/en/apm/get-started/7.9/whats-new.html

Hi, internal and external questions are arising about what Ingest Manger and Agent are doing in terms of where data is coming from and how it gets to where it goes for setup, etc. Users familiar with Beats may have an incorrect inherited assumption, and so we can document it all together, in one place, and in a way to clarify and inform new-to-Beats users, too.

Content is below, tho you can frame it all as desired in the Docs, I don't think it needs to be in the FAQ or listed as Q&A necessarily:

Does the Elastic Agent download or does Kibana download the Integration packages needed from the internet?

answer: Kibana connects to the Elastic package registry at epr.elastic.co and downloads the Packages and stores their assets in Elasticsearch. This done for the user now in Ingest Manager as opposed to configuring and executing it manually as in stand-alone Elastic Beats usage.

Does Elastic Agent download anything at all from the Internet?

answer: No, with one special exception. Since Metricbeat and Filebeat are bundled with the Elastic Agent they are on the host when Agent is installed. The special case is the Elastic Endpoint, which is downloaded by the Agent based OS and architecture from the Elastic download site, when Elastic Agent Policy is set to include it. Bundling Elastic Endpoint together with Elastic Agent is a known feature request scoped for a future release (link to issue?)

Do the Beats that Agent configures on the host send assets into Elasticsearch like stand-alone beats do?

answer: No, this is handled by Ingest Manager now. For the example of the stand-alone Elastic Beat Nginx module, the Integrations packages on epr.elastic.co contain what used to be in the Filebeat or Metricbeat module. That is, the Elasticsearch and Kibana assets. As the assets are now managed by Ingest Manager, there is no need for the Elastic Agent configured Beats to run their setup step and send assets to Elasticsearch and/or Kibana.

The Uptime monitoring guide currently lives in the Kibana repo. We should move this content to the observability-docs repo to live with the other Observability content.

• All branches, from master-7.2 need to be moved.
• Unfortunately, there will not be a way to maintain the history of the content.
• Once moved, the master content should be deleted from the Kibana repo.

cc @elastic/uptime for any concerns or feedback

As a prerequisite to the Observability documentation overhaul, we should begin to:

• Separate non-reference content from reference content
• Improve TOC headings to be easier to scan
• Sort lists in a consistent manner (alphabetically)

Beats prereqs

• Make update beats docs elastic/apm-server#3459
• Simplify Beats documentation headers (and use consistent wording) elastic/beats#16825

APM Server work

• Break configuration section into two different sections: Configuration & How to
• Sort lists alphabetically where feasible
• manage storage section is still a mix of reference/guide
• Improve scanability of the TOC by adding shorter nav titles with title abbreviations to APM specific topics:

== Long descriptive page title for SEO
++++
<titleabbrev>Short TOC header</titleabbrev>
++++

Related Issues

• Beats Reference content: elastic/beats#16821
• Beats restructure and refactor: elastic/beats#16825

The Kibana writers would like us to move the Observability products under the Observability heading in the Kibana reference. The APM content cannot be moved, but Logs, Metrics, and Uptime can. This move will help up clean up their long TOC, without requiring us to refactor any APM content.

The idea discussed would be to have all Obseravability solutions on one page, with links to the docs:

• Observability
  • Observability overview
  • APM (link to kib ref)
  • Logs (link to obs docs)
  • Metrics (link to obs docs)
  • Uptime (link to obs docs)

The new Kibana TOC would then look something like this:

• ...
• Graph
• Visualize
• Observability
• APM
• Elastic Security
• ...

Summary

We should add a how-to guide to the APM Overview that explains how to create custom transactions and spans to instrument custom methods.

Prerequisites

• Tab widget
• Unified nav

Related links

• Java custom instrumentation: https://www.elastic.co/guide/en/apm/agent/java/master/java-method-monitoring.html
• RUM: https://www.elastic.co/guide/en/apm/agent/rum-js/current/custom-transactions.html
• Go: https://www.elastic.co/guide/en/apm/agent/go/current/custom-instrumentation.html
• .NET: (API only) https://www.elastic.co/guide/en/apm/agent/dotnet/current/public-api.html
• Node.js: https://www.elastic.co/guide/en/apm/agent/nodejs/current/custom-transactions.html
• Python: https://www.elastic.co/guide/en/apm/agent/python/current/instrumenting-custom-code.html
• Ruby: https://www.elastic.co/guide/en/apm/agent/ruby/current/custom-instrumentation.html

General release tasks

• What's new in Observability 7.11
• Stack docs copy of What's new
• APM Server
  • Release notes
  • Breaking changes
  • Upgrade guide
• Beats
  • Release notes
  • Breaking changes
  • Upgrade guide

Brandon's list of tasks

• Node.js agent dynamic updates elastic/apm-agent-nodejs#1883 elastic/apm-agent-nodejs#1975
• Python agent "added in" elastic/apm-agent-python#983 elastic/apm-agent-python#984
• Update agent instructions in the APM app elastic/kibana#77739 elastic/kibana#85855
• [APM] Document known issue with Dependencies Table for RUM services elastic/kibana#88202
• Reupdate APM screenshots -- elastic/kibana#88101 changed some of the terminologies used in the APM app. Screenshots + docs should be updated again
• Working from elastic/beats#23116, add application logging to the Filebeat quickstart.
• If ifdef/screenshots for videos: #323
• Update APM Kubernetes content (if applicable)

Create documentation for experimental synthetics monitoring. TBD what this will look like. It could live in the Observability documentation and look like this :

• Quickstart
• Examples / cookbook
• Partial CLI reference
• App explanation

My notes are below.

Implementation

• Real browser synthetic agent (experimental)
  • Deployed as a library (so you can invoke from CL) and also as a docker container, that combines Heartbeat with Real Browser Agent.
    • Implemented as a new Heartbeat monitor. HB is responsible for invocation, but then hands off to a node.js library. Can also run in standalone mode without HB/Kib -- write JS scripts, test them locally, and then when you're ready to run on a schedule, put it together in HB which will run and send to Kib.
  • Scripted multi-step journeys that interact with a webpage
  • Up/down status per step with screenshots in the Uptime app. Error steps will have a comparison to what it looked like when passing, and a stack trace.

Why?

Uptime checks that ensure your app is up-and-running are only part of the equation. Key user journeys, like signing up, adding items to a cart, and checking out should also be monitored to prevent bugs from taking down your system and costing you $.

Synthetic monitoring alerts you when these key user journeys break -- so you can fix them quickly and avoid costly downtime or lost sales.

Synthetic monitoring can also alert you when these user journeys are slower than normal. Engineers can then navigate to the User Experience tab to determine where things might be going wrong, and what might be causing that slowdown. Finally, you can manually navigate to the browser to debug locally.

Synthetic monitoring can also be used for tests and maintenance of APIs. These tests monitor product code paths, ensuring that the functionality and performance of the APIs don't degrade.

Proactive. Controlled. Stable. Enforce SLAs. Find regressions. Ensure a reliable and consistent user experience, free of broken features and long loading times.

We added a custom banner to the Observability Guide to help users understand that the content was for a future a release (our current branch/version strategy made it necessary to mark master as current).

At 7.9 GA, we need to remove the banner added by #79.

See style issue 169 and updated term usage guide

It looks like we can no longer include the generic ess-getting-started file from the docs repo. There is now an additional step of selecting Elastic Observability, which is not included in the file.

Should we write the steps involved or do we need to create a specific 'set up on Cloud' topic for Observability, that we then include from the Observability repo?

cc @bmorelli25 @dedemorton

As part of the 7.10 release, the logs/metrics/uptime content is included in the Observability Guide. This issue is to create redirects from the monitoring guides to the new locations.

Redirects are listed here.

Related PRs

#162
#134
#129
elastic/kibana#79978