Code Monkey home page Code Monkey logo

escli's Introduction

BACKGROUND

This repo includes a CLI tool to simplify interacting with the Elasticsearch REST API. It makes a couple of assumptions:

  • On OSX you have the following installed via brew:

    • brew install coreutils provides:
      • greadlink
      • gdate
      • gpaste
    • brew install gnu-sed provides:
      • gsed
    • brew install bash for bash version 4 or later
      • grep -q "/usr/local/bin/bash" /etc/shells || echo '/usr/local/bin/bash' | sudo tee -a /etc/shells to add the brew installed bash to the list of approved shells
      • chsh -s /usr/local/bin/bash to switch to the brew installed bash shell permanently
  • (Preferred) Your credentials are stored in LastPass

  • (Preferred) You have the LastPass CLI tool installed, lpass via brew (brew install lastpass-cli)

  • (Alternative) Use some other method to provide credentials (see escli.conf.sample for other methods)

  • Make a copy of escli.conf.sample to escli.conf and customize (follow guidances within file)

  • Optionally make a copy of escli_c.conf.sample to escli_c.conf and customize

STRUCTURE

There is 1 CLI tool escli.bash which is then linked as esl and esp to point to a 'lab' and 'production' instances of Elasticsearch clusters.

The CLI tool escli.bash has a single configuration file, escli.conf. Within the config file are the base URL for interacting with the 2 ES clusters, the "Content Type" header to use, and 2 commands for acquiring the username + password to use when interacting with the ES clusters.

The es_funcs.bash includes Bash functions which can be sourced into your shell and executed. These functions all leverage the base CLI tool, escli.bash.

Finally there's a file cmds.md which includes examples with output showcasing how to use the escli.bash CLI tool.

 $ ▶ tree -I '*ARCHIVE*|*WIP*' -L 3
.
├── EXAMPLES.md
├── LICENSE
├── README.md
├── cmds.md
├── es_funcs.bash
├── esc -> escli.bash
├── escli.bash
├── escli.conf
├── escli.conf.sample
├── escli_c.conf.sample
├── esl -> escli.bash
├── esp -> escli.bash
├── gen_EXAMPLES.bash
├── kbc -> escli.bash
├── kbl -> escli.bash
├── kbp -> escli.bash
├── zencli.bash
├── zencli.conf.sample
├── zsl -> zencli.bash
└── zsp -> zencli.bash

1 directory, 20 files

USAGE

escli.bash

The escli.bash has several "personalities" on which it can operate against Elastic Search & Kibana URLs. Additionally it can operate against different instances' environments. For example, when you want to use the "lab" persona, you'd invoke it using the ./esl symbolic link to target the Elastic lab instance.

 $ ▶ ./esl

    USAGE: ./esl [HEAD|GET|PUT|POST] '...ES/KB REST CALL...'

    EXAMPLES:

        ./esl GET  '_cat/shards?pretty'
        ./esl GET  '_cat/indices?pretty&v&human'
        ./esl GET  '_cat'
        ./esl GET  ''
        ./esl PUT  '_all/_settings'   -d "$DATA"
        ./esl POST '_cluster/reroute' -d "$DATA"


Alternatively, when wanting to use the "production" persona, you'd invoke it using the ./esp symbolic link.

$ ▶ ./esp

    USAGE: ./esp [HEAD|GET|PUT|POST] '...ES/KB REST CALL...'

    EXAMPLES:

        ./esp GET  '_cat/shards?pretty'
        ./esp GET  '_cat/indices?pretty&v&human'
        ./esp GET  '_cat'
        ./esp GET  ''
        ./esp PUT  '_all/_settings'   -d "$DATA"
        ./esp POST '_cluster/reroute' -d "$DATA"


Additionally if you maintain a cluster that's hosted via Elastic's found.io domain you can use the ./esc symbolic link.

$ ▶ ./esc

    USAGE: ./esc [HEAD|GET|PUT|POST] '...ES/KB REST CALL...'

    EXAMPLES:

        ./esc GET  '_cat/shards?pretty'
        ./esc GET  '_cat/indices?pretty&v&human'
        ./esc GET  '_cat'
        ./esc GET  ''
        ./esc PUT  '_all/_settings'   -d "$DATA"
        ./esc POST '_cluster/reroute' -d "$DATA"


When targeting the Kibana "lab" or "production" environments you'd use either ./kbl or ./kbp.

 $ ▶ ./kbl

    USAGE: ./kbl [HEAD|GET|PUT|POST] '...ES/KB REST CALL...'

    EXAMPLES:

        ./kbl GET  '_cat/shards?pretty'
        ./kbl GET  '_cat/indices?pretty&v&human'
        ./kbl GET  '_cat'
        ./kbl GET  ''
        ./kbl PUT  '_all/_settings'   -d "$DATA"
        ./kbl POST '_cluster/reroute' -d "$DATA"


 $ ▶ ./kbp

    USAGE: ./kbp [HEAD|GET|PUT|POST] '...ES/KB REST CALL...'

    EXAMPLES:

        ./kbp GET  '_cat/shards?pretty'
        ./kbp GET  '_cat/indices?pretty&v&human'
        ./kbp GET  '_cat'
        ./kbp GET  ''
        ./kbp PUT  '_all/_settings'   -d "$DATA"
        ./kbp POST '_cluster/reroute' -d "$DATA"


es_funcs.bash

If you'd like to make use of the helper functions within the file es_funcs.bash you simply source this file into your existing shell.

$ ▶ . es_funcs.bash
$ ▶ 

This is a list of the functions it provides:

$ ▶ escli_ls
gen_README
cmp_README
mk_README
gen_EXAMPLES
cmp_EXAMPLES
mk_EXAMPLES
calc_date
calc_hour
calc_date_1daybefore
calc_date_1dayafter
calc_millis_date
julian_day
ceiling_divide
escli_ls
escli_lsl
list_node_name_suffixes_usage_helper
help_cat
help_indices
list_nodes
list_nodes_storage
list_nodes_zenoss_alarms
show_nodes_fs_details
show_nodes_circuit-breaker_summary
show_nodes_circuit-breaker_details
show_nodes_threadpools_active_rejected
show_nodes_threadpools_details
show_nodes_threadpools_summary
show_shards
show_big_shards
show_small_shards
show_hot_shards
show_shard_usage_by_node
relo_shard
cancel_relo_shard
cancel_relo_shards_all
retry_unassigned_shards
show_shard_distribution_by_node_last3days
show_hot_idxs_shard_distribution_by_node
show_shards_biggerthan55gb
show_idx_with_oversized_shards_summary
show_idx_with_oversized_shards_details
show_rebalance_throttle
show_node_concurrent_recoveries
show_cluster_concurrent_rebalance
increase_rebalance_throttle_XXXmb
increase_node_concurrent_recoveries
increase_cluster_concurrent_rebalance
reset_rebalance_throttle
reset_node_concurrent_recoveries
reset_cluster_concurrent_rebalance
change_allocation_threshold
increase_node_recovery_allocations
reset_node_recovery_allocations
show_recovery
show_recovery_full
enable_readonly_idx_pattern
disable_readonly_idx_pattern
enable_readonly_idxs
disable_readonly_idxs
show_readonly_idxs
show_readonly_idxs_full
clear_readonlyallowdel_idxs
set_idx_default_field
set_template_default_field
set_idx_shards_per_node
set_idx_max_docvalue_fields_search
set_idx_num_replicas_to_X
estop
estop_recovery
estop_relo
estop_tasks
estop_rejected_writes
estop_active_threads
estop_idx_indexing
estop_node_indexing
estop_unassigned_shards
show_health
show_cluster_stats
show_watermarks
show_state
showcfg_cluster
showrecov_stats
showrecov_hot_threads
showrecov_idx_shard_stats
show_stats_cluster
show_tasks_stats
show_tasks_pending
show_tasks_descriptions
verify_idx_retentions
show_idx_retention_violations
show_idx_doc_sources_1st_10k
show_idx_doc_sources_all_cnts
show_idx_doc_sources_all_k8sns_cnts
show_idx_doc_sources_all_k8sns_cnts_hourly
show_es_eol
show_es_ecs
showcfg_num_shards_per_idx
showcfg_shard_allocations
explain_allocations
explain_allocations_hddinfo
show_shard_routing_allocation
enable_shard_allocations
disable_shard_allocations
clear_shard_allocations
show_idx_sizes
show_idx_stats
show_idx_create_timestamps_utc
show_idx_create_timestamps_localtz_last20
show_idx_types
show_idx_last10
delete_idx
showcfg_idx_cfgs
showcfg_idx_stats
show_idx_version_cnts
show_idx_mappings
clear_idx_cache_fielddata
clear_idx_cache_query
clear_idx_cache_request
clear_idx_cache_all
list_index_metric_types
show_field_capabilities
show_fields_multiple_defs_summary
show_fields_multiple_defs_details
show_field_X_multiple_defs_details
show_field_names
show_field_counts
list_node_name_suffixes
show_excluded_nodes
exclude_node_name
clear_excluded_nodes
eswhoami
showcfg_auth_roles
showcfg_auth_rolemappings
list_auth_roles
list_auth_rolemappings
evict_auth_cred_cache
create_bearer_token
del_docs_k8s_ns_range
forcemerge_to_expunge_deletes
estail_deletebyquery
estail_forcemerge
calc_total_docs_hdd_overXdays
calc_daily_docs_hdd_overXdays
calc_idx_type_avgs_overXdays
calc_num_nodes_overXdays
list_ilm_policies
show_ilm_policy
show_ilm_policies
list_aliases
show_alias_details
show_alias_details_excludeEmpty
show_alias_for_idxs
list_writable_ilm_idxs_on_alias
show_writable_ilm_idxs_on_alias_details
explain_indexes_ilm
show_ilm_components_for_idx
bootstrap_ilm_idx
trigger_ilm_rollover
list_templates
show_template
show_template_idx_patterns
show_template_ilm_idx_alias_details
list_plugins
chk_s3_plugin_nodes

You can also get that list with a short description of each function:

$ ▶ escli_lsl

#0-----------------------------------------------
# helper funcs
##-----------------------------------------------
gen_README                                     # generate contents of README.md
cmp_README                                     # sdiff new README.md vs. existing README.md
mk_README                                      # save new README.md over existing README.md
gen_EXAMPLES                                   # generate content of EXAMPLES.md
cmp_EXAMPLES                                   # sdiff new EXAMPLES.md vs. existing EXAMPLES.md
mk_EXAMPLES                                    # save new EXAMPLES.md over existing EXAMPLES.md

#1-----------------------------------------------
# date & math funcs
##-----------------------------------------------
calc_date                                      # print UTC date X "days | days ago"
calc_hour                                      # print UTC date X "hours | hours ago"
calc_date_1daybefore                           # print UTC date 1 day before given date (YYYY-mm-dd)
calc_date_1dayafter                            # print UTC date 1 day after given date (YYYY-mm-dd)
calc_millis_date                               # convert UTC millis date to human format
julian_day                                     # calculate julian day based on a YYYYmmdd
ceiling_divide                                 # ceiling divide 2 numbers

#2-----------------------------------------------
# usage funcs
##-----------------------------------------------
escli_ls                                       # list function names
escli_lsl                                      # list function names + desc.
list_node_name_suffixes_usage_helper         local env="$1"

#3-----------------------------------------------
# help funcs
##-----------------------------------------------
help_cat                                       # print help for _cat API call
help_indices                                   # print help for _cat/indices API call

#4-----------------------------------------------
# node funcs
##-----------------------------------------------
list_nodes                                     # list ES nodes along w/ a list of data node suffixes for use by other cmds.
list_nodes_storage                             # list ES nodes HDD usage
list_nodes_zenoss_alarms                       # list ES node HDD usage alarms in Zenoss
show_nodes_fs_details                          # list ES nodes filesystem details
show_nodes_circuit-breaker_summary             # list ES nodes circuit breaker tripped summaries
show_nodes_circuit-breaker_details             # list ES nodes circuit breaker details
show_nodes_threadpools_active_rejected         # list ES nodes thread pool counts (_cat/thread_pool) ... any all zeros filtered out
show_nodes_threadpools_details                 # list ES nodes thread pool details (_cat/thread_pool)
show_nodes_threadpools_summary                 # list ES nodes thread pool (_cat/thread_pool)

#5-----------------------------------------------
# shard mgmt funcs
##-----------------------------------------------
show_shards                                    # list all the index shards sorted by size (big->small)
show_big_shards                                # list top 20 shards for a given node's suffix (1a, 1b, etc.)
show_small_shards                              # list smallest 20 shards for a given node's suffix (1a, 1b, etc.)
show_hot_shards                                # list today's "hot" shards for a given node's suffix (1a, 1b, etc.)
show_shard_usage_by_node                       # list all the index shards sorted by size (big->small)
relo_shard                                     # move an indices' shard from node suffix X to node suffix Y
cancel_relo_shard                              # cancel move of an index shard from node suffix X
cancel_relo_shards_all                         # cancel all shard RELOCATIONS in recovery queue
retry_unassigned_shards                        # reallocate all unassignable shards (elapsed past 5 retries)

#6-----------------------------------------------
# shard size analysis funcs
##-----------------------------------------------
show_shard_distribution_by_node_last3days      # show distribution of day X's shards across nodes
show_hot_idxs_shard_distribution_by_node       # show distribution of today's hot index shards across nodes
show_shards_biggerthan55gb                     # show shards which are > 55GB (too big)
show_idx_with_oversized_shards_summary         # show summary of indexes w/ shards > 55GB (too big)
show_idx_with_oversized_shards_details         # show detailed view of indexes w/ shards > 55GB (too big)

#7-----------------------------------------------
# increase/decrease relo/recovery throttles
##-----------------------------------------------
show_rebalance_throttle                        # show routing allocations for rebalancing & recoveries (current)
show_node_concurrent_recoveries                # show cluster.routing.allocation.node_concurrent_recoveries
show_cluster_concurrent_rebalance              # show cluster.routing.allocation.cluster_concurrent_rebalance
increase_rebalance_throttle_XXXmb              # change bytes_per_sec routing allocations for rebalancing & recoveries (throttle, just b/w)
increase_node_concurrent_recoveries            # change cluster.routing.allocation.node_concurrent_recoveries
increase_cluster_concurrent_rebalance          # change cluster.routing.allocation.cluster_concurrent_rebalance
reset_rebalance_throttle                       # reset routing allocations for rebalancing & recoveries (throttle default)
reset_node_concurrent_recoveries               # reset cluster.routing.allocation.node_concurrent_recoveries
reset_cluster_concurrent_rebalance             # reset cluster.routing.allocation.cluster_concurrent_rebalance
change_allocation_threshold                    # override the allocation threshold (cluster.routing.allocation.balance.threshold)

#8-----------------------------------------------
# node recovery funcs
##-----------------------------------------------
increase_node_recovery_allocations             # optimal recovery/rebalance settings when a node gets re-introduced to cluster
reset_node_recovery_allocations                # resets to default recovery/rebalance settings

#9-----------------------------------------------
# recovery funcs
##-----------------------------------------------
show_recovery                                  # show a summary of recovery queue
show_recovery_full                             # show full details of recovery queue
enable_readonly_idx_pattern                    # set index read_only flag for pattern of indices
disable_readonly_idx_pattern                   # clear index read_only flag for pattern of indices
enable_readonly_idxs                           # set index read_only flag
disable_readonly_idxs                          # clear index read_only flag
show_readonly_idxs                             # show indexes' read_only setting which are enabled (true)
show_readonly_idxs_full                        # show indexes' read_only setting for all indices
clear_readonlyallowdel_idxs                    # clear read_only_allow_delete flag
set_idx_default_field                          # set index.query.default_field => [ "*" ]
set_template_default_field                     # set template index.query.default_field => [ "*" ]
set_idx_shards_per_node                        # set index.routing.allocation.total_shards_per_node = X
set_idx_max_docvalue_fields_search             # set index.max_docvalue_fields_search = X
set_idx_num_replicas_to_X                      # set an index's number_of_replicas to X

#10-----------------------------------------------
# health/stat funcs
##-----------------------------------------------
estop                                          # mimics `top` command, watching ES nodes CPU/MEM usage
estop_recovery                                 # watches the ES recovery queue
estop_relo                                     # watches ES relocations
estop_tasks                                    # watches ES tasks
estop_rejected_writes                          # watches ES write thread pools for rejected writes (EsRejectedExecutionException)
estop_active_threads                           # watches ES thread pools for active/rejected activities
estop_idx_indexing                             # watches ES indexing activities for indexes
estop_node_indexing                            # watches ES indexing activities for nodes
estop_unassigned_shards                        # watches ES shards that are UNASSIGNED
show_health                                    # cluster's health stats
show_cluster_stats                             # show cluster stats (_cluster/stats?pretty&human)
show_watermarks                                # show watermarks when storage marks readonly
show_state                                     # shows the state of the indicies' shards (RELO, Translog, etc.)
showcfg_cluster                                # show all '_cluster/settings' configs
showrecov_stats                                # show recovery stats (_recovery)
showrecov_hot_threads                          # show hot thread details
showrecov_idx_shard_stats                      # show an index's shard stats
show_stats_cluster                             # shows the _stats for entire cluster
show_tasks_stats                               # shows the tasks queue
show_tasks_pending                             # shows the pending tasks queue
show_tasks_descriptions                        # shows the description of queued tasks
verify_idx_retentions                          # shows the distribution of index retentions (days per index type & version)
show_idx_retention_violations                  # shows the indexes which fall outside a given retention window (days)
show_idx_doc_sources_1st_10k                   # show the hostnames that sent documents to an index
show_idx_doc_sources_all_cnts                  # show the total num. docs each hostname sent to an index
show_idx_doc_sources_all_k8sns_cnts            # show the total num. docs each namespace sent to an index
show_idx_doc_sources_all_k8sns_cnts_hourly     # show the total num. docs each namespace sent to an index (last 3 hours, top 50 NS')
show_es_eol                                    # show EOL for various ES products such as *beats, etc.
show_es_ecs                                    # show EOL for various ES products such as *beats, etc.

#11----------------------------------------------
# shard funcs
##-----------------------------------------------
showcfg_num_shards_per_idx                     # show number of shards configured per index template
showcfg_shard_allocations                      # show cluster level shard allocation configs
explain_allocations                            # show details (aka. explain) cluster allocation activity
explain_allocations_hddinfo                    # show details (aka. explain) cluster allocation activity (full)
show_shard_routing_allocation                  # show status (cluster.routing.allocation.enable)
enable_shard_allocations                       # allow the allocator to route shards (cluster.routing.allocation.enable)
disable_shard_allocations                      # disallow the allocator to route shards (cluster.routing.allocation.enable)
clear_shard_allocations                        # clear the allocator to route shards (cluster.routing.allocation.enable)

#12----------------------------------------------
# index stat funcs
##-----------------------------------------------
show_idx_sizes                                 # show index sizes sorted (big -> small)
show_idx_stats                                 # show index stats sorted (big -> small)
show_idx_create_timestamps_utc                 # show index creation timestamps sorted (oldest -> newest) for indexes created
show_idx_create_timestamps_localtz_last20      # show index creation timestamps sorted (oldest -> newest) for last 20 indexes created
show_idx_types                                 # show idx types [beat type] - [retention period] - [beat version]
show_idx_last10                                # show last 10 indexes (by date) for a given idx pattern
delete_idx                                     # delete an index, asks for confirmation to delete indices
showcfg_idx_cfgs                               # show all '<index name>/_settings' configs
showcfg_idx_stats                              # show all '<index name>/_stats'
show_idx_version_cnts                          # show index sizes sorted (big -> small)
show_idx_mappings                              # show an index's _mappings (flattened) '<index name>/_mapping'
clear_idx_cache_fielddata                      # clear /_cache/clear?fielddata=true
clear_idx_cache_query                          # clear /_cache/clear?query=true
clear_idx_cache_request                        # clear /_cache/clear?request=true
clear_idx_cache_all                            # clear /_cache/clear
list_index_metric_types                        # list ES index metric types

#13----------------------------------------------
# field funcs
##-----------------------------------------------
show_field_capabilities                        # show field capabilities (type, searchable, aggregatable) for index pattern
show_fields_multiple_defs_summary              # list of fields with multiple capabilities defs. for index pattern
show_fields_multiple_defs_details              # detailed view of fields with multiple capabilities defs. for index pattern
show_field_X_multiple_defs_details             # detailed view of a single field's multiple capabilities defs. for index pattern
show_field_names                               # Return a list of fields in a given index pattern
show_field_counts                              # Return a count of fields in a given index pattern

#14----------------------------------------------
# node exclude/include funcs
##-----------------------------------------------
list_node_name_suffixes                        # show node name suffixes
show_excluded_nodes                            # show excluded nodes from cluster
exclude_node_name                              # exclude a node from cluster (node suffix)
clear_excluded_nodes                           # clear any excluded cluster nodes

#15----------------------------------------------
# auth funcs
##-----------------------------------------------
eswhoami                                       # show auth info about who am i
showcfg_auth_roles                             # show auth info about roles
showcfg_auth_rolemappings                      # show auth info about role mappings
list_auth_roles                                # list all roles
list_auth_rolemappings                         # list all rolemappings
evict_auth_cred_cache                          # evict/clear users from the user cache
create_bearer_token                            # create bearer token for user

#16----------------------------------------------
# k8s namespace funcs
##-----------------------------------------------
del_docs_k8s_ns_range                          # delete k8s namespace docs over a specific time range
forcemerge_to_expunge_deletes                  # force merge of shards to expunge deleted docs
estail_deletebyquery                           # watch deletebyquery tasks
estail_forcemerge                              # watch forcemerges in tasks queue

#17----------------------------------------------
# capacity planning functions
##-----------------------------------------------
calc_total_docs_hdd_overXdays                  # calc. the total docs & HDD storage used by all indices over X days
calc_daily_docs_hdd_overXdays                  # calc. the individual daily total docs & HDD storage used by all indices over X days
calc_idx_type_avgs_overXdays                   # calc. the avg number of docs & HDD storage used per idx types over X days
calc_num_nodes_overXdays                       # calc. the HDD storage required based on idx types usage over X days

#18----------------------------------------------
# ilm funcs
##-----------------------------------------------
list_ilm_policies                              # show all _ilm/policy names
show_ilm_policy                                # show a single _ilm/policy/<policy> details
show_ilm_policies                              # show all _ilm/policy details
list_aliases                                   # show all _alias names
show_alias_details                             # show all _alias details
show_alias_details_excludeEmpty                # show all _alias that are not '"aliases": {}'
show_alias_for_idxs                            # shows alias name & which index is writable for a given idx pattern
list_writable_ilm_idxs_on_alias                # show names of idxs where 'is_write_index: true' on aliases
show_writable_ilm_idxs_on_alias_details        # show verbose which idxs are 'is_write_index: true' on aliases
explain_indexes_ilm                            # explain ilm for given indexes '<index pattern>/_ilm/explain'
show_ilm_components_for_idx                    # show ilm for given index '<index pattern>/_ilm/explain'
bootstrap_ilm_idx                              # creates an index and designates it as the write index for an alias
trigger_ilm_rollover                           # trigger ILM to rollover current index via alias

#19----------------------------------------------
# template funcs
##-----------------------------------------------
list_templates                                 # show all template details
show_template                                  # show template X's details
show_template_idx_patterns                     # show index_patterns for templates that match provided pattern
show_template_ilm_idx_alias_details            # show index_patterns, ilm-policy, & rollover alias for templates that match provided pattern

#20----------------------------------------------
# plugin funcs
##-----------------------------------------------
list_plugins                                   # show all plugins installed on cluster
chk_s3_plugin_nodes                            # check that each node reports having access to s3 plugin


Each function includes a 'show usage' if you run it without any arguments. For example:

$ ▶ list_nodes

USAGE: list_nodes [l|p|c]

Most of the functions will take a single argument, either l or p or c to denote which cluster you want it to target. A handful of functions can take additional items, such as relo_shard & delete_idx. Consult their usage for more details.

WORKFLOWS

Deleting docs from an index

$ ▶ del_docs_k8s_ns_range

USAGE: del_docs_k8s_ns_range [l|p|c] <idx pattern> <k8s namespace> <start time> <end time>


  * TIME FORMAT: 2019-07-10T00:00:00.000Z

  * INDX FORMAT:
      -- filebeat-*
      -- -or- filebeat-6.5.1-2019.07.04,filebeat-6.5.1-2019.07.05,....
      -- -or- filebeat-*-2019.07*


    ------------------------------------------------------------------------------------------------------

    Example
    =======
    $ del_docs_k8s_ns_range l filebeat-* big-dipper-perf 2019-07-11T11:57:20.968Z 2019-07-12T04:26:38.757Z
    {"task":"vudQxvnfSQuxMtdkq8ZTUQ:844209600"}

    ------------------------------------------------------------------------------------------------------

        Source: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html


$ del_docs_k8s_ns_range l filebeat-6.5.1-2019.07.31 big-dripper 2019-07-31T13:58:29.145Z 2019-07-31T17:40:00.000Z
{"task":"vudQxvnfSQuxMtdkq8ZTUQ:2390166372"}

$ del_docs_k8s_ns_range l filebeat-6.5.1-2019.07.31 flex-generator 2019-07-31T13:58:29.145Z 2019-07-31T17:40:00.000Z
{"task":"vudQxvnfSQuxMtdkq8ZTUQ:2390297564"}
$ ▶ estail_deletebyquery l
estail_deletebyquery
===================================
indices:data/write/delete/byquery  transport  43m    lab-rdu-es-data-01a
indices:data/write/delete/byquery  transport  42.8m  lab-rdu-es-data-01a
===================================
estail_deletebyquery
===================================
indices:data/write/delete/byquery  transport  43.2m  lab-rdu-es-data-01a
indices:data/write/delete/byquery  transport  43m    lab-rdu-es-data-01a
===================================
estail_deletebyquery
===================================
indices:data/write/delete/byquery  transport  43.3m  lab-rdu-es-data-01a
===================================
estail_deletebyquery
===================================
indices:data/write/delete/byquery  transport  43.5m  lab-rdu-es-data-01a
===================================
estail_deletebyquery
===================================
done
$ ▶ forcemerge_to_expunge_deletes l filebeat-6.5.1-2019.07.31
...
... Ctrl-C at any time, it's scheduled
...
$ ▶ estail_forcemerge l
estail_forcemerge
===================================
indices:admin/forcemerge     transport  15.1m  lab-rdu-es-data-01a
indices:admin/forcemerge[n]  direct     15.1m  lab-rdu-es-data-01a
indices:admin/forcemerge[n]  transport  15.1m  lab-rdu-es-data-01c
indices:admin/forcemerge[n]  transport  15.1m  lab-rdu-es-data-01b
===================================
estail_forcemerge
===================================
indices:admin/forcemerge     transport  15.3m  lab-rdu-es-data-01a
indices:admin/forcemerge[n]  direct     15.3m  lab-rdu-es-data-01a
indices:admin/forcemerge[n]  transport  15.3m  lab-rdu-es-data-01b
indices:admin/forcemerge[n]  transport  15.3m  lab-rdu-es-data-01c
===================================
...
...
estail_forcemerge
===================================
indices:admin/forcemerge     transport  24.6m  lab-rdu-es-data-01a
indices:admin/forcemerge[n]  direct     24.6m  lab-rdu-es-data-01a
indices:admin/forcemerge[n]  transport  24.6m  lab-rdu-es-data-01b
indices:admin/forcemerge[n]  transport  24.6m  lab-rdu-es-data-01c
===================================
estail_forcemerge
===================================
done

OTHER USES

Loading a template

$ ./esp PUT '_template/metrics-template' -d "$(cat metrics.template.json)" | jq .

Dumping all the templates to files

$ escli=~/projects/escli/esp
$ tmplList="$(${escli} GET '_cat/templates' | awk '/^[mfp].*beat|f5|syslog|messaging/ {print $1}')"
$ for i in $tmplList; do
    ${escli} GET "_template/${i}?pretty" > ${i}.tmp
done

REFERENCES

escli's People

Contributors

akrzos-bw avatar ecmingol avatar geoffmore avatar jaymingol avatar jeffspahr avatar jmainguy avatar odinu avatar slmingol avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

escli's Issues

show_hot_shards is wrong

Output is wrong:

$ show_hot_shards p 1a
index                                    shard prirep state           docs   store ip             node
filebeat-60d-6.5.1-2020.08.31-000072     1     r      STARTED     71001105  47.5gb 192.168.33.195 rdu-es-data-01a
f5-30d-2020.10.11-000081                 0     r      STARTED      2177348   1.1gb 192.168.33.195 rdu-es-data-01a
.monitoring-es-7-2020.10.11              0     r      STARTED       825922 663.9mb 192.168.138.72 rdu-es-data-01ab
.monitoring-logstash-7-2020.10.11        0     p      STARTED      3022696 624.5mb 192.168.138.72 rdu-es-data-01ab
.watcher-history-10-2020.10.11           0     r      STARTED         7678   6.5mb 192.168.138.76 rdu-es-data-01ad

Improve exclude_nodes cmds to manage a list of nodes and take add/remove lists of nodes

Today these cmds only can deal w/ 1 node. Improve em so you can give a single or csv list of short node prefixes and it'll go figure out IPs and add them to the exclude lists.

show_excluded_nodes                            # show excluded nodes from cluster
exclude_node_name                              # exclude a node from cluster (node suffix),
clear_excluded_nodes                           # clear any excluded cluster nodes

calc_daily_docs_hdd_overXdays doesn't work

$ calc_daily_docs_hdd_overXdays p 30


Indexes' Primmary Shard Analysis (Daily)
========================================
[DATE RANGE: 2020.09.12 - 2020.10.11]
========================================

-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
-bash: 09: value too great for base (error token is "09")
Date        JulianDay  TotalShards  TotalDocs   TotalStorageGBs  IdxCounts
====        =========  ===========  =========   ===============  =========
2020.09.12  35         1745704184   917         19
2020.09.13  48         2934427012   1521        22
2020.09.14  43         2386186156   1402        19
2020.09.15  53         3258441496   1855        21
2020.09.16  56         3487543046   2004        22
2020.09.17  45         2514423785   1446        21
2020.09.18  53         3305149871   1836        21
2020.09.19  44         2621189901   1382        20
2020.09.20  38         2040743109   1084        20
2020.09.21  64         3519911999   1856        24
2020.09.22  61         3577252809   1862        21
2020.09.23  64         3565482635   2120        22
2020.09.24  42         2646915452   1544        16
2020.09.25  50         3094615641   1750        22
2020.09.26  40         2240731492   1240        20
2020.09.27  50         3169991141   1709        22
2020.09.28  47         2591238167   1503        21
2020.09.29  55         3162834636   1898        23
2020.09.30  50         2848689173   1718        22
2020.10.01  2459124    62           3419021728  2190             25
2020.10.02  2459125    55           2698196306  1785             22
2020.10.03  2459126    51           2780930379  1562             22
2020.10.04  2459127    53           2639992933  1426             28
2020.10.05  2459128    66           3840777642  2165             31
2020.10.06  2459129    65           3374931882  1917             30
2020.10.07  2459130    68           4316970594  2501             33
2020.10.08  2459131    63           3713377580  2152             30
2020.10.09  2459132    58           3568554870  2043             29
2020.10.10  2459133    48           2467959221  1269             27
2020.10.11  2459134    15           14139435    3                15


show_idx_doc_sources_all_k8sns_cnts_hourly show range of hour usage instead of just 3hrs

Make this do a range of hours, say 3-6 hours instead of static 3hrs.

$ show_idx_doc_sources_all_k8sns_cnts_hourly p filebeat-60d*


k8s document sources (counts - hourly)
======================================



k8sns                             totalDocs   2020-10-11T13:00:00.000Z  2020-10-11T14:00:00.000Z  2020-10-11T15:00:00.000Z  2020-10-11T16:00:00.000Z
==========                        ==========  ==========                ==========                ==========                ==========
message-search                    30044371    3163261                   10570848                  9478162                   6832100
metrics-elasticsearch             29726277    2784447                   10303094                  10050351                  6588385
mdr-generator                     15793921    898376                    4926714                   5876174                   4092657
mdr-generator-uat                 15684453    895643                    4878013                   5830285                   4080512
stirrer                           14457999    818285                    4071963                   5127993                   4439758
http-voice-v2                     11154926    573337                    3497223                   4098046                   2986320
big-dipper                        9321138     556520                    2670072                   3441963                   2652583
vapor                             7887501     446068                    2218096                   2800567                   2422770
one-id                            2459552     201454                    821325                    823098                    613675
one-id-uat                        1400018     121914                    467735                    467498                    342871
gcac                              1215816     70591                     382601                    446618                    316006
kafka-c3                          1148928     144257                    416836                    332248                    255587
one-id-ldap                       1039210     90544                     347089                    347049                    254528
one-id-ldap-uat                   1039186     90571                     347063                    347073                    254479
datadog                           765584      65963                     255331                    257042                    187248
trunkgroup-consumer               714077      63832                     239532                    238260                    172453
kube-service-catalog              658243      55581                     222142                    218196                    162324
kafka-connect                     632554      53654                     224461                    222970                    131469
correlator-staging                309099      26597                     103149                    103344                    76009
ksql-server                       305850      4885                      71257                     163578                    66130
voice-media-service               281898      20572                     89036                     96238                     76052
tooling-pollers                   257364      22219                     85981                     85963                     63201
correlator                        248236      21635                     82716                     83189                     60696
ultratrace                        170062      14382                     54999                     58038                     42643
one-id-jfk-debug                  137935      11710                     46589                     46330                     33306
metricbeat                        110055      9593                      36757                     36759                     26946
openshift-sdn                     103233      8885                      33969                     34079                     26300
nirds                             92783       5668                      30660                     32167                     24288
kube-system                       84567       7375                      28150                     28073                     20969
filebeat                          78349       6234                      25882                     26663                     19570
vps                               77286       6732                      25800                     25817                     18937
moneymaker-uep                    70099       5929                      23390                     23615                     17165
clec-kafka-search                 60167       5224                      20107                     20103                     14733
moneymaker-provisioning           46098       4018                      15406                     15392                     11282
moneymaker-proxy                  43790       3793                      14425                     14658                     10914
one-id-okta-uat                   41715       3621                      13952                     13952                     10190
one-id-okta                       41640       3606                      13903                     13918                     10213
cnam                              41596       2227                      12419                     14333                     12617
gsx-sbc-poller                    38406       3285                      12822                     12830                     9469
openshift-monitoring              38283       3274                      12701                     12685                     9623
heartbeat                         35210       3058                      11597                     11853                     8702
apdialog                          34936       3150                      11826                     11533                     8427
awx                               31763       2758                      10604                     10636                     7765
backoffice-kafka-search           30623       2691                      10287                     10155                     7490
openshift-ansible-service-broker  27706       2427                      9248                      9255                      6776
metrics                           26079       2924                      8489                      8467                      6199
activecalls                       24747       2880                      8015                      7694                      6158
sysdig-agent                      20632       1613                      6488                      7273                      5258
moneymaker-discounting            20464       1786                      6832                      6832                      5014
identity-mgmt                     18303       1220                      6103                      6101                      4879


Adjust show_idx_with_oversized_shards_summary to 55GB instead of 50GB

With ILM indexes will typically be slightly bigger than 50GB. That's OK, should change threshold on this cmd to 55GB instead of 50GB to catch shards that are grossly larger than 50GB.

$ show_idx_with_oversized_shards_summary p


Daily Indicies w/ > 50GB shards

days  IdxType
----  -------
64    filebeat-60d-6.5.1
33    filebeat-60d-7.6.2
19    messaging-7d-6.5.1
22    packetbeat-60d-6.5.1
54    syslog-90d

`show_hot_idxs_shard_distribution_by_node` shows incorrect shards from old indexes

$ show_hot_idxs_shard_distribution_by_node p


[DATE: 2020.10.07]



node              indexType                               #shards
----              ---------                               -------
rdu-es-data-01a   filebeat-60d-7.6.2-000098               1
rdu-es-data-01a   filebeat-60d-7.6.2-000099               1
rdu-es-data-01a   metricbeat-45d-7.6.2-000079             1
rdu-es-data-01a   metricbeat-45d-7.8.0-2020.10.03-000075  1
rdu-es-data-01ab  filebeat-60d-7.6.2-000100               2
rdu-es-data-01ab  filebeat-60d-7.6.2-000101               3
rdu-es-data-01ab  syslog-90d-000093                       1
rdu-es-data-01ad  filebeat-30d-6.5.1-000076               1
rdu-es-data-01ad  filebeat-60d-7.6.2-000100               3
rdu-es-data-01ad  filebeat-60d-7.6.2-000101               2
rdu-es-data-01ad  messaging-7d-6.5.1-000127               1
rdu-es-data-01ad  syslog-90d-000093                       1
-----------------------------------------------------------------
                                                          4
...

This should only show the most recent instance in the ILM sequence.

rdu-es-data-01a   filebeat-60d-7.6.2-000098               1
rdu-es-data-01a   filebeat-60d-7.6.2-000099               1

Should be

rdu-es-data-01a   filebeat-60d-7.6.2-000099               1

`show_nodes_circuit-breaker_summary` shows incorrect summary of circuit breakers

Math in the summary cmd is broken:

$ show_nodes_circuit-breaker_summary p

node circuit breakers tripped counts
---------------------------------------------------
rdu-es-data-01a:    request:  0  fielddata:  0  in_flight_requests:  0  accounting:  0  parent:  0
rdu-es-data-01ab:   request:  0  fielddata:  0  in_flight_requests:  0  accounting:  0  parent:  0
rdu-es-data-01ad:   request:  0  fielddata:  0  in_flight_requests:  0  accounting:  0  parent:  0
rdu-es-data-01b:    request:  0  fielddata:  0  in_flight_requests:  0  accounting:  0  parent:  0
rdu-es-data-01c:    request:  0  fielddata:  0  in_flight_requests:  0  accounting:  0  parent:  0
rdu-es-data-01d:    request:  0  fielddata:  0  in_flight_requests:  0  accounting:  0  parent:  0
rdu-es-data-01e:    request:  0  fielddata:  0  in_flight_requests:  0  accounting:  0  parent:  0
rdu-es-data-01f:    request:  0  fielddata:  0  in_flight_requests:  0  accounting:  0  parent:  0
rdu-es-data-01g:    request:  0  fielddata:  0  in_flight_requests:  0  accounting:  0  parent:  0
rdu-es-data-01h:    request:  0  fielddata:  0  in_flight_requests:  0  accounting:  0  parent:  0
rdu-es-data-01i:    request:  0  fielddata:  0  in_flight_requests:  0  accounting:  0  parent:  0
rdu-es-data-01j:    request:  0  fielddata:  0  in_flight_requests:  0  accounting:  0  parent:  0
rdu-es-data-01k:    request:  0  fielddata:  0  in_flight_requests:  0  accounting:  0  parent:  0
rdu-es-data-01l:    request:  0  fielddata:  0  in_flight_requests:  0  accounting:  0  parent:  0

`show_idx_doc_sources_all_k8sns_cnts_hourly` has wrapping error when a NS stops showing up logs

show_idx_doc_sources_all_k8sns_cnts_hourly has wrapping error when a NS stops showing up logs.

$ show_idx_doc_sources_all_k8sns_cnts_hourly p filebeat-*


k8s document sources (counts - hourly)
======================================



k8sns              totalDocs   2020-10-14T15:00:00.000Z  2020-10-14T16:00:00.000Z  2020-10-14T17:00:00.000Z          2020-10-14T18:00:00.000Z
==========         ==========  ==========                ==========                ==========                        ==========
stirrer            112812489   23880880                  37491000                  37719489                          13721120
vapor              61037642    12914713                  20275057                  20410211                          7437661
http-voice-v2      52660251    11976760                  17857842                  17210808                          5614841
mdr-generator      27406411    5193058                   9096832                   9381291                           3735230
mdr-generator-uat  27239228    5122232                   9036371                   9340096                           3740529
fraud-staging      24777913    22862320                  1915593                   mdr-generator-prod                24416603
4662995            8117025     8363799                   3272784                   message-search                    21106360
...

calc_idx_type_avgs_overXdays has a tally bug with ILM

calc_idx_type_avgs_overXdays has a tally bug with ILM. Doesn't avg. correctly over days. when ILM rolls an idx more than 1x a day.

$ calc_idx_type_avgs_overXdays p 10
...
...
===> [idxType: filebeat-60d-7.6.2]
filebeat-60d-7.6.2-2020.11.04-000197       7   1  398105175            352
filebeat-60d-7.6.2-2020.11.04-000198       7   1  393007924            352
filebeat-60d-7.6.2-2020.11.04-000199       7   1  419045751            356
filebeat-60d-7.6.2-2020.11.05-000200       7   1  441903120            362
filebeat-60d-7.6.2-2020.11.05-000201       7   1  399156468            350
filebeat-60d-7.6.2-2020.11.05-000202       7   1  410136305            356
filebeat-60d-7.6.2-2020.11.06-000203       7   1  431044699            350
filebeat-60d-7.6.2-2020.11.06-000204       7   1  419037986            364
filebeat-60d-7.6.2-2020.11.06-000205       7   1  409329467            355
filebeat-60d-7.6.2-2020.11.06-000206       7   1  433272852            352
filebeat-60d-7.6.2-2020.11.07-000207       7   1  507818376            353
filebeat-60d-7.6.2-2020.11.07-000208       7   1  480168119            350
filebeat-60d-7.6.2-2020.11.08-000209       7   1  444031200            350
filebeat-60d-7.6.2-2020.11.08-000210       7   1  439273546            351
filebeat-60d-7.6.2-2020.11.09-000211       7   1  416453430            363
filebeat-60d-7.6.2-2020.11.09-000212       7   1  413265381            360
filebeat-60d-7.6.2-2020.11.09-000213       7   1  429659401            353
filebeat-60d-7.6.2-2020.11.10-000214       7   1  417432629            365
filebeat-60d-7.6.2-2020.11.10-000215       7   1  436695594            362
filebeat-60d-7.6.2-2020.11.10-000216       7   1  441189560            355
filebeat-60d-7.6.2-2020.11.11-000217       7   1  431687032            358
filebeat-60d-7.6.2-2020.11.11-000218       7   1  453179097            365
filebeat-60d-7.6.2-2020.11.11-000219       7   1  448501450            358
filebeat-60d-7.6.2-2020.11.11-000220       7   1  443677724            361
filebeat-60d-7.6.2-2020.11.12-000221       7   1  427833881            364
filebeat-60d-7.6.2-2020.11.12-000222       7   1  420583182            358
filebeat-60d-7.6.2-2020.11.12-000223       7   1  480610151            352
filebeat-60d-7.6.2-2020.11.13-000224       7   1  489899874            367
filebeat-60d-7.6.2-2020.11.13-000225       7   1  425129446            362
filebeat-60d-7.6.2-2020.11.13-000226       7   1  423586673            353
filebeat-60d-7.6.2-2020.11.14-000227       7   1  377464575            308
filebeat-60d-7.6.2	432328389 355 25939703357 21323 31
...
...

The avg. 355 should be over days not indexes.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.