$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 9.5G 0 rom
nvme0n1 259:6 0 1.5T 0 disk
├─nvme0n1p1 259:7 0 200M 0 part /boot/efi
├─nvme0n1p2 259:8 0 1G 0 part /boot
└─nvme0n1p3 259:9 0 1.5T 0 part
├─centos_init500--c11-root 253:0 0 50G 0 lvm /
├─centos_init500--c11-swap 253:1 0 4G 0 lvm [SWAP]
└─centos_init500--c11-home 253:2 0 1.4T 0 lvm /home
nvme1n1 259:10 0 1.5T 0 disk
├─nvme1n1p1 259:13 0 100M 0 part
├─nvme1n1p5 259:14 0 4G 0 part
├─nvme1n1p6 259:15 0 4G 0 part
├─nvme1n1p7 259:17 0 119.9G 0 part
└─nvme1n1p8 259:18 0 1.3T 0 part
$ fdisk -l
[..]
Disk /dev/nvme1n1: 1600.3 GB, 1600321314816 bytes, 3125627568 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x7f33a7d2
Device Boot Start End Blocks Id System
/dev/nvme1n1p1 1 3125627534 1562813767 ee GPT
[..]
Delete the partitions
Select the disk that contains the partitions you intend to delete:
$ fdisk /dev/nvme1n1
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Delete the partition:
Command (m for help): d
Selected partition 1
Partition 1 is deleted
NOTE: The partition is automatically selected if there are no other partitions on disk. If the disk contains multiple partitions, select a partition that you want to delete.
Verify the paritions:
Command (m for help): p
Disk /dev/nvme1n1: 1600.3 GB, 1600321314816 bytes, 3125627568 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x7f33a7d2
Device Boot Start End Blocks Id System
Save the change and quit fdisk:
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Verify the partitions with lsblk:
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 9.5G 0 rom
nvme0n1 259:6 0 1.5T 0 disk
├─nvme0n1p1 259:7 0 200M 0 part /boot/efi
├─nvme0n1p2 259:8 0 1G 0 part /boot
└─nvme0n1p3 259:9 0 1.5T 0 part
├─centos_init500--c11-root 253:0 0 50G 0 lvm /
├─centos_init500--c11-swap 253:1 0 4G 0 lvm [SWAP]
└─centos_init500--c11-home 253:2 0 1.4T 0 lvm /home
nvme1n1 259:10 0 1.5T 0 disk
Start the benchmark on the load generator node (remember to set the race id there)
[es@node1 ~]$ export RACE_ID=734bb4b3-8b7a-4c0b-9fa6-aaeb4659569f
[es@node1 ~]$ esrally race --pipeline=benchmark-only --target-host=10.10.10.2:39200,10.10.10.3:39200,10.10.10.4:39200 --track=geonames --challenge=append-no-conflicts --on-error=abort --race-id=${RACE_ID}
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
[INFO] Race id is [734bb4b3-8b7a-4c0b-9fa6-aaeb4659569f]
[INFO] Racing on track [geonames], challenge [append-no-conflicts] and car ['external'] with version [7.17.0].
[WARNING] merges_total_time is 420149 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] merges_total_throttled_time is 81765 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] indexing_total_time is 825388 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] refresh_total_time is 76340 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] flush_total_time is 10787 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
Running delete-index [100% done]
Running create-index [100% done]
Running check-cluster-health [100% done]
Running index-append [100% done]
Running refresh-after-index [100% done]
Running force-merge [100% done]
Running refresh-after-force-merge [100% done]
Running wait-until-merges-finish [100% done]
Running index-stats [100% done]
Running node-stats [100% done]
Running default [100% done]
Running term [100% done]
Running phrase [100% done]
Running country_agg_uncached [100% done]
Running country_agg_cached [100% done]
Running scroll [100% done]
Running expression [100% done]
Running painless_static [100% done]
Running painless_dynamic [100% done]
Running decay_geo_gauss_function_score [100% done]
Running decay_geo_gauss_script_score [100% done]
Running field_value_function_score [100% done]
Running field_value_script_score [100% done]
Running large_terms [100% done]
Running large_filtered_terms [100% done]
Running large_prohibited_terms [100% done]
Running desc_sort_population [100% done]
Running asc_sort_population [100% done]
Running asc_sort_with_after_population [100% done]
Running desc_sort_geonameid [100% done]
Running desc_sort_with_after_geonameid [100% done]
Running asc_sort_geonameid [100% done]
Running asc_sort_with_after_geonameid [100% done]
------------------------------------------------------
_______ __ _____
/ ____(_)___ ____ _/ / / ___/_________ ________
/ /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \
/ __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/
/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/
------------------------------------------------------
| Metric | Task | Value | Unit |
|---------------------------------------------------------------:|-------------------------------:|---------------:|--------:|
| Cumulative indexing time of primary shards | | 13.3055 | min |
| Min cumulative indexing time across primary shards | | 0 | min |
| Median cumulative indexing time across primary shards | | 2.68572 | min |
| Max cumulative indexing time across primary shards | | 2.74885 | min |
| Cumulative indexing throttle time of primary shards | | 0 | min |
| Min cumulative indexing throttle time across primary shards | | 0 | min |
| Median cumulative indexing throttle time across primary shards | | 0 | min |
| Max cumulative indexing throttle time across primary shards | | 0 | min |
| Cumulative merge time of primary shards | | 4.82182 | min |
| Cumulative merge count of primary shards | | 57 | |
| Min cumulative merge time across primary shards | | 0 | min |
| Median cumulative merge time across primary shards | | 0.984917 | min |
| Max cumulative merge time across primary shards | | 1.06472 | min |
| Cumulative merge throttle time of primary shards | | 0.978367 | min |
| Min cumulative merge throttle time across primary shards | | 0 | min |
| Median cumulative merge throttle time across primary shards | | 0.195508 | min |
| Max cumulative merge throttle time across primary shards | | 0.265933 | min |
| Cumulative refresh time of primary shards | | 1.20573 | min |
| Cumulative refresh count of primary shards | | 148 | |
| Min cumulative refresh time across primary shards | | 3.33333e-05 | min |
| Median cumulative refresh time across primary shards | | 0.258775 | min |
| Max cumulative refresh time across primary shards | | 0.283433 | min |
| Cumulative flush time of primary shards | | 0.172783 | min |
| Cumulative flush count of primary shards | | 11 | |
| Min cumulative flush time across primary shards | | 1.66667e-05 | min |
| Median cumulative flush time across primary shards | | 0.0345 | min |
| Max cumulative flush time across primary shards | | 0.0385167 | min |
| Total Young Gen GC time | | 16.263 | s |
| Total Young Gen GC count | | 2821 | |
| Total Old Gen GC time | | 2.312 | s |
| Total Old Gen GC count | | 41 | |
| Store size | | 3.03867 | GB |
| Translog size | | 3.58559e-07 | GB |
| Heap used for segments | | 0.701981 | MB |
| Heap used for doc values | | 0.0314178 | MB |
| Heap used for terms | | 0.54541 | MB |
| Heap used for norms | | 0.0736694 | MB |
| Heap used for points | | 0 | MB |
| Heap used for stored fields | | 0.0514832 | MB |
| Segment count | | 102 | |
| Total Ingest Pipeline count | | 0 | |
| Total Ingest Pipeline time | | 0 | s |
| Total Ingest Pipeline failed | | 0 | |
| error rate | index-append | 0 | % |
| Min Throughput | index-stats | 90.01 | ops/s |
| Mean Throughput | index-stats | 90.02 | ops/s |
| Median Throughput | index-stats | 90.02 | ops/s |
| Max Throughput | index-stats | 90.04 | ops/s |
| 50th percentile latency | index-stats | 5.16153 | ms |
| 90th percentile latency | index-stats | 6.00114 | ms |
| 99th percentile latency | index-stats | 6.61081 | ms |
| 99.9th percentile latency | index-stats | 10.4064 | ms |
| 100th percentile latency | index-stats | 10.8105 | ms |
| 50th percentile service time | index-stats | 4.00402 | ms |
| 90th percentile service time | index-stats | 4.6339 | ms |
| 99th percentile service time | index-stats | 5.10083 | ms |
| 99.9th percentile service time | index-stats | 9.17415 | ms |
| 100th percentile service time | index-stats | 9.22474 | ms |
[..]
[WARNING] No throughput metrics available for [index-append]. Likely cause: The benchmark ended already during warmup.
----------------------------------
[INFO] SUCCESS (took 4008 seconds)
----------------------------------
Note: If you only want to shutdown the node but don’t want to delete the node and the data, add the option “–preserve-install” additionally.
Troubleshooting
Elasticsearch start failure due to max virtual memory is too low
$ cat /home/es/.rally/benchmarks/races/aa826112-d371-4f09-9b68-f9084e7c9e0b/rally-node-0/logs/server/rally-benchmark.log
[..]
[2022-10-28T16:58:41,041][ERROR][o.e.b.Bootstrap ] [rally-node-0] node validation exception
[1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
bootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[..]
$ sysctl -a | grep max_map_count
vm.max_map_count = 65530
To fix this issue, change the kernel parameter
$ vim /etc/sysctl.conf
vm.max_map_count=1048576
$ sysctl -p
vm.max_map_count = 1048576
Restart Elasticsearch and verify the process and port
Elasticsearch start failure due to max file descriptors is too low
$ cat /home/es/.rally/benchmarks/races/aa826112-d371-4f09-9b68-f9084e7c9e0b/rally-node-0/logs/server/rally-benchmark.log
[..]
[2022-10-28T18:02:09,107][ERROR][o.e.b.Bootstrap ] [rally-node-1] node validation exception
[1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
bootstrap check failure [1] of [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[..]
To fix this issue, change the max open files value in /etc/security/limits.conf
$ vim /etc/security/limits.conf
* soft nofile 1048576
* hard nofile 1048576
Exit and login back to the shell to see the changed value
ulimit -a | grep "open files"
open files (-n) 1048576
Rally is the macrobenchmarking framework for Elasticsearch. This post follows the instructions here to install Rally and run the very first benchmark(aka race).
Note: If you do not uncomment the 4 lines in Modules/Setup, you would fail to install Rally with the ssl module unavailable issue as mentioned in the following troubleshooting section.
Install git
Git is not required if all of the following conditions are met:
You are using Rally only as a load generator (–pipeline=benchmark-only) or you are referring to Elasticsearch configurations with –team-path.
You create your own tracks and refer to them with –track-path.
In all other cases, Rally requires git 1.9 or better. Verify with git –version
$ yum -y remove git $ yum -y remove git-* $ yum install git $ git version git version 2.38.1
Install JDK
A JDK is required on all machines where you want to launch Elasticsearch. If you use Rally just as a load generator to benchmark remote clusters, no JDK is required. Refer to here to determine the appropriate JDK version to run Elasticsearch.
$ yum install java
$ java -version
openjdk version "1.8.0_345"
OpenJDK Runtime Environment (build 1.8.0_345-b01)
OpenJDK 64-Bit Server VM (build 25.345-b01, mixed mode)
$ wget https://download.oracle.com/java/17/archive/jdk-17.0.5_linux-x64_bin.rpm
$ rpm -ivh jdk-17.0.5_linux-x64_bin.rpm
$ java -version
java version "17.0.5" 2022-10-18 LTS
Java(TM) SE Runtime Environment (build 17.0.5+9-LTS-191)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.5+9-LTS-191, mixed mode, sharing)
$ rpm -qi jdk-17-17.0.5-ga.x86_64
Name : jdk-17
Epoch : 2000
Version : 17.0.5
Release : ga
Architecture: x86_64
Install Date: Wed 02 Nov 2022 03:58:42 PM PDT
Group : Development/Tools
Size : 316751437
License : https://java.com/freeuselicense
Signature : RSA/SHA256, Tue 13 Sep 2022 09:36:17 AM PDT, Key ID 72f97b74ec551f03
Source RPM : jdk-17-17.0.5-ga.src.rpm
Build Date : Tue 13 Sep 2022 09:35:27 AM PDT
Build Host : java.com
Relocations : /usr/java
Vendor : Oracle Corporation
URL : http://www.oracle.com/technetwork/java/javase/overview/index.html
Summary : Java Platform Standard Edition Development Kit
Description :
The Java Platform Standard Edition Development Kit (JDK) includes both
the runtime environment (Java virtual machine, the Java platform classes
and supporting files) and development tools (compilers, debuggers,
tool libraries and other tools).
With java 1.8.0, you can run Elasticsearch 7.17.x or lower version.
To find the JDK, Rally expects the environment variable JAVA_HOME to be set on all targeted machines. To have more specific control, for example when you want to benchmark across a wide range of Elasticsearch releases, you can also set JAVAx_HOME where x is the major version of a JDK (e.g. JAVA8_HOME would point to a JDK 8 installation). Rally will then choose the highest supported JDK per version of Elasticsearch that is available.
$ pip3.8 install --upgrade pip
$ pip3.8 install esrally
$ esrally -h
usage: esrally [-h] [--version] {race,list,info,create-track,generate,compare,download,install,start,stop} ...
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
You Know, for Benchmarking Elasticsearch.
optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
subcommands:
{race,list,info,create-track,generate,compare,download,install,start,stop}
race Run a benchmark
list List configuration options
info Show info about a track
create-track Create a Rally track from existing data
generate Generate artifacts
compare Compare two races
download Downloads an artifact
install Installs an Elasticsearch node locally
start Starts an Elasticsearch node locally
stop Stops an Elasticsearch node locally
Find out more about Rally at https://esrally.readthedocs.io/en/2.6.0/
$ esrally list tracks
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
Available tracks:
Name Description Documents Compressed Size Uncompressed Size Default Challenge All Challenges
---------------- ----------------------------------------------------------------------- ----------- ----------------- ------------------- ----------------------- --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
dense_vector Benchmark for dense vector indexing and search 10,000,000 7.2 GB 19.5 GB index-and-search index-and-search
elastic/endpoint Endpoint track 0 0 bytes 0 bytes default default
elastic/logs Track for simulating logging workloads 14,009,078 N/A N/A logging-indexing cross-clusters-search,logging-disk-usage,logging-indexing-querying,logging-indexing,logging-querying,logging-snapshot-mount,logging-snapshot-restore,logging-snapshot,many-shards-quantitative,many-shards-snapshots
elastic/security Track for simulating Elastic Security workloads 77,513,777 N/A N/A security-querying index-alert-source-events,security-indexing-querying,security-indexing,security-querying
eql EQL benchmarks based on endgame index of SIEM demo cluster 60,782,211 4.5 GB 109.2 GB default default,index-sorting
geonames POIs from Geonames 11,396,503 252.9 MB 3.3 GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts,significant-text
geopoint Point coordinates from PlanetOSM 60,844,404 482.1 MB 2.3 GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts
geopointshape Point coordinates from PlanetOSM indexed as geoshapes 60,844,404 470.8 MB 2.6 GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts
geoshape Shapes from PlanetOSM 84,220,567 17.0 GB 58.7 GB append-no-conflicts append-no-conflicts,append-no-conflicts-big
http_logs HTTP server log data 247,249,096 1.2 GB 31.1 GB append-no-conflicts append-no-conflicts,runtime-fields,append-no-conflicts-index-only,append-sorted-no-conflicts,append-index-only-with-ingest-pipeline,update,append-no-conflicts-index-reindex-only
metricbeat Metricbeat data 1,079,600 87.7 MB 1.2 GB append-no-conflicts append-no-conflicts
nested StackOverflow Q&A stored as nested docs 11,203,029 663.3 MB 3.4 GB nested-search-challenge nested-search-challenge,index-only
noaa Global daily weather measurements from NOAA 33,659,481 949.4 MB 9.0 GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,aggs,filter-aggs
nyc_taxis Taxi rides in New York in 2015 165,346,692 4.5 GB 74.3 GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts-index-only,update,append-ml,aggs
percolator Percolator benchmark based on AOL queries 2,000,000 121.1 kB 104.9 MB append-no-conflicts append-no-conflicts
pmc Full text benchmark with academic papers from PMC 574,199 5.5 GB 21.7 GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts,append-fast-with-conflicts,indexing-querying
so Indexing benchmark using up to questions and answers from StackOverflow 36,062,278 8.9 GB 33.1 GB append-no-conflicts append-no-conflicts,transform,frequent-items
so_vector Benchmark for vector search with StackOverflow data 2,000,000 12.3 GB 32.2 GB index-and-search index-and-search
sql SQL query performance based on NOAA Weather data 33,659,481 949.4 MB 9.0 GB sql sql
tsdb metricbeat information for elastic-app k8s cluster 116,633,698 N/A 123.0 GB append-no-conflicts append-no-conflicts,downsample
-------------------------------
[INFO] SUCCESS (took 3 seconds)
-------------------------------
$ esrally list cars
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
Available cars:
Name Type Description
----------------------- ------ --------------------------------------
16gheap car Sets the Java heap to 16GB
1gheap car Sets the Java heap to 1GB
24gheap car Sets the Java heap to 24GB
2gheap car Sets the Java heap to 2GB
4gheap car Sets the Java heap to 4GB
8gheap car Sets the Java heap to 8GB
defaults car Sets the Java heap to 1GB
basic-license mixin Basic License
debug-non-safepoints mixin More accurate CPU profiles
ea mixin Enables Java assertions
fp mixin Preserves frame pointers
g1gc mixin Enables the G1 garbage collector
parallelgc mixin Enables the Parallel garbage collector
trial-license mixin Trial License
unpooled mixin Enables Netty's unpooled allocator
x-pack-ml mixin X-Pack Machine Learning
x-pack-monitoring-http mixin X-Pack Monitoring (HTTP exporter)
x-pack-monitoring-local mixin X-Pack Monitoring (local exporter)
x-pack-security mixin X-Pack Security
zgc mixin Enables the ZGC garbage collector
-------------------------------
[INFO] SUCCESS (took 6 seconds)
-------------------------------
Run the first race with Rally
A “race” in Rally is the execution of a benchmarking experiment. You can choose different benchmarking scenarios (called tracks) for your benchmarks.
Rally should be run as a non-root user. We create a user “es” to run the following race.
$ groupadd es
$ useradd es -g es
$ passwd es
$ cd /home/es
$ su - es
$ export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre
$ esrally race --distribution-version=7.17.0 --track=geonames
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
[INFO] Race id is [d3102b91-ac10-4383-b6a4-7b98d2831af7]
[INFO] Preparing for race ...
[INFO] Downloading Elasticsearch 7.17.0 (297.0 MB total size) [100%]
[INFO] Downloading track data (252.9 MB total size) [100.0%]
[INFO] Decompressing track data from [/home/es/.rally/benchmarks/data/geonames/documents-2.json.bz2] to [/home/es/.rally/benchmarks/data/geonames/documents-2.json] (resulting size: [3.30] GB) ... [OK]
[INFO] Preparing file offset table for [/home/es/.rally/benchmarks/data/geonames/documents-2.json] ... [OK]
[INFO] Racing on track [geonames], challenge [append-no-conflicts] and car ['defaults'] with version [7.17.0].
Running delete-index [100% done]
Running create-index [100% done]
Running check-cluster-health [100% done]
Running index-append [100% done]
Running refresh-after-index [100% done]
Running force-merge [100% done]
Running refresh-after-force-merge [100% done]
Running wait-until-merges-finish [100% done]
Running index-stats [100% done]
Running node-stats [100% done]
Running default [100% done]
Running term [100% done]
Running phrase [100% done]
Running country_agg_uncached [100% done]
Running country_agg_cached [100% done]
Running scroll [100% done]
Running expression [100% done]
Running painless_static [100% done]
Running painless_dynamic [100% done]
Running decay_geo_gauss_function_score [100% done]
Running decay_geo_gauss_script_score [100% done]
Running field_value_function_score [100% done]
Running field_value_script_score [100% done]
Running large_terms [100% done]
Running large_filtered_terms [100% done]
Running large_prohibited_terms [100% done]
Running desc_sort_population [100% done]
Running asc_sort_population [100% done]
Running asc_sort_with_after_population [100% done]
Running desc_sort_geonameid [100% done]
Running desc_sort_with_after_geonameid [100% done]
Running asc_sort_geonameid [100% done]
Running asc_sort_with_after_geonameid [100% done]
------------------------------------------------------
_______ __ _____
/ ____(_)___ ____ _/ / / ___/_________ ________
/ /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \
/ __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/
/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/
------------------------------------------------------
| Metric | Task | Value | Unit |
|---------------------------------------------------------------:|-------------------------------:|----------------:|--------:|
| Cumulative indexing time of primary shards | | 12.7087 | min |
| Min cumulative indexing time across primary shards | | 0.0196167 | min |
| Median cumulative indexing time across primary shards | | 2.5376 | min |
| Max cumulative indexing time across primary shards | | 2.58153 | min |
| Cumulative indexing throttle time of primary shards | | 0.0108667 | min |
| Min cumulative indexing throttle time across primary shards | | 0 | min |
| Median cumulative indexing throttle time across primary shards | | 0 | min |
| Max cumulative indexing throttle time across primary shards | | 0.00731667 | min |
| Cumulative merge time of primary shards | | 7.13778 | min |
| Cumulative merge count of primary shards | | 52 | |
| Min cumulative merge time across primary shards | | 0 | min |
| Median cumulative merge time across primary shards | | 1.20084 | min |
| Max cumulative merge time across primary shards | | 2.20237 | min |
| Cumulative merge throttle time of primary shards | | 1.59453 | min |
| Min cumulative merge throttle time across primary shards | | 0 | min |
| Median cumulative merge throttle time across primary shards | | 0.2181 | min |
| Max cumulative merge throttle time across primary shards | | 0.569333 | min |
| Cumulative refresh time of primary shards | | 2.89785 | min |
| Cumulative refresh count of primary shards | | 256 | |
| Min cumulative refresh time across primary shards | | 0.00186667 | min |
| Median cumulative refresh time across primary shards | | 0.577325 | min |
| Max cumulative refresh time across primary shards | | 0.599333 | min |
| Cumulative flush time of primary shards | | 0.2239 | min |
| Cumulative flush count of primary shards | | 14 | |
| Min cumulative flush time across primary shards | | 0.0023 | min |
| Median cumulative flush time across primary shards | | 0.0471833 | min |
| Max cumulative flush time across primary shards | | 0.05055 | min |
| Total Young Gen GC time | | 18.57 | s |
| Total Young Gen GC count | | 2243 | |
| Total Old Gen GC time | | 3.541 | s |
| Total Old Gen GC count | | 66 | |
| Store size | | 2.82211 | GB |
| Translog size | | 3.07336e-07 | GB |
| Heap used for segments | | 0.733753 | MB |
| Heap used for doc values | | 0.0489769 | MB |
| Heap used for terms | | 0.557007 | MB |
| Heap used for norms | | 0.0753784 | MB |
| Heap used for points | | 0 | MB |
| Heap used for stored fields | | 0.0523911 | MB |
| Segment count | | 103 | |
| Total Ingest Pipeline count | | 0 | |
| Total Ingest Pipeline time | | 0 | s |
| Total Ingest Pipeline failed | | 0 | |
| Min Throughput | index-append | 87315.7 | docs/s |
| Mean Throughput | index-append | 87373.7 | docs/s |
| Median Throughput | index-append | 87368.1 | docs/s |
| Max Throughput | index-append | 87440.9 | docs/s |
| 50th percentile latency | index-append | 316.067 | ms |
| 90th percentile latency | index-append | 458.448 | ms |
| 99th percentile latency | index-append | 1152.53 | ms |
| 100th percentile latency | index-append | 1316.07 | ms |
| 50th percentile service time | index-append | 316.067 | ms |
| 90th percentile service time | index-append | 458.448 | ms |
| 99th percentile service time | index-append | 1152.53 | ms |
| 100th percentile service time | index-append | 1316.07 | ms |
| error rate | index-append | 0 | % |
| Min Throughput | index-stats | 89.91 | ops/s |
| Mean Throughput | index-stats | 89.95 | ops/s |
| Median Throughput | index-stats | 89.95 | ops/s |
| Max Throughput | index-stats | 89.97 | ops/s |
| 50th percentile latency | index-stats | 4.36948 | ms |
| 90th percentile latency | index-stats | 5.06188 | ms |
| 99th percentile latency | index-stats | 5.51726 | ms |
| 99.9th percentile latency | index-stats | 7.79772 | ms |
| 100th percentile latency | index-stats | 9.64821 | ms |
| 50th percentile service time | index-stats | 3.16338 | ms |
| 90th percentile service time | index-stats | 3.67796 | ms |
| 99th percentile service time | index-stats | 3.86689 | ms |
| 99.9th percentile service time | index-stats | 4.13559 | ms |
| 100th percentile service time | index-stats | 6.9374 | ms |
[..]
----------------------------------
[INFO] SUCCESS (took 4199 seconds)
----------------------------------
You can save this report also to a file by using –report-file=/path/to/your/report.md and save it as CSV with –report-format=csv.
What did Rally just do?
It downloaded and started Elasticsearch 7.17.0
It downloaded the relevant data for the geonames track
It ran the actual benchmark
And finally it reported the results
Rally Configuration
Rally stores its configuration in the file .rally/rally.ini which is automatically created the first time Rally is executed.
$ pwd
/home/es/.rally/benchmarks
$ ls
data distributions races teams tracks
$ ls distributions/
elasticsearch-7.17.0-linux-x86_64.tar.gz
$ ls tracks/default/
download.sh elastic eql geonames geopoint geopointshape geoshape http_logs metricbeat nested noaa nyc_taxis percolator pmc README.md so sql
$ ls races/d3102b91-ac10-4383-b6a4-7b98d2831af7/
race.json rally-node-0
$ ls teams/default/
cars LICENSE NOTICE plugins README.md
$ ls -l data/geonames/
total 3723472
-rw-rw-r-- 1 es es 3547613828 Oct 27 17:01 documents-2.json
-rw-rw-r-- 1 es es 265208777 Oct 27 17:00 documents-2.json.bz2
-rw-rw-r-- 1 es es 4250 Oct 27 17:02 documents-2.json.offset
$ cat /home/es/.rally/rally.ini
[meta]
config.version = 17
[system]
env.name = local
[node]
root.dir = /home/es/.rally/benchmarks
src.root.dir = /home/es/.rally/benchmarks/src
[source]
remote.repo.url = https://github.com/elastic/elasticsearch.git
elasticsearch.src.subdir = elasticsearch
[benchmarks]
local.dataset.cache = /home/es/.rally/benchmarks/data
[reporting]
datastore.type = in-memory
datastore.host =
datastore.port =
datastore.secure = False
datastore.user =
datastore.password =
[tracks]
default.url = https://github.com/elastic/rally-tracks
[teams]
default.url = https://github.com/elastic/rally-teams
[defaults]
preserve_benchmark_candidate = false
[distributions]
release.cache = true
The benchmark data directory can be changed by modifying root.dir in rally.ini.
root.dir (default: “~/.rally/benchmarks”): Rally uses this directory to store all benchmark-related data. It assumes that it has complete control over this directory and any of its subdirectories.
src.root.dir (default: “~/.rally/benchmarks/src”): The directory where the source code of Elasticsearch or any plugins is checked out. Only relevant for benchmarks from sources.
Uninstall python3
The following are the optional commands in the case you need to uninstall python3 in CentOS.
“WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.”
$ pip3.8 install esrally WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available. WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”)’: /simple/esrally/ WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”)’: /simple/esrally/ WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”)’: /simple/esrally/ WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”)’: /simple/esrally/ WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”)’: /simple/esrally/ Could not fetch URL https://pypi.org/simple/esrally/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host=’pypi.org’, port=443): Max retries exceeded with url: /simple/esrally/ (Caused by SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”)) - skipping ERROR: Could not find a version that satisfies the requirement esrally (from versions: none) ERROR: No matching distribution found for esrally WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available. Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host=’pypi.org’, port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”)) - skipping WARNING: There was an error checking the latest version of pip.
To fix this, install the python after un-commenting the four lines in Modules/Setup.
$ curl -O https://www.python.org/ftp/python/3.8.1/Python-3.8.1.tgz
$ tar zxf Python-3.8.1.tgz
$ mv Python-3.8.1 /usr/src
$ cd /usr/src/Python-3.8.1/
$ vim Modules/Setup
SSL=/usr/local/ssl
_ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
-L$(SSL)/lib -lssl -lcrypto
$ ./configure --enable-optimizations
$ make altinstall
$ python3.8 -m ssl
Each track allows to overwrite the corresponding parameters using –track-params.
For example, the track geonames allows to overwrite the following parameters.
bulk_size (default: 5000)
bulk_indexing_clients (default: 8): Number of clients that issue bulk indexing requests.
ingest_percentage (default: 100): A number between 0 and 100 that defines how much of the document corpus should be ingested.
conflicts (default: “random”): Type of id conflicts to simulate. Valid values are: ‘sequential’ (A document id is replaced with a document id with a sequentially increasing id), ‘random’ (A document id is replaced with a document id with a random other id).
conflict_probability (default: 25): A number between 0 and 100 that defines the probability of id conflicts. This requires to run the respective challenge. Combining conflicts=sequential and conflict-probability=0 makes Rally generate index ids by itself, instead of relying on Elasticsearch’s automatic id generation https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#_automatic_id_generation_.
on_conflict (default: “index”): Whether to use an “index” or an “update” action when simulating an id conflict.
recency (default: 0): A number between 0 and 1 that defines whether to bias towards more recent ids when simulating conflicts. See the Rally docs for the full definition of this parameter. This requires to run the respective challenge.
number_of_replicas (default: 0)
number_of_shards (default: 5)
max_num_segments: The maximum number of segments to force-merge to.
source_enabled (default: true): A boolean defining whether the _source field is stored in the index.
index_settings: A list of index settings. Index settings defined elsewhere (e.g. number_of_replicas) need to be overridden explicitly.
cluster_health (default: “green”): The minimum required cluster health.
error_level (default: “non-fatal”): Available for bulk operations only to specify ignore-response-error-level.
The default track parameters can be verified in the index.json.
A pipeline is a series of steps that are performed to get benchmark results. This is not intended to customize the actual benchmark but rather what happens before and after a benchmark.
An example will clarify the concept: If you want to benchmark a binary distribution of Elasticsearch, Rally has to download a distribution archive, decompress it, start Elasticsearch and then run the benchmark. However, if you want to benchmark a source build of Elasticsearch, it first has to build a distribution using the Gradle Wrapper. So, in both cases, different steps are involved and that’s what pipelines are for.
You can get a list of all pipelines with esrally list pipelines
Name Description ----------------- --------------------------------------------------------------------------------------------- from-sources Builds and provisions Elasticsearch, runs a benchmark and reports results. from-distribution Downloads an Elasticsearch distribution, provisions it, runs a benchmark and reports results. benchmark-only Assumes an already running Elasticsearch instance, runs a benchmark and reports results
This is intended if you want to provision a cluster by yourself. Do not use this pipeline unless you are absolutely sure you need to. As Rally has not provisioned the cluster, results are not easily reproducable and it also cannot gather a lot of metrics (like CPU usage).
To benchmark a cluster, you also have to specify the hosts to connect to.
A track is the description of one or more benchmarking scenarios with a specific document corpus. It defines for example the involved indices, data files and which operations are invoked. List the available tracks with esrally list tracks. Although Rally ships with some tracks out of the box, you should usually create your own track based on your own data.
Geonames: for evaluating the performance of structured data.
Geopoint: for evaluating the performance of geopoint queries.
Geopointshape: for evaluating the performance of geopointshape queries.
Geoshape: for evaluating the performance of geoshape data.
Percolator: for evaluating the performance of percolation queries.
PMC: for evaluating the performance of full text search.
NYC taxis: for evaluating the performance for highly structured data.
NYC taxis (ARM): for evaluating the performance for highly structured data and detect ARM-specific regressions.
Nested: for evaluating the performance for nested documents.
HTTP Logs: for evaluating the performance of (Web) server logs.
NOAA: for evaluating the performance of range fields.
EQL: for evaluating the performance of EQL.
SQL: for evaluating the performance of SQL.
SO (Transform): for evaluating the performance of the transform feature.
SO (Frequent Items): for evaluating the performance of the frequent items aggregation.
Logging: for evaluating the performance of the Log Monitoring part of Elastic’s Observability solution.
Logging (Many Shards): for evaluating the performance of the Log Monitoring part of Elastic’s Observability solution with a large amount of shards.
Logging (Snapshots): for evaluating snapshot performance when taking snapshot with a large of amount of shards.
Dense Vector: for evaluating the performance of vectors search.
Security: for evaluating the performance of Elastic’s Security solution.
TSDB: for evaluating the performance of TSDB (time series database).
Logging (CCS): for evaluating the performance of CCS (cross cluster search).
$ esrally list tracks
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
Available tracks:
Name Description Documents Compressed Size Uncompressed Size Default Challenge All Challenges
---------------- ----------------------------------------------------------------------- ----------- ----------------- ------------------- ----------------------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
elastic/logs Track for simulating logging workloads 14,009,078 N/A NA logging-indexing logging-indexing-querying,logging-indexing,logging-querying,logging-snapshot-mountlogging-snapshot-restore,logging-snapshot,cross-clusters-search,logging-disk-usage,many-shards-quantitative,many-shards-snapshots
elastic/security Track for simulating Elastic Security workloads 77,513,777 N/A NA security-querying security-indexing-querying,security-indexing,security-querying,index-alert-source-events
elastic/endpoint Endpoint track 0 0 bytes 0bytes default default
eql EQL benchmarks based on endgame index of SIEM demo cluster 60,782,211 4.5 GB 109.2GB default default,index-sorting
geonames POIs from Geonames 11,396,503 252.9 MB 3.3GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflictssignificant-text
geopoint Point coordinates from PlanetOSM 60,844,404 482.1 MB 2.3GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts
geopointshape Point coordinates from PlanetOSM indexed as geoshapes 60,844,404 470.8 MB 2.6GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts
geoshape Shapes from PlanetOSM 84,220,567 17.0 GB 58.7GB append-no-conflicts append-no-conflicts,append-no-conflicts-big
http_logs HTTP server log data 247,249,096 1.2 GB 31.1GB append-no-conflicts append-no-conflicts,runtime-fields,append-no-conflicts-index-onlyappend-sorted-no-conflicts,append-index-only-with-ingest-pipeline,update,append-no-conflicts-index-reindex-only
metricbeat Metricbeat data 1,079,600 87.7 MB 1.2GB append-no-conflicts append-no-conflicts
nested StackOverflow Q&A stored as nested docs 11,203,029 663.3 MB 3.4GB nested-search-challenge nested-search-challenge,index-only
noaa Global daily weather measurements from NOAA 33,659,481 949.4 MB 9.0GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,aggs,filter-aggs
nyc_taxis Taxi rides in New York in 2015 165,346,692 4.5 GB 74.3GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts-index-onlyupdate,append-ml,aggs
percolator Percolator benchmark based on AOL queries 2,000,000 121.1 kB 104.9MB append-no-conflicts append-no-conflicts
pmc Full text benchmark with academic papers from PMC 574,199 5.5 GB 21.7GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflictsappend-fast-with-conflicts,indexing-querying
so Indexing benchmark using up to questions and answers from StackOverflow 36,062,278 8.9 GB 33.1GB append-no-conflicts append-no-conflicts,transform,frequent-items
sql SQL query performance based on NOAA Weather data 33,659,481 949.4 MB 9.0GB sql sql
dense_vector Benchmark for dense vector indexing and search 10,000,000 7.2 GB 19.5GB index-and-search index-and-search
so_vector Benchmark for vector search with StackOverflow data 2,000,000 12.3 GB 32.2GB index-and-search index-and-search
tsdb metricbeat information for elastic-app k8s cluster 116,633,698 N/A 123.0GB append-no-conflicts append-no-conflicts,downsample
-------------------------------
[INFO] SUCCESS (took 2 seconds)
-------------------------------
$ esrally info --track=geonames
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
Showing details for track [geonames]:
* Description: POIs from Geonames
* Documents: 11,396,503
* Compressed Size: 252.9 MB
* Uncompressed Size: 3.3 GB
================================================
Challenge [append-no-conflicts] (run by default)
================================================
Indexes the whole document corpus using Elasticsearch default settings. We only adjust the number of replicas as we benchmark asingle node cluster and Rally will only start the benchmark if the cluster turns green. Document ids are unique so all indexoperations are append only. After that a couple of queries are run.
Schedule:
----------
1. delete-index
2. create-index
3. check-cluster-health
4. index-append (8 clients)
5. refresh-after-index
6. force-merge
7. refresh-after-force-merge
8. wait-until-merges-finish
9. index-stats
10. node-stats
11. default
12. term
13. phrase
14. country_agg_uncached
15. country_agg_cached
16. scroll
17. expression
18. painless_static
19. painless_dynamic
20. decay_geo_gauss_function_score
21. decay_geo_gauss_script_score
22. field_value_function_score
23. field_value_script_score
24. large_terms
25. large_filtered_terms
26. large_prohibited_terms
27. desc_sort_population
28. asc_sort_population
29. asc_sort_with_after_population
30. desc_sort_geonameid
31. desc_sort_with_after_geonameid
32. asc_sort_geonameid
33. asc_sort_with_after_geonameid
==========================================
Challenge [append-no-conflicts-index-only]
==========================================
Indexes the whole document corpus using Elasticsearch default settings. We only adjust the number of replicas as we benchmark asingle node cluster and Rally will only start the benchmark if the cluster turns green. Document ids are unique so all indexoperations are append only.
Schedule:
----------
1. delete-index
2. create-index
3. check-cluster-health
4. index-append (8 clients)
5. force-merge
6. wait-until-merges-finish
======================================
Challenge [append-fast-with-conflicts]
======================================
Indexes the whole document corpus using a setup that will lead to a larger indexing throughput than the default settings. Rallywill produce duplicate ids in 25% of all documents (not configurable) so we can simulate a scenario with appends most of the timeand some updates in between.
Schedule:
----------
1. delete-index
2. create-index
3. check-cluster-health
4. index-update (8 clients)
5. force-merge
6. wait-until-merges-finish
============================
Challenge [significant-text]
============================
Indexes the whole document corpus using Elasticsearch default settings. We only adjust the number of replicas as we benchmark asingle node cluster and Rally will only start the benchmark if the cluster turns green. Document ids are unique so all indexoperations are append only.
Schedule:
----------
1. delete-index
2. create-index
3. check-cluster-health
4. index-append (8 clients)
5. force-merge
6. wait-until-merges-finish
7. significant_text_selective
8. significant_text_sampled_selective
9. significant_text_unselective
10. significant_text_sampled_unselective
-------------------------------
[INFO] SUCCESS (took 1 seconds)
-------------------------------
A track is specified in a JSON file. A track JSON file can include the following sections:
indices/templates: define the relevant indices and index templates
data-streams: define the relevant data streams
composable-templates/component-templates: define the relevant composable and component templates
corpora: define all document corpora (i.e. data files) that Rally should use for this track
challenge(s): describe more than one set of operations, in the event your track needs to test more than one set of scenarios
schedule: describe the workload for the benchmark, for example index with two clients at maximum throughput while searching with another two clients with ten operations per second
operations: describe which operations are available for this track and how they are parametrized
A challenge describes one benchmarking scenario, for example indexing documents at maximum throughput with 4 clients while issuing term and phrase queries from another two clients rate-limited at 10 queries per second each. It is always specified in the context of a track. See the available challenges by listing the corresponding tracks with esrally list tracks.
car
A car is a specific configuration of an Elasticsearch cluster that is benchmarked, for example the out-of-the-box configuration, a configuration with a specific heap size or a custom logging configuration. List the available cars with esrally list cars.
$ esrally list cars
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
Available cars:
Name Type Description
----------------------- ------ --------------------------------------
16gheap car Sets the Java heap to 16GB
1gheap car Sets the Java heap to 1GB
24gheap car Sets the Java heap to 24GB
2gheap car Sets the Java heap to 2GB
4gheap car Sets the Java heap to 4GB
8gheap car Sets the Java heap to 8GB
defaults car Sets the Java heap to 1GB
basic-license mixin Basic License
debug-non-safepoints mixin More accurate CPU profiles
ea mixin Enables Java assertions
fp mixin Preserves frame pointers
g1gc mixin Enables the G1 garbage collector
parallelgc mixin Enables the Parallel garbage collector
trial-license mixin Trial License
unpooled mixin Enables Netty's unpooled allocator
x-pack-ml mixin X-Pack Machine Learning
x-pack-monitoring-http mixin X-Pack Monitoring (HTTP exporter)
x-pack-monitoring-local mixin X-Pack Monitoring (local exporter)
x-pack-security mixin X-Pack Security
zgc mixin Enables the ZGC garbage collector
-------------------------------
[INFO] SUCCESS (took 1 seconds)
-------------------------------
telemetry
Telemetry is used in Rally to gather metrics about the car, for example CPU usage or index size.
$ esrally list telemetry
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
Available telemetry devices:
Command Name Description
-------------------------- -------------------------- --------------------------------------------------------------------
jit JIT Compiler Profiler Enables JIT compiler logs.
gc GC log Enables GC logs.
jfr Flight Recorder Enables Java Flight Recorder (requires an Oracle JDK or OpenJDK 11+)
heapdump Heap Dump Captures a heap dump.
node-stats Node Stats Regularly samples node stats
recovery-stats Recovery Stats Regularly samples shard recovery stats
ccr-stats CCR Stats Regularly samples Cross Cluster Replication (CCR) related stats
segment-stats Segment Stats Determines segment stats at the end of the benchmark.
transform-stats Transform Stats Regularly samples transform stats
searchable-snapshots-stats Searchable Snapshots Stats Regularly samples searchable snapshots stats
shard-stats Shard Stats Regularly samples nodes stats at shard level
data-stream-stats Data Stream Stats Regularly samples data stream stats
ingest-pipeline-stats Ingest Pipeline Stats Reports Ingest Pipeline stats at the end of the benchmark.
disk-usage-stats Disk usage of each field Runs the indices disk usage API after benchmarking
Keep in mind that each telemetry device may incur a runtime overhead which can skew results.
-------------------------------
[INFO] SUCCESS (took 0 seconds)
-------------------------------
race
A race is one invocation of the Rally binary. Another name for that is one “benchmarking trial”. During a race, Rally runs one challenge on a track with the given car.
A tournament is a comparison of two races. Looks like Rally doesn’t have the tournament mode support yet. Instead, the comparison can be made as the following command between two races. Note that, we should NOT run the same benchmark multiple times without data cleanup between the benchmarks. It will give us unreproducible results.
$ esrally compare --baseline=066e02fa-a71a-4239-b515-984f705d5f02 --contender=734bb4b3-8b7a-4c0b-9fa6-aaeb4659569f
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
Comparing baseline
Race ID: 066e02fa-a71a-4239-b515-984f705d5f02
Race timestamp: 2022-10-31 23:00:44
Challenge: append-no-conflicts
Car: external
User tags: intention=3shards1replica
with contender
Race ID: 734bb4b3-8b7a-4c0b-9fa6-aaeb4659569f
Race timestamp: 2022-10-31 21:42:54
Challenge: append-no-conflicts
Car: external
User tags: intention=3shards0replica
------------------------------------------------------
_______ __ _____
/ ____(_)___ ____ _/ / / ___/_________ ________
/ /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \
/ __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/
/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/
------------------------------------------------------
| Metric | Task | Baseline | Contender | Diff | Unit | Diff % |
|--------------------------------------------------------------:|-------------------------------:|----------------:----------------:|-------------:|--------:|---------:|
| Cumulative indexing time of primary shards | | 14.3493 | 140125 | -0.33683 | min | -2.35% |
| Min cumulative indexing time across primary shard | | 0 | 0 | 0 | min | 0.00% |
| Median cumulative indexing time across primary shard | | 0.00696667 | 000696667 | 0 | min | 0.00% |
| Max cumulative indexing time across primary shard | | 4.954 | 473868 | -0.21532 | min | -4.35% |
| Cumulative indexing throttle time of primary shards | | 0 | 0 | 0 | min | 0.00% |
| Min cumulative indexing throttle time across primary shard | | 0 | 0 | 0 | min | 0.00% |
| Median cumulative indexing throttle time across primary shard | | 0 | 0 | 0 | min | 0.00% |
| Max cumulative indexing throttle time across primary shard | | 0 | 0 | 0 | min | 0.00% |
| Cumulative merge time of primary shards | | 9.5478 | 486773 | -4.68007 | min | -49.02% |
| Cumulative merge count of primary shards | | 93 | 70 | -23 | | -24.73% |
| Min cumulative merge time across primary shard | | 0 | 0 | 0 | min | 0.00% |
| Median cumulative merge time across primary shard | | 0.00276667 | 000276667 | 0 | min | 0.00% |
| Max cumulative merge time across primary shard | | 3.61232 | 171572 | -1.8966 | min | -52.50% |
| Cumulative merge throttle time of primary shards | | 2.62635 | 0569683 | -2.05667 | min | -78.31% |
| Min cumulative merge throttle time across primary shard | | 0 | 0 | 0 | min | 0.00% |
| Median cumulative merge throttle time across primary shard | | 0 | 0 | 0 | min | 0.00% |
| Max cumulative merge throttle time across primary shard | | 1.06415 | 021315 | -0.851 | min | -79.97% |
| Cumulative refresh time of primary shards | | 1.37218 | 045075 | -0.92143 | min | -67.15% |
| Cumulative refresh count of primary shards | | 421 | 327 | -94 | | -22.33% |
| Min cumulative refresh time across primary shard | | 0 | 0 | 0 | min | 0.00% |
| Median cumulative refresh time across primary shard | | 0.0191417 | 00191417 | 0 | min | 0.00% |
| Max cumulative refresh time across primary shard | | 0.4631 | 0152433 | -0.31067 | min | -67.08% |
| Cumulative flush time of primary shards | | 0.144633 | 0185383 | 0.04075 | min | +28.17% |
| Cumulative flush count of primary shards | | 14 | 14 | 0 | | 0.00% |
| Min cumulative flush time across primary shard | | 0 | 0 | 0 | min | 0.00% |
| Median cumulative flush time across primary shard | | 0.000191667 | 0000191667 | 0 | min | 0.00% |
| Max cumulative flush time across primary shard | | 0.0575833 | 00674167 | 0.00983 | min | +17.08% |
| Total Young Gen GC time | | 30.494 | 14511 | -15.983 | s | -52.41% |
| Total Young Gen GC count | | 3888 | 2381 | -1507 | | -38.76% |
| Total Old Gen GC time | | 7.329 | 3247 | -4.082 | s | -55.70% |
| Total Old Gen GC count | | 109 | 48 | -61 | | -55.96% |
| Store size | | 5.78897 | 301044 | -2.77854 | GB | -48.00% |
| Translog size | | 8.19564e-07 | 665896e-07 | -0 | GB | -18.75% |
| Heap used for segments | | 0.513947 | 0385025 | -0.12892 | MB | -25.08% |
| Heap used for doc values | | 0.037796 | 00160522 | -0.02174 | MB | -57.53% |
| Heap used for terms | | 0.383759 | 0296478 | -0.08728 | MB | -22.74% |
| Heap used for norms | | 0.0499268 | 00380859 | -0.01184 | MB | -23.72% |
| Heap used for points | | 0 | 0 | 0 | MB | 0.00% |
| Heap used for stored fields | | 0.0424652 | 00344086 | -0.00806 | MB | -18.97% |
| Segment count | | 82 | 66 | -16 | | -19.51% |
| Total Ingest Pipeline count | | 0 | 0 | 0 | | 0.00% |
| Total Ingest Pipeline time | | 0 | 0 | 0 | ms | 0.00% |
| Total Ingest Pipeline failed | | 0 | 0 | 0 | | 0.00% |
| error rate | index-append | 0 | 0 | 0 | % | 0.00% |
| Min Throughput | index-stats | 90.001 | 900126 | 0.01165 | ops/s | +0.01% |
| Mean Throughput | index-stats | 90.0055 | 900241 | 0.01854 | ops/s | +0.02% |
| Median Throughput | index-stats | 90.0048 | 900219 | 0.01715 | ops/s | +0.02% |
| Max Throughput | index-stats | 90.0142 | 900422 | 0.02796 | ops/s | +0.03% |
| 50th percentile latency | index-stats | 5.80729 | 540389 | -0.40339 | ms | -6.95% |
| 90th percentile latency | index-stats | 6.71673 | 622462 | -0.49211 | ms | -7.33% |
| 99th percentile latency | index-stats | 7.26607 | 68574 | -0.40867 | ms | -5.62% |
| 99.9th percentile latency | index-stats | 11.3788 | 111477 | -0.23112 | ms | -2.03% |
| 100th percentile latency | index-stats | 13.6309 | 140192 | 0.38829 | ms | +2.85% |
| 50th percentile service time | index-stats | 4.61601 | 424021 | -0.37579 | ms | -8.14% |
| 90th percentile service time | index-stats | 5.40477 | 493823 | -0.46654 | ms | -8.63% |
| 99th percentile service time | index-stats | 5.85565 | 534821 | -0.50744 | ms | -8.67% |
| 99.9th percentile service time | index-stats | 9.8608 | 774023 | -2.12057 | ms | -21.51% |
| 100th percentile service time | index-stats | 12.6355 | 944686 | -3.18866 | ms | -25.24% |
| error rate | index-stats | 0 | 0 | 0 |
[..]
-------------------------------
[INFO] SUCCESS (took 0 seconds)
-------------------------------
[root@vm1 home]# groupadd es
[root@vm1 home]# useradd es -g es
[root@vm1 home]# passwd es
[root@vm1 home]# cd es
Download elasticsearch
[root@vm1 es]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.4.3-linux-x86_64.tar.gz
[root@vm1 es]# tar zxf elasticsearch-8.4.3-linux-x86_64.tar.gz
[root@vm1 es]# chown -R es:es /home/es
[root@vm1 es]# su es
[es@vm1 ~]$ cd elasticsearch-8.4.3/
Start elasticsearch
[es@vm1 elasticsearch-8.4.3]$ bin/elasticsearch
[...]
[2022-10-17T17:07:49,984][INFO ][o.e.h.AbstractHttpServerTransport] [vm1] publish_address {127.0.0.1:9200}, bound_addresses {[::]:9200}
[...]
✅ Elasticsearch security features have been automatically configured!
✅ Authentication is enabled and cluster connections are encrypted.
ℹ️ Password for the elastic user (reset with `bin/elasticsearch-reset-password -u elastic`):
[...]
ℹ️ HTTP CA certificate SHA-256 fingerprint:
[...]
ℹ️ Configure Kibana to use this cluster:
• Run Kibana and click the configuration link in the terminal when Kibana starts.
• Copy the following enrollment token and paste it into Kibana in your browser (valid for the next 30 minutes):
[...]
ℹ️ Configure other nodes to join this cluster:
• On this node:
⁃ Create an enrollment token with `bin/elasticsearch-create-enrollment-token -s node`.
⁃ Uncomment the transport.host setting at the end of config/elasticsearch.yml.
⁃ Restart Elasticsearch.
• On other nodes:
⁃ Start Elasticsearch with `bin/elasticsearch --enrollment-token <token>`, using the enrollment token that you generated.
Verify elasticsearch
[root@vm2 es]# curl 10.10.10.1:9200
curl: (52) Empty reply from server
Modify elastic config file:
[es@vm1 elasticsearch-8.4.3]$ vim config/elasticsearch.yml
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
network.host: 10.10.10.1
# Enable security features
xpack.security.enabled: false
[root@vm2 es]# wget https://artifacts.elastic.co/downloads/kibana/kibana-8.4.3-linux-x86_64.tar.gz
[root@vm2 es]# tar zxf kibana-8.4.3-linux-x86_64.tar.gz
[root@vm2 es]# cd kibana-8.4.3/
[root@vm2 kibana-8.4.3]# chown -R es:es /home/es
Start Kibana
[root@vm2 kibana-8.4.3]# bin/kibana
Kibana should not be run as root. Use --allow-root to continue.
[root@vm2 kibana-8.4.3]# su es
[es@vm2 kibana-8.4.3]$ bin/kibana
[2022-10-17T17:41:59.539-07:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
[2022-10-17T17:42:06.604-07:00][INFO ][http.server.Preboot] http server running at http://localhost:5601
[2022-10-17T17:42:06.644-07:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2022-10-17T17:42:06.646-07:00][INFO ][preboot] "interactiveSetup" plugin is holding setup: Validating Elasticsearch connection configuration…
[2022-10-17T17:42:06.681-07:00][INFO ][root] Holding setup until preboot stage is completed.
i Kibana has not been configured.
Go to http://localhost:5601/?code=263178 to get started.
Allow connections from remote users:
[root@vm2 kibana-8.4.3]# vim config/kibana.yml
# To allow connections from remote users, set this parameter to a non-loopback address.
#server.host: "localhost"
server.host: "10.10.10.2"
Restart the kibana to reflect the change:
[root@vm2 kibana-8.4.3]# su es
[es@vm2 kibana-8.4.3]$ bin/kibana
[...]
Go to http://10.10.10.2:5601/?code=293334 to get started.
If you run into the following issue when to connect to elasticsearch server from kibana web UI, you can change the URL from “https://10.10.10.1:9200“ to “http://10.10.10.1:9200“ for testing purpose.
[2022-11-09T10:55:02.691-08:00][ERROR][plugins.interactiveSetup.elasticsearch] Unable to connect to host "https://10.10.10.1:9200": write EPROTO 139880583923648:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:
Jay Berry, a fourteen years old boy, was living an ideal life when he unexpectedly met some monkeys during a mission to find his old milk cow, Sally Gooden. From his grandpa, he learned that there was a big reward for catching these monkeys alive. After many failures, Jay Berry finally captures the monkeys. When he received the money, he chose to use it to finance Daisy’s surgery instead. Throughout the story, Jay Berry learned to make life-enhancing choices and got happiness in the end.
The story is set in the late 1800s. The Berry family lived in Missouri as sharecroppers and later moved to a farm called Cherokee Land, which was in the middle of the Cherokee Nation. It was located in a strip from the foothills of the Ozark Mountains to the banks of the Illinois River in northeastern Oklahoma. They lived in a log house which was in the foothills overlooking the river bottoms. IThe setting influenced the plot and the events that took place. In the Cherokee bottoms, Jay Berry was likely to find numerous animals. He followed a game trail and found the sycamore tree full of wild monkeys. This led to the plot, which all started from this event. Also it affected the character background. Jay lived in a place where most people survive by hunting. He liked to shoot little animals in the beginning of the story, but changed as the story progressed. The setting could be the reason that the protagonist was unable to solve his problems and this was shown multiple times throughout the story.
The main plot of the story was Jay Berry’s struggle to capture the monkeys. He discovered a tree full of monkeys during his trip to find Sally Gooden. He is determined to capture them, for the reward would be enough money to achieve his desires. But this was much harder than he imagined and he greatly underestimates the intelligence of Jimbo, a very trained chimpanzee. His grandpa supported him and came up with ideas that might work. The first trap was a complete failure but it helped Jay Berry understand that the animals he was catching were not like any fox or racoon. They were way smarter and Jimbo was going to make sure that Jay was not going to get the satisfaction of capturing a single monkey. He lost his gunny sack and traps, which infuriated him so much that he shot Jimbo two times in the belly. But he quickly realized his fatal mistake because Jimbo was awfully angry and bared his sharp teeth. He ordered the monkeys to chase Jay and the poor hunter and dog ran straight through a brier patch. This does not dishearten Jay Berry and now he is determined to get back his possessions. On the second attempt, he almost captured two monkeys but Jimbo set them free while Rowdy and Jay fought off the monkeys. They came back with a lot of scratches and bites, but no monkeys. Daisy, his little sister, was scared that they might have gotten hydrophobia, and was bound to put both his brother and the dog in chains if their father hadn’t stepped in. The monkeys were from the circus so they were all vaccinated. All the Berry family had to worry about was that the wounds didn’t get infected.
After a few days of Daisy practicing her red cross, Jay Berry and Rowdy were healthy again. Grandpa advised Jay to befriend Jimbo so they will listen to him. Jay kindly approached the monkeys and offered Jimbo an apple. Jimbo decided to also give Jay something. So, he offered him a can of sour mash. Jay knew the stuff was nasty and politely refused the offer, which enraged Jimbo. Realizing that he had no choice, he reluctantly took the sour mash and the taste was enough to make him sick. Then, the bad taste left and Jay decided to take another can of sour mash. Soon, both Rowdy and he were drunk and fell asleep. When they woke up, Jay found out that his britches were stolen. He returned home so ill and unsteady that he fell unconscious the moment he was put in bed. Once again, he received the “Red Cross Treatment.” When Grandpa heard about this, he declared that they needed to go to the library to find some useful information on catching the monkeys. This showed that they were determined to the extent that they were ready to cross any limit. They had never gone to a library before, but were sure they would be provided with an answer there.The librarian gave them a book, which said that monkeys love coconuts. However, the whole half-bushel of coconuts were stolen by the monkeys, who returned the britches, beanshooter, and gunny sack.
Just when Jay was about to lose hope, Daisy found a fairy ring and wished that he got the gun and paint pony. Jay and his parents wished that Daisy would get her leg fixed up. This showed that they all cared about one another. That night, there was a fierce storm which frightened the monkeys so much that they willingly followed Jay Berry to the corn crib. They were provided with a comfortable shelter and plenty of food. Jay Berry receives one hundred fifty-six dollars, which he at first wanted to spend on his pony and rifle. But after taking one look at Daisy’s crippled leg, he made up his mind and financed the surgery to get her leg fixed up. While at town, Daisy used the money she had to buy his brother the rifle he wanted. She gave it to him on the condition that he would never use it to shoot little animals. Grandpa also used the money he saved to buy Jay the paint pony he always desired. The conclusion was that Jay’s self-sacrificing yet rational choice made all of his family happy.
Jay Berry Lee is the main character of the story. At the beginning of the novel, he was the happiest boy on Earth up till he reached fourteen-years old. He had lived an idyllic childhood with his little sister Daisy and his parents. The Berry family were very poor because they were sharecroppers until they moved to Oklahoma when Grandpa offered them free land. Here, Jay discovered a group of monkeys, whose capture could change his whole life. He set out to capture every one of the monkeys and devoted himself to finding ways to catch them. He needed the help of his grandfather, who made traps that might do the job.
Sadly, none of the traps work and Jay Berry gained nothing but a lot of scratches and painful monkey bites. He was constantly outsmarted by Jimbo, who simply broke every trap with his fingers and laughed rudely at Jay Berry whenever a trap failed. This made Jay more determined to capture them because he had spent all his valuable time in doing it. He does not want to simply give up on something he had worked so hard for. His opportunity finally arrived when the fierce storm struck and he thought that the monkeys must have had a difficult time finding shelter. He found all of them freezing and soaking wet. They were quivering from the cold and struggled to keep warm. Despite how they harmed him in the past, Jay kindly dried them off and took care of them. No monkey made the effort to bite him Jimbo surprisingly laid his head on Jay’s shoulder. Jay soothingly said that everything would be alright and he would help them. Finally, they willingly followed Jay home and into the corn crib. Later on, Jay was tempted to use the money to achieve his desires but it suddenly hit him that grandpa was secretly asking him to change his mind. Even if he bought the piny, he could never ride it because it reminded himself too much of the crippled leg of Daisy. He decided to use all of the money for Daisy to get a surgery in town, so the crippled leg would heal. He does not regret it, but suddenly it felt lonesome and gloomy at the farm. He was relieved that his efforts had not gone to waste when Mama wrote that the operation was a success and her sister was learning to walk. Jay changed from a boy who was focused on hunting to a boy who cared for animals and learned to place other’s priority higher than his own. This caused him to be the best Challenger student for he persevered through hardships. He thought of problems as challenges that needed to be solved. In this case, he thought of capturing the monkeys as a challenge, since the reward could greatly help him. He analyzed the problem and found the steps to solve it. He used the traps to devise plants that could capture the monkeys. Also, he showed kindness to the monkeys and saved their lives. The life-enhancing values he demonstrated improved his life and others.
Jimbo was the antagonist in the story. He made sure that all the other monkeys didn’t get caught in the traps and worked against Jay Berry during his several attempts. He acted rudely whenever he snatched a two dollar award from Jay and once, he got him drunk. He stole his britches, which was not his. His impudent remarks and cruel actions caused him to be the worst Challenger student. He did not respect others’ rights. Grandpa was a key supporting character and guided Jay Berry throughout the story. He helped design many traps which Jay used. For example, he made a trap with a net and it caught the old goose Gandy. It even caught two little monkeys, but Jimbo freed them during the chaos. Grandpa went as far as going to the library himself to get information. This showed that he really cared about Jay catching the monkeys. He always supported his grandson and helped him through hardships. Daisy is also a supporting character. She always warned Jay to not catch little animals because the Old Man of the Mountains was constantly watching. She was a protector of wildlife and disliked how boys captured poor animals just for the fun of it. Even though she was annoying sometimes, she cared greatly for her brother. She nursed him when he was bitten and when he was drunk. Also, she wished that her brother finally received the rifle and pony he so long wanted. This showed that she loved her brother and helped him achieve true happiness. One example was when she used the money she saved to purchase the rifle.From these characters I learned that I should persevere through challenges and be optimistic.
The main theme of the story is that anyone can achieve his or her goals as long as he or she sets their mind to it and does not give up. Throughout the course of the novel, Jay Berry did this by demonstrating the value of perseverance, courage, and kindness. When he faced failure after failure, he still pursued his goals and was determined. For example, he asked for the help of his grandfather and they created new traps to catch the monkeys. The net actually worked, though Jimbo freed the monkeys in the end. They made several attempts and finally Jay persuaded the gang of monkeys to follow him into the corn crib, where they would be provided with a comfortable home and plentiful food.He showed rationality when he thought about what would happen to the monkeys during the storm. He knew that these monkeys did not know how to react to this fierce storm and this was an ideal opportunity for him. This was how he got the monkeys to willingly follow him. Jay Berry was also very kind. One example was when he dried off the pitiful monkeys, who were freezing. His act of kindness allowed Jimbo to realize that he could trust Jay, who was just trying to help them. This taught me that a kind act would also help you in the end.
My favorite part of the story was when Jay Berry first discovered the tree full of monkeys. It was from here that he learned about the reward and was determined to capture them. He showed many values in his attempts and achieved true satisfaction in the end. My least favorite part was when he had to depart with his beloved pony. Even though he knew that he had done a good deed and his sister would no longer be a cripple, it broke his heart to acknowledge the fact that Dolly and he would be separated. Though, of course, he met Dolly again when his grandfather bought it using the money he saved. Yes, I would recommend this book to a friend because it taught how people could achieve their goals by putting their mind into it and persevering through challenges.
$ docker run --restart=always --name nd-export -d -p 9100:9100 -v /proc:/host/proc -v /sys:/host/sys -v /:/rootfs:ro,rslave --net=host quay.io/prometheus/node-exporter
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d8c7fc8a4e40 quay.io/prometheus/node-exporter "/bin/node_exporter" 32 minutes ago Up 32 minutes nd-export
Deploy Prometheus on a VM
$ vim /etc/prometheus/prometheus.yml
global:
scrape_configs:
- job_name: 'node'
scrape_interval: 10s
static_configs:
- targets: ['localhost:9100','host1-ip:9100','host2-ip:9100','host3-ip:9100','host4-ip:9100']
$ docker run --restart=always --name prometheus -d -p 9090:9090 -v /etc/prometheus:/etc/prometheus prom/prometheus:v1.8.2
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3700700b61c9 prom/prometheus:v1.8.2 "/bin/prometheus -co…" 57 minutes ago Up 22 minutes 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp prometheus
Verify the Prometheus by accessing http://vm-ip:9090/targets. You should be able to see the target nodes from the web UI.
Deploy Grafana on a VM
$ docker run --restart=always --name grafana -d -p 3000:3000 grafana/grafana
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
da48a274ecc0 quay.io/prometheus/node-exporter "/bin/node_exporter" 27 minutes ago Up 27 minutes nd-export
814511410bff grafana/grafana "/run.sh" 57 minutes ago Up 57 minutes 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp grafana
3700700b61c9 prom/prometheus:v1.8.2 "/bin/prometheus -co…" 57 minutes ago Up 22 minutes 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp prometheus
Configure the Promethous data source in Grafana
Login to the Grafana web UI by accessing http://vm-ip:3000/. The default username and password is admin/admin.
On the left corner, click “Add sources”.
Then add “Prometheus” as data source and fill in the prometheus data source name and the URL. Click Save & test. The data source name will be used when you create the Grafana dashboard later.
Create the Grafana dashboard
The Grafana dashboard can be created by importing an json template and binding to the previous prometheus data source.
