TL;DR Summary: Instructions for setting up an ElasticSearch version 7 cluster using Ubuntu 16.04 are provided, including server configuration and software installation steps. Troubleshooting tips and next steps for production readiness are also included.
- Install ElasticSearch and Kibana with specific configurations.
- Open necessary firewall ports for operation.
- Troubleshooting options for cluster issues are detailed.
- Next steps include memory adjustments and proxy server configuration.
Here we will install ElasticSearch and Kibana using Ubuntu 16.04 and ElasticSearch 7.0.1. At the bottom are some troubleshooting notes.
ElasticSearch has become a public company now, having made an IPO and sold shares. One of its first actions was to create ElasticSearch version 7. Unfortunately the installation instructions for how to set up a cluster with version have changed. So we have written new instructions. The main difference from what we wrote below is there is no need to add any discovery plugins or configure those.
So these instructions replace the version 6 instructions we wrote here.
Server names
For this example we have two servers. Change the names and IP addresses to match your environment. Don't use loopback address as we will make a cluster of two servers. Instead use their internal IP addresses.
172.31.46.15 parisx
172.31.47.43 paris2x
And we have this public IP address so we can access Kibana from the internet:
ec2-35-180-186-122.eu-west-3.compute.amazonaws.com:5601
Firewall Ports
Open firewall ports 9200 (http interface), 9300 (transport), and 5601 (KIbana.)
Install Software
Execute these instructions:
Troubleshooting
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.0.1-amd64.deb
sudo dpkg -i elasticsearch-7.0.1-amd64.deb
https://artifacts.elastic.co/downloads/kibana/kibana-7.0.1-linux-x86_64.tar.gz
sudo dpkg -i kibana-7.0.1-amd64.deb
Create ElasticSearch Configuration File
On the paris server replace this file /etc/elasticsearch/elasticsearch.yml with:
cluster.name: paris
node.name: parisx
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.31.46.15
discovery.seed_hosts: ["172.31.46.15", "172.31.47.43"]
cluster.initial_master_nodes: ["172.31.46.15"]
node.master: true
On the paris2 server replace this file /etc/elasticsearch/elasticsearch.yml with the contents shown below.
cluster.name: paris
node.name: paris2x
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.31.47.43
discovery.seed_hosts: ["172.31.47.43","172.31.46.15"]
cluster.initial_master_nodes: ["172.31.46.15"]
Start ElasticSearch on each server:
sudo service elasticsearch start
Check the cluster status. You should see 2 nodes if everything is working.
curl -XGET http://172.31.46.15:9200/_cluster/health?pretty
{
"cluster_name" : "paris",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 2,
"active_shards" : 4,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
Configure Kibana
Replace the config file /etc/kibana/kibana.yml with this:
server.host: "172.31.46.15"
server.name: "paris"
elasticsearch.hosts: ["http://172.31.46.15:9200"]
elasticsearch.preserveHost: false
logging.dest: stdout
logging.verbose: true
Start Kibana. It might take a minute or longer to start.
sudo service kibana start
Open the Kibana Console
http://ec2-35-180-186-122.eu-west-3.compute.amazonaws.com:5601/app/kibana
It might take a few minutes to download the graphics and to start Kibana as well. The dashboard will appear:
- If either node cannot join the cluster then erase the /var/lib/elasticsearch folder as the cluster UUIDs probably don't match. This will happen if you make mistakes and restart the servers with different settings.
- Make sure port 5601 is open in the outbound direction. In Europe, because of the GDPR requirements, companies have stopped opening all ports in the outbound direction by default.
- If you want to run either ElasticSearch or Kibana in the foreground you can do that. This is only for troubleshooting purposes. The Kibana configuration above is set to log output to stdout so you would have to run it in the foreground to see that or set the logs to write to disk. You can run Kibana as root. But you cannot run ElasticSearch as root. So you would have to change the ownership of /usr/share/elasticsearch, /var/log/elasticsearch, and /var/lib/elasticsearch to your userid or login as user elasticsearch to run elasticsearch in the foreground. You will find both binaries in /usr/share/kibana and /usr/share/elasticsearch folders.
- Look for ElasticSearch errors in /var/log/elasticsearch/paris.log. The file name is the node name.
curl http://172.31.47.43:9200
{
"name" : "paris2x",
"cluster_name" : "paris",
"cluster_uuid" : "uB_1NLmYRbKcVbKGPzsNSQ",
"version" : {
"number" : "7.0.1",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "e4efcb5",
"build_date" : "2019-04-29T12:56:03.145736Z",
"build_snapshot" : false,
"lucene_version" : "8.0.0",
"minimum_wire_compatibility_version" : "6.7.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
Next Steps
The next steps before going to production would be to increase the memory size to ½ of the memory of the machine (recommended by ElasticSearch) in /etc/elasticsearch/jvm.options by changing the values below.
-Xms1g
-Xmx1g
You would also want to put a proxy server in front of this so that you can configure basic authentication and ssh. (You can configure ssh in ElasticSearch as well.)
ElasticSearch Guide
- ElasticSearch Tutorial for Beginners: ElasticSearch Basics
- Using Apache Hive with ElasticSearch
- Using ElasticSearch with Apache Spark
- Curl ElasticSearch Commands Cheat Sheet
- Spark ElasticSearch Hadoop Update and Upsert Example and Explanation
- How To Write Apache Spark Data to ElasticSearch Using Python
- ElasticSearch Search Syntax and Boolean and Aggregation Searches
- ElasticSearch Nested Queries: How to Search for Embedded Documents
- ElasticSearch Joins: Has_Child, Has_parent query
- How to Setup up an Elastic Version 7 Cluster
- How to Configure Filebeat for nginx and ElasticSearch
- Using Beats and Logstash to Send Logs to ElasticSearch
- How to Load CSV File into ElasticSearch with Logstash
- ElasticSearch Aggregations Explained
- What is the ELK Stack?
- Logstash 101: Using Logstash in a Data Processing Pipeline
- Using Kibana to Execute Queries in ElasticSearch using Lucene and Kibana Query Language
- ElasticSearch Machine Learning
- How to Install Elastic Enterprise Search
- How To Use Elastic Enterprise Search with GitHub
These postings are my own and do not necessarily represent BMC's position, strategies, or opinion.