Saturday, August 19, 2017

Introduction To Elastic Search, Kibana and X-Pack

Introduction :

- Elasticsearch is a search engine. It is powered by Lucene, an open-source full-text search library.
It provides a distributed  full-text search engine with an HTTP web interface and schema-free JSON documents.
- Kibana is the data visualization plugin for elasticsearch.
- X-Pack is an Elastic Stack extension that bundles security, alerting, monitoring, reporting, and graph capabilities into one easy-to-install package.

Installation :

Required:
Elasticsearch requires at least Java 8
$ java -version
java version "1.8.0_92"
Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)

Steps:

1) Download elastic search and kibana from https://www.elastic.co/start

* Following example is for Mac

$ ls
elasticsearch-5.5.0.tar.gz kibana-5.5.0-darwin-x86_64.tar.gz

2) Extract ElasticSearch and verify files and folders

$ tar zxf elasticsearch-5.5.0.tar.gz 
$ ls
elasticsearch-5.5.0 elasticsearch-5.5.0.tar.gz kibana-5.5.0-darwin-x86_64.tar.gz
$ cd elasticsearch-5.5.0
$ ls
LICENSE.txt NOTICE.txt README.textile bin config lib modules plugins

3) Run elasticsearch by typing command bin/elasticsearch from the parent folder.

$ bin/elasticsearch

[2017-08-19T02:41:02,805][INFO ][o.e.n.Node               ] [] initializing ...
….
[2017-08-19T02:41:08,001][INFO ][o.e.n.Node               ] [CkWGb2R] starting ...
[2017-08-19T02:41:08,233][INFO ][o.e.t.TransportService   ] [CkWGb2R] publish_address {127.0.0.1:9300}, bound_addresses {[fe80::1]:9300}, {[::1]:9300}, {127.0.0.1:9300}
[2017-08-19T02:41:11,427][INFO ][o.e.c.s.ClusterService   ] [CkWGb2R] new_master {CkWGb2R}{CkWGb2RYSG2kDz1hQdwRZw}{naQylIySRB-8gk5ZZXNbOw}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-08-19T02:41:11,509][INFO ][o.e.h.n.Netty4HttpServerTransport] [CkWGb2R] publish_address {127.0.0.1:9200}, bound_addresses {[fe80::1]:9200}, {[::1]:9200}, {127.0.0.1:9200}
[2017-08-19T02:41:11,509][INFO ][o.e.n.Node               ] [CkWGb2R] started

* Elastic is running on the node CkWGb2R which is created by default. Node name can be changed in config/elasticsearch.yml (node.name) before staring the elasticsearch.

* Its running on 127.0.0.1(localhost) port 9200

4) Open browser and load http://localhost:9200. You can see the following information.

{
  "name" : "CkWGb2R",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "OxfxTDEJSju3VUlX8GFpqw",
  "version" : {
    "number" : "5.5.0",
    "build_hash" : "260387d",
    "build_date" : "2017-06-30T23:16:05.735Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}

Cluster :
- A cluster is a collection of one or more nodes (servers) that together holds entire data and provides federated indexing and search capabilities across all nodes. 
- A cluster is identified by a unique name which by default is "elasticsearch". 

Node:
- A node is a single server that is part of the cluster, stores your data, and participates in the cluster’s indexing and search capabilities. 
- A node is identified by a name which by default is a random UUID that is assigned to the node at startup. One can define any node name if do not want the default.

5) Extract Kibana

$ tar zxf kibana-5.5.0-darwin-x86_64.tar.gz
$ ls
elasticsearch-5.5.0 elasticsearch-5.5.0.tar.gz kibana-5.5.0-darwin-x86_64 kibana-5.5.0-darwin-x86_64.tar.gz
$

6) Run Kibana in another terminal by typing command bin/kibana from the parent folder.

$ bin/kibana

  log   [21:21:53.232] [info][status][plugin:kibana@5.5.0] Status changed from uninitialized to green - Ready
  ….
  log   [21:21:53.907] [info][listening] Server running at http://localhost:5601
  log   [21:21:53.909] [info][status][ui settings] Status changed from uninitialized to yellow - Elasticsearch plugin is yellow
 ….
  log   [21:21:59.907] [info][status][ui settings] Status changed from yellow to green - Ready

7) Access http://localhost:5601 in browser





Kibana Dev Tools

- Go to http://localhost:5601, kibana UI -> Dev Tools -> Console. 
- Left panel to write query and execution. 
- Right panel will show the result.

Type "GET /" and run. 
This will fetch the current node and cluster information that we have already seen.



Terminology

Index:
- An index is a collection of documents having more or less similar characteristics. For ex, index for product data, index for customer data etc. 
- And index is identified by a lowercase name which is used to refer to the index when performing indexing, search, update, and delete operations against the documents in it.

Type:
- A type is a logical category/partition of your index whose semantics is completely up to you. 
- A type is defined for documents that have a set of common fields. 
- For example, let’s assume you run a e-commerce platform and store all your data in a single index. In this index, you may define a type for user data, another type for product data, and yet another type for feedback data.

Document:
- A document is a basic unit of information that can be indexed. 
- For example, you can have a document for a single customer, another document for a single product, and yet another for a single order. 
- This document is expressed in JSON

Create Document

Kibana UI -> Dev Tools -> Console

Type the following and RUN.

POST /my-index/my-type/1
{
  "body" : "test"
}

* This will create a document of id 1 of type 'my-type' under index 'my-index'. In order to index a document, we must tell Elasticsearch which type in the index it should go to.
* Elasticsearch does not require you to explicitly create an index first before you can index documents into it. In the previous example, Elasticsearch automatically created the 'my-index' index





* Elasticsearch has fast search responses because, instead of searching the text directly, it searches an index like retrieving pages in a book related to a keyword by scanning the index at the back of a book.

Get Document

GET /my-index/my-type/1



Search (API)

GET my-index/_search
{
  "query": {
    "match": {
      "body": "test"
    }
  }
}

Output
{
  "took": 24,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "my-index",
        "_type": "my-type",
        "_id": "1",
        "_score": 0.2876821,
        "_source": {
          "body": "test"
        }
      }
    ]
  }
}

Delete Document

DELETE /my-index/my-type/1

Output:
{
  "found": true,
  "_index": "my-index",
  "_type": "my-type",
  "_id": "1",
  "_version": 2,
  "result": "deleted",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  }
}

GET /my-index/my-type/1
{
  "_index": "my-index",
  "_type": "my-type",
  "_id": "1",
  "found": false
}


Install X-Pack

* Install at Elastic Side

- stop elastic search by Ctrl+C

- Install x-pack plugin

$ bin/elasticsearch-plugin install x-pack
-> Downloading x-pack from elastic
….
-> Installed x-pack

-Start elastic again

$ bin/elasticsearch
….
[2017-08-19T03:22:28,975][INFO ][o.e.p.PluginsService     ] [CkWGb2R] loaded plugin [x-pack]
….
[2017-08-19T03:22:33,927][INFO ][o.e.n.Node               ] [CkWGb2R] starting ...
….
[2017-08-19T03:22:37,479][INFO ][o.e.n.Node               ] [CkWGb2R] started
….

* Install x-pack UI at kibana side

- stop kibana by Ctrl+C
- Install x-pack plugin

$ bin/kibana-plugin install x-pack
Attempting to transfer from x-pack
Attempting to transfer from https://artifacts.elastic.co/downloads/kibana-plugins/x-pack/x-pack-5.5.0.zip
Transferring 119276235 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...
Plugin installation complete

- Restart kibana

$ bin/kibana
log   [22:01:43.618] [info][listening] Server running at http://localhost:5601
log   [22:01:43.620] [info][status][ui settings] Status changed from uninitialized to green - Ready

* Go to http://localhost:5601

You can find the login screen. This comes because of x-pack which gives role based permissions and authentication.



* Enter default username as ‘elastic’ and password as ‘changeme



* You can find more menu options in left side bar.
* Click on Monitoring and check elastic search and Kibana nodes.