Cara menggunakan mongodb plus elasticsearch

MongoDB is a database which stores data without the need for a pre-established model ("strict description") of this data. In practice: I can save something into MongoDB without spending time creating tables and stuff. Just save a JSON doc, that’s it.

• A blog post which details how Elasticsearch helped on performances for Mongo: http://blog.quarkslab.com/mongodb-vs-elasticsearch-the-quest-of-the-holy-performances.html

• And Elasticsearch can then integrate with Kibana, a way to visualize query results. Awesome!

3. Installing MongoDB

source: https://docs.mongodb.com/manual/tutorial/install-mongodb-on-debian/

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 0C49F3730359A14518585931BC711F9BA15703C6
echo "deb http://repo.mongodb.org/apt/debian jessie/mongodb-org/3.4 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.4.list

sudo apt-get update

sudo apt-get install -y mongodb-org

disable Transparent Huge Pages as per https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/

create a new file:

sudo vi /etc/init.d/disable-transparent-hugepages

paste this in the text editor:

#!/bin/bash
### BEGIN INIT INFO
# Provides:          disable-transparent-hugepages
# Required-Start:    $local_fs
# Required-Stop:
# X-Start-Before:    mongod mongodb-mms-automation-agent
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description:       Disable Linux transparent huge pages, to improve
#                    database performance.
### END INIT INFO

case $1 in
  start)
    if [ -d /sys/kernel/mm/transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/transparent_hugepage
    elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/redhat_transparent_hugepage
    else
      return 0
    fi

    echo 'never' > ${thp_path}/enabled
    echo 'never' > ${thp_path}/defrag

    re='^[0-1]+$'
    if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]]
    then
      # RHEL 7
      echo 0  > ${thp_path}/khugepaged/defrag
    else
      # RHEL 6
      echo 'no' > ${thp_path}/khugepaged/defrag
    fi

    unset re
    unset thp_path
    ;;
esac

Make the file executable:

sudo chmod 755 /etc/init.d/disable-transparent-hugepages

Make the file to be ran on reboot:

sudo update-rc.d disable-transparent-hugepages defaults

Start Mongo:

sudo service mongod start

Check that it runs fine:

sudo cat /var/log/mongodb/mongod.log

→ There should be a line "[initandlisten] waiting for connections on port <port>"

And now stop it, as we will need to run it differently for elasticsearch:

sudo service mongod stop

3. Install Elasticsearch

source: https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

Download the public signing key:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Then:

sudo apt-get update

sudo apt-get update

sudo apt-get update

sudo apt-get update

sudo apt-get update

sudo apt-get update

4. Config Elasticsearch

sudo apt-get update

→ switch this param to true: bootstrap.memory_lock: true

You then need to make sure the JVM Heap size is no more than half the RAM. First fix a memory param:

sudo apt-get update

Add these lines:

LimitMEMLOCK=infinity

Adjust resource limits:

sudo apt-get update

Add line:

elasticsearch - nofile 65536

Add a jvm parameter:

sudo vi /etc/elasticsearch/jvm.options

Add this line:

-Djava.io.tmpdir=/var/tmp

5. Install the mongo to elasticsearch connection

a. elastic2-doc-manager

This is a doc manager by mongodb labs.

Source: https://github.com/mongodb-labs/elastic2-doc-manager

sudo apt-get update

b. run Mongo as a replicaset

sudo service mongod stop

Create the path for your db (if needed)

sudo mkdir -p /data/db

sudo vi /etc/init.d/disable-transparent-hugepages

sudo vi /etc/init.d/disable-transparent-hugepages

Then:

sudo vi /etc/init.d/disable-transparent-hugepages

sudo vi /etc/init.d/disable-transparent-hugepages

sudo vi /etc/init.d/disable-transparent-hugepages

6. Install kibana

Kibana is the visualization engine for elastic.

sudo vi /etc/init.d/disable-transparent-hugepages

Configure Kibana to start automatically at boot:

sudo vi /etc/init.d/disable-transparent-hugepages

7. Install X-pack

sudo vi /etc/init.d/disable-transparent-hugepages

• it might need to create an empty file named /et/default/elasticsearch)

• see https://discuss.elastic.co/t/installing-x-pack-with-nonstandard-conf-dir/76448/3

  INFO

  the second command (x-pack install for kibana) takes long minutes, that’s normal.

cd /usr/share/elasticsearch sudo bin/elasticsearch-plugin install x-pack

cd /usr/share/kibana sudo bin/kibana-plugin install x-pack

8. Disable the security component of X-Pack

This security component is hard to configure, and we don’t need it if we run elasticsearch behind a web server and a reverse proxy, on a single machine.

Add xpack.security.enabled: false

to /etc/elasticsearch/elasticsearch.yml

and to /etc/kibana/kibana.yml

Also in the same kibana.conf file, change the default username and passwd to "elastic" and "changeme" and leave the quotes

• start Elasticsearch: sudo /usr/share/elasticsearch/bin elasticsearch

• start Kibana: sudo /usr/share/kibana/bin kibana

9. Install the Mongo-connector for ElasticSearch

Source: https://blog.jixee.me/how-to-use-mongo-connector-with-elasticsearch/

sudo vi /etc/init.d/disable-transparent-hugepages

Edit the conf of Mongo to turn on replicasets:

#!/bin/bash
### BEGIN INIT INFO
# Provides:          disable-transparent-hugepages
# Required-Start:    $local_fs
# Required-Stop:
# X-Start-Before:    mongod mongodb-mms-automation-agent
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description:       Disable Linux transparent huge pages, to improve
#                    database performance.
### END INIT INFO

case $1 in
  start)
    if [ -d /sys/kernel/mm/transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/transparent_hugepage
    elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/redhat_transparent_hugepage
    else
      return 0
    fi

    echo 'never' > ${thp_path}/enabled
    echo 'never' > ${thp_path}/defrag

    re='^[0-1]+$'
    if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]]
    then
      # RHEL 7
      echo 0  > ${thp_path}/khugepaged/defrag
    else
      # RHEL 6
      echo 'no' > ${thp_path}/khugepaged/defrag
    fi

    unset re
    unset thp_path
    ;;
esac

Uncomment "replication", add two lines:

#!/bin/bash
### BEGIN INIT INFO
# Provides:          disable-transparent-hugepages
# Required-Start:    $local_fs
# Required-Stop:
# X-Start-Before:    mongod mongodb-mms-automation-agent
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description:       Disable Linux transparent huge pages, to improve
#                    database performance.
### END INIT INFO

case $1 in
  start)
    if [ -d /sys/kernel/mm/transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/transparent_hugepage
    elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/redhat_transparent_hugepage
    else
      return 0
    fi

    echo 'never' > ${thp_path}/enabled
    echo 'never' > ${thp_path}/defrag

    re='^[0-1]+$'
    if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]]
    then
      # RHEL 7
      echo 0  > ${thp_path}/khugepaged/defrag
    else
      # RHEL 6
      echo 'no' > ${thp_path}/khugepaged/defrag
    fi

    unset re
    unset thp_path
    ;;
esac

#!/bin/bash
### BEGIN INIT INFO
# Provides:          disable-transparent-hugepages
# Required-Start:    $local_fs
# Required-Stop:
# X-Start-Before:    mongod mongodb-mms-automation-agent
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description:       Disable Linux transparent huge pages, to improve
#                    database performance.
### END INIT INFO

case $1 in
  start)
    if [ -d /sys/kernel/mm/transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/transparent_hugepage
    elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/redhat_transparent_hugepage
    else
      return 0
    fi

    echo 'never' > ${thp_path}/enabled
    echo 'never' > ${thp_path}/defrag

    re='^[0-1]+$'
    if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]]
    then
      # RHEL 7
      echo 0  > ${thp_path}/khugepaged/defrag
    else
      # RHEL 6
      echo 'no' > ${thp_path}/khugepaged/defrag
    fi

    unset re
    unset thp_path
    ;;
esac

10. Start elasticsearch and Kibana

#!/bin/bash
### BEGIN INIT INFO
# Provides:          disable-transparent-hugepages
# Required-Start:    $local_fs
# Required-Stop:
# X-Start-Before:    mongod mongodb-mms-automation-agent
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description:       Disable Linux transparent huge pages, to improve
#                    database performance.
### END INIT INFO

case $1 in
  start)
    if [ -d /sys/kernel/mm/transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/transparent_hugepage
    elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/redhat_transparent_hugepage
    else
      return 0
    fi

    echo 'never' > ${thp_path}/enabled
    echo 'never' > ${thp_path}/defrag

    re='^[0-1]+$'
    if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]]
    then
      # RHEL 7
      echo 0  > ${thp_path}/khugepaged/defrag
    else
      # RHEL 6
      echo 'no' > ${thp_path}/khugepaged/defrag
    fi

    unset re
    unset thp_path
    ;;
esac

You can check that the connection is made here, your Mongo collections should be listed on this page: