MongoDB is a database which stores data without the need for a pre-established model ("strict description") of this data. In practice: I can save something into MongoDB without spending time creating tables and stuff. Just save a JSON doc, that’s it.
MongoDB alone is great, but I will store gigabytes of data, with several text fields and some simple graph logic as well. Elasticsearch is known for managing well the indexes and queries related to these data types.
A blog post which details how Elasticsearch helped on performances for Mongo: //blog.quarkslab.com/mongodb-vs-elasticsearch-the-quest-of-the-holy-performances.html
And Elasticsearch can then integrate with Kibana, a way to visualize query results. Awesome!
3. Installing MongoDB
source: //docs.mongodb.com/manual/tutorial/install-mongodb-on-debian/
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 0C49F3730359A14518585931BC711F9BA15703C6 echo "deb //repo.mongodb.org/apt/debian jessie/mongodb-org/3.4 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.4.list
sudo apt-get update
sudo apt-get install -y mongodb-org
disable Transparent Huge Pages as per //docs.mongodb.com/manual/tutorial/transparent-huge-pages/
create a new file:
sudo vi /etc/init.d/disable-transparent-hugepages
paste this in the text editor:
#!/bin/bash ### BEGIN INIT INFO # Provides: disable-transparent-hugepages # Required-Start: $local_fs # Required-Stop: # X-Start-Before: mongod mongodb-mms-automation-agent # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Disable Linux transparent huge pages # Description: Disable Linux transparent huge pages, to improve # database performance. ### END INIT INFO case $1 in start) if [ -d /sys/kernel/mm/transparent_hugepage ]; then thp_path=/sys/kernel/mm/transparent_hugepage elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then thp_path=/sys/kernel/mm/redhat_transparent_hugepage else return 0 fi echo 'never' > ${thp_path}/enabled echo 'never' > ${thp_path}/defrag re='^[0-1]+$' if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]] then # RHEL 7 echo 0 > ${thp_path}/khugepaged/defrag else # RHEL 6 echo 'no' > ${thp_path}/khugepaged/defrag fi unset re unset thp_path ;; esac
Make the file executable:
sudo chmod 755 /etc/init.d/disable-transparent-hugepages
Make the file to be ran on reboot:
sudo update-rc.d disable-transparent-hugepages defaults
Start Mongo:
sudo service mongod start
Check that it runs fine:
sudo cat /var/log/mongodb/mongod.log
→ There should be a line "[initandlisten] waiting for connections on port <port>"
And now stop it, as we will need to run it differently for elasticsearch:
sudo service mongod stop
3. Install Elasticsearch
source: //www.elastic.co/guide/en/elasticsearch/reference/current/deb.html
Download the public signing key:
wget -qO - //artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Then:
sudo apt-get update0
sudo apt-get update1
sudo apt-get update
sudo apt-get update3
sudo apt-get update4
sudo apt-get update5
4. Config Elasticsearch
sudo apt-get update6
→ switch this param to true: bootstrap.memory_lock: true
You then need to make sure the JVM Heap size is no more than half the RAM. First fix a memory param:
sudo apt-get update7
Add these lines:
LimitMEMLOCK=infinity
Adjust resource limits:
sudo apt-get update8
Add line:
elasticsearch - nofile 65536
Add a jvm parameter:
sudo vi /etc/elasticsearch/jvm.options
Add this line:
-Djava.io.tmpdir=/var/tmp
5. Install the mongo to elasticsearch connection
a. elastic2-doc-manager
This is a doc manager by mongodb labs.
Source: //github.com/mongodb-labs/elastic2-doc-manager
sudo apt-get update9
b. run Mongo as a replicaset
sudo service mongod stop
Create the path for your db (if needed)
sudo mkdir -p /data/db
sudo vi /etc/init.d/disable-transparent-hugepages1
sudo vi /etc/init.d/disable-transparent-hugepages2
Then:
sudo vi /etc/init.d/disable-transparent-hugepages3
sudo vi /etc/init.d/disable-transparent-hugepages4
sudo vi /etc/init.d/disable-transparent-hugepages5
6. Install kibana
Kibana is the visualization engine for elastic.
sudo vi /etc/init.d/disable-transparent-hugepages6
Configure Kibana to start automatically at boot:
sudo vi /etc/init.d/disable-transparent-hugepages7
7. Install X-pack
sudo vi /etc/init.d/disable-transparent-hugepages8
it might need to create an empty file named /et/default/elasticsearch)
see //discuss.elastic.co/t/installing-x-pack-with-nonstandard-conf-dir/76448/3
INFO
the second command (x-pack install for kibana) takes long minutes, that’s normal.
cd /usr/share/elasticsearch sudo bin/elasticsearch-plugin install x-pack
cd /usr/share/kibana sudo bin/kibana-plugin install x-pack
8. Disable the security component of X-Pack
This security component is hard to configure, and we don’t need it if we run elasticsearch behind a web server and a reverse proxy, on a single machine.
Add xpack.security.enabled: false
to /etc/elasticsearch/elasticsearch.yml
and to /etc/kibana/kibana.yml
Also in the same kibana.conf file, change the default username and passwd to "elastic" and "changeme" and leave the quotes
start Elasticsearch: sudo /usr/share/elasticsearch/bin elasticsearch
start Kibana: sudo /usr/share/kibana/bin kibana
9. Install the Mongo-connector for ElasticSearch
Source: //blog.jixee.me/how-to-use-mongo-connector-with-elasticsearch/
sudo vi /etc/init.d/disable-transparent-hugepages9
Edit the conf of Mongo to turn on replicasets:
#!/bin/bash ### BEGIN INIT INFO # Provides: disable-transparent-hugepages # Required-Start: $local_fs # Required-Stop: # X-Start-Before: mongod mongodb-mms-automation-agent # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Disable Linux transparent huge pages # Description: Disable Linux transparent huge pages, to improve # database performance. ### END INIT INFO case $1 in start) if [ -d /sys/kernel/mm/transparent_hugepage ]; then thp_path=/sys/kernel/mm/transparent_hugepage elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then thp_path=/sys/kernel/mm/redhat_transparent_hugepage else return 0 fi echo 'never' > ${thp_path}/enabled echo 'never' > ${thp_path}/defrag re='^[0-1]+$' if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]] then # RHEL 7 echo 0 > ${thp_path}/khugepaged/defrag else # RHEL 6 echo 'no' > ${thp_path}/khugepaged/defrag fi unset re unset thp_path ;; esac0
Uncomment "replication", add two lines:
#!/bin/bash ### BEGIN INIT INFO # Provides: disable-transparent-hugepages # Required-Start: $local_fs # Required-Stop: # X-Start-Before: mongod mongodb-mms-automation-agent # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Disable Linux transparent huge pages # Description: Disable Linux transparent huge pages, to improve # database performance. ### END INIT INFO case $1 in start) if [ -d /sys/kernel/mm/transparent_hugepage ]; then thp_path=/sys/kernel/mm/transparent_hugepage elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then thp_path=/sys/kernel/mm/redhat_transparent_hugepage else return 0 fi echo 'never' > ${thp_path}/enabled echo 'never' > ${thp_path}/defrag re='^[0-1]+$' if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]] then # RHEL 7 echo 0 > ${thp_path}/khugepaged/defrag else # RHEL 6 echo 'no' > ${thp_path}/khugepaged/defrag fi unset re unset thp_path ;; esac1
#!/bin/bash ### BEGIN INIT INFO # Provides: disable-transparent-hugepages # Required-Start: $local_fs # Required-Stop: # X-Start-Before: mongod mongodb-mms-automation-agent # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Disable Linux transparent huge pages # Description: Disable Linux transparent huge pages, to improve # database performance. ### END INIT INFO case $1 in start) if [ -d /sys/kernel/mm/transparent_hugepage ]; then thp_path=/sys/kernel/mm/transparent_hugepage elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then thp_path=/sys/kernel/mm/redhat_transparent_hugepage else return 0 fi echo 'never' > ${thp_path}/enabled echo 'never' > ${thp_path}/defrag re='^[0-1]+$' if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]] then # RHEL 7 echo 0 > ${thp_path}/khugepaged/defrag else # RHEL 6 echo 'no' > ${thp_path}/khugepaged/defrag fi unset re unset thp_path ;; esac2
10. Start elasticsearch and Kibana
#!/bin/bash ### BEGIN INIT INFO # Provides: disable-transparent-hugepages # Required-Start: $local_fs # Required-Stop: # X-Start-Before: mongod mongodb-mms-automation-agent # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Disable Linux transparent huge pages # Description: Disable Linux transparent huge pages, to improve # database performance. ### END INIT INFO case $1 in start) if [ -d /sys/kernel/mm/transparent_hugepage ]; then thp_path=/sys/kernel/mm/transparent_hugepage elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then thp_path=/sys/kernel/mm/redhat_transparent_hugepage else return 0 fi echo 'never' > ${thp_path}/enabled echo 'never' > ${thp_path}/defrag re='^[0-1]+$' if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]] then # RHEL 7 echo 0 > ${thp_path}/khugepaged/defrag else # RHEL 6 echo 'no' > ${thp_path}/khugepaged/defrag fi unset re unset thp_path ;; esac3
You can check that the connection is made here, your Mongo collections should be listed on this page: