In this post ...
Retrieve and query the remote linux machine message logging by a visualization platform

GOTO

Previous post: Retrieve and query the remote linux machine message logging
Next step ... ;)
Github repository

Note:

$ --> execute the command as cloudera user
# --> execute the command as root user
Remote server = Big Data server = localhost . In this example the remote machine to monitoring is the Big Data server in order to simplified the architecture.

Install elasticsearch

1. Download elasticsearch version 1.7.3. I choose this version in order to avoid library conflicts between Cloudera Virtual Machine and elasticsearch dependencies.

2. Extract elasticsearch software

    
   
           $ unzip ~cloudera/Downloads/elasticsearch-1.7.3.zip -d ~cloudera/ $ mv ~cloudera/elasticsearch-1.7.3 ~cloudera/elasticsearch  

   
   

3. Start elasticsearch

    
   
           $ ~cloudera/elasticsearch/bin/elasticsearch  

   
   

4. Run a test with the following command

    
   
           $ curl -X GET http://localhost:9200/  

   
   

Install kibana

1. Download kibana version 4.1.1

2. Extract kibana

    
   
           $ cd ~cloudera $ tar -xvf ~cloudera/Downloads/kibana-4.1.1-linux-x64.tar.gz $ mv ~cloudera/kibana-4.1.1-linux-x64 ~cloudera/kibana  

   
   

3. Start Kibana

    
   
           $ ~cloudera/kibana/bin/kibana  

   
   

4. Open in the browser http://quickstart:5601

Set up Flume agent

1. Create a directory from a user directory and change the current directory

    
   
           $ mkdir ~cloudera/monitoring-with-big-data-technologies-part-three $ cd ~cloudera/monitoring-with-big-data-technologies-part-three  

   
   

2. Copy previous flume agent files

    
   
           cp ../monitoring-with-big-data-technologies-part-two/step-3/* ./  

   
   

3. Replace in flume.properties and log4j.properties with the new flume agent

    
   
           $ sed -i 's/syslogAgentStep3/syslogAgentPartThree/g' flume.properties $ sed -i 's/flume-step3.log/flume-part-three.log/g' log4j.properties $ sed -i 's/monitoring-with-big-data-technologies-part-two\/step-3/monitoring-with-big-data-technologies-part-three/g' *.* $ sed -i 's/monitoring-with-big-data-technologies-part-two/monitoring-with-big-data-technologies-part-three/g' log4j.properties  

   
   

4. Add elasticsearch sink to flume agent by Replace line 74-76 with the following lines in flume.properties file:

    
   
           # Define ElasticSearch (es) as a new flume sink syslogAgentPartThree.sinks = es syslogAgentPartThree.sinks.es.type = elasticsearch # Comma separated list of hostname:port syslogAgentPartThree.sinks.es.hostNames = localhost:9200,localhost:9300 # The name of the index which the date will be appended to syslogAgentPartThree.sinks.es.indexName = monitoring # The type to index the document to syslogAgentPartThree.sinks.es.indexType = syslog # Default elastic search cluster syslogAgentPartThree.sinks.es.clusterName = elasticsearch # Number of events to be written per txn syslogAgentPartThree.sinks.es.batchSize = 10 # TTL in days syslogAgentPartThree.sinks.es.ttl = 10d syslogAgentPartThree.sinks.es.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer 
    
    # Binding source/channel/sink syslogAgentPartThree.sources.r1.channels = c1 syslogAgentPartThree.sinks.s1.channel = c1 syslogAgentPartThree.sinks.es.channel = c1  

   
   

5. Start elasticsearch

    
   
           $ ~cloudera/elasticsearch/bin/elasticsearch  

   
   

6. Add in flume-env.sh in order to include in flume classpath elastichsearch library

    
   
           export FLUME_CLASSPATH=$FLUME_CLASSPATH:`find ~cloudera/elasticsearch/lib/ -type f -name *.jar | tr "\n" ":"`  

   
   

7. Start Flume agent with the following command

    
   
           $ flume-ng agent --conf "./" --conf-file "./flume.properties" -n syslogAgentPartThree -Dlog4j.configuration=file:./log4j.properties  

   
   

8. Monitoring Flume agent log with the following command from another tab

    
   
           $ tail -F ~cloudera/monitoring-with-big-data-technologies-part-three/log/flume-part-three.log  

   
   

9. Send from another tab a test message using the logger command to test it

    
   
           $ logger -t test 'Testing Flume with rsyslog!'  

   
   

10. Retrieve via curl syslog message from elastichsearch

     
    
            $ curl 'http://localhost:9200/_search?pretty=true'  

    
    

Create a simple dashboard

I just prepared a video in oder to show how to create a dashboard.
Please visit elasticsearch for more informations.

GOTO

Previous post: Retrieve and query the remote linux machine message logging
Next step ... ;)
Github repository

___---*** That's all ***---___

Happy coding ;)