DevTech101

DevTech101

elk

Collectd , logstash and Elasticsearch

Install Collectd from opencsw /opt/csw/bin/pkgutil -i collectd /opt/csw/bin/pkgutil -i collectd-utils Modify collectd.conf vi /etc/opt/csw/collectd.conf Hostname “yourhost” FQDNLookup false LoadPlugin syslog LoadPlugin cpu LoadPlugin df LoadPlugin disk LoadPlugin interface LoadPlugin load LoadPlugin memory LoadPlugin network LoadPlugin rrdtool LoadPlugin zfs_arc Server “10.10.10.11” “3333” Restart or enable collectd svcadm enable svc:/network/cswcollectd:default forward the logs to elasticsearch with logstash …

Collectd , logstash and Elasticsearch Read More »

Configuring Elasticsearch betas/topbeats on linux

First, lets Download the beats template. curl -O https://raw.githubusercontent.com/elastic/topbeat/master/etc/topbeat.template.json Upload to elastic template curl -XPUT ‘http://elk3.domain.com:9200/_template/topbeat’ -d@topbeat.template.json Now, lets install topbeat, add elastic beat repo, and run yum -y install topbeat Modify /etc/topbeat/topbeat.yml, under elasticsearch add elastic (or logstash) hosts hosts: [“10.10.3.48:9204”, “10.10.3.48:9205”, “10.10.3.48:9206”] Start the beat service systemctl start topbeat Install topbeats dashboards curl …

Configuring Elasticsearch betas/topbeats on linux Read More »

Upgrading Elasticsearch To A Newer Version

How to upgrading Elasticsearch To A Newer Version The key is exporting the instance. Disable shard movement curl -XPUT http://elk2.domain.com:9200/_cluster/settings -d ‘ { “transient”: { “cluster.routing.allocation.enable”: “none” } }’ Migrate to new version (on linux) export INS=1 alias cp=cp cd /opt /opt/elasticsearch-ins${INS}/bin/start_elastic.sh stop unzip -qq /var/tmp/elasticsearch-2.3.0.zip mv elasticsearch-ins${INS} elasticsearch-ins${INS}-2.2 mv elasticsearch-2.3.0 elasticsearch-ins$INS cd elasticsearch-ins${INS} cp …

Upgrading Elasticsearch To A Newer Version Read More »

How to modify an active index type in elasticsearch

Elasticsearch: updating the mappings and settings of an existing index Note: This was written using elasticsearch 0.9. Elasticsearch will automatically create an index (with basic settings and mappings) for you if you post a first document: $ curl -X POST ‘http://localhost:9200/thegame/weapons/1’ -d \ ‘{ “_id”: 1, “name”: “Longsword”, “description”: “The Longsword can be wielded in …

How to modify an active index type in elasticsearch Read More »

Elasticsearch datatypes

Field datatypes edit On this page Core datatypes Complex datatypes Geo datatypes Specialised datatypes Multi-fields Elasticsearch Reference: Getting Started Setup Breaking changes API Conventions Document APIs Search APIs Aggregations Indices APIs cat APIs Cluster APIs Query DSL Mapping Field datatypes Array datatype Binary datatype Boolean datatype Date datatype Geo-point datatype Geo-Shape datatype IPv4 datatype Nested …

Elasticsearch datatypes Read More »

Logstash Nagios Configuration Grok example (logstash.conf)

Nagios logstash yml conf file input { file { # Wildcards work, here 🙂 path => [ “/var/log/messages” ] start_position => “beginning” type => “nagios-alert” } } #input { #tcp { #host => “10.10.10.1” #port => 3333 #type => “nagios-alert” #} #} filter { if [type] == “nagios-alert” { if [message] =~ /nagios3 nagios:/ { …

Logstash Nagios Configuration Grok example (logstash.conf) Read More »

elasticserach mappings

sql-log mappings “mappings”: { “sql-log”: { “properties”: { “sql_session_id”: { “store”: true, “type”: “string” }, “sql_duration_min”: { “store”: false, “index”: “not_analyzed”, “type”: “integer” }, “sql_duration_subsec”: { “store”: true, “index”: “analyzed”, “type”: “integer” }, “hostname”: { “store”: true, “type”: “string” }, “type”: { “type”: “string”, “store”: true }, “message”: { “type”: “string”, “index”: “not_analyzed”, “store”: false …

elasticserach mappings Read More »

Solaris Elasticsearch Forwarder(Logstash) Setup

Create elastic data pool zpool create data1 c1d1 Add elastic user groupadd elastic useradd -d /export/home/elastic -g elastic -m -s /bin/bash -c “Elastic Search” elastic Install elastic binary’s cd /opt;unzip -qq /install/elasticsearch-2.2.0.zip;mv elasticsearch-2.2.0 elasticsearch mkdir /data1 /data1/data /data1/plugins /data1/log mkdir /opt/elasticsearch/current /opt/elasticsearch/current/logs /opt/elasticsearch/current/data /opt/elasticsearch/current/tmp chown -R elastic:elastic /opt/elasticsearch /data1 Grant access for elastic user echo …

Solaris Elasticsearch Forwarder(Logstash) Setup Read More »