ELK and Kafka Zookeeper Configuration on Oracle BDA

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...

First lets configure Kafka

Note: In Kafka 8.1 (and lower) you can not delete a topic, make sure to update to a latter version then 8.2+

Modify the below Kafka settings

To make kafka listing on all interfaces

First lets create a topic

Note: The instructions below is missing initial zookeeper setup which is alrady done on the Oracle BDA.

To watch the active queue

To delete a topic (with all messages)

Make sure to stop all logstash instances using this topic
Also good to stop all kafka brokers

Newer versions, full clean up

Note: if delete is not working, stop Kafka, then

Kafka helpful hints

To get all messages offset

To see the number of PartitionCount, ReplicationFactor, etc

To modify/add a PartitionCount

The below will change it to two partitions

Logstash and Kafka configuration

On the Client side, lets create a logstash output with Kafka output

logstash.conf

Start logstash with SMF or at commend line

For SMF details check here

On the server side (Kafka consumer), lets create a logstash input with Kafka input and elasticsearch output

logstash.conf
Note: In Linux its in /etc/logstash/conf.d/50-logstash.conf

Start logstash with SMF or at commend line

To increase logstash Memory, add below

Log locations

Default logstash logs

Default Kafka logs (on BDA)

For SMF details check here

Leave a Reply

avatar
  Subscribe  
Notify of