The below post describes how to configure Flum to grab data from a Kafaka topic as the source then send the data to an HDFS target. Pipeline flow Original source > Kafka topic > Flum from (Kafka topic) > HDFS Create a run file agent_name=flume1 flume-ng agent -n $agent_name -c conf -f conf/flume.conf # Debug …
Kafka To Flume PipeLing To HDFS And Elasticsearch Read More »
First lets configure Kafka Note: In Kafka 8.1 (and lower) you can not delete a topic, make sure to update to a latter version then 8.2+ Modify the below Kafka settings message.max.bytes: 5000000 replica.fetch.max.bytes: 5 max.connections.per.ip: 500 To make kafka listing on all interfaces Added the following to the Advanced Snippet text field: listeners=PLAINTEXT://0.0.0.0:9092,SSL://127.0.0.1:9093 #Note: …
ELK and Kafka Zookeeper Configuration on Oracle BDA Read More »
To update Kafka on an Oracle BDA (or on CM), follow the process below Log into CM as admin user and check if KAFKA parcel 0.8.2.0-1.kafka1.3.1.p0.9 is available for download. Go to Hosts –> Parcels Check if KAFKA parcel 0.8.2.0-1.kafka1.3.1.p0.9 is available for download Note: Version may differ depending on CDH 5.4.* release If your …