APACHE FLUME HOW TO

A Sink specifies where the data obtained from the source should be output. Now I have connected to a data source, I need somewhere to write the data that it collects. There are some additional settings that have been set: maxBatchSize means that upto 1000 tweets will be collected before being processed as a batch, maxBatchDurationMillis sets the max number of milliseconds to wait before a batch is considered done. The consumerKey, consumerSecret, accessToken and accessTokenSecret should be filled in using the values given to you by Twitter. keywords = Hadoop, bigdata, spark, hive, Hbase, flume TwitterAgent. maxBatchDurationMillis = 100000 TwitterAgent. accessTokenSecret = 0 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TwitterAgent. accessToken = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TwitterAgent. consumerSecret = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TwitterAgent. consumerKey = XXXXXXXXXXXXXXXX TwitterAgent.

The mandatory properties for the flume source are: channels, type, consumerKey, consumerSecret, accessToken and accessTokenSecret. This can be done by going to and creating a new app. To connect to a twitter source first we need to obtain a set of keys by registering as a twitter app developer. Some of these attributes are mandatory, others are optional meaning their default value will be set if they arent specified within the config file. There are many different sources that can be connected to, for this example I will be connecting to Twitter.Įach source will have a different set of attributes that can be set. What is a Source?Ī Source referes to a data source that can be connected to using Flume.

APACHE FLUME HOW TO

I will demonstrate how to configure the Source and Sink in the next few sections. Here I have defined a Channel named TwitterChannel, a Source named Twitter and a Sink named HDFS. I will create a Flume configuration file to demonstrate how to set up a channel followed by a source and sink. It has only one default property and that is its type. An agent is started as follows:įlume-ng agent -n -c conf -f What is a Channel?Ī Channel is used to stage events on an agent using an in memory queue. This gets passed to the flume application as a parameter when it’s started. An Agent is a JVM process that contains a Source, a Channel and a Sink that all contribute to obtaining an event and storing it on the target data store.Īgents are configured using a configuration file that is like a Java properties file. What is an Agent?įlume data flows are defined as events and all events pass through an Agent. So what does this mean? Essentially, flume can efficiently stream data from a user defined source to a user defined target, continuously! This allows continued ingestion of data from a source that is always producing data, hence why log capturing is a main use case, however mining social media data is also a good use case. For the purposes of this blog, the central storage location will be a Hadoop cluster. Apache Flumeįlume is a distributed service dedicated to aggregating and transporting large amount of event data (namely log data) from many different sources into a central storage location. Users who are interested in Apache Flume may also be interested in Confluent and Apache Kafka, since it can be deployed to solve a lot of the same ingestion problems in addition to offering other capabilities such as real-time stream processing, broad connectivity, distributed publish / subscribe, and event-driven microservices architectures.In this post I will be discussing some simple use cases and important features of Apache Flume. This speaks to the more generalized utility of Kafka as a framework. When Flume and Kafka are integrated, data is usually moving from Flume to Kafka. It’s worth noting that although it is called a “sink” connector from the perspective of Flume, Flume usually serves as a source from the point of view of Kafka. Likewise, there is a source connector for Flume in the Kafka Connect ecosystem. Organizations that use both Flume and Kafka together have a variety of choices, such as a Kafka sink in Flume (dubbed “Flafka”) where Flume can be configured as a Kafka producer.

Moreover, Kafka’s capabilities are often considered as a superset of Flume’s. While Flume is primarily used for log ingestion, specifically to Hadoop, for batch processing, Kafka can also be used for this.

Apache flume