Flume demo with source,sink and channel

The basics of flume has already been covered in the previous blog post
http://www.maninmanoj.com/2018/10/flume-architecture-and-its-use-case.html

In this post, I'll explain about the use cases with simple example.

Scenario: 

An example conf file of flume with agent name "a1" this will run a webserver on port "44444" in local machine.

Step 1:

Create a sample file "example.conf" in the directory "/home/manoj/flume". The content of the file will look as below:


-----
# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
-------

Step 2:

Now start the webserver on port 44444 and flume agent with the example.conf file and keep it running on one SSH terminal.

-------
 flume-ng agent --name a1 --conf /home/manoj/flume/conf --conf-file /home/manoj/flume/conf/example.conf
------


Step 3:

Open a new SSH connect to the same server and connect to port "44444" as below:

Step 4:

Now if you type anything on telnet connection this will appear at same time on the SSH terminal where flume agent is running.

Output on the flume agent will be as below:

In the next post, I'll explain the flume agent with sink as HDFS.