Task 7.1. (A): 🔅Integrating LVM with Hadoop and
providing Elasticity to DataNode Storage
What is LVM?
In Linux, Logical Volume Manager is a device mapper framework that provides logical volume management for the Linux kernel. Most modern Linux distributions are LVM-aware to the point of being able to have their root file systems on a logical volume.
What is Hadoop?
Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation.
- For this, I have created two external HDD
right now we have a device that has a size of 10GB and 4GB respectively
Device Name1: /dev/sdb
Device Name2: /dev/sdc
- Create physical volume.
To do this we have the following command
# pvcreate “device 1” “device2”
you can add as many devices as you want
Here you can see we have created two physical volume
you can see these volumes using the following command
# pvdisplay “device names”
so, our PV(physical volumes) are ready to use
2. Now create a group of PV’s and it is called VOLUME GROUP(vg)
To create VG we have the following command
# vgcreate “name for the group” “devices that will part of the group”
And using the following command you can see VG created
# vgdisplay
3. Activate VG so we can work on this further otherwise it will give some errors
To do this we have the following command
# vgchange -a y “Volume group name”
-a: for activate
y: for yes
n: for no
4. Now from VG we need to create Logical Volumes (LV)
To do this we have the following command
# lvcreate -L “size” -n “any lv name” “VG ”
-L : for size
n : for name
we can see logical volume created byusing the following command
# lvdisplay “VG”
5.Now our Logical volumes are created now we have to format it and mount it on the Slave Node directory
To format use the following command
# mkfs.ext4 “LVM device name”
To mount this Logical Volume run the following command
# mount “device name”
6.Start the Hadoop cluster to see whether our setup is working fine
Increase The Storage On The Fly
LVM has the feature that without umounting the device we are able to increase storage on the fly
and this is the main key using this we can actually able to give Elasticity to our Slave Nodes so we can increase our storage up to any extent
so we have 2GB free so we can again extent current storage up to 15Gb
1. To increase LVM we have the following command
# lvextend — size + “size in GB” “device name”
Note: “ — ” represents two hyphens
2.Resize our file system so we get new memory blocks
To do this we have the following command
# resize2fs “device name”
so Finally we are able to increase our Slave Node storage on the fly without stopping cluster
To decrease the size of LVM:
As we know that we never reduce our storage in a live environment if you have such a case so I added this method also but we need to stop that data node otherwise will face a data loss issue
- umount the LVM device deactivate VG
To do this we have the following command
vgchange -a n “VG”
2. Then reduce the storage of VG by lvreduce cmd and again mount it on slave node directory
so we have changed our storage from 12GB to 11GB
- To remove the LVM use the cmds shown below:
Conclusion:-
So with help of the concept of Linux LVM we are able to increase/decrease the storage dynamically of the slave node that will be used in theHadoop cluster.
Thanks everyone for reading my article…..😇✍️✍️
