Apache Hadoop Review

Handle Explosions In Data With Big Data Technology

Apache Hadoop is a big data technology which means it offers an ecosystem, framework, and technology built to process large amounts of data. As companies and organizations evolve and grow, they also have to deal with explosions in data. These are situations or instances wherein they need to process and manage large data sets, and meet the challenges associated with a technological world which is becoming  more information-driven.

Highly-Scalable Framework That Ensures High-Availability

This big data technology is a highly-scalable solution. Apache Hadoop can automatically scale up as the number of servers and machines required to process, store, and analyze large data sets expands. What’s great about this is that the  computing technology eliminates reliance on hardware whenever it needs to scale up. It distributes large data sets across clusters of servers and machines and handles intensive parallel computing on those clusters. In case, errors or failures happen within  each cluster of servers or computers, Apache Hadoop can immediately detect them and provides ways to remediate the issues to ensure high-availability.

Reliable Distributed File System

Apache Hadoop delivers a distributed file system known as HDFS or Hadoop Distributed File System. How does this file system work? The HDFS splits large data files into blocks that are arranged sequentially. Once it’s done dividing the data files into blocks, it distributes and stores the blocks across large clusters of servers or machines. One noteworthy characteristic of this file system is that it is very reliable. The HDFS has a fault tolerance capability, an attribute or property that allows a system to maintain continuous operation despite experiencing failures or faults within its components. It can replicate the blocks of data files it stored and distributed across the clusters; so that in the event that failures occur, tasks and processes can still be executed on the data sets in their replicates.

A Distributed Parallel Computing Component Built Based On Apache YARN

Aside from its reliable distributed file system, Apache Hadoop also has a main component called Map/Reduce. This a framework that utilizes the Apache YARN system to handle distributed parallel computing across Hadoop clusters. Apache YARN system is a cluster management and job scheduling tool which is also developed by The Apache Software Foundation.

Understanding The Map/Reduce Architecture

To understand the reliable and powerful features of Map/Reduce, let us examine its architecture. Map/Reduce comes with an architecture that uses a master/slave structure. Computation operations or tasks are first organized in a single master server called jobtracker. These computation operations or tasks are also referred to as map/reduce jobs. The jobtracker allows users to directly interact with the Apache Hadoop framework. It enables them to send map/reduce jobs to this master server. Then, the jobtracker puts the submitted jobs in queue of pending map/reduce jobs. The jobtracker executes these jobs, prioritizing the execution of the jobs that were submitted earlier – first-come/first-served basis.

The jobtracker assigns the map/reduce jobs to several slave servers known as tasktrackers. Each node in the cluster of servers or computers is linked to a single tasktracker. The tasktrackers are the ones responsible for executing computation operations and tasks on the data sets distributed across the nodes contained in the cluster of servers or computers. However, the manner they execute such operations or tasks depends on the instructions they are getting from the master server or the jobtracker. When the tasktrackers detect failures while they are running computation operations or tasks on the nodes they are assigned to, it redistribute the tasks across other available nodes that are functioning and working properly. In other words, they have the capability to perform good load balancing and can re-execute map/reduce tasks without requiring large runtime overhead.