Explain Hadoop File System along with its architecture.

-

Hadoop File System(HDFS)

 The Hadoop File System was created with a distributed file system design. It runs on standard hardware. In contrast to other distributed systems, HDFS is extremely faulted tolerant and built with low-cost hardware. HDFS stores a big quantity of data and makes it easy to access. To store such large amounts of da the files are spread over numerous computers. These files are kept in a redundant form to protect the system from data loss in the event of a breakdown. HDFS also enables parallel processing of applications.

Hadoop applications use the Hadoop Distributed File Solution (HDFS) as their primary data storage system. It implements a distributed file system that allows high-performance access to data across highly scalable Hadoop clusters using a NameNode and DataNode architecture. HDFS is an important component of the numerous Hadoop ecosystem technologies since it provides a dependable way of maintaining massive data pools and supporting associated big data analytics applications.


Features of HDFS

  • It is suitable for distributed storage and processing. 
  • Hadoop provides a command interface to interact with HDFS.
  • The built-in servers of namenode and datanode help users to easily check the status of the cluster. 
  • Streaming access to file system data.
  • HDFS provides file permissions and authentication.


 HDFS Architecture

The architecture of a Hadoop File System is shown in figure 7.2. Those components of the architecture are described as below:



Namenode

The namenode is the commodity hardware that contains the GNU/Linux operating system and the namenode software. It is software that can be run on commodity hardware. The system having the namenode acts as the master server and it does the following tasks:

•Manages the file system namespace. 

•Regulates client's access to files.

•It also executes file system operations such as renaming, closing, and opening files and directories.


Datanode

The datanode is a piece of commodity hardware that runs the GNU/Linux operating system as well as the datanode software. A datanode will exist for each node (Commodity hardware/System) in a cluster. These nodes are in charge of their system's data storage.

Datanodes perform read-write operations on the file systems, as per the client's request. 

They also perform operations such as block creation, deletion, and replication according to the instructions of the namenode.


Block

In general, user data is saved in HDFS files. In a file system, the file will be partitioned into one or more segments and/or stored in separate data nodes. Blocks are the name given to these file chunks. In othe terms, a Block is the smallest quantity of data that HDFS can read or write. The default block size is 64MB, although it may be adjusted in the HDFS setup as needed.


Goals of HDFS

Fault detection and recovery: Because HDFS relies on a large number of commodity hardware components, component failure is common. As a result, HDFS should have techniques for rapid and automated failure detection and recovery.

Huge datasets: To manage applications with large datasets, HDFS should have hundreds of nodes per cluster.

Hardware at data: When the computation takes place near the data, a requested job can be completed quickly. It decreases network traffic while increasing throughput, especially when large datasets are involved.

Comments

Post a Comment

Popular posts from this blog

Suppose that a data warehouse for Big-University consists of the following four dimensions: student, course, semester, and instructor, and two measures count and avg_grade. When at the lowest conceptual level (e.g., for a given student, course, semester, and instructor combination), the avg_grade measure stores the actual course grade of the student. At higher conceptual levels, avg_grade stores the average grade for the given combination. a) Draw a snowflake schema diagram for the data warehouse. b) Starting with the base cuboid [student, course, semester, instructor], what specific OLAP operations (e.g., roll-up from semester to year) should one perform in order to list the average grade of CS courses for each BigUniversity student. c) If each dimension has five levels (including all), such as “student < major < status < university < all”, how many cuboids will this cube contain (including the base and apex cuboids)?

-
Image
a) A Snowflake Shema is shown in figure below: b) The specific OLAP operations to be performed: 1. Roll-up on course from course_id to department. 2. Roll-up on a student from student_id to university. 3. Dice on course, student with department ="CS" and university = "biguniversity" 4. Drill-down on a student from the university to student_name. c) N = 4 dimensions The cube will contain 54 = 625 cuboids.
Read more

Suppose that a data warehouse consists of the four dimensions; date, spectator, location, and game, and the two measures, count and charge, where charge is the fee that a spectator pays when watching a game on a given date. Spectators may be students, adults, or seniors, with each category having its own charge rate. a) Draw a star schema diagram for the data b) Starting with the base cuboid [date; spectator; location; game], what specific OLAP operations should perform in order to list the total charge paid by student spectators at GM Place in 2004?

-
Image
  b) The specific OLAP operations to be performed are: 1. Roll-up on date from date id to year. 2. Roll-up on spectator from spectator id to status. 3. Roll-up on location from location id to location name. 4. Roll-up on the game from game id to all. 5. Dice with status= “students”, location name= “GM Place”, and year=2004
Read more

Explain network topology .Explain tis types with its advantages and disadvantges.

-
Network topology Network topology refers to how the nodes and links in a network are arranged. A network node is a device that can send, receive, store, or forward data. A network link connects nodes and may be either cabled or wireless links. Understanding topology types provides the basis for building a successful network. There are a number of topologies but the most common are bus, ring, star, and mesh: A bus network topology is when every network node is directly connected to a main cable. In a ring topology, nodes are connected in a loop, so each device has exactly two neighbors. Adjacent pairs are connected directly; non-adjacent pairs are connected indirectly through multiple nodes. In a star network topology, all nodes are connected to a single, central hub and each node is indirectly connected through that hub.  A mesh topology is defined by overlapping connections between nodes. You can create a full mesh topology, where every node in the network is connected to every other
Read more