Explain benefits and challenges of Hadoop.

-

 Benefits of Hadoop

Scalability: Unlike traditional systems, which have storage limitations, Hadoop is scalable since it functions in a distributed environment. This enabled data architects to create early Hadoop data lakes. 

Resilience: The Hadoop Distributed File System (HDFS) is intrinsically robust. To prepare for the risk of hardware or software failures, data stored on each node of a Hadoop cluster is also duplicated on other nodes of the cluster. This design is purposely redundant to ensure failure tolerance. If one node fails, there is always a backup of the data in the cluster.

Flexibility: Unlike typical relational database management systems, Hadoop allows you to store data in any format, including semi-structured or unstructured data. Hadoop allows organizations to readily access new data sources and access various sorts of data. 


Challenges with Hadoop Architectures

Complexity: Hadoop is a low-level, Java-based platform that might be too complicated to deal with for end users. Hadoop infrastructures can also need a substantial amount of skill and resources to set up, maintain, and update.

Performance: Hadoop performs calculations by doing frequent reads and writes to disk, which is time-consuming and inefficient when compared to frameworks that attempt to store and process data in memory as much as possible, such as Apache Spark.

Viability in the long run: The world witnessed a tremendous unraveling inside the Hadoop realm in 2019. Google, whose groundbreaking 2004 article on MapReduce served as the foundation for the construction of Apache Hadoop, has discontinued using MapReduce entirely. In the Hadoop industry, there were also some high-profile mergers and acquisitions. Furthermore, in 2020, a prominent Hadoop supplier switched its product set away from being Hadoop-centric, since Hadoop is now considered to be "more of a mindset than a technology."



Comments

Post a Comment

Popular posts from this blog

Suppose that a data warehouse for Big-University consists of the following four dimensions: student, course, semester, and instructor, and two measures count and avg_grade. When at the lowest conceptual level (e.g., for a given student, course, semester, and instructor combination), the avg_grade measure stores the actual course grade of the student. At higher conceptual levels, avg_grade stores the average grade for the given combination. a) Draw a snowflake schema diagram for the data warehouse. b) Starting with the base cuboid [student, course, semester, instructor], what specific OLAP operations (e.g., roll-up from semester to year) should one perform in order to list the average grade of CS courses for each BigUniversity student. c) If each dimension has five levels (including all), such as “student < major < status < university < all”, how many cuboids will this cube contain (including the base and apex cuboids)?

-
Image
a) A Snowflake Shema is shown in figure below: b) The specific OLAP operations to be performed: 1. Roll-up on course from course_id to department. 2. Roll-up on a student from student_id to university. 3. Dice on course, student with department ="CS" and university = "biguniversity" 4. Drill-down on a student from the university to student_name. c) N = 4 dimensions The cube will contain 54 = 625 cuboids.
Read more

Suppose that a data warehouse consists of the four dimensions; date, spectator, location, and game, and the two measures, count and charge, where charge is the fee that a spectator pays when watching a game on a given date. Spectators may be students, adults, or seniors, with each category having its own charge rate. a) Draw a star schema diagram for the data b) Starting with the base cuboid [date; spectator; location; game], what specific OLAP operations should perform in order to list the total charge paid by student spectators at GM Place in 2004?

-
Image
  b) The specific OLAP operations to be performed are: 1. Roll-up on date from date id to year. 2. Roll-up on spectator from spectator id to status. 3. Roll-up on location from location id to location name. 4. Roll-up on the game from game id to all. 5. Dice with status= “students”, location name= “GM Place”, and year=2004
Read more

What is national data warehouse? What is census data?

-
Image
National Data warehouse   A large number of national data warehouses can be identified from the existing data resources within the central government ministries. These are potential subject areas in which data warehouses may be developed at present and in the future. Census Data Census data can help planners and others understand their communities' social, economic, and demographic conditions. Census data is the primary data used by planners to understand the social, economic, and demographic conditions locally and nationally. We sometimes think the census is unique to the United States, but census data is collected in other countries.  The total process of collecting, compiling, and publishing demographic, economic, and social data about a specific time to all people in a country or fixed or defined area part of a country. Most countries also include a housing census. A census is a procedure of systematically acquiring and recording information about the members of a given populat
Read more