How Map Reduce work? Explain

-

  •  MapReduce performs two critical functions: it filters and distributes work to various nodes within the cluster or map, a function known as the mapper, and it organizes and reduces the data from each node into a coherent answer to a query, known as the reducer.


  • MapReduce runs in parallel over huge cluster sizes to spread input data and collate outcomes. Because cluster size does not affect the final output of a processing operation, workloads can be distributed a practically any number of computers. As a result, MapReduce makes software development easier MapReduce is accessible in a variety of programming languages, including C, C++, Java, Ruby, Perl, and Python. MapReduce libraries allow programmers to build jobs without having to worry about communication or coordination between nodes.
  • MapReduce is also fault-tolerant, with each node sending its status to a master node regularly. If a node fails to reply as expected, the master node reassigns that portion of the work to other nodes in the cluster that are available. This enables resilience and makes it viable for MapReduce to run on affordable commodity servers.


  • MapReduce's strength is in its ability to handle large data sets by distributing processing over many nodes and then combining or reducing the outputs of those nodes. Users might, for example, use a single server application to list and tally the number of times each word appears in a novel, but this is time-consuming. Users, on the other hand, can divide the workload among 26 individuals, such that each person takes a page, writes a word on a separate piece of paper, and then takes a new page when they are completed. This is MapReduce's map component. And if someone leaves, someone else takes his or her position. This highlights the fault-tolerant nature of MapReduce. When all of the pages have been processed, users organize their single-word pages into 26 boxes, one for each letter of the word. Each user takes a box and alphabetically organizes each word in the stack. The number of pages with the same term is an example of MapReduce's reduce feature.

  • Reduce might be used by a social networking site to assess users' possible friends, coworkers, and other contacts based on on-site activity, names, localities, employers, and a variety of other data variables. A booking website may utilize MapReduce to assess customers' search criteria and history activity, and en produces personalized options for each.
  • The computation paradigm described by MapReduce is relatively simple, allowing for improved efficiency for the developers who write methods for processing massive amounts of data. In the case of Google, where the majority of the information that has to be processed is kept in textual form and is represented by Web pages or log files, this strategy has shown to be successful. Distributed grep, count of URL-access frequency, reverse web-link graph, term vector per host, inverted index, and distributed sort are some of the examples that show the flexibility of MapReduce. These examples are mostly focused on text-based processing. MapReduce may also be utilized to address a broader range of issues with some modifications. An exciting use is in the field of machine learning, where statistical algorithms such as Support Vector Machines (SVM), Linear Regression (LR), Neural Network (NN), etc. are expressed as a map and reduce functions. Other intriguing applications may be found in the realm of compute-intensive applications, such as the high-precision calculation of Pi.

Comments

Post a Comment

Popular posts from this blog

Suppose that a data warehouse for Big-University consists of the following four dimensions: student, course, semester, and instructor, and two measures count and avg_grade. When at the lowest conceptual level (e.g., for a given student, course, semester, and instructor combination), the avg_grade measure stores the actual course grade of the student. At higher conceptual levels, avg_grade stores the average grade for the given combination. a) Draw a snowflake schema diagram for the data warehouse. b) Starting with the base cuboid [student, course, semester, instructor], what specific OLAP operations (e.g., roll-up from semester to year) should one perform in order to list the average grade of CS courses for each BigUniversity student. c) If each dimension has five levels (including all), such as “student < major < status < university < all”, how many cuboids will this cube contain (including the base and apex cuboids)?

-
Image
a) A Snowflake Shema is shown in figure below: b) The specific OLAP operations to be performed: 1. Roll-up on course from course_id to department. 2. Roll-up on a student from student_id to university. 3. Dice on course, student with department ="CS" and university = "biguniversity" 4. Drill-down on a student from the university to student_name. c) N = 4 dimensions The cube will contain 54 = 625 cuboids.
Read more

Suppose that a data warehouse consists of the four dimensions; date, spectator, location, and game, and the two measures, count and charge, where charge is the fee that a spectator pays when watching a game on a given date. Spectators may be students, adults, or seniors, with each category having its own charge rate. a) Draw a star schema diagram for the data b) Starting with the base cuboid [date; spectator; location; game], what specific OLAP operations should perform in order to list the total charge paid by student spectators at GM Place in 2004?

-
Image
  b) The specific OLAP operations to be performed are: 1. Roll-up on date from date id to year. 2. Roll-up on spectator from spectator id to status. 3. Roll-up on location from location id to location name. 4. Roll-up on the game from game id to all. 5. Dice with status= “students”, location name= “GM Place”, and year=2004
Read more

Suppose that a data warehouse consists of the three dimensions time, doctor, and patient, and the two measures count and charge, where a charge is the fee that a doctor charges a patient for a visit. a) Draw a schema diagram for the above data warehouse using one of the schemas. [star, snowflake, fact constellation] b) Starting with the base cuboid [day, doctor, patient], what specific OLAP operations should be performed in order to list the total fee collected by each doctor in 2004? c) To obtain the same list, write an SQL query assuming the data are stored in a relational database with the schema fee (day, month, year, doctor, hospital, patient, count, charge)

-
Image
Solution: a) Star Schema is shown in the figure below: b) First, we should use the roll-up operation to get the year 2004(rolling-up from the day then month to a year). After getting that, we need to use the slice operation to select (2004). Second, we should use roll-up operation again to get all patients. Then, we need to use the slice operation to select (all). Finally, we get a list of the total fee collected by each doctor in 2004. So, 1. roll up from day to month to year 2. slice for year = “2004” 3. roll up on patient from the individual patient to all 4. slice for patient = “all” 5. get the list of the total fees collected by each doctor in 2004 c) Select doctor, Sum(charge) From fee Where year = 2004 Group by doctor
Read more