Define the concepts of support and confidence and Lift for an association rule mining.

-

 SUPPORT, CONFIDENCE AND LIFT

Imagine a Stationery store with tens of thousands of different products. We wouldn't want to calculate all associations between every possible combination of products. Instead, we would want to select only potentially relevant rules from the set of all possible rules. Therefore, we use the measures support, confidence, and lift to reduce the number of associations we need to analyze

1. Support (S): The metric support tells us how popular an itemset is, as measured by the proportion of transactions in which the itemset appears. Support of an itemset calculated as

Number of transactions that contains an itemset Support (s) /Total number of transaction in the given dataset


In our data shown in table 5.1, the support of itemset (Pencil) is,Support (Pencil) / Total hunder of transaction in the given dataset =4/5=0.8-=80% 

The support of itemset (Pencil, Paper) is

Support ((Pencil, Paper]) =Number of transactions that contain an (Pencil, Paper )/Total number of transactions in the given dataset = 3/5=0.6=60%

2. Confidence: Confidence tells us how likely an itemset Y is purchased given that itemset X is purchased expressed as [X-Y]. It is measured by the proportion of transactions with itemset X, in which itemset Y also appears. 

Confidence of an association rule X→Y calculated as:

Confidence (c)=  Number of transactions that contain an itemset (X, Y)/Number of transactions that contain an itemset (X)

For example, the confidence of the association rule (Paper, Book) →(Rubber) is,

Confidence (Paper, Book) → (Rubber)) = Number of transactions that contain an itemset (Paper, Book, Rubber)/ Number of transactions that contain an itemset (Paper, book

Confidence ([Paper, Book) → (Beer)) =2/3=2/3*100=67%

Hence, the confidence can be interpreted as an estimate of the probability P(Y|X). In other words, this is the probability of finding the itemset Y in transactions under the condition that these transactions also contain the itemset X. Confidence is directed and therefore usually gives different values for the rules X-Y and Y→ X. In our example, the confidence that (Rubber) is purchased given that (Paper, Book) is purchased ([Paper, Book)→ (Rubber]) is 2 out of 3, or 67%. This means the conditional probability P((Rubber) | (Paper, Book)) = 67%.

Note that the confidence measure might misrepresent the importance of an association. This is because it only accounts for how popular itemset X is (in our case (Paper, Book)) but not Y (in our case (Rubber)). If (Rubber) is also very popular in general, there will be a higher chance that a transaction containing (Paper, Book) will also contain (Rubber), thus inflating the confidence measure. To account for the base popularity of both items, we use a third measure called lift.

3. Lift: Lift tells us how likely item Y is purchased when item X is purchased while controlling for how popular items Y and X are. It measures how many times more often X and Y occur together than expected if they were statistically independent.

 Lift (X→Y)=; number of transactions that contain an itemset (X, Y)/Support (X) *Support(Y)

If the value of Lift = 1: implies no association between itemsets.

If the value of Lift > 1: greater than 1 means that itemset Y is likely to be  if  itemset X is bought, 

If the value of Lift < 1: less than 1 means that itemset Y is unlikely to be bought if itemset X is bought.

For example, lift for association rule (Paper, Book)-> (Rubber) is calculated as:





Comments

Post a Comment

Popular posts from this blog

Suppose that a data warehouse for Big-University consists of the following four dimensions: student, course, semester, and instructor, and two measures count and avg_grade. When at the lowest conceptual level (e.g., for a given student, course, semester, and instructor combination), the avg_grade measure stores the actual course grade of the student. At higher conceptual levels, avg_grade stores the average grade for the given combination. a) Draw a snowflake schema diagram for the data warehouse. b) Starting with the base cuboid [student, course, semester, instructor], what specific OLAP operations (e.g., roll-up from semester to year) should one perform in order to list the average grade of CS courses for each BigUniversity student. c) If each dimension has five levels (including all), such as “student < major < status < university < all”, how many cuboids will this cube contain (including the base and apex cuboids)?

-
Image
a) A Snowflake Shema is shown in figure below: b) The specific OLAP operations to be performed: 1. Roll-up on course from course_id to department. 2. Roll-up on a student from student_id to university. 3. Dice on course, student with department ="CS" and university = "biguniversity" 4. Drill-down on a student from the university to student_name. c) N = 4 dimensions The cube will contain 54 = 625 cuboids.
Read more

Suppose that a data warehouse consists of the four dimensions; date, spectator, location, and game, and the two measures, count and charge, where charge is the fee that a spectator pays when watching a game on a given date. Spectators may be students, adults, or seniors, with each category having its own charge rate. a) Draw a star schema diagram for the data b) Starting with the base cuboid [date; spectator; location; game], what specific OLAP operations should perform in order to list the total charge paid by student spectators at GM Place in 2004?

-
Image
  b) The specific OLAP operations to be performed are: 1. Roll-up on date from date id to year. 2. Roll-up on spectator from spectator id to status. 3. Roll-up on location from location id to location name. 4. Roll-up on the game from game id to all. 5. Dice with status= “students”, location name= “GM Place”, and year=2004
Read more

Suppose that a data warehouse consists of the three dimensions time, doctor, and patient, and the two measures count and charge, where a charge is the fee that a doctor charges a patient for a visit. a) Draw a schema diagram for the above data warehouse using one of the schemas. [star, snowflake, fact constellation] b) Starting with the base cuboid [day, doctor, patient], what specific OLAP operations should be performed in order to list the total fee collected by each doctor in 2004? c) To obtain the same list, write an SQL query assuming the data are stored in a relational database with the schema fee (day, month, year, doctor, hospital, patient, count, charge)

-
Image
Solution: a) Star Schema is shown in the figure below: b) First, we should use the roll-up operation to get the year 2004(rolling-up from the day then month to a year). After getting that, we need to use the slice operation to select (2004). Second, we should use roll-up operation again to get all patients. Then, we need to use the slice operation to select (all). Finally, we get a list of the total fee collected by each doctor in 2004. So, 1. roll up from day to month to year 2. slice for year = “2004” 3. roll up on patient from the individual patient to all 4. slice for patient = “all” 5. get the list of the total fees collected by each doctor in 2004 c) Select doctor, Sum(charge) From fee Where year = 2004 Group by doctor
Read more