Short note on Text mining/Mining Text Data.

-

 Mining Text Databases/Text mining

  • Text mining is an interdisciplinary field that draws on information retrieval, data- mining, machine learning, statistics, and computational linguistics. Text mining has been very active. An important goal is to derive high-quality information from the text. Text databases consist of a huge collection of documents. They collect this information from several sources such as news articles, books, digital libraries, e-mail messages, web pages, etc. 
  • This is typically done through the discovery of patterns and trends by means such as statistical M pattern learning, topic modeling, and statistical language modeling. Text mining usually requires structuring the input text (e.g., parsing, along with the addition of some derived linguistic features and the removal of others, and subsequent insertion into a database). This is followed by deriving patterns within the structured data, and evaluation and interpretation of the output. "High quality" in text mining usually refers to a combination of relevance, novelty, and interestingness.
  • Typical text mining tasks include text categorization, text clustering, concept/entity extraction, production of granular taxonomies, sentiment analysis, document summarization, and entity-relation modeling (i.e., learning relations between named entities).
  •  Other examples include multilingual data mining, multidimensional text analysis, contextual text mining, and trust and evolution analysis in text data, as well as text mining applications in security, biomedical literature analysis, online media analysis, and analytical customer relationship management. 
  • Various kinds of text mining and analysis software and tools are available in academic institutions, open-source forums, and industry. Text mining often also uses WordNet, Semantic Web, Wikipedia, and other information sources to enhance the understanding and mining of text data.
  • Due to the increase in the amount of information, text databases are growing rapidly. In many of the text databases, the data is semi-structured. For example, a document may contain a few structured fields, such as title, author, publishing date, etc. But along with the structure data, the document also contains unstructured text components, such as abstract and contents. Without knowing what could be in the documents, it is difficult to formulate effective queries for analyzing and extracting useful information from the data. Users require tools to compare the documents and rank their importance and relevance. Therefore, text mining has become a popular and essential theme in data mining.

                                 OR,

Text mining is the procedure of synthesizing information, by analyzing relations, patterns, and rules among textual data. These procedures contain text summarization, text categorization, and text clustering.

1. Text summarization is the procedure to extract its partial content reflecting its whole contents automatically.

2. Text categorization is the procedure of assigning a category to the text among categories predefined by users

3. Text clustering is the procedure of segmenting texts into several clusters, depending on the substantial relevance.



Text Representation Issues

  •  Each word has a dictionary meaning or meanings

– Run – (1) the verb. (2) the noun, in cricket

– Cricket – (1) The game. (2) The insect.

– Apple (the company) or apple (the fruit)


  •  Ambiguity and context-sensitivity - Each word is used in various “senses”

– Tendulkar made 100 runs

– Because of an injury, Tendulkar can not run and will need a runner between the wickets


  • Capturing the “meaning” of sentences is an important issue as well. Grammar, parts of speech, time sense could be easy!
  •  Order of words in the query

– hot dog stand in the amusement park

– hot amusement stand in the dog park




Comments

Post a Comment

Popular posts from this blog

Suppose that a data warehouse for Big-University consists of the following four dimensions: student, course, semester, and instructor, and two measures count and avg_grade. When at the lowest conceptual level (e.g., for a given student, course, semester, and instructor combination), the avg_grade measure stores the actual course grade of the student. At higher conceptual levels, avg_grade stores the average grade for the given combination. a) Draw a snowflake schema diagram for the data warehouse. b) Starting with the base cuboid [student, course, semester, instructor], what specific OLAP operations (e.g., roll-up from semester to year) should one perform in order to list the average grade of CS courses for each BigUniversity student. c) If each dimension has five levels (including all), such as “student < major < status < university < all”, how many cuboids will this cube contain (including the base and apex cuboids)?

-
Image
a) A Snowflake Shema is shown in figure below: b) The specific OLAP operations to be performed: 1. Roll-up on course from course_id to department. 2. Roll-up on a student from student_id to university. 3. Dice on course, student with department ="CS" and university = "biguniversity" 4. Drill-down on a student from the university to student_name. c) N = 4 dimensions The cube will contain 54 = 625 cuboids.
Read more

Discuss classification or taxonomy of virtualization at different levels.

-
Image
Taxonomy of virtualization Virtualization is mainly used to emulate the execution environment, storage, and networks. The execution environment is classified into two: – Process-level – implemented on top of an existing operating system.  – System-level – implemented directly on hardware and does not or minimum requirement of the existing operating system. OR,  Virtualization covers a wide range of emulation techniques that are applied to different areas of computing. A classification of these techniques helps us better understand their characteristics and use  V irtualization is mainly used to emulate  ● Execution Environments: To provide support for the execution of the programs eg. OS, and Application.  ○ Process Level: Implemented on top of an existing OS that has full control of the hardware  ○ System Level: Implemented directly on Hardware and do not require support from existing OS.  ● Storage: Storage vi...
Read more

Short note on E-Government Architecture

-
Image
1)  Presentation layer The presentation layer identifies and describes the system users, who require access to government information at different capacities, and the channels through which information can be accessed.  During system development, one is required to explicitly identify the government user, the system is intended to serve, and also the means through which this information is to be accessed so that the system can be tailored to meet these requirements. It manages the user’s interface with the system. If a pro- project is to be successful, different stakeholders need to be identified at the beginning, involved in the initial stages and kept involved throughout development and implementation. 2) E-government layer E-government public services utilize very specialized applications that are only available to certain agencies and not all agencies participating in the consortium. The main goal of the e-government layer is to achieve a government that;  does not as...
Read more