Short note on Stop Word Removal

-

 Stop-Word Removal

Stop-word removal refers to the process of removing stop words from the list of tokens or stemmed words. Stop words are grammatical words that are irrelevant to text contents, so they need to be removed for more efficiency. The stop-word list is loaded from a file, and if they are registered in the list, they are removed. The stemming and the stop-word removal may be swapped; stop words are removed before stemming the tokens. Therefore, in this subsection, we provide a detailed description of stop words and the stop-word removal.

The stop word refers to the word which functions only grammatically and is irrelevant to the given text contents. Prepositions, such as "in," "on," "to"; and so on, typically belong to the stop-word group. Conjunctions such as "and," "or," "but," and "however" also belong to the group. The definite article, "the," and the infinite articles, "a" and "an," are also more frequent stop words. The stop words occur dominantly in all texts in the collection; removing them causes to improve very much efficiency in processing texts.

Let us explain the process of removing stop words in the ing. The stop-word list is prepared as a file and is loaded from it. For each word, if it is registered in the list, it is removed. The remaining words after removing stop words are usually nouns, verbs, and adjectives. Instead of loading the stop-word list from a file, we may consider using the classifier which decides whether each word is a stop word or not.

Some nouns, verbs, and adjectives as the remaining words may occur in other texts as well as in the current one. The words are called common words, and they are not useful for identifying text contents, so they need to be removed like stop words. The TF-IDF (Term Frequency Inverse Term Frequency) weight is the criteria for deciding whether each word is a common word, or not. The remaining words after removing stop words are useful in the news article collection, but not all of them are so in the technical one medical text collections. We may consider the further removal, depending on text lengths; it is applied to long texts.

Comments

Post a Comment

Popular posts from this blog

What are different steps used in JDBC? Write down a small program showing all steps.

-
 T he fundamental steps involved in the process of connecting to a database and executing a query consist of the following: Import JDBC packages. Load and register the JDBC driver. Open a connection to the database. Create a statement object to perform a query. Execute the statement object and return a query resultset. Process the resultset. Close the resultset and statement objects. Close the connection. These steps are described in detail in the sections that follow. 1. Import JDBC Packages This is for making the JDBC API classes immediately available to the application program. The following import statement should be included in the program irrespective of the JDBC driver being used: import java.sql.*; Additionally, depending on the features being used, Oracle-supplied JDBC packages might need to be imported. For example, the following packages might need to be imported while using the Oracle extensions to JDBC such as using advanced data types such as BLOB, and so on. import o...
Read more

Discuss classification or taxonomy of virtualization at different levels.

-
Image
Taxonomy of virtualization Virtualization is mainly used to emulate the execution environment, storage, and networks. The execution environment is classified into two: – Process-level – implemented on top of an existing operating system.  – System-level – implemented directly on hardware and does not or minimum requirement of the existing operating system. OR,  Virtualization covers a wide range of emulation techniques that are applied to different areas of computing. A classification of these techniques helps us better understand their characteristics and use  V irtualization is mainly used to emulate  ● Execution Environments: To provide support for the execution of the programs eg. OS, and Application.  ○ Process Level: Implemented on top of an existing OS that has full control of the hardware  ○ System Level: Implemented directly on Hardware and do not require support from existing OS.  ● Storage: Storage vi...
Read more

Pure Versus Partial EC

-
Image
Pure Versus Partial EC EC takes several forms depending on the degree of digitization (the transformation from physical to digital). Companies utilizing pure EC conduct all of their business online. Businesses utilizing partial EC conduct a portion of their business online and a portion of their business off-line OR,    Pure E-electronic commerce refers to a situation where a business transacts its business activities PURELY online. that is, the goods or services sold do not have any physical presence. take a business that sells software, for example, you just pay and download, or one that sells music files online. Partial E-Commerce on the other hand refers to a slightly opposite situation where a business has an online presence, but still has a physical location for the goods and services it sells. For example, Amazon, Walmart, and any other you can think of. you buy the goods online but receive them physically.                   ...
Read more