Part B: Subject-specific skills (theory units) · Lesson 5 of 5All lessons
AI CourseCBSE Class 10 AI (417): Unit-wise Notes & MCQsPart B: Subject-specific skills (theory units)
Lesson 5

Unit 6: Natural Language Processing

14 min read40 practice questions10-question quiz

Natural Language Processing is the part of AI that helps computers work with human languages. These notes cover natural languages, NLP applications, NLP stages, chatbots, text normalisation, Bag of Words, TFIDF, and the use-case walkthrough from the unit.

Natural Language and NLP

A natural language is a human language. French, Spanish, English, and Japanese are examples of natural languages.

Features of Natural Languages:

  • They are governed by set rules that include syntax, lexicon, and semantics.
  • All natural languages are redundant, i.e., the information can be conveyed in multiple ways.
  • All natural languages change over time.
  • Different meanings in different contexts.

Natural language is not always simple for a computer. Same-sounding words may have different meanings. A word may also change its meaning according to the context. In the given examples, the word red is used in different ways. A red face may show shame, anger, an allergic reaction, or another meaning depending on the sentence. A red car mainly refers to the colour of the car.

Context is very important in natural language. Humans understand sentences almost naturally because they have experience and memories of using that language. Computers need help to understand such meaning and context.

Computer languages are used to interact with computers. Python, C++, Java, and HTML are examples of computer languages. Computers need a specific set of instructions called programs to understand human input. To communicate with a computer, natural language has to be converted into a form that the computer can process.

Natural Language Processing, or NLP, is the sub-field of AI that focuses on enabling computers to analyse, understand and process human languages to derive meaningful information from human language.

NLP is important because computers process electronic signals in binary language. NLP helps convert natural language into digital form. Its purpose is to make communication between humans and computer systems possible. It also includes tools and techniques that help in better communication of intent and context.

Everyday example: The same word can sound simple in a sentence, but its meaning becomes clear only when the full sentence is understood.

Applications of Natural Language Processing

Artificial Intelligence is used in many daily-life situations. Natural Language Processing is used in several real-life applications.

Applications of Natural Language Processing:

  • Autogenerated captions: Captions are created by changing natural speech into text in real time. This helps make video content more accessible. Examples given are auto-generated captions on YouTube and Google Meet.
  • Voice assistants: Voice assistants take natural speech, process it, and give an output. They use NLP to understand natural language and carry out tasks efficiently. Examples given are Hey Google, set an alarm at 3.30 pm; Hey Alexa, play some music; Hey Siri, what’s the weather today.
  • Language Translation: This deals with generating translation from another language. It converts text or speech from one language to another. This helps cross-linguistic communication and global connectivity. The example given is Google Translate.
  • Sentiment Analysis: Sentiment Analysis is a tool used to express whether an opinion has a positive, negative, or neutral sentiment. Customer sentiment analysis helps automatically detect emotions when customers interact with products, services, or a brand.
  • Text Classification: Text classification is a tool that classifies a sentence or document category-wise. In the given example, news articles from Food, Sports, and Politics are placed into categories. Raw text is classified into predefined groups or categories.
  • Keyword Extraction: Keyword extraction is a tool that automatically extracts the most used and important words and expressions from a text. It can give useful insights into people’s opinions about any business on social media. Customer Service can be improved using a Keyword extraction tool.

In the keyword extraction activity, keyword extraction in NLP means automatically finding and extracting the most important words or phrases from a text. These keywords represent the main topics or themes in the text. They are useful for document summarization, information retrieval, and content analysis.

Everyday example: If a short review says that a service is fast and helpful, keyword extraction can focus on important words like fast and helpful.

Stages of Natural Language Processing

The stages of NLP help in understanding and processing human language. Each stage has a different role.

Stages of Natural Language Processing:

  • Lexical Analysis: NLP begins by identifying the structure of the input words. A large group of words is divided into structural paragraphs, sentences, and words. Lexicon means a collection of the various words and phrases used in a language. In this stage, lengthy text is broken into chunks.
  • Syntactic Analysis / Parsing: This stage checks the grammar of sentences and phrases. It creates relationships among words and removes sentences that are logically incorrect. This stage focuses on whether the grammar is correct.
  • Semantic Analysis: In this stage, the input text is checked for meaning. Every word and phrase is checked for meaningfulness. The example given is that a sentence containing hot ice cream would be rejected. This stage checks whether sentences make actual sense.
  • Discourse Integration: This stage forms the story of the sentence. Every sentence should have a relationship with the sentences before and after it. This checks whether the flow of words makes sense.
  • Pragmatic Analysis: In this stage, sentences are checked for their relevance in the real world. Pragmatic means practical or logical. This step needs knowledge of the intent in a sentence. It may discard the actual word meaning taken after semantic analysis and use the intended meaning instead.

These stages show how NLP moves from words and grammar to meaning, flow, and intended meaning.

Everyday example: A sentence may be grammatically correct, but the intended meaning becomes clear only when the situation around it is understood.

Chatbots: Script-bot and Smart-bot

One of the most common applications of Natural Language Processing is a chatbot. A chatbot is a computer program designed to simulate human conversation through voice commands, text chats, or both.

A chatbot can learn over time how to interact with humans in a better way. It can answer questions and troubleshoot customer problems. It can also evaluate and qualify prospects, generate sales leads, and increase sales on an ecommerce site.

The unit mentions different chatbots for interaction:

  • Elizabot
  • Mitsuki
  • Cleverbot
  • Singtel

The purpose of trying chatbots is to identify and interact with different chatbots. Chatbots are created so that the interaction feels like talking to a real human, because humans are comfortable with that style of communication.

After interacting with different chatbots, it becomes clear that there are 2 types of chatbots around us:

  • Script-bot
  • Smart-bot

Script-bots are also described as scripted or traditional chatbots. Smart-bots are described as AI-powered and having more knowledge.

This distinction comes from observing that some chatbots follow a scripted style, while others are AI-powered.

Everyday example: A chatbot on a support page may reply in a fixed style, while another chatbot may seem to use more knowledge in the conversation.

Text Normalisation and Text Processing

Text Normalisation helps clean textual data. It brings the text to a lower level of complexity than the original data.

In this part, work is done on a collection of written text. Text from multiple documents taken together is called a corpus.

Steps of Text Normalisation:

  • Sentence Segmentation
  • Tokenization
  • Removing Stop words, Special Characters and Numbers
  • Converting Text to a Common Case
  • Stemming
  • Lemmatization

Sentence Segmentation: In sentence segmentation, the whole corpus is divided into sentences. Each sentence is treated as different data. The corpus is reduced to sentences.

Tokenization: After sentence segmentation, each sentence is divided into tokens. A token can be any word, number, or special character in a sentence. In tokenisation, every word, number, and special character is considered separately.

Removing Stop words, Special Characters and Numbers: In this step, tokens that are not needed are removed from the token list. Stop words are words that occur very frequently in the corpus but do not add value to it. Humans use grammar to make sentences meaningful for other people. However, grammatical words do not add essence to the information being passed through the statement. So, they are treated as stop words. Stop words occur very often but say very little or nothing about the context or meaning. Removing them helps the computer focus on meaningful terms.

Special characters and numbers may also be present in a corpus. Whether they should be removed depends on the type of corpus. If the document contains email IDs, special characters and numbers may be kept. In other text data, if they do not make sense, they may be removed with stop words.

Converting Text to a Common Case: After stop words are removed, the whole text is converted into a similar case, preferably lowercase. This prevents the machine from treating the same word as different words only because the case is different. In the given example, all forms of hello are converted to lowercase and treated as the same word.

Stemming: In stemming, the remaining words are reduced to their root words. It removes affixes and converts words to their base form. The words produced after stemming may not be meaningful. For example, healed, healing, and healer are reduced to heal, but studies becomes studi. Stemming does not check whether the final word has meaning. It only removes affixes, so it is faster.

Lemmatization: Stemming and lemmatization are alternative processes because both remove affixes. The difference is that in lemmatization, the word obtained after affix removal, called the lemma, is meaningful. Lemmatization makes sure the lemma is a meaningful word, so it takes longer than stemming. In the same example, studies becomes study instead of studi.

After text normalisation, the text is reduced to tokens, which are the simplest form of words present in the corpus. The next task is to convert these tokens into numbers.

Bag of Words

Bag of Words is a Natural Language Processing model. It helps extract features from text that can be useful in machine learning algorithms. In Bag of Words, the occurrences of each word are found and the vocabulary for the corpus is constructed.

Bag of Words gives two things:

  • A vocabulary of words for the corpus
  • The frequency of these words, meaning the number of times each word occurs in the whole corpus

It is called a bag of words because the sequence of sentences or tokens does not matter. The important things are the unique words and their frequency.

Step-by-step approach to implementing the Bag of Words algorithm:

  • Text Processing: Collect data and pre-process it
  • Create a Dictionary: Make a list of all the unique words occurring in the corpus. (Vocabulary)
  • Create document vectors: For each document in the corpus, find out how many times the word from the unique list of words has occurred.
  • Create document vectors for all the documents.

In the given example, there are three documents:

  • Document 1: Aman and Avni are stressed
  • Document 2: Aman went to a therapist
  • Document 3: Avni went to download a health chatbot

After text normalisation, the documents are shown as tokens:

  • Document 1: [aman, and, avni, are, stressed]
  • Document 2: [aman, went, to, a, therapist]
  • Document 3: [avni, went, to, download, a, health, chatbot]

Then a dictionary is created. It contains the unique words from all three documents. Even if a word is repeated in different documents, it is written only once in the dictionary.

Next, document vectors are created. The vocabulary is written in the top row. For each word in a document, if it matches the vocabulary, a 1 is placed under it. If the same word appears again, the value is increased by 1. If the word does not occur in that document, 0 is placed under it.

The same process is done for all documents. The final table has the vocabulary in the header row and one row for each document. This table is the document vector table for the corpus.

Bag of Words helps count words, but the tokens still need to be converted into useful number values. This leads to TFIDF.

TFIDF: Term Frequency and Inverse Document Frequency

TFIDF stands for Term Frequency and Inverse Document Frequency. It helps identify the value of each word.

Bag of Words gives the frequency of words in each document. If a word occurs more often in a document, it may have more value for that document. For example, in a document on air pollution, air and pollution may occur many times and give context about the document.

Some words occur very frequently in almost all documents. Examples given are and, this, is, the, etc. These words are important for humans because they make statements understandable. For the machine, they do not provide useful information about the corpus. These are called stop words and are mostly removed during pre-processing.

Words can be understood in three groups:

  • Stop words: These occur the most in all documents and have negligible value.
  • Frequent words: These have adequate occurrence in the corpus and usually talk about the document’s subject.
  • Rare or valuable words: These occur the least but add the most value to the corpus.

Term Frequency: Term Frequency is the frequency of a word in one document. It can be found in the document vector table, where the frequency of each vocabulary word is recorded for each document.

Document Frequency: Document Frequency is the number of documents in which a word occurs. It does not depend on how many times the word occurs inside those documents.

Inverse Document Frequency: For Inverse Document Frequency, the document frequency is placed in the denominator and the total number of documents is placed in the numerator.

Formula:

  • TFIDF(W) = TF(W) * log(IDF(W))
  • The log is to the base of 10.

TF values are for each document. IDF values are for the whole corpus. So, IDF values are multiplied with each row of the document vector table.

Important points about TFIDF:

  • Words that occur in all the documents with high term frequencies have the lowest values and are considered to be the stop words.
  • For a word to have a high TFIDF value, it needs a high term frequency but less document frequency. This shows that the word is important for one document but is not common for all documents.
  • These values help the computer understand which words should be considered while processing natural language. The higher the value, the more important the word is for a given corpus.

Applications of TFIDF:

  • Document Classification: Helps in classifying the type and genre of a document.
  • Topic Modelling: It helps in predicting the topic for a corpus.
  • Information Retrieval System: To extract the important information out of a corpus.
  • Stop word filtering: Helps in removing unnecessary words from a text body.

Natural Language Processing: Use Case Walkthrough

The use-case walkthrough introduces Code and No-Code NLP tools. It also explains Sentiment Analysis as an NLP application using the No-code tool Orange Data Mining.

Examples of Code and No-code NLP Tools:

Code NLP:

  • NLTK package: Natural Language Tool Kit or NLTK is a package available for text processing in Python. It contains functions and modules that can be used for Natural Language Processing.
  • SpaCy: SpaCy is an open-source natural language processing library designed to build NLP applications. It offers features such as tokenization, part-of-speech tagging, named entity recognition, dependency parsing, and more.

No-Code NLP:

  • Orange Data Mining: It is a machine learning tool for data analysis through Python and visual programming. Operations on data can be performed through simple drag-and-drop steps.
  • MonkeyLearn: MonkeyLearn is a text analysis platform that offers NLP tools and machine learning models for text analysis. It supports tasks such as classification, sentiment analysis, and entity recognition. Users can create custom models or use pre-trained ones for tasks like social media monitoring and customer feedback analysis.

Sentiment Analysis is a Natural Language Processing technique used to analyse whether given textual data is positive, negative, or neutral.

Applications of Sentiment Analysis-Customer Service:

  • Customer sentiment analysis helps in automatic detection of emotions when customers interact with products, services, or brands.

Applications of Sentiment Analysis –Voice of the Customer:

  • Voice of the customer analysis helps to analyze customer feedback and gain actionable insights from it.
  • It measures the gap between what customers expect and what they actually experience when they use the products or services.

The walkthrough then uses Orange Data Mining to understand Sentiment Analysis in detail.

About these notes

These notes and questions were written by Syllabax from the official CBSE Class X Artificial Intelligence (417) Facilitator Handbook and the 2026-27 curriculum. This is not an official CBSE publication.

✦ Natural Language✦ Computer Language✦ Natural Language Processing✦ Syntax✦ Lexicon✦ Corpus✦ Token✦ Stop words✦ Stemming✦ Lemmatization✦ Bag of Words✦ Term Frequency✦ Document Frequency✦ Inverse Document Frequency✦ TFIDF✦ Chatbot✦ Sentiment Analysis