Enhanced Product Categorization with Retrieval-Augmented Generation (RAG)

Github Jupyter Notebook

Outline

• Introduction
• Achievements
• Project Development
  • Traditional_NLP
  • RAG with GPT3.5
• Challenges
  • Chunk Strategies
  • Hallucination

Introduction

Project Description:

This project aimed to revolutionize the way product titles are matched with category trees by integrating NLP techniques with Retrieval-Augmented Generation (RAG).

Objective:

while analyze competitor’s products, it is neccessary to map the product to the internal categorization, which is crucial for competitor analysis.

while uploading new product from our supplies, with this enhanced method to find product category can increase the accuracy and efficiency of product information, which are crucial for inventory management.

Achievements

Accuracy: 96%+

• Implemented a hybrid model using Retrieval-Augmented Generation (RAG) and GPT, which surpassed traditional NLP models by 23% in accuracy.
• High Precision Retrieval: Achieved a remarkable 99% accuracy rate in identifying the top 20 candidate sets for product categorization, demonstrating the efficacy of the optimized retrieval process.
• Effective Matching Generation: The use of prompt engineering with GPT resulted in reaching 96% accuracy in matching products to their correct categories.

ProjectDevelopment

Traditional_NLP

Steps:

• Tokenization and preprocessing. for example, performing stemming or lemmatization.

• IDF Calculation: Calculate the inverse document frequency for each term.

• TF-IDF Calculation:Calculate the TF-IDF score for each term in the product title based on TF and IDF values.

• TF-IDF Vectorization:Construct a TF-IDF vector for the product title based on the calculated TF-IDF scores.

• Cosine Similarity Calculation:Calculate the cosine similarity between the TF-IDF vector of the product title and the category vectors in the hierarchical tree.

• Prediction: Based on the highest cosine similarity score

Achieve 80% accuracy in top 5 candidates. Typical badcases fails to capture contextual information of product and category.

RAG

Steps:

• Indexing: Construct the hierarchical category tree with jsonl format.

• Vectorization: Employed the text-embedding-3-small model for optimal vector dimensionality to ensure robust and scalable indexing.

• Retrieval: Fine-tuned the retrieval algorithms using cosine distance to balance cost and precision effectively.

• Query and Generation:: Find the most matching category from the candicates by use of the capacity of llms. Significantly lower the hallucination rate by limiting the candidate token quantity.

Challenges_Faces

Chunk_Strategies

raise several chunk strategies:

1. use product title as bridge to connect category with new product titles.

2. first, stem and simplify product title only keep main info, then do the step above.

3. directly use category jsonl as chunk then prompt choose from candidate. [ best performance and lowest cost] above

Hallucination

In the multi-classification tasks involving long prompts, the language model (e.g., GPT) often generated hallucinated or irrelevant responses. This issue was particularly challenging as it compromised the accuracy and reliability of the product categorization.

Addressing Hallucination: To mitigate this issue, we integrated Retrieval-Augmented Generation (RAG) and utilized cosine similarity measures. This approach helped narrow down the classification candidates, effectively reducing the potential for hallucination by focusing the language model’s responses on more probable and relevant categories.