AI-900 12,13

Module : 12 

Overview of Information Extraction

* Information extraction is a process that uses AI to pull structured data from unstructured or semi-structured documents, like receipts, invoices, forms, and scanned files. It combines computer vision, OCR, and machine learning or generative AI.

* Key Steps in an Information Extraction Pipeline

1. Text Detection & Extraction (OCR)

- It uses computer vision to find text areas in an image.

- The system extracts raw text from scanned or image-based documents.

2. Value Identification & Mapping

It maps the OCR text to specific data fields using:

- Machine learning

- Rule-based mapping
- Generative AI (in newer solutions)

Example

From a scanned receipt, the system extracts fields such as:

- Vendor

- Date
- Subtotal
- Tax
- Total

* Choosing the Right Approach

When designing an information extraction solution, consider:

Document Characteristics

- Layout consistency

- Fixed templates lead to simple, rule-based extraction
- Many formats require machine learning or generative AI methods

Volume

- High volume needs automated machine learning or scalable cloud services

Accuracy Requirements

- Critical data may need human verification

Technical Infrastructure

Security & Privacy

- Sensitive documents need strong data protection

Processing Power

- AI and large language models need significant computing power

Latency Requirements

- For real-time processing, use lighter, faster models

Scalability

- Cloud solutions can adjust to different workloads

Integration Complexity

- Check API formats, data outputs, and system compatibility

* Recommendation

Most solutions use tools like:

- Azure Document Intelligence (formerly Form Recognizer)

- Azure Content Understanding

These tools cut down development time and offer:

- Reliability

- Scalability

- High accuracy

- Easy API integration

Optical Character Recognition (OCR) 

* OCR converts visual text in images, such as scanned documents, photos, PDFs, and screenshots, into editable and searchable text data.

Used for:

- Invoices and receipts

- Forms

- Handwritten notes

- Scanned PDFs

- Photos of documents

* OCR works through a five-stage pipeline:

1. Image Acquisition and Input

Sources:

- Smartphone photos

- Scanned pages

- Video frames

- PDF images

- Image quality at this stage heavily affects accuracy.

2. Preprocessing and Image Enhancement

- This step improves image clarity before detecting text.

- Noise Reduction

- Traditional methods: Gaussian filter, median filter, morphology

- Machine learning methods: Denoising autoencoders, CNNs

- Contrast Adjustment

- Traditional methods: Histogram equalization, thresholding

- Machine learning methods: Deep enhancement models

- Skew Correction

- Traditional methods: Hough transform, projection profiles

- Machine learning methods: Regression CNNs to detect rotation

- Resolution Optimization

- Traditional methods: Bicubic/bilinear scaling

- Machine learning methods: Super-resolution GANs, ResNet models

3. Text Region Detection

- This stage identifies where the text is located.

Layout Analysis

- Traditional methods: Connected components, RLE

- Deep learning methods: U-Net, Mask R-CNN, LayoutLM

Text Block Identification

- This groups characters into words, lines, and paragraphs.

- Traditional methods: Clustering, whitespace analysis

- Machine learning methods: GNNs, transformers

Reading Order Detection

- Uses rule-based geometrical logic

- Machine learning uses sequence models

Region Classification

- Traditional methods: SVMs using font and position features

- Machine learning methods: CNNs, Vision Transformers

4. Character Recognition and Classification

- This is the core of OCR.

Feature Extraction

- Traditional methods: Statistical and structural features

- Deep learning methods: CNN-learned features

Pattern Matching

- Template matching

- HMMs, SVMs, KNN

- Deep neural networks (LeNet, ResNet, EfficientNet)

Context Analysis

- This improves accuracy using language understanding:

- N-gram models

- Dictionary correction (edit distance)

- LSTMs, Transformers (BERT-like models)

- Attention mechanisms

Confidence Scoring

- Uses Bayesian scoring

- Softmax probabilities

- Ensemble outputs

5. Output Generation and Post-Processing

- This step converts recognized characters into finished text.

- Text Compilation

- Uses rule-based assembly

- RNN/LSTM sequence models

- Transformers for complex layouts

- Format Preservation

- This keeps paragraphs, spacing, and tables.

Coordinate Mapping

- This stores the location of each text piece.

Quality Validation

- Uses dictionaries and grammar checks

- Statistical language models

- Neural validation (GPT/BERT)

- Ensemble validation

OCR Pipeline = I-P-D-R-O  

1. Image input  
2. Preprocess  

3. Detect text regions 

4. Recognize characters  

5. Output and refine  

Field Extraction & Mapping

* Field extraction takes the OCR output and identifies what each text value means, like Date, Total, and Invoice Number. It then organizes this information into a structured format used in business systems.

* OCR tells you what the text is. Field extraction tells you what the text means.

* Field Extraction Pipeline (5 Stages)

1. OCR Output Ingestion

This stage takes OCR results, including:
- Raw text
- Bounding box positions
- Reading order
- Confidence scores
- Layout information

Field extraction relies heavily on position and context, not just text.

2. Field Detection & Candidate Identification

A. Template-Based Detection

- Uses anchor keywords, such as "Date:" and "Total:"

- Applies regular expressions

- Utilizes fixed layout templates

This method is very accurate for known document formats but weak when the layout varies.

B. Machine Learning-Based Detection

- This method trains models using labeled examples. It employs transformers, GNNs, and layout-based AI. It also uses sequence-to-sequence models. 

- This approach handles varying layouts and learns patterns, but requires training data.

C. Generative AI-Based Detection

- This method prompts an LLM with document text and schema. It uses few-shot examples and chain-of-thought reasoning. 

- It requires minimal training and offers a flexible approach.

3. Field Mapping & Association

- Once candidate values are identified, they are mapped to schema fields.

A. Key-Value Pairing

Techniques include:

- Proximity analysis to find the nearest label and value

- Reading order analysis

- Geometric relationships like alignment and indentation

B. Linguistic Analysis

- This includes Named Entity Recognition (NER) to detect dates, amounts, and names, along with POS tagging and dependency parsing.

C. Table / Structured Content Extraction

This is used for invoices, receipts, and line items, employing techniques such as:

- CNN-based table detection

- Object detection for cells

- Graph models for grid structure

- Header detection

- Row-column associations

D. Confidence & Validation

This stage checks:

- OCR confidence

- Pattern match confidence

- Cross-field validation, ensuring the subtotal equals the sum of items

- Context validation, like date format and currency checks

4. Data Normalization & Standardization

- This process standardizes extracted values before they are integrated.

A. Format Standardization

This involves:

- Converting dates to a single format

- Normalizing currency formats

- Standardizing decimal and number formats

- Unifying text casing and encoding

B. Data Validation & QA

- This includes rule-based validation to check patterns, ranges, and required fields, as well as statistical validation to identify outliers and analyze distributions. It also involves cross-document validation between invoices and purchase orders.

5. Integration with Business Systems

- After extraction, the values are prepared for use in downstream systems.

A. Schema Mapping

- Values are mapped to database fields, data types are converted, field names are changed, and business rules are applied.

B. Quality Reporting

- This stage reports field-level confidence, document-level accuracy, and classifies errors with analytics.

* Field Extraction Pipeline = I-D-M-N-I

- Ingest OCR output

- Detect fields

- Map fields

- Normalize values

- Integrate into systems

Module : 13 
AI-Powered Information Extraction

Azure AI Services for Information Extraction

* Azure AI offers several cloud-based services to help extract, analyze, and understand information from images, documents, audio, video, and other unstructured data.

* You can use these services individually or combine them to create effective automation and data-processing solutions.

* Core Azure AI Services for Information Extraction  

1) Azure Vision Image Analysis

This service helps extract insights from images by:

- Detecting common objects

- Generating captions

- Tagging image content

- Extracting text with OCR

Useful for:

- Image search

- Digital asset tagging

- Data extraction from product labels, receipts, and more

2) Azure Content Understanding

- This is a generative AI service that performs multimodal analysis.

It can extract insights from:

- Documents

- Images

- Audio

- Video

Useful for:

- Meeting summarization

- Policy analysis

- Processing various types of content

- Automated knowledge extraction

3) Azure Document Intelligence

- This service specializes in reading and extracting data from digital or scanned forms.

It extracts:

- Fields

- Values

- Tables

- Key-value pairs

It works with:

- Invoices

- Receipts

- Purchase orders

- Tax forms

- Business documents

Useful for:

- Automating business processes

- Handling accounts payable workflows

- Digitizing documents

4) Azure AI Search

- This service provides AI-powered indexing using a series of cognitive skills.

It can extract, enrich, and index information from:

- Structured content

- Unstructured content

- Images

- PDFs

- Scanned documents

Useful for:

- Knowledge mining

- Creating searchable digital repositories

- Enterprise search solutions

* Common Use Cases  

1) Data Capture

- Automatically capture data from images or documents.  

- Example: Extracting contact details from a business card using a phone camera.

2) Business Process Automation

- Trigger workflows by reading data from forms.  

- Example: Extracting fields from invoices and sending them to accounts payable.

3) Meeting Summarization & Analysis

- Extract key points, decisions, and action items from recorded calls or videos.  

- Example: AI-generated meeting notes and summaries.

4) Digital Asset Management (DAM)

- Automatically tag, classify, and index images and videos.  

- Example: Creating a searchable library of stock photos.

5) Knowledge Mining

- Extract information from large amounts of structured and unstructured data.  

- Example: Reading census forms to create a searchable database.

* Quick Summary

- Vision Image Analysis: Objects, captions, tags, OCR.  

- Content Understanding: Multimodal AI for documents, images, audio, and video.  

- Document Intelligence: Reads structured forms like invoices and receipts.  

- AI Search: Index and enrich content with AI skills.  

- Use Cases: Automation, search, data capture, meeting analysis, DAM, and knowledge mining.

Extract Information with Azure Vision - Simple Notes

* Azure Vision Image Analysis is great for getting insights from photos, business cards, menus, and small scanned documents. It generates captions, tags, detects objects, and extracts text from images.

1. Automated Caption & Tag Generation

- Azure Vision can analyze an image and produce:

- A main caption, which is a short description of the whole image

- Dense captions, which focus on key objects

- Tags, which are keywords that classify the image

Example

- Image: A man walking a dog in a busy street

Caption:

- A man walking a dog on a leash

Dense Captions:

- A man walking a dog on a leash

- A man walking on the street

- A yellow car on the street

- A green telephone booth

Tags:

- outdoor, vehicle, building, road, street, taxi, person, dog, yellow, walking, city, car, clothing

These captions and tags help with:

- Search indexing

- Photo organization

- Automated content labeling

2. Object Detection

- Azure Vision can recognize common objects and people in an image and return:

- Object name

- Bounding box, which shows the location coordinates

- Confidence score

Example

- Image: Apple, banana, and orange

Azure Vision detects:

- Apple

- Banana

- Orange

- Each object is outlined with bounding boxes to show where they are in the image.

Useful for:

- Inventory apps

- Object tracking

- Image-based classification

3. Optical Character Recognition (OCR)

Azure Vision can read text from:

- Printed documents

- Handwritten notes

- Business cards

- Menus

- Signboards

OCR extracts:

- Lines of text

- Individual words

- Locations of each text region

Example: Business Card

Extracted text:

- Adventure Works Cycles

- Roberto Tamburello

- Engineering Manager

- roberto@adventure-works.com

- 555-123-4567

Useful for:

- Digitizing contacts

- Translating menus

- Reading labels

- Small document extraction

* Quick Summary

- Captioning: Describes the image.

- Dense Captions: Describes key objects.

- Tags: Keywords for categorization.

- Object Detection: Identifies and locates objects.

- OCR: Extracts text from images.

- Azure Vision Image Analysis is perfect for capturing data, understanding images, and processing small documents.

Extract Information from Forms with Azure Document Intelligence

* Azure Document Intelligence is designed for processing complex documents and forms. It can manage everything from simple receipts to detailed tax forms and multi-page applications.

It offers:

- Prebuilt models for common document types

- Custom model training for specialized business documents

1. Using Prebuilt Models

Azure Document Intelligence includes many ready-to-use models, such as:

- Receipts

- Invoices

- Purchase orders

- Tax forms

- Mortgage applications

- ID documents

- And more.

- Example: Mortgage Application

A loan company processes hundreds of mortgage forms each day. Azure’s prebuilt mortgage model can automatically extract fields like:

- Borrower name

- Address

- Telephone number

- Social Security Number

- Date of birth

- Marital status

- Employment status

- Employer name

- Employer address

- Income

- Citizenship

- This saves time, reduces manual data entry, and decreases errors.

2. Creating Custom Models

- If your documents do not match prebuilt templates, you can create custom models.

- How Custom Model Creation Works

1) Collect Samples  

- Gather several examples of your document.

2) Use OCR to Identify Layout  

- Azure OCR will detect text, tables, and structure.

3) Label Fields  

You mark the fields you want to extract:

- Names

- Dates

- IDs

- Custom business fields

4) Train the Model  

- Azure learns your document layout and field positions.

5) Deploy & Use  

- The model extracts data from future documents automatically.

Custom models are great for:

- Proprietary business forms

- Industry-specific templates

- Multi-page documents

- Scanned archives

* Quick Summary

- Prebuilt models = fast extraction for common forms (receipts, invoices, mortgage forms, tax papers).

- Custom models = tailor extraction to your own documents using labeled training samples.

- OCR + labeling = identifies layout + fields to extract.

- Ideal for high-volume processing and automation.

Create a Knowledge Mining Solution with Azure AI Search

* What is Azure AI Search?

- Azure AI Search is a cloud service for indexing, searching, and improving data with AI skills. It helps build digital asset management and knowledge mining solutions.

* Key Components

1. Indexer

- An indexer runs a repeatable process that:

- Ingests data from sources like Azure Storage and databases.

- Breaks down documents to extract contents such as text, images, and metadata.

- Uses AI skills to enrich the data.

- Creates fields for the search index.

- Stores the results in an index.

- AI Skills Used in the Indexer

- AI skills help pull useful information from unstructured content:

Azure Vision

- Generates image tags and captions.

Azure Language

- Extracts sentiment, key phrases, and entities.

Azure Document Intelligence

- Pulls structured fields from forms and documents, like names and dates.

Knowledge Store

- Along with a searchable index, Azure AI Search can store enriched data in a knowledge store using Azure Storage.

* Types of assets stored:

Tables

- Structured field values extracted by the skill set.

Images

- Extracted or generated from documents.

JSON documents

- Detailed hierarchical representations of the enriched data.

- This supports downstream analytics, dashboards, or custom applications.

* Purpose

Azure AI Search helps create:

- Powerful search experiences.

- Knowledge mining platforms.

- Enterprise-wide digital asset management systems.