file_loader: Raw PDF/TXT
cleaner: Remove TOC/Footers
chunker: 120-word Overlapping Chunks
prompts: Contextual Prompting
generator: Phi-2 Generation with Retry
qa_parser: Pattern Matching Parsing
evidence: Sentence-level Embedding Match
validators: Reference & Vague Check
dedupe: Semantic Cosine Similarity
interim: Grounded QA Dataset
SimpleRetriever: TF-IDF Vectorization
SFT: Supervised Fine-Tuning
Discriminators: Fact / Style / Safety
RL Loop: PPO / Policy Gradient
Reward Calculation: Fact-Check + Critic Probabilities
evaluation: Final Benchmarked Model
DATA INGESTION & PRE
QA PARSING & VALIDATION
HALLUCINATION REDUCTION - RL
DATASET RETRIEVAL
by Saad