I was building a pipeline to automate funding proposal creation for the SBIR program when I hit a wall that completely changed my understanding of document processing. What started as a straightforward integration project turned into a two-day sprint to solve a problem I didn't know existed—and one that's likely breaking AI pipelines across the industry.
The Setup: SBIR Automation Gone Wrong
The Small Business Innovation Research (SBIR) program provides funding opportunities for tech companies, but their application materials come as PDFs with wildly inconsistent formatting. My goal was simple: convert these documents to text, feed them into a RAG pipeline, and automate proposal preparation. Standard engineering approach, proven tools, should work perfectly.
Except it didn't.
When I tested the first batch of conversions, I discovered that standard PDF-to-text libraries were losing up to 20% of the content. Not just minor formatting issues—entire sections were mangled, tables became incomprehensible, and critical details simply vanished. For an AI system designed to help create funding proposals, this wasn't just inconvenient. It was catastrophic.
The Hidden Problem with PDF Conversion
Here's what caught me off guard: this appears to be a widespread but little-discussed problem. Most developers assume PDF-to-text conversion is a solved problem. We grab a popular library, pipe documents through it, and move on to the "interesting" parts of our AI pipeline.
But SBIR documents—like many real-world PDFs—use non-standard formatting, complex layouts, embedded tables, and inconsistent text encoding. When your conversion tool hits a formatting edge case, it doesn't throw an error. It just silently loses data.
For AI and RAG systems, this creates a classic garbage-in-garbage-out scenario. Your language model might be perfectly tuned, your embedding strategy flawless, but if 20% of your source material is missing or corrupted, your entire pipeline is compromised. You're not just losing data—you're losing context, relationships, and the subtle details that often determine whether an AI system provides useful output or hallucinated nonsense.
Engineering a Real Solution
Rather than accept lossy conversion as an inevitable limitation, I decided to build a proper solution. The engineering requirements were clear:
- High fidelity: Preserve all content, including tables, images, and complex layouts
- Robust architecture: Handle edge cases without crashing
- Production ready: Fast processing with both GUI and CLI interfaces
- AI-friendly output: Generate clean, markdownlint-compliant markdown suitable for downstream processing
The solution uses a multi-library extraction approach with intelligent fallbacks. Instead of relying on a single PDF library that might choke on unusual formatting, the system tries multiple extraction methods—marker-pdf, PyMuPDF, and pdfplumber—and uses the best result for each document.
I implemented a subprocess-based, memory-safe architecture to prevent crashes when processing problematic PDFs. Each document is processed in isolation, so a corrupted file can't bring down the entire batch. The system processes documents in under 5 seconds per file while preserving images, tables, and code blocks in properly formatted markdown.
Two Days, Start to Finish
The entire converter was built over a two-day period in August 2025 using my custom agentic software development workflows. This is where modern AI-assisted development really shines—not in replacing engineering judgment, but in accelerating the implementation of well-defined solutions.
The agentic workflow handled the repetitive coding tasks while I focused on architecture decisions, library selection, and testing edge cases with real SBIR documents. By the end of day two, I had a production-grade tool that solved the original problem and could handle the broader challenge of reliable PDF processing.
The Bigger Picture: Document Processing as Infrastructure
This experience highlighted a fundamental issue in AI development: we're often so focused on model performance and prompt engineering that we underestimate the importance of high-quality data preparation. Document processing isn't glamorous, but it's foundational.
If you're building AI systems that work with real-world documents—legal contracts, research papers, government forms, technical specifications—you need to audit your conversion pipeline. Take a random sample of your documents, convert them with your current tools, and manually compare the output to the originals. You might be surprised by what you're losing.
The implications go beyond individual projects. As AI systems become more prevalent in business processes, the quality of document processing will directly impact decision-making, compliance, and operational effectiveness. A 20% data loss rate isn't just a technical inconvenience—it's a business risk.
Lessons for AI Pipeline Design
This project reinforced several principles I apply across all my AI development work:
- Test with real data early: Synthetic or clean test documents won't reveal the edge cases that break production systems
- Build redundancy into critical paths: Single points of failure in data processing will eventually fail
- Measure data quality, not just system performance: Fast processing means nothing if the output is corrupted
- Solve infrastructure problems properly: Quick fixes in foundational components create technical debt that compounds over time
The PDF converter is now a core component of my development toolkit, and the SBIR automation project got back on track. But the real value was the reminder that in AI development, the unglamorous infrastructure work often determines whether your sophisticated algorithms succeed or fail in the real world.
Sometimes the most important engineering problems are the ones hiding in plain sight.