Case Study: Cimbology
An AI Knowledge Assistant for Utility Engineers
By Shawn Weekly | Deep Research Project | 2024-2025
By Shawn Weekly | Deep Research Project | 2024-2025
The electrical utility industry loses decades of irreplaceable expertise every time a senior engineer retires. How do you capture and make accessible 30+ years of tacit knowledge, specifications, and domain understanding?
Can combining Knowledge Graphs with RAG (Retrieval-Augmented Generation) across unstructured, information rich documentation ( called GraphRAG) create a verifiable, trustworthy AI assistant for engineering decisions?
I built a full-stack GraphRAG system and tested it against a real Professional Engineering (PE) exam. Results: 84.85% accuracy with full citation traceability vs. 77% for baseline LLM with zero traceability.
"A baseline LLM behaves like a 'dumb-but-fast' intern who has read every book but understood nothing. Add structured knowledge via a Knowledge Graph and you get an assistant that can reason, cite sources, and be trusted."
Meet Jack and Susan, two pillars of expertise at their electrical engineering firm.
A master substation physical designer with over 30 years of experience.
Jack creates the physical layouts and ensures that substations are properly designed for construction and maintenance.
With exceptional skills in AutoCAD and Autodesk Inventor, he's the go-to expert that colleagues rely on daily.
An electrical engineer with 30+ years designing protection and control schemes for the substations her company designs and builds.
Susan knows AutoCAD Electrical like the back of her hand.
She always knows exactly where to find the right standards, specifications, and information needed for any project.
Jack and Susan are retiring next month. Although they've shared all their specifications, drawings, files, and documents with their team, it's an overwhelming amount of information.
How will the team digest this wealth of knowledge?
How can they find what they need when they need it?
This scenario plays out across the electrical utility industry every day. Organizations lose not just employees, but decades of hard-won knowledge about:
Traditional knowledge & document management systems (SharePoint, Autodesk Vault, ProjectWise, file servers) preserve the documents but not the expertise to navigate and apply them effectively. And many of the systems above are often improperly configured and don't (or simply can't) deliver good search results.
I architected Cimbology to test whether a hybrid approach that combined the structured reasoning of Knowledge Graphs, the proven capabilities of RAG based search, and the flexibility of Large Language Models could solve the "verifiable expertise" problem.
Uses GraphDB and SPARQL to store structured domain knowledge based on the IEC 61970/61968 CIM standard. This provides the "reasoning backbone"; the means for the system to understand the relationships between Substations, ConductingEquipment, Measurements, etc.
Why it matters: Structured knowledge is verifiable and can be reasoned over. RADICAL reduction in hallucinations.
Stores embeddings of document chunks (PDF pages, sections, tables) enabling semantic search over unstructured content. Uses advanced chunking strategies (layout-aware, table extraction, multimodal image descriptions) to preserve context.
Why it matters: Unstructured documents contain the bulk of engineering knowledge. RAG makes them searchable and retrievable. Advanced chunking gets ALL of the information from the documents, not just the text in the paragraphs.
Coordinates queries across both the Knowledge Graph and Vector Database, then synthesizes results using an LLM (Google Gemini or Azure OpenAI). Provides prompt engineering, context management, and citation tracking.
Why it matters: The orchestrator ensures the LLM receives high-quality, verifiable context—not just "best guess" vectors.
RESTful API built with ASP.NET Core, Entity Framework Core, and dotnetRDF. Handles chat sessions, query routing, session persistence (SQLite), and multi-modal client support (web, AutoCAD plugin).
Why it matters: Enterprise-grade reliability, testability, and maintainability.
To prove the effectiveness of GraphRAG, I needed a real-world test. I purchased and used a Professional Engineering (PE) exam from 2025 and tested four different approaches against a corpus of 2017-era engineering standards documents.
The Challenge: If the system can answer 2025 exam questions using outdated 2017 documents, it proves the system truly understands context and relationships; it's not just keyword matching.
77.27% Accuracy
Sent questions directly to GPT-5-Mini on Azure OpenAI with zero context. Fast and surprisingly accurate, but completely unverifiable. A "black box."
Verdict: Untrustworthy for engineering decisions
53.03% Accuracy
Used basic page-level embeddings and naive vector search against Google Gemini 2.5 Pro. Performed worse than baseline! Proved that "adding RAG" doesn't automatically improve results.
Verdict: Naive RAG is actively harmful
83.33% Accuracy
Used layout-aware chunking, table extraction, and multimodal image descriptions and went back to Azure OpenAI and GPT-5-Mini. Major improvement, proves context quality matters more than model size.
Verdict: Much better, but still missing structured, semantic reasoning
84.85% Accuracy
Combined Advanced RAG with Knowledge Graph reasoning. Achieved highest accuracy and provided full citation traceability.
Verdict: Winner. Accurate and trustworthy.
Full Methodology: Tested against 66 PE exam questions. Documents included IEEE standards, NESC code, and utility design guides (all from 2017 or earlier). Used Google Gemini AND Azure OpenAI to ensure comparison across different vendor models. Accuracy measured by correct answer selection (A/B/C/D). See the full interactive analysis for question-by-question breakdown.
Unlike many "research projects" that remain theoretical, Cimbology is fully implemented, tested, and deployed. Every component described above exists as working software.
Watch this video demonstration showing Cimbology in action, including the GraphRAG query process, citation tracking, and real-time knowledge retrieval:
NOTE: If you see the infamous 153 error below, it's because YouTube wants folks to PAY them extra to embed content. Since I am not rich or a content creator, please click the link to watch directly on YouTube if you can't see it here.
The PE exam validation testing is fully documented and reproducible:
My biggest learning: LLMs are incredibly powerful pattern matchers, but they don't "understand" in the way humans do. When I tested naive RAG and saw a 24% accuracy drop, it crystallized the problem: throwing vectors at an LLM without structure is like giving a student a library card but no dewey decimal system.
Knowledge Graphs provide the structured reasoning layer that makes AI trustworthy. They enforce relationships, validate data against schemas (SHACL), and enable transparent citation of sources.
Managing four containerized services (Qdrant, GraphDB, API, Web) taught me the importance of health checks, service dependencies, and volume management.
Key lesson: Infrastructure as Code only works if it's reproducible.
Every developer should be able to run docker compose up and have a
working environment in under 2 minutes.
Using a real PE exam as a test set was crucial. It forced me to confront real-world questions with objectively correct answers. This is vastly different from "testing" an AI with subjective questions where any plausible answer feels acceptable.
Key lesson: If you can't measure it, you can't improve it. The 66-question test set became my north star and every architectural change was validated against it.
Engineering documents are rich with tables, diagrams, and figures. My early RAG attempts ignored these, treating them as "just more text." When I added table extraction and multimodal image descriptions, accuracy jumped significantly.
Key lesson: Engineering knowledge isn't just prose. Ignoring structured data (tables) and visual data (diagrams) means ignoring the most information-dense parts of a document.
I'm seeking opportunities to apply this expertise in a technical leadership role within the electrical utility industry. If you're facing similar knowledge management challenges, let's talk.