The Problem
Why integration is harder than it looks.
The Export-to-Excel Workflow
Data exists in your ERP. But the only way to get it into Power BI is: run a report, download to Excel, clean up the headers, upload to SharePoint. Someone does this every week. It takes 4 hours. This is your "integration."
The One-Time Load That Became Critical
Someone did a one-time data pull for a board presentation. It worked, so it became the "official" process. Now critical Power BI dashboards depend on a Python script that runs on someone's laptop.
Full Refresh at Scale
The Data Factory pipeline does a full refresh every night — truncate and reload. It worked with 100K rows. Now you have 15M rows. The refresh takes 6 hours and blocks morning reports. You need incremental loads, but the pipeline wasn't built for CDC.
Deliverables
Integration Architecture
Documented approach for each source: connectivity method, refresh pattern, schedule, and monitoring approach.
Data Factory Pipelines
Production pipelines that extract, load, and transform data into your Fabric Lakehouse. Parameterized and incremental.
Monitoring & Alerting
Pipeline monitoring integrated with Teams or your incident system. You know when something fails within minutes.
Documentation & Runbooks
Technical documentation: data flows, Notebook transformation logic, schedules, and troubleshooting procedures.
Our process.
Discovery
Analyze source systems: data structures, volumes, API capabilities, refresh requirements.
Pipeline Development
Build pipelines iteratively, starting with priority sources. Each integration tested and validated.
Testing & Optimization
Test under production conditions: full volumes, concurrent loads, failure scenarios.
Deployment & Handoff
Deploy to production, configure monitoring, train your team.
Integrations in action.
How we automated vendor spend categorization with Fabric Notebooks
Situation
A construction company's procurement team struggled with credit card spend analysis. Vendor names appeared in inconsistent formats across transactions — "Home Depot," "THE HOME DEPOT #4521," "HD Supply." This made it impossible to track total vendor spend, enforce preferred vendor usage, or analyze spending by category.
What We Built
- Fabric Notebook Pipeline: Python-based vendor name normalization using intelligent string matching and fuzzy logic.
- Categorization Engine: Automated Level 1 and Level 2 spend categorization.
- Power BI Semantic Model: Clean, categorized data landing in a semantic model for self-service analysis.
- Power App for Exceptions: Embedded Power App in Power BI allowing business users to normalize unmatched vendors.
Technical Details
- Fabric Notebook with PySpark for scalable processing.
- Fuzzy matching using Levenshtein distance with configurable threshold.
- Lookup tables maintained in SharePoint for business-owned categorization.
- Incremental processing of new transactions only.
Outcome
Unified view of vendor spend across all credit card transactions. Procurement identified $180K in spend fragmentation that could be consolidated to fewer vendors for volume discounts. Business users now self-serve categorization updates.
How we built real-time inventory tracking across 12 job sites
Situation
A construction firm managed material inventory across 12 active job sites using Excel spreadsheets. Project managers tracked their own inventory locally. Central operations had no visibility into what materials existed where. Result: over-ordering at some sites while others delayed projects waiting for materials.
What We Built
- Data Integration: Connected procurement system, delivery tracking, and usage logs into Fabric Lakehouse.
- Fabric Notebooks: Transformation pipeline creating inventory snapshots with movement history.
- Power BI Dashboard: Real-time stock levels, reorder alerts, vendor delivery performance.
- Mobile Access: Power BI mobile app for project managers to check inventory on-site.
Outcome
Single source of truth for inventory across all sites. Over-ordering reduced 25%. Project delays from material shortages reduced 40%. Operations now redistributes excess materials between sites proactively.