Back to Projects AI Pipelines

Data Source Automator

End-to-end pipeline: research → specs → code → deploy → verify with automated feedback loops and self-healing.

The Problem

Data engineering teams waste weeks researching data sources, writing specifications, building extraction pipelines, and deploying them. Each service needs a schema contract with the frontend, infrastructure provisioning, and end-to-end verification — a manual process that doesn't scale beyond a handful of services.

The Solution

A multi-agent pipeline that automates the entire lifecycle: research agents explore extraction methods, spec agents generate data models, coding agents build microservices with JSON schema contracts (search, detail, report, settings, config) that bridge backend and frontend, deployment agents push to Kubernetes via ArgoCD, and verification agents run automated checks across infrastructure, data contracts, and API responses. When tests fail, a code-fixer agent patches the code and redeploys automatically — with a LoopGuard that escalates to humans after 3 failed attempts on the same bug.

Architecture

%%{init: {'theme': 'dark', 'themeVariables': { 'fontFamily': 'Inter', 'secondaryColor': '#1e293b', 'primaryColor': '#3b82f6', 'primaryBorderColor': '#60a5fa' }}}%% graph TB subgraph Research ["Research Phase"] A["Supervisor Agent"] --> B["API Researcher"] A --> B2["Download Analyst"] A --> C["Scraping Analyst"] B --> D["Method Selector"] B2 --> D C --> D end subgraph Spec ["Specification Phase"] D --> E["Data Model Gen"] E --> F["Requirements Writer"] F --> G["Human Review"] end subgraph Impl ["Implementation Phase"] G --> H["Tech Spec Agent"] H --> I["Coding Agent"] I --> J["Testing Agent"] J --> L["Schema Contract"] L --> K["Human Review"] end subgraph Deploy ["Deployment"] K --> M["Deploy Agent"] M --> N["ArgoCD Sync"] end subgraph Verify ["Verification"] N --> O["Verify Agent"] O --> O1["Infra: Pods, Health, ArgoCD"] O --> O2["Data: Swagger, Schemas, DB"] O --> O3["API: Search, Detail"] end subgraph Fix ["Feedback Loop"] O1 & O2 & O3 -->|FAIL| P["Code Fixer"] P --> Q["LoopGuard"] Q -->|retry| M Q -->|max reached| R["HITL Escalation"] end O1 & O2 & O3 -->|PASS| S["✓"] classDef default fill:#0f172a,stroke:#334155,color:#fff,stroke-width:1px; classDef review fill:#3b0764,stroke:#a855f7,stroke-width:2px; classDef agent fill:#0f172a,stroke:#3b82f6,color:#fff; classDef feedback fill:#0f172a,stroke:#f59e0b,color:#fff,stroke-width:2px; classDef success fill:#064e3b,stroke:#10b981,color:#fff,stroke-width:2px; class G,K,R review; class A,B,B2,C,H,I,J,M,O,O1,O2,O3,P agent; class Q feedback; class S success;
AI Agent
Process Step
Human Review
Feedback Loop

Tags

PythonMulti-AgentHITL

Outcomes

  • 50+ microservices deployed through the pipeline
  • Automated deploy → verify → fix → redeploy feedback loop
  • JSON schema contract system bridging backend APIs and low-code frontend
  • Self-healing with LoopGuard: max 3 iterations per bug, then HITL escalation