About the role

Senior Lead Data & AI Engineer developing scalable data engineering frameworks for Vanguard. Managing AI deployments and orchestration using Databricks and cloud-native services.

Responsibilities

Architect and build reusable, metadata-driven data and AI engineering frameworks
Leverage Databricks, lakehouse architecture, declarative pipelines, and cloud-native services
Architect and deploy Delta Live Tables and Lakeflow jobs on Databricks
Leverage Databricks Workflows and Job Orchestration to schedule and monitor AI agent deployments
Integrate Lakeflow for real-time data stream processing
Ensure seamless orchestration between AI models and data pipelines
Implement and orchestrate AI agents using frameworks such as Agentic systems, AgentOps tooling
Hands-on experience deploying AI agents using RAG, Graph RAG, MCP-enabled integrations
Manage AI agent lifecycles, monitoring, and scaling using tools like SageMaker
Ensure robust data governance and AI observability through Unity Catalog, AWS Glue

9–12+ years in data engineering, specializing in AI deployment within cloud ecosystems (Databricks, AWS)
Hands-on experience deploying AI agents using frameworks like RAG, graph RAG, and orchestrating agents (AgentOps, Agent-bricks, etc.)
Proficient in AWS AI/ML services (SageMaker, Bedrock) and orchestration tools (MWAA, Step Functions)
Strong knowledge of lakehouse architecture, Unity Catalog, and data modeling best practices
Deep experience in data orchestration, monitoring, and scalable AI-driven workflows.