Lead Databricks Engineer designing, building, and scaling cloud-based lakehouse platform. Collaborating with data teams for ML and analytical workloads.
Responsibilities
Architect, build, and maintain scalable ETL/ELT pipelines on the Databricks Lakehouse Platform using PySpark, Spark SQL, and Delta Lake.
Design and implement medallion (bronze/silver/gold) data architectures and enforce data quality, governance, and lineage standards.
Optimize Spark jobs and cluster configurations for performance and cost, including partitioning, caching, and autoscaling strategies.
Implement and manage Unity Catalog for access control, data governance, and cross-workspace asset sharing.
Build and orchestrate workflows using Databricks Workflows, Delta Live Tables, and CI/CD pipelines.
Collaborate with data scientists, analysts, and business stakeholders to translate requirements into reliable data products.
Establish engineering best practices, conduct code reviews, and mentor junior data engineers.
Monitor production pipelines, troubleshoot failures, and drive root-cause analysis and continuous improvement.
Requirements
5+ years of data engineering experience, with 3+ years building production solutions on Databricks and Apache Spark.
Expert proficiency in Python (PySpark) and advanced SQL.
Deep hands-on experience with Delta Lake, Unity Catalog, and the medallion architecture pattern.
Strong experience with at least one major cloud platform (AWS, Azure, or GCP) and its core data services.
Proven track record optimizing Spark performance and managing cluster cost.
Experience with data modeling, warehousing concepts, and building dimensional/analytics-ready datasets.
Proficiency with Git-based version control, CI/CD, and infrastructure-as-code.
Bachelor's degree in Computer Science, Engineering, or equivalent practical experience.
Water resource engineer - in - training supporting flood, river, stormwater, and watercourse projects at Onterris. Assisting modelling, design, field investigations, and regulatory work.
Senior Process Engineer delivering mineral - processing design and improvements for Sedgman’s global resources business. Supporting mining projects from feasibility through detailed engineering in Montreal’s hybrid office.
Lead threat intelligence for Clio’s legal AI platform, tracking adversaries, fraud, and abuse. Building intelligence capabilities that protect law firms and Clio Payments.
Quantum Engineer improving T - centre qubit quality and scalability at Photonic, a quantum computing company. Collaborating across nanofabrication, hardware, software, and modelling teams.
Co - op ASIC Digital Verification Engineer contributing to verification of SiTime’s mixed - signal PLL integrated circuits. Learning testbench and test - case development within the ASIC verification team.
Co - op engineer validating SiTime’s MEMS timing ASIC devices through PCB debugging, lab characterization, and Python test automation. Supporting pre - silicon drivers and production - readiness verification.
Traffic signal and roadway lighting engineer designing transportation infrastructure for Arcadis. Preparing compliant drawings, specifications, photometrics, reports and field deliverables.
Staff Forward Deployed Engineer building AI - powered client integrations for OpenLoop’s telehealth infrastructure. Creating reusable SDKs, agents, API tooling, and deployment practices across U.S. virtual care.
Principal Geotechnical Engineer leading geotechnical infrastructure projects for AtkinsRéalis, an engineering and nuclear services organization. Mentoring teams and expanding transportation, tunnel, rail, and municipal geotechnics capability.