Databricks Data Engineer developing and optimizing data pipelines using the Databricks platform. Collaborating with cross-functional teams to deliver high-quality data solutions for analytics and reporting.
Responsibilities
Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
Build and optimize ETL/ELT workflows to ingest, transform, and curate large datasets.
Implement data solutions following Medallion Architecture (Bronze, Silver, Gold).
Develop high-performance PySpark and Spark SQL transformations.
Optimize Delta Lake tables using partitioning, optimization, vacuuming, and time travel capabilities.
Orchestrate and monitor Databricks Workflows and scheduled jobs.
Collaborate with analytics, engineering, and product teams to deliver reliable data products.
Apply software engineering best practices, including version control, automated testing, and CI/CD.
Utilize AI-assisted development tools for code generation, debugging, documentation, and productivity while validating outputs to ensure quality and accuracy.
Contribute to data quality, performance tuning, and operational excellence across the platform.
Requirements
7+ years of experience in data engineering
Bachelor's degree in Computer Science, Software Engineering, or a related field (or equivalent work experience)
Hands-on experience with Databricks Workspaces, Clusters, Jobs, and Unity Catalog
Strong knowledge of Delta Lake, including ACID transactions, Time Travel, OPTIMIZE, and VACUUM
Experience building and managing Databricks Workflows
Strong experience with PySpark, Spark SQL, and Scala
Expertise designing and building ETL/ELT pipelines
Programming Languages: Python (Advanced), SQL (Advanced), Scala (Strong working knowledge)
Experience with at least one of the following cloud platform: AWS (Amazon S3, AWS Glue, Redshift), Azure (Azure Data Lake Storage Gen2 (ADLS Gen2),Azure Data Factory (ADF), Azure DevOps)
Experience using AI-powered developer tools for coding, debugging, testing, and documentation
Understanding of responsible AI usage, code validation, and quality assurance.
Git version control, CI/CD pipelines, Unit testing and automated validation for data transformations, Strong software engineering and code quality practices.
Benefits
Join one of the world’s fastest-growing AI-first digital engineering companies and make a real impact at scale.
Lead and collaborate with a high-energy team of talented, driven individuals solving complex, meaningful challenges.
Work with Fortune 500 companies and disruptive innovators in a research-driven environment with 60+ patents.
Stay ahead of the curve by gaining hands-on experience with cutting-edge AI, ML, data, and cloud technologies while continuously upskilling.
Manufacturing Development Engineer scaling Arbutus Medical’s procedure - kit production from prototypes to reliable manufacturing. Bridging product development, suppliers, quality, and operations for surgical medical devices.
Forward Deployed Engineer building production AI systems for Sperton’s operating businesses. Discovering high - impact workflows, shipping end - to - end software, and measuring operational impact.
Autonomy Engineer developing hybrid behavior - tree and learned - policy systems for Dominion Dynamics’ sovereign Canadian combat platform. Deploying validated autonomy from simulation onto test aircraft.
Ingénieur responsable de projets électriques du bâtiment chez Norda Stelo, concevant des systèmes électromécaniques. Gestion de projets institutionnels, commerciaux et industriels, budgets et échéanciers.
Substation Owner’s Engineer overseeing utility power - delivery designs, standards, and construction quality for Leidos. Advising project owners across planning, procurement, construction, and commissioning.
Water resource engineer - in - training supporting flood, river, stormwater, and watercourse projects at Onterris. Assisting modelling, design, field investigations, and regulatory work.
Senior Process Engineer delivering mineral - processing design and improvements for Sedgman’s global resources business. Supporting mining projects from feasibility through detailed engineering in Montreal’s hybrid office.
Lead threat intelligence for Clio’s legal AI platform, tracking adversaries, fraud, and abuse. Building intelligence capabilities that protect law firms and Clio Payments.