PySpark/Databricks Developer, Junior to Intermediate

Posted 5 days ago

Apply Now

Resume Score

Check how well your resume matches this job before you apply.

Sign in to check score

About the role

  • Junior-to-intermediate developer building PySpark and Databricks data pipelines for financial services and wealth management platforms. Debugging transformations, validating data, and maintaining cloud-based environments.

Responsibilities

  • Write and maintain hand-written PySpark transforms for silver and gold layer tables, including customer, account, and transaction data models
  • Work with Lakeflow Declarative Pipelines and Auto CDC / SCD Type 2 patterns for change-data-capture merges
  • Query and validate data in Unity Catalog across dev/qa/uat environments using SQL warehouses
  • Debug failed pipeline runs by reading event logs, tracing bad records, and fixing schema drift or data quality issues
  • Maintain reference/lookup tables and PowerShell/Python utility scripts used to operate the platform
  • Write and update unit and integration tests for transform logic
  • Participate in code reviews using Git feature branches and GitLab merge requests
  • Keep documentation current when system behavior changes
  • Build and maintain Databricks data pipelines for a financial services/wealth management data platform
  • Help ingest, transform, and publish data through bronze, silver, and gold layers, file exports, and Kafka
  • Work alongside a senior engineer and help keep environments healthy across dev/qa/uat
  • In the first 90 days, independently handle small transform bugs or enhancements, test changes, open merge requests, run operational scripts, query tables, diagnose failures, and begin taking ownership of Databricks environments

Requirements

  • Solid Python fundamentals; ability to write clean, readable code using functions, modules, and basic OOP
  • Some exposure to Apache Spark / PySpark, or strong SQL skills plus willingness to learn Spark quickly
  • Working knowledge of SQL, including joins, aggregations, and window functions
  • Basic Git workflow knowledge: branches, commits, and pull/merge requests
  • Ability to read other people's code and stack traces and debug methodically
  • Direct Databricks experience is nice to have
  • Familiarity with Delta Lake, medallion architecture, or CDC/SCD concepts is nice to have
  • Azure exposure is nice to have
  • Kafka or other streaming/event systems experience is nice to have
  • CI/CD pipeline experience is nice to have
  • Prior financial services or regulated data environment experience is nice to have

Benefits

  • Inclusive employer committed to diversity
  • Accommodation provided during the recruitment and selection process
  • Opportunity to grow fast and collaborate with exceptional teammates
  • Exposure to high-impact projects for scaling startups and global enterprises
  • Opportunity to learn and grow into a data engineering specialist

Job type

Full Time

Experience level

Junior

Salary

Not specified

Degree requirement

No Education Requirement

Tech skills

ApacheAzureKafkaPySparkPythonSparkSQLUnity

Location requirements

HybridTorontoCanada

Report this job

Found something wrong with the page? Please let us know by submitting a report below.