Consultant – PySpark, Databricks Developer, Intermediate – Senior

Posted 1 hour ago

Apply Now

Resume Score

Check how well your resume matches this job before you apply.

Sign in to check score

About the role

  • PySpark/Databricks developer building and maintaining wealth-management data pipelines at Electric Mind. Developing transforms, debugging data quality issues, and supporting Databricks environments.

Responsibilities

  • Build and maintain Databricks data pipelines for a financial services/wealth management data platform
  • Write and maintain hand-written PySpark transforms for silver- and gold-layer customer, account, and transaction tables
  • Work with Lakeflow Declarative Pipelines and Auto CDC/SCD Type 2 patterns
  • Query and validate data in Unity Catalog across dev/qa/uat using SQL warehouses
  • Debug failed pipeline runs by reading event logs, tracing bad records, and fixing schema drift or data quality issues
  • Maintain reference/lookup tables and PowerShell/Python utility scripts
  • Write and update unit and integration tests
  • Participate in GitLab code reviews using feature branches and merge requests
  • Keep technical documentation current
  • Work alongside a senior engineer and help maintain healthy development, QA, and UAT environments
  • In the first 90 days, independently handle small transform bugs or enhancements, operational scripts, pipeline diagnosis, and begin taking ownership of Databricks environments and new transforms

Requirements

  • Comfortable writing clean, readable Python code; understands functions, modules, and basic OOP
  • Databricks knowledge
  • Some exposure to Apache Spark/PySpark, or strong SQL skills plus willingness to learn Spark quickly
  • Working knowledge of SQL, including joins, aggregations, and window functions
  • Basic Git workflow: branches, commits, and pull/merge requests
  • Comfortable reading others' code and stack traces and debugging methodically
  • Direct Databricks experience, Delta Lake, medallion architecture, CDC/SCD concepts, Azure, Kafka, CI/CD, and financial services experience are nice-to-have
  • Must be able to learn quickly, ask good questions, and write clean code

Job type

Full Time

Experience level

Senior

Salary

CA$100,000 - CA$140,000 per year

Degree requirement

No Education Requirement

Tech skills

ApacheAzureKafkaPySparkPythonSparkSQLUnity

Location requirements

HybridTorontoCanada

Report this job

Found something wrong with the page? Please let us know by submitting a report below.