Databricks Data Engineer building scalable ingestion and ETL pipelines for Avanquest’s global software products. Ensuring data quality, lineage, reliability, and optimized storage.
About the role
- Data Engineer responsible for designing and maintaining enterprise data warehouse for various projects. Collaborating with stakeholders to ensure efficient data flow and integration.
Responsibilities
- Design, develop, and maintain scalable Synapse pipelines for data ingestion, processing, and transformation in Azure data platform.
- Develop and optimize T-SQL and Python-based data processing scripts and integrate them with Synapse pipelines for ETL tasks and data transformations.
- Design and develop mapping dataflow solutions for complex data transformations.
- Develop and maintain data storage solutions, ensuring efficient data retrieval and storage on Azure Data Lake and databases.
- Implement and maintain API integrations, including REST APIs, for seamless data flow between systems.
- Design and test the end-to-end process for data ingestion, data transformation, data cleansing, data delivery, data quality and data timeliness.
- Implement and ensure data security and privacy measures, complying with industry standards and regulations.
- Monitor and optimize the performance of Azure data services, Azure Synapse pipelines, Function Apps, and Logic Apps.
- Improve the scalability, efficiency, and cost-effectiveness of data pipelines.
Requirements
- Bachelor's degree or equivalent in the field of Computer Science or Engineering
- 3+ years of Proven experience as a Data Engineer, specifically working with Microsoft Azure technologies.
- Proficient in T-SQL, Pyspark, Mapping Dataflow and Python programming language
- Hands-on experience with Azure data services, including Azure Data Factory, Azure Data Lake, Azure Function App, Azure Logic App, Azure SQL Database, Azure Synapse Analytics, mapping dataflow, and Event Hub.
- Understanding of data warehousing concepts and best practices.
- Experience with Data modeling.
- Experience in working with structured, semi-structured and unstructured data.
- Excellent written and verbal communication skills
- Excellent multi-tasking and organizational skills
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Report this job
Found something wrong with the page? Please let us know by submitting a report below.
EC
Administrateur de plateforme de données chez EDC, société d’État aidant les entreprises canadiennes à réussir à l’étranger. Soutien, sécurisation et amélioration de plateformes infonuagiques de données et d’analyse.
EC
Associate Data Platform Administrator supporting secure cloud data platforms at EDC, Canada’s export credit agency. Monitoring, provisioning, troubleshooting, and improving analytics infrastructure.
Data engineering manager leading Arc’teryx’s Databricks platform, ingestion, and QA infrastructure. Developing engineers and delivering reliable data solutions for retail, digital, and business operations.
Data Engineering Advisor building and maintaining Desjardins’ cybersecurity data lake. Developing Azure - based ingestion, governance, and data - quality automation.
Senior Data Engineer building scalable data pipelines and platforms for Sleep Country Canada, a Canadian sleep - products retailer. Enabling reliable reporting, analytics, and operational decision - making across business teams.
Vice President leading data strategy, architecture, and engineering for PICTON Investments, a Canadian alternative investment manager. Building financial data platforms supporting portfolios, analytics, risk, and AI.
LP
Senior Data Engineer – Databricks | Python | SQL | AWS
LinkedIn Recruiter Post
Senior Data Engineer needed with 7+ years in Databricks, Python, SQL, AWS. Experience with Unity Catalog or Banking/Financial Services is a plus.
LP
Azure Data Engineer-Databricks /ADF
LinkedIn Recruiter Post
Azure Data Engineer needed for a financial client to develop pipelines using ADF, Databricks, and Azure Functions, and migrate to Unity Catalog.
LP
Databricks Architect
LinkedIn Recruiter Post
Hiring Databricks Architect in Brampton, ON (Hybrid). 10+ years experience required. Skills: Databricks, Informatica, Spark, PySpark, SQL, Delta Lake, Unity Catalog, Azure.