Lead Data Engineer // Azure & Kafka

  • Australia
  • Sydney
  • Contract
  • AU$900 - AU$1000 per day

They are seeking an experienced and motivated Lead Data Engineer with 8-10 years of expertise in building and managing data platforms. The ideal candidate will be instrumental in designing and implementing scalable data solutions while leveraging cutting-edge technologies and tools. This role offers an opportunity to work on complex, real-time data integration projects and develop state-of-the-art data lakes to support organizational goals.

Responsibilities:

  • Design, develop, and manage data platforms utilizing Microsoft Azure services, including Azure Blob Storage, Azure Kubernetes Service (AKS), and Azure Data Factory (ADF).
  • Build and optimize Data Lakes using Databricks, with proficiency in PySpark and Python.
  • Develop and implement real-time data integration solutions using data streaming technologies such as Kafka, Azure Functions, ADF, and Flink.
  • Architect and maintain CI/CD pipelines for seamless deployment and integration of data systems.
  • Write, debug, and optimize complex PL/SQL and T-SQL procedures to ensure efficient database operations.
  • Collaborate with cross-functional teams to identify and resolve data challenges, ensuring data integrity and availability.
  • Continuously improve the performance and scalability of data systems through innovative practices and tools.

Must Have:

  • 8-10 years of experience in data engineering with a focus on modern data platforms and integrations.
  • Hands-on expertise with Microsoft Azure ecosystem, including Azure Blob, AKS, and ADF.
  • Strong experience in Databricks, with advanced knowledge of PySpark and Python.
  • Proven ability to design and implement real-time data streaming solutions using Kafka, Azure Functions, ADF, and Flink.
  • Solid understanding and practical experience in building and managing Data Lakes on Databricks.
  • Exceptional SQL skills, with hands-on experience in PL/SQL and T-SQL for writing and debugging complex SQL procedures.
  • Strong problem-solving skills and a collaborative approach to working with stakeholders.
  • Experience leading projects and mentoring junior engineers is a plus.

If you think you have above skills and experiences, click ‘Apply’ button or send your resume to alex.nguyen@talentinternational.com

Apply now

Submit your details and attach your resume below. Hint: make sure all relevant experience is included in your CV and keep your message to the hiring team short and sweet - 2000 characters or less is perfect.