Lead Data Engineer - Bangalore Location

Responsibilities:

  • Design and Build Data pipelines in collaboration with Product Owners / BSA, Data Analysts and Business stake holders..
  • Work with Architecture / Practice team to ensure quality solutions are implemented with adherence to engineering best practices.
  • Build Reusable utilities, frameworks and user defined functions
  • Evaluate and utilize new technologies/tools/frameworks centered around Orchestration Pipelines.
  • Provide technical leadership on Orchestration Pipelines to the project teams.

 

 

 

 Basic requirements:

  • Overall 7+ Years of Experience in Data Modelling and ETL processing projects.
  • 4+ Years of Experience in designing and implementing large data pipelines using Spark, Hive on AWS with Python and pySpark.
  • 1+ Years of Experience in building end to end pipelines using Airflow DAG.
  • Hands on Experience in Data Ingestion, Data Cleansing, Data Transformation tools and techniques.
  • Hands on Experience with Data Quality and Data Lineage tools and frameworks.
  • Hands on Experience in Spark Batch and Streaming Data pipelines with Kafka.
  • Experience of working in Agile model.
  • Expert level knowledge on Coding and Design Best Practices.
  • Expert level experience in SQL.
  • Expert level Experience in Trouble shooting.
  • Experience supporting and working with cross-functional teams in a dynamic environment.
  • Excellent verbal and written communication skills in English
  • Ability to communicate effectively with team members and business stakeholders

Nice to have:

  • General knowledge/experience of AWS snowflake is an added advantage
  • Degree:

    Any Graduate

  • Job Type:

    Full time

  • Salary:

    3000000 - 3500000 P.A.

  • Location:

    Bangalore, Karnataka

  • Industry:

    Information Technology

  • Primary Skills:

    Airflow, Pyspark, Hive, Kafka

Get the latest jobs delivered to your inbox

Be the first to hear about the latest openings every Saturday by subscribing to our weekly newsletter. Never miss an opportunity!

Ultimate Virtual Interview Checklist [FREE DOWNLOAD]

Get prepared for your virtual interview with the help of this by ticking off items from the checklist. Download for FREE!

Get It Now

Looking for job opportunities? Look no further, join QWEEN and start applying for jobs. It's 100% FREE!