Job Description

Hi,

Our client is looking Technical Lead/Data Librarian with Python and Pyspark and AWS in Remote below is the detailed requirements.

Job Title : Technical Lead/Data Librarian with Python and Pyspark and AWS

Location : Remote

Duration : Long-Term Contract

The ideal candidate will have a strong background in data engineering and development, with expertise in various data skills and tools. The role involves working on complex data projects, collaborating with cross-functional teams, and ensuring the efficient processing and management of data.

Key Responsibilities:

• SQL skills to understand the transformation and perform analysis
• PySpark and MWAA DAGs skills to understand the data pipelines
• AWS Glue Data Catalog/ LF skills to understand the access control
• Lakehouse concepts and databases to understand consumption patterns Architectural ownership
• Cross functional collaboration
• Strategic vision
• Develop and maintain data pipelines using Python and Pyspark.
• Design and implement efficient SQL queries for data extraction and manipulation.
• Work with Aurora Postgres and Redshift for data storage and management.
• Utilize Informatica or other ETL tools for data integration and transformation.
• Implement and manage AWS cloud services, including Athena, S3, ECS/Docker, EMR, EC2, Lambda, CloudWatch, and EventBridge.
• Use Airflow or other orchestration tools for workflow management.
• Ensure data quality and integrity through rigorous testing and validation.
• Collaboxxxvith other teams to understand data requirements and deliver solutions.

Required Skills:

• Proficiency in Python and Pyspark.
• Strong SQL skills.
• Experience with Aurora Postgres and Redshift.
• Knowledge of Informatica or other ETL tools.
• Experience with Al.JS cloud services, including Athena, S3, Lambda, CloudWatch, and EventBridge.
• Ability to work with other cloud technologies (GCP, Azure) and understand AWS touch points.

Preferred Skills:

• ECS/ Docker (or any docker experience/concepts)
• EMR, EC2 (or knowledge of any other distributed computing)
• Familiarity with Airflow or other orchestration tools.

Job Tags

Similar Jobs