Senior Data Engineer

Location: 

Pune, MH, IN, 411026

Country/Region:  India
Job Function:  Engineering
Requisition ID:  34601
Employment Type:  Salary

Job Description: Senior Data Engineer (Palantir Foundry)

Overview:

As a Senior Data Engineer at Lear, you will take a leadership role in designing, building, and maintaining robust data pipelines within the Foundry platform. Your expertise will drive the seamless integration of data and analytics, ensuring high-quality datasets and supporting critical decision-making processes. If you’re passionate about data engineering and have a track record of excellence, this role is for you!

Responsibilities:

  1. Manage Execution of Data-Focused Projects:
    • As a senior member of the LEAR foundry team, support in designing, building and maintaining data-focused projects using Lear’s data analytics and application platforms.
    • Participate in projects from conception to root cause analytics and solution deployment.
    • Understand program and product delivery phases, contributing expert analysis across the lifecycle. Ensure Project deliverables are met as per agreed timeline.
  2. Tools and Technologies:
    • Utilize key tools within Palantir Foundry, including:
      • Pipeline Builder: Author data pipelines using a visual interface.
      • Code Repositories: Manage code for data pipeline development.
      • Data Lineage: Visualize end-to-end data flows.
    • Leverage programmatic health checks to ensure pipeline durability.
    • Work with both new and legacy technologies to integrate separate data feeds and transform them into new scalable datasets.
    • Mentor junior data engineers on best practices.
  3. Data Pipeline Architecture and Development:
    • Lead the design and implementation of complex data pipelines.
    • Collaborate with cross-functional teams to ensure scalability, reliability, and efficiency and utilize Git concepts for version control and collaborative development.
    • Optimize data ingestion, transformation, and enrichment processes.
  4. Big Data, Dataset Creation and Maintenance:
    • Utilize pipeline or code repository to transform big data into manageable datasets and produce high-quality datasets that meet the organization’s needs.
    • Implement optimum build time to ensure effective utilization of resource.
  5. High-Quality Dataset Production:
    • Produce and maintain datasets that meet organizational needs.
    • Optimize the size and build scheduled of datasets to reflect the latest information.
    • Implement data quality health checks and validation.
  6. Collaboration and Leadership:
    • Work closely with data scientists, analysts, and operational teams.
    • Provide technical guidance and foster a collaborative environment.
    • Champion transparency and effective decision-making.
  7. Continuous Improvement:
    • Stay abreast of industry trends and emerging technologies.
    • Enhance pipeline performance, reliability, and maintainability.
    • Contribute to the evolution of Foundry’s data engineering capabilities.
  8. Compliance and data security:
    • Ensure documentation and procedures align with internal practices (ITPM) and Sarbanes Oxley requirements, continuously improving them.
  9. Team Development and Collaboration:
    • Mentor junior team members and contribute to their growth.
    • Foster collaboration within cross-functional teams.
    • Share best practices and encourage knowledge sharing.
  10. Quality Assurance & Optimization:
    • Optimize data pipelines and their impact on resource utilization of downstream processes.
    • Continuously test and improve data pipeline performance and reliability.
    • Optimize system performance for all deployed resources.

Qualifications:

  • Bachelor’s or master’s degree in Computer Science, Engineering, or a related field.
  • Minimum 5 years of experience in data engineering, ETL, and data integration.
  • Proficiency in Python and libraries like Pyspark, Pandas, Numpy.
  • Strong understanding of Palantir Foundry and its capabilities.
  • Familiarity with big data technologies (e.g., Hadoop, Spark, Kafka).
  • Excellent problem-solving skills and attention to detail.
  • Effective communication and leadership abilities.

 

 

Location Code: