Senior Data Engineer

Job Description: Senior Data Engineer (Palantir Foundry)

As a Senior Data Engineer at Lear, you will take a leadership role in designing, building, and maintaining robust data pipelines within the Foundry platform. Your expertise will drive the seamless integration of data and analytics, ensuring high-quality datasets and supporting critical decision-making processes. If you’re passionate about data engineering and have a track record of excellence, this role is for you!

Responsibilities:

Manage Execution of Data-Focused Projects:
- As a senior member of the LEAR foundry team, support in designing, building and maintaining data-focused projects using Lear’s data analytics and application platforms.
- Participate in projects from conception to root cause analytics and solution deployment.
- Understand program and product delivery phases, contributing expert analysis across the lifecycle. Ensure Project deliverables are met as per agreed timeline.
Tools and Technologies:
- Utilize key tools within Palantir Foundry, including:
  - Pipeline Builder: Author data pipelines using a visual interface.
  - Code Repositories: Manage code for data pipeline development.
  - Data Lineage: Visualize end-to-end data flows.
- Leverage programmatic health checks to ensure pipeline durability.
- Work with both new and legacy technologies to integrate separate data feeds and transform them into new scalable datasets.
- Mentor junior data engineers on best practices.
Data Pipeline Architecture and Development:
- Lead the design and implementation of complex data pipelines.
- Collaborate with cross-functional teams to ensure scalability, reliability, and efficiency and utilize Git concepts for version control and collaborative development.
- Optimize data ingestion, transformation, and enrichment processes.
Big Data, Dataset Creation and Maintenance:
- Utilize pipeline or code repository to transform big data into manageable datasets and produce high-quality datasets that meet the organization’s needs.
- Implement optimum build time to ensure effective utilization of resource.
High-Quality Dataset Production:
- Produce and maintain datasets that meet organizational needs.
- Optimize the size and build scheduled of datasets to reflect the latest information.
- Implement data quality health checks and validation.
Collaboration and Leadership:
- Work closely with data scientists, analysts, and operational teams.
- Provide technical guidance and foster a collaborative environment.
- Champion transparency and effective decision-making.
Continuous Improvement:
- Stay abreast of industry trends and emerging technologies.
- Enhance pipeline performance, reliability, and maintainability.
- Contribute to the evolution of Foundry’s data engineering capabilities.
Compliance and data security:
- Ensure documentation and procedures align with internal practices (ITPM) and Sarbanes Oxley requirements, continuously improving them.
Team Development and Collaboration:
- Mentor junior team members and contribute to their growth.
- Foster collaboration within cross-functional teams.
- Share best practices and encourage knowledge sharing.
Quality Assurance & Optimization:
- Optimize data pipelines and their impact on resource utilization of downstream processes.
- Continuously test and improve data pipeline performance and reliability.
- Optimize system performance for all deployed resources.

Qualifications:

Bachelor’s or master’s degree in Computer Science, Engineering, or a related field.
Minimum 5 years of experience in data engineering, ETL, and data integration.
Proficiency in Python and libraries like Pyspark, Pandas, Numpy.
Strong understanding of Palantir Foundry and its capabilities.
Familiarity with big data technologies (e.g., Hadoop, Spark, Kafka).
Excellent problem-solving skills and attention to detail.
Effective communication and leadership abilities.