Location:
Pune, MH, IN, 411026
Senior Data Engineer
- Job Description: Senior Data Engineer (Palantir Foundry)
As a Senior Data Engineer at Lear, you will take a leadership role in designing, building, and maintaining robust data pipelines within the Foundry platform. Your expertise will drive the seamless integration of data and analytics, ensuring high-quality datasets and supporting critical decision-making processes. If you’re passionate about data engineering and have a track record of excellence, this role is for you!
- Manage Execution of Data-Focused Projects:
- As a senior member of the LEAR foundry team, support in designing, building and maintaining data-focused projects using Lear’s data analytics and application platforms.
- Participate in projects from conception to root cause analytics and solution deployment.
- Understand program and product delivery phases, contributing expert analysis across the lifecycle. Ensure Project deliverables are met as per agreed timeline.
- Tools and Technologies:
- Utilize key tools within Palantir Foundry, including:
- Pipeline Builder: Author data pipelines using a visual interface.
- Code Repositories: Manage code for data pipeline development.
- Data Lineage: Visualize end-to-end data flows.
- Leverage programmatic health checks to ensure pipeline durability.
- Work with both new and legacy technologies to integrate separate data feeds and transform them into new scalable datasets.
- Mentor junior data engineers on best practices.
- Utilize key tools within Palantir Foundry, including:
- Data Pipeline Architecture and Development:
- Lead the design and implementation of complex data pipelines.
- Collaborate with cross-functional teams to ensure scalability, reliability, and efficiency and utilize Git concepts for version control and collaborative development.
- Optimize data ingestion, transformation, and enrichment processes.
- Big Data, Dataset Creation and Maintenance:
- Utilize pipeline or code repository to transform big data into manageable datasets and produce high-quality datasets that meet the organization’s needs.
- Implement optimum build time to ensure effective utilization of resource.
- High-Quality Dataset Production:
- Produce and maintain datasets that meet organizational needs.
- Optimize the size and build scheduled of datasets to reflect the latest information.
- Implement data quality health checks and validation.
- Collaboration and Leadership:
- Work closely with data scientists, analysts, and operational teams.
- Provide technical guidance and foster a collaborative environment.
- Champion transparency and effective decision-making.
- Continuous Improvement:
- Stay abreast of industry trends and emerging technologies.
- Enhance pipeline performance, reliability, and maintainability.
- Contribute to the evolution of Foundry’s data engineering capabilities.
- Compliance and data security:
- Ensure documentation and procedures align with internal practices (ITPM) and Sarbanes Oxley requirements, continuously improving them.
- Team Development and Collaboration:
- Mentor junior team members and contribute to their growth.
- Foster collaboration within cross-functional teams.
- Share best practices and encourage knowledge sharing.
- Quality Assurance & Optimization:
- Optimize data pipelines and their impact on resource utilization of downstream processes.
- Continuously test and improve data pipeline performance and reliability.
- Optimize system performance for all deployed resources.
- Bachelor’s or master’s degree in Computer Science, Engineering, or a related field.
- Minimum 5 years of experience in data engineering, ETL, and data integration.
- Proficiency in Python and libraries like Pyspark, Pandas, Numpy.
- Strong understanding of Palantir Foundry and its capabilities.
- Familiarity with big data technologies (e.g., Hadoop, Spark, Kafka).
- Excellent problem-solving skills and attention to detail.
- Effective communication and leadership abilities.