Senior Data Engineer - Unity Catalog

Full-time Developing,architecture,data,engineering,etl,pyspark,python,spark,sql

KPI Partners is seeking a skilled Senior Data Engineer with expertise in Databricks, Unity Catalogue, Pyspark, Python, and SQL. As a member of our dynamic team, you will be responsible for designing, developing, and maintaining scalable data pipelines and solutions that support our data-driven decision-making processes.

Key Responsibilities:

• Develop and maintain data pipelines using Databricks and Pyspark to ensure efficient data processing and transformation.
• Utilize Unity Catalogue to manage and govern data assets and ensure data quality and security.
• ollaborate with data analysts, data scientists, and other stakeholders to understand data requirements and deliver actionable insights.
• Write efficient and optimized queries using SQL to extract and manipulate data from various sources.
• Monitor and troubleshoot data pipeline performance, implement improvements, and provide support for data-related ...