Senior Consultant_ Azure Databricks

Full-time Computer Occupations

Job Description

Technical requirement:•Design and develop scalable data pipelines that ingest, transform, and deliver data from multiple source systems to downstream consumers.•Work with large and complex datasets using distributed data processing frameworks and cloud-based data platforms.•Translate business and technical requirements into reliable data solutions that are maintainable, reusable, and production-ready.•Build data transformation logic for cleansing, standardization, deduplication, enrichment, aggregation, and preparation of analytical datasets.•Apply sound data engineering principles around modular design, reusability, automation, testing, and operational reliability.•Handle common real-world data challenges such as late-arriving data, duplicate records, source inconsistencies, schema changes, and reprocessing requirements.•Improve performance and efficiency of data processing workloads by analyzing bottlenecks and applying appropriate optimization tec...