Senior Data Engineer – Databricks Lakehouse Platform

This role is for professionals seeking a long-term career with us; subcontractor applications will not be considered.

We are seeking an experienced Senior Data Engineer to design, build, and optimize modern

data platforms using the Databricks Lakehouse Platform . The ideal candidate will have 10+

years of data engineering experience

years of data engineering experience

and deep expertise in Lakehouse

Architecture , Medallion Architecture , cloud-native data solutions, and enterprise-scale data

governance practices.

Key Responsibilities

Data Platform Architecture & Engineering

Design and implement scalable, high-performance data platforms using the Databricks Lakehouse architecture.

Lead the development of enterprise data solutions leveraging Bronze, Silver, and Gold layers following Medallion Architecture principles.

Build and maintain batch and streaming data pipelines for large-scale data processing.

Develop reusable data frameworks, accelerators, and engineering best practices.

Optimize data storage, processing, and query performance across cloud environments.

Databricks Development

Design, develop, and maintain Databricks workflows using PySpark, SQL, and Python.

Implement and manage Delta Live Tables (DLT) pipelines.

Utilize Unity Catalog for centralized governance, metadata management, access controls, and lineage tracking.

Support advanced analytics, machine learning, and AI workloads on the Databricks platform.

Monitor and troubleshoot production data pipelines and platform performance.

Data Governance & Quality

Establish and enforce enterprise data governance standards and policies.

Implement data quality frameworks, validation rules, reconciliation processes, and monitoring solutions.

Ensure compliance with security, privacy, regulatory, and audit requirements.

Drive metadata management, lineage, cataloging, and data stewardship initiatives.

Collaborate with business and governance teams to improve trust and usability of enterprise data assets.

Design and deploy cloud-native data solutions on Azure, AWS, or GCP.

Implement scalable and secure cloud architectures aligned with enterprise standards.

Optimize cloud resource utilization, performance, and cost management.

Integrate cloud-native services with Databricks-based solutions.

DevOps & Automation

Automate deployment, testing, monitoring, and operational processes.

Apply Infrastructure as Code (IaC) practices using tools such as Terraform or similar.

Promote engineering excellence through code reviews, testing strategies, and release management processes.

Provide technical leadership and mentorship to junior and mid-level engineers.

Collaborate with architects, analysts, data scientists, product owners, and business stakeholders.

Contribute to platform roadmaps, architectural decisions, and technology evaluations.

Lead technical discussions and establish best practices across the data engineering organization.

Required Qualifications

Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or a related field.

10+ years of experience in Data Engineering, Data Warehousing, or Big Data platforms.

5+ years of hands-on experience with Databricks in enterprise environments.

Strong expertise in:

Lakehouse Architecture

Medallion Architecture

Unity Catalog

Data Governance and Data Quality frameworks

Data Modeling and Data Warehousing concepts

Extensive experience with at least one major cloud platform:

Amazon Web Services (AWS)

Advanced programming experience with:

Python

SQL

PySpark

Strong understanding of CI/CD methodologies and DevOps practices.

Experience implementing Infrastructure as Code (Terraform).

Expertise in performance tuning, optimization, and troubleshooting of large-scale data pipelines.

Strong understanding of security, access controls, and compliance requirements in cloud environments.

Preferred Qualifications

Databricks certifications such as 'Databricks Certified Data Engineer Professional'

Cloud certifications (Azure, AWS, or GCP).

Experience with real-time streaming technologies and event-driven architectures.

Experience supporting AI/ML and Generative AI data platforms.

Familiarity with data observability tools and modern data quality frameworks.

Experience implementing enterprise metadata management and data lineage solutions.

Hiring Process

We aim to complete the interview process in October-November. The targeted onboarding is January 2027 or earlier.

#J-18808-Ljbffr