Databricks / PySpark Data Engineer

We are seeking a hands-on Data Engineer with strong Databricks and PySpark experience to build scalable data pipelines and analytics applications within a modern data platform.

This role focuses on modernizing legacy ETL and reporting systems (Teradata, Informatica, Tableau) into Databricks-native pipelines, dashboards, and Python-based data applications. The ideal candidate operates across both data engineering and lightweight application development.

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark on Databricks


• Build analytics data models and transformation workflows for enterprise reporting and analytics


• Migrate legacy ETL workloads from platforms such as Informatica and Teradata to Databricks


• Develop Databricks-native dashboards and analytics applications to replace traditional BI tools


• Build lightweight Python-based data applications (e.g., FastAPI) to expose and interact with data


• Integrate Databricks pipelines with APIs and application services


• Implement Slowly Changing Dimensions (SCD) and dimensional data modeling techniques


• Develop reusable data engineering frameworks and standardized pipelines


• Optimize Spark workloads for performance, scalability, and cost efficiency


• Collaborate with analytics and business teams to deliver user-facing data solutions


• Leverage AI-assisted coding tools (e.g., Copilot, ChatGPT) to improve development productivity


• Contribute to best practices for modern data engineering and analytics application development

Required Experience

• 5+ years of hands-on experience building data pipelines using PySpark in production environments


• Strong experience with the Databricks platform (workspaces, clusters, Jobs & Workflows, Unity Catalog)


• Experience building analytics dashboards within Databricks (Databricks SQL)


• Proven experience designing and building scalable ETL/ELT data pipelines


• Strong Python development skills, including building REST APIs or data services


• Experience building or supporting data-driven applications (not just traditional ETL pipelines)


• Solid understanding of data modeling, including dimensional modeling and transformation patterns


• Experience using AI-assisted development tools (e.g., Copilot, ChatGPT) in engineering workflows


• Exposure to LLM integration or an AI-powered data application


• Familiarity with cloud platforms (AWS, Azure, or GCP)

Preferred Experience

• Experience migrating from legacy platforms: Informatica → Databricks / Spark; Teradata → cloud-native data platforms; Tableau (or similar) → Databricks-native dashboards


• Experience with FastAPI or similar Python frameworks for data applications


• Exposure to CI/CD pipelines for data engineering workflows


• Understanding of microservices architecture and scalable application design


• Experience in Healthcare Payor domain (e.g., claims processing, member data, provider data, eligibility, or billing systems)

McLaren Strategic Solutions is a leading-edge global technology consulting firm, addressing critical challenges across industries such as retail, financial services, and healthcare. Integrating a powerful ecosystem of platforms with capital-efficient execution, McLaren specializes in digital transformation to help businesses optimize operations, accelerate revenue, and achieve scalable outcomes. McLaren’s expertise spans the development of customer-centric applications, modernizing systems for cost-effectiveness and security, and leveraging cloud scalability for future-ready architectures. With a deep commitment to operational excellence, McLaren provides comprehensive managed services, including application maintenance, cybersecurity, platform solutions, and AI-optimized operations, ensuring seamless, secure, and efficient performance.

From supply chain automation to compliance and analytics, McLaren drives measurable impact: improving workforce productivity, reducing inventory costs, and cutting technology ownership expenses. With its emphasis on automation and zero business downtime, McLaren facilitates seamless migrations from legacy systems to modern platforms, enabling organizations to harness the full potential of digital transformation. Backed by strategic partnerships and a proven delivery model, McLaren empowers clients to innovate, modernize, and achieve lasting success in today’s digital economy.

McLaren is a certified minority owned business through the NMSDC and has a mission to place more people from non-traditional backgrounds into sustainable technology careers. Through partnerships with non-profit technology programs in underserved communities and Veteran organizations, candidates transition from tech training programs into real IT careers at McLaren. Our unique recruitment policy allows us to create exceptional teams, bringing a broad spectrum of experience to our company and creating anything but a traditional consulting firm.

Visit McLaren Strategic Solutions to learn more!

Doran Jones Inc. is proud to be part of the McLaren Strategic Ventures Group.