Azure Databricks Engineer (Dallas, TX)
- Azure Databricks Architecture & Design :
- Design, build, and optimize scalable, high-performance data pipelines and data lakes using Azure Databricks.
- Architect and implement end-to-end analytics solutions leveraging Databricks and other Azure services (e.g., Azure Data Lake, Azure SQL, Azure Blob Storage).
- Lead the design of cloud-based architectures using Azure Databricks for data processing, transformation, and reporting.
- Data Engineering & Integration :
- Design and implement ETL/ELT processes to ingest, process, and transform data across various sources (structured and unstructured).
- Collaborate with data scientists, analysts, and other stakeholders to understand business requirements and develop data solutions.
- Manage data integration workflows between Databricks and other platforms like Power BI, Azure SQL, Synapse, etc.
- Optimization & Performance Tuning :
- Identify performance bottlenecks in Databricks environments and optimize clusters, queries, and code.
- Continuously improve and scale data pipelines to accommodate growing data volumes and business needs.
- Collaboration & Leadership :
- Mentor and guide junior engineers and team members in best practices related to Azure Databricks and data engineering.
- Work closely with cross-functional teams (data science, analytics, business intelligence) to deliver integrated solutions.
- Provide technical leadership in cloud data architecture and data engineering.
- Monitoring & Security :
- Implement monitoring and alerting systems to ensure data pipelines and workflows are running smoothly.
- Ensure compliance with data security and governance standards across the data ecosystem.
- Continuous Improvement & Innovation :
- Stay up to date with the latest advancements in Azure Databricks, cloud technologies, and data engineering trends.
- Continuously evaluate and introduce new technologies and methodologies to enhance the data platform.
Experience :
- Minimum 10 years of hands-on experience in data engineering and cloud-based data platforms.
- At least 5 years of experience working with Azure Databricks, building data pipelines, and performing data engineering tasks.
- Strong experience with Azure services such as Azure Data Lake, Azure Synapse Analytics, Azure Blob Storage, Azure SQL Database, etc.
Technical Skills :
- Proficiency in Spark, PySpark, Scala, or SQL for large-scale data processing.
- Expertise in Databricks notebooks, clusters, job scheduling, and libraries.
- Deep understanding of cloud-native data engineering practices and architecture on Azure.
- Familiarity with data modeling, data lakes, and data warehouse concepts.
- Expertise in Azure services: Data Factory, Data Lake, Synapse Analytics, Event Hubs, Cosmos DB.
Data Pipeline and ETL/ELT Development :
- Experience building and optimizing ETL/ELT workflows on Databricks.
- Knowledge of data orchestration tools such as Azure Data Factory, Apache Airflow, or similar.
Big Data Technologies :
- Proficiency in handling large-scale distributed data processing with tools like Apache Spark.
- Experience with technologies such as Kafka, Delta Lake, and Databricks Runtime.
- Strong understanding of Delta Lake and Lakehouse architecture .
Programming Languages :
- Strong programming skills in Python, Scala, or Java.
- Experience with SQL-based querying and optimizations.
Cloud & DevOps :
- Strong understanding of Azure cloud services and architecture.
- Familiarity with DevOps principles, CI/CD pipelines, and automation tools. Terraform required.
- Knowledge of Git, Azure DevOps, or similar version control and deployment systems.
Collaboration & Leadership :
- Excellent communication skills to work with both technical and non-technical stakeholders.
- Proven track record of mentoring and leading teams of data engineers.
- Experience managing complex projects and working with cross-functional teams.
Certifications :
- Microsoft Certified: Azure Data Engineer Associate or equivalent certification.
- Databricks Certified Associate Developer for Apache Spark.
Education :
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
Recommended Jobs
Accounts Receivable Specialist
Job Summary: The Accounts Receivable Specialist manages and maintains all aspects of the accounts receivable function. This role ensures timely, accurate invoicing and collection, reconciliation o…
Training and Compliance Instructor - First Aid and Safety
Requisition Number: 214583 Job Description Cintas is seeking a Training and Compliance Instructor - First Aid and Safety. Responsibilities include facilitating training courses and instruction…
Software Engineer, Link
Who we are About the Organization Link is a digital wallet designed for effortless and secure online payments and digital transactions. With Link, consumers enjoy convenience and peace of m…
Physician Assistant-Certified
Provide patient services under the direct supervision and responsibility of the supervising physician, including obtaining medical and personal histories, performing physical exams, educating patient…
Data Architect
Data Architect - Remote Intertek, a leading provider of quality and safety solutions to many of the world's top-recognized brands and companies, is actively seeking a Data Architect to join our …
Accounting Clerk - DoubleTree Suites Austin
An Accounting Clerk is responsible for responding to guest inquiries, resolving guest billing disputes, performing month-end duties and preparing daily receivables in the hotel's continuing effort to…
Regional Finance Director
Talarico for Texas is hiring a Regional Finance Director to join James Talarico’s US Senate campaign. This is a full-time, paid position perfect for someone interested in starting their career in Dem…
Recruiting Sourcer
At Serenity Healthcare, we’re redefining what mental wellness looks and feels like. As a family-owned and people-first organization, we’re passionate about building teams that are not only highly ski…
Nanny
Must have a US visa.