Data Engineer 2
Position: Data Engineer 2
Location: Austin, TX 78751 (Remote within TX)
Duration: 9 months Contract with possibility to extension
Seeking Data Engineer 2 who will be responsible for developing, maintaining, and optimizing big data solutions using the Databricks Unified Analytics Platform. This role supports data engineering, machine learning, and analytics initiatives within this organization that relies on large-scale data processing.
Duties include:
- Designing and developing scalable data pipelines
- Implementing ETL/ELT workflows
- Optimizing Spark jobs
- Integrating with Azure Data Factory
- Automating deployments
- Collaborating with cross-functional teams
- Ensuring data quality, governance, and security.
Candidate Skills and Qualifications:
Years | Required/Preferred | Experience |
4 | Required | Implement ETL/ELT workflows for both structured and unstructured data |
4 | Required | Automate deployments using CI/CD tools |
4 | Required | Collaborate with cross-functional teams including data scientists, analysts, and stakeholders |
4 | Required | Design and maintain data models, schemas, and database structures to support analytical and operational use cases |
4 | Required | Evaluate and implement appropriate data storage solutions, including data lakes (Azure Data Lake Storage) and data warehouses |
4 | Required | Implement data validation and quality checks to ensure accuracy and consistency |
4 | Required | Contribute to data governance initiatives, including metadata management, data lineage, and data cataloging |
4 | Required | Implement data security measures, including encryption, access controls, and auditing; ensure compliance with regulations and best practices |
4 | Required | Proficiency in Python and R programming languages |
4 | Required | Strong SQL querying and data manipulation skills |
4 | Required | Experience with Azure cloud platform |
4 | Required | Experience with DevOps, CI/CD pipelines, and version control systems |
4 | Required | Working in agile, multicultural environments |
4 | Required | Strong troubleshooting and debugging capabilities |
3 | Required | Design and develop scalable data pipelines using Apache Spark on Databricks |
3 | Required | Optimize Spark jobs for performance and cost-efficiency |
3 | Required | Integrate Databricks solutions with cloud services (Azure Data Factory) |
3 | Required | Ensure data quality, governance, and security using Unity Catalog or Delta Lake |
3 | Required | Deep understanding of Apache Spark architecture, RDDs, DataFrames, and Spark SQL |
3 | Required | Hands-on experience with Databricks notebooks, clusters, jobs, and Delta Lake |
1 | Preferred | Knowledge of ML libraries (MLflow, Scikit-learn, TensorFlow) |
1 | Preferred | Databricks Certified Associate Developer for Apache Spark |
1 | Preferred | Azure Data Engineer Associate |
---
Recommended Jobs
Full Time Radiology Job Plano, TX
This opportunity is in Plano, located 19 miles north of downtown Dallas, It is the 9th-largest city in Texas and 4th-largest city in the Dallas/Fort Worth region. Plano is home to many leading global…
General Manager
Fuzzy's Taco Shop is hiring a Highly Skilled General Manager Are you a dynamic leader who thrives in a fast-paced environment? Do you enjoy working with a fun and loving team? Fuzzy's Taco Shop is…
Line cook experience
Job Title : Line Cook Location: This position is based in Bedford, NY. Schedule : Full-time or Part-time; Must be available for weekends and evenings About Us: Mast Market is a neighb…
Full Time ObGyn Job Houston, TX
Come live and practice in Houston, TX. It is known for attractions that include a world-famous historic park, professional sports, and the largest rodeo in the country. You will enjoy restaurants of e…
Senior Backend Engineer
Senior Backend Engineer Minimum Years of Experience Required: 6+ Years Salary range: $140,000 - $180,000 Eligibility to Work: To be eligible for this role, applicants should be a Permanen…
Bilingual Software Developer
Client (type/ industry): Software Engineering Company Working Location: San Jose, CA or Dallas, TX ~ Employment Type: Full-time (40hours/week) ~ Language: Japanese and English ~ Salary: …
Surgery Scheduling Coordinator - Orthopedics (Sugar Land)
At Houston Methodist, the Surgery Scheduling Coordinator position is responsible for coordinating the scheduling and related processes for inpatient and outpatient surgical procedures. This position…
Credentialing Customer Success Manager
About Plutus Health Inc.: Plutus Health Inc. is a leading provider of Revenue Cycle Management (RCM) services, certified in SOC2 compliance and recognized among the Inc. 5000 fastest-growing priva…
Staff Site Reliability Engineer - Big data Platform and Cloud Engineering
Company Description Visa is a world leader in payments and technology, with over 259 billion payments transactions flowing safely between consumers, merchants, financial institutions, and govern…
Part-Time Dishwasher
D iscover your Purpose! The Conservatory at Champion Forest , a Discovery Senior Living community is growing, thriving and has meaningful job opportunities available in a safe and supportive work …