Senior developer

Job Description

We are seeking a highly experienced Senior AI Developer with strong Databricks expertise to lead the design, architecture, and implementation of scalable AI and data solutions. This role goes beyond development and requires ownership of end-to-end solution architecture within the Databricks Lakehouse ecosystem, ensuring high performance, scalability, and enterprise readiness.

Key Responsibilities

Design and architect end-to-end AI/ML solutions on the Databricks Lakehouse Platform.

Develop and optimize scalable data pipelines using Apache Spark and Databricks.

Define and implement data architecture standards, including Delta Lake and Unity Catalog.

Build and manage end-to-end ML pipelines (data ingestion, training, deployment, monitoring).

Establish and implement MLOps frameworks using MLflow and CI/CD pipelines.

Collaborate with business stakeholders to translate requirements into scalable AI solutions.

Lead technical design discussions and provide architectural guidance across teams.

Ensure performance optimization, cost efficiency, and reliability of Databricks workloads.

Design batch and real-time processing systems (streaming, event-driven architectures).

Integrate advanced AI capabilities such as LLMs and Generative AI into enterprise solutions.

Enforce data governance, security, and compliance best practices.

Mentor junior developers and contribute to team capability building.

Pay Rate: 70hr

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [email protected] learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy:

Skills and Requirements

Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or related field.

7+ years of experience in AI/ML, data engineering, or distributed systems. 3+ years of hands-on experience with Databricks.

Strong expertise in Apache Spark (PySpark/Scala).

Advanced programming skills in Python (Scala is a plus).

Proven experience in solution design and architecture.

Deep understanding of data lakehouse architecture, ETL/ELT pipelines, and data modeling.

Experience with ML lifecycle management (MLflow, deployment, monitoring).

Experience with cloud platforms (Azure, AWS, or GCP).