Databricks Platform Lead
Job Reference: 250702001• Job Title: Databricks Platform Lead• Type : Hybrid / Contract• Location: Coventry• Salary: £300-350/day (Outside IR35)• Experience : 5+ years About the RoleWe’re looking for an experienced and motivated Databricks Platform Engineer to provide critical support and optimization for Databricks environments on AWS. If you’re passionate about cloud infrastructure, big data, and automation, this role is an excellent opportunity to work on cutting-edge technology alongside dynamic engineering and DevOps teams.Key ResponsibilitiesPlatform Support• Provide daily operational support for Databricks on AWS• Monitor and troubleshoot clusters, jobs, workspaces, and other Databricks re-sources• Manage access controls and data security policies• Implement and maintain Unity Catalog and Medallion architecture• Handle incidents, requests, and environment changes.• Collaborate with cross-functional teams to resolve complex issues• Monitor and support Databricks ML flowsPerformance & Optimization• Analyze workloads and optimize cluster performance and cost-efficiency• Recommend best practices for Apache Spark job execution and data processing pipelinesAWS Infrastructure Management• Configure and manage AWS services including S3, EC2, IAM, Lambda, and VPC• Ensure robust security and compliance within AWS-Databricks integrationsAutomation & DevOps• Create and maintain automation scripts for monitoring and operational maintenance• Work with DevOps to automate infrastructure provisioning, scaling, and CI/CD pipelines using tools like Jenkins, Git, and TerraformQualifications• Bachelor’s degree in Computer Science, IT, or related field• 5+ years of experience supporting Databricks or similar big data platforms• Expertise in Databricks on AWS (cluster tuning, Spark execution, job management)• Strong working knowledge of AWS services (EC2, S3, IAM, Lambda, RDS, VPC)• Proficient in Python, Scala, or SQL for data engineering tasks• Deep understanding of Apache Spark, ETL pipelines, and data architectures• Familiar with DevOps practices and tools (Terraform, Ansible, Jenkins)• Experience with monitoring platforms such as CloudWatch or Datadog• Excellent analytical, problem-solving, and communication skills. Please provide your recent CV with the below detailsNotice Period –Residency Status in UK –Visa Expiry Date (if applicable) –Consent to share your data with the client – Yes/NoAgreed to work on the given rate – Yes/No