Databricks Data Engineer (Remote)

Indicium AI is a global AI-native consultancy trusted by leading enterprises to deliver AI into production at scale. The Databricks Data Engineer will be responsible for developing, implementing, and maintaining scalable data solutions, ensuring data integration, security, and reliability to support advanced analytics and strategic decision-making.

Responsibilities:

• Perform data ingestion/integration from various sources (relational and NoSQL databases, internal and external service APIs, files, and others) and ensure data quality and consistency
• Implement data storage solutions (Data Warehouses, Data Lakes) and optimize the performance of data queries and processing
• Managing data loading in distributed storage (whether relational or non-relational)
• Aggregate data using distributed tools that can handle large volumes of data
• Design, develop, and maintain robust and efficient Electronic Logistics (ELT) pipelines using data engineering best practices
• Ensure the entire ELT process is functioning correctly through monitoring and metrics such as SLAs
• Monitor the execution of ELT applications hosted in the cloud and on-premises
• To ensure data security and governance by implementing data access and quality policies
• To provision and/or maintain the data infrastructure, ensuring scalability, availability, and security
• Automate the provisioning and management of infrastructure using Infrastructure as Code (IaC) tools
• Disseminating DevOps and DataOps best practices within the team
• Collaborate with teams to deliver data solutions that meet business needs
• Research and implement new technologies and tools to improve the efficiency and scalability of data solutions
• Having the freedom and critical thinking skills to propose and question solutions related to data engineering
• Documenting processes, architectures, and data solutions

Requirements:

• At least 4 years of experience in at least one programming language (Python, Java, Ruby, JavaScript, Scala, etc.)
• Experience with version control systems such as GitHub, GitLab, Bitbucket, etc
• Advanced knowledge of SQL
• Advanced knowledge in DBT
• Experience or knowledge working with Databricks
• Knowledge of algorithms and data structures
• Have experience with technologies such as Spark, Kafka, Presto, and/or Airflow and feel confident creating aggregated datasets
• Experience with data warehouses such as Google BigQuery, Redshift, and/or Snowflake is required
• Have experience with Infrastructure as Code (IaC) tools, such as Terraform
• Have experience with cloud infrastructure (AWS, GCP, etc.)
• Have experience with cloud data processing (AWS, GCP, Azure, Snowflake, Databricks)
• Certification in public cloud platforms (AWS, Azure, Snowflake, Databricks, GPC) at the Associate level or equivalent
• Applicants must be U.S. citizens to be eligible for this role