Key skills
PythonScalaSQLNoSQLGenAIOpenAIAnthropicGeminiETLAnalyticsSnowflakeBigQuerySparkApache SparkKafkaAirflowApache AirflowTDDAWSAzureGCPIAMBigTablePub/SubCloud StorageMongoDB
About this role
Brooksource is a trusted IT and Engineering services provider that supports Fortune 500 organizations through various staffing solutions. They are seeking a Senior Data Engineer to develop, automate, and maintain ETL pipelines, design data models, and optimize data performance across cloud-based ecosystems.
Responsibilities:
- Develop, automate, and maintain batch and streaming ETL pipelines using Apache Airflow, Apache Spark, Python, and Scala
- Build and manage cloud-based data ecosystems on GCP (BigQuery, Bigtable, Dataproc, Pub/Sub, Cloud Storage, IAM, VPC)
- Design and optimize SQL and NoSQL data models for data lakes and warehouses (BigQuery, MongoDB, Snowflake)
- Write complex SQL queries for advanced data transformation, aggregation, and analytics optimization within BigQuery or equivalent platforms
- Apply modern Test-Driven Development (TDD) methodologies for big data pipelines, ensuring test automation across Airflow workflows, Spark jobs, and transformation logic
- Apply data mesh and data-as-a-product principles to enable reusable and domain-driven datasets
- Implement real time ingestion with Kafka Connect and process streaming data using Spark Streaming, Apache Flink, or similar technologies
- Optimize data performance, scalability, and cost efficiency across GCP components
- Ensure compliance with PCI and PII data with standards such as GDPR, PCI DSS, SOX, and CCPA
- Integrate GenAI tools such as OpenAI, Gemini, and Anthropic LLMs for intelligent data quality and analytics enhancement
- Collaborate with stakeholders, data scientists, and full stack engineers to deliver trusted, documented, and reusable data products
Requirements:
- 4+ years of experience in software development
- Understanding of application design patterns, event-driven architecture, database, schemas and testing strategies
- In-depth knowledge and experience with continuous integration, continuous deployment and test-driven development
- Bachelor's Degree or equivalent in MIS, Computer Science or related field
- Experience with large-scale application troubleshooting and performance tuning
- Exposure working with major cloud platforms (GCP, AWS, or Azure)
- Familiarity and experience with XP (Extreme Programming)