Home
Jobs
Saved
Resumes
Senior Data Engineer at Effectual | JobVerse
JobVerse
Home
Jobs
Recruiters
Companies
Pricing
Blog
Jobs
/
Senior Data Engineer
Effectual
Remote
Website
LinkedIn
Senior Data Engineer
United States
Full Time
2 hours ago
$150,000 - $180,000 USD
Visa Sponsor
Apply Now
Key skills
Apache
AWS
Cloud
ETL
Kafka
Microservices
Spark
Terraform
Data Engineering
Analytics
Apache Spark
Apache Kafka
CloudFormation
Lambda
Kinesis
Amazon Kinesis
CI/CD
Leadership
About this role
Role Overview
Design, build, and maintain scalable streaming data architectures using Kafka, MSK, and Kinesis
Develop real-time data pipelines that handle high-volume, high-velocity data streams
Implement event-driven architectures and microservices patterns for streaming data processing
Create and optimize data streaming topologies for complex event processing scenarios
Design fault-tolerant streaming systems with proper error handling and data recovery mechanisms
Configure, deploy, and manage Apache Kafka clusters and AWS MSK environments
Implement Kafka Connect pipelines for streaming data integration
Design optimal Kafka topic partitioning strategies and replication configurations
Monitor and optimize Kafka cluster performance, throughput, and latency
Implement Kafka security configurations including SSL/TLS, SASL, and ACLs
Manage Kafka Schema Registry for data serialization and evolution
Design and implement Amazon Kinesis Data Streams and Kinesis Data Firehose solutions
Configure Kinesis Analytics applications for real-time stream processing
Optimize Kinesis shard management and auto-scaling configurations
Implement Kinesis data retention and archival strategies
Integrate Kinesis with other AWS services for comprehensive streaming solutions
Develop real-time stream processing applications using Apache Spark Streaming, Kafka Streams, or AWS Lambda
Implement complex event processing (CEP) patterns for real-time analytics
Build streaming ETL pipelines that transform data in motion
Create real-time aggregations, windowing operations, and stateful stream processing
Optimize streaming query performance and resource utilization
Ensure seamless integration between streaming systems and data lakes, data warehouses, and operational databases
Implement data lineage and monitoring for streaming data pipelines
Create automated data quality checks and validation for streaming data
Manage data serialization formats (Avro, JSON, Protobuf) and schema evolution
Coordinate with data scientists and analysts to ensure streaming data meets analytical requirements
Implement Infrastructure as Code (IaC) for streaming data platforms using Terraform or CloudFormation
Automate deployment and management of streaming infrastructure through CI/CD pipelines
Monitor streaming system health, performance metrics, and alerting
Implement disaster recovery and high availability strategies for streaming systems
Stay current with emerging trends in streaming technologies and cloud-native solutions
Collaborate with data architects, data scientists, and application teams on streaming data requirements
Support rigorous project governance through daily progress reviews and time tracking
Provide technical leadership and mentorship to junior data engineers
Communicate complex streaming concepts to technical and non-technical stakeholders
Operate with transparency and responsiveness to support high-performing teams.
Requirements
7+ years of experience in the data engineering field with significant streaming data specialization
Bachelor's degree in Computer Science, Engineering, or related STEM field
Extensive hands-on experience with Apache Kafka including cluster management, performance tuning, and ecosystem tools
Proven experience with AWS MSK and Amazon Kinesis services in production environments
Strong background in real-time data processing and stream analytics.
Tech Stack
Apache
AWS
Cloud
ETL
Kafka
Microservices
Spark
Terraform
Benefits
Medical, dental, and vision health insurances,
Short term disability, long term disability and life insurances,
401k with Company match
Paid time off (PTO) (120 hours PTO that accrue over one year)
Paid time off for major holidays (14 days per year)
These and any other employee benefit offerings are subject to management’s discretion and may change at any time.
Apply Now
Home
Jobs
Saved
Resumes