Senior Platform Engineer (DataOps)
United States of America
Full Time
1 hour ago
H1B Sponsor Likely
Key skills
PythonC++CSparkApache SparkKafkaApache KafkaAirflowApache AirflowAWSAmazon Web ServicesAzureGCPGoogle CloudTerraformKubernetesDockerVaultArgoCDTeamCityKinesisPub/SubInfluxDBElasticsearchPrometheusDatadogSentrySplunkJaegerKibanaGitGitHubGitLabBitbucketGitOpsSource ControlRabbitMQPulsarAmazon KinesisAgileCI/CDLeadership
About this role
Black Canyon Consulting (BCC) is searching for a Senior Platform Engineer (DataOps) to support the National Center for Biotechnology Information (NCBI). The role involves developing and improving the DataOps platform, maintaining tools, and ensuring compliance with security regulations.
Responsibilities:
- Develops and continuously improves DataOps platform
- Develops and maintains common tools and libraries
- Evaluates new technologies and practices
- Helps NCBI developers with adoption of platform
- Ensures compliance with the Federal application security regulations and standards by providing automated solutions and compliance pipelines
- Embraces agile development and continuous improvement
- Encourages growth mindset and offers leadership opportunities at any level
Requirements:
- 7+ years of experience is the field
- Strong coding skills in at least one programing language are required (Python, C++, ...)
- Kubernetes, containerization
- Google Cloud Platform (GCP), Amazon Web Services (AWS), Microsoft Azure or equivalent cloud services
- Apache Kafka, Google Cloud Pub/Sub or equivalent
- Apache AirFlow or equivalent
- Experience with data processing applications and modern cloud-based data processing infrastructure
- Linux command-line skills
- B.S. in a STEM field (Engineering, Computer Science, Mathematics, Physics) or equivalent industry experience in Systems Engineering
- Google Anthos
- Docker
- GitOps tools: ArgoCD or equivalent
- Infrastructure as code tools: Terraform or equivalent
- GitLab, GitHub, Bitbucket, Teamcity, Artifactory, or equivalent products for management of Git source control, CI/CD pipelines and artifact lifecycle management
- Modern observability and logging tools: Prometheus, EFK (ElasticSearch, fluentd, Kibana), TIGK (Telegraph, InfluxDB, Graphana, Kapacitor)
- DataDog, Sensu, Jaeger, Sentry, OpsGenie, PagerDuty, Splunk, or equivalent
- Secret Management tools such as Hashicorp Vault, CyberArk, Azure Key Vault, Google Cloud Secret Manager or equivalent
- Data transfer tools: AWS DataSync, Aspera, MinIO, CloudSoda or equivalent
- Apache Pulsar, RabbitMQ, Amazon Kinesis, Apache Flume, Apache Storm, Apache Spark Streaming, Google Cloud Pub/Sub
- Experience with best-practice design patterns in coding and architecture
- Experience working in Agile environment