Platform Engineer (DataOps)
United States of America
Full Time
6 hours ago
$115,000 - $150,000 USD
H1B Sponsor Likely
Key skills
PythonC++KubernetesContainerizationGoogle Cloud PlatformAmazon Web ServicesMicrosoft AzureApache KafkaGoogle Cloud Pub/SubApache AirFlowData processing applicationsCloud-based data processing infrastructureLinux command-lineCSparkApache SparkKafkaAirflowApache AirflowAWSAzureGCPGoogle CloudTerraformDockerVaultArgoCDTeamCityKinesisPub/SubInfluxDBElasticsearchPrometheusDatadogSentrySplunkJaegerKibanaGitGitHubGitLabBitbucketGitOpsSource ControlRabbitMQPulsarAmazon KinesisAgileCI/CDLeadership
About this role
Black Canyon Consulting (BCC) is searching for a Platform Engineer (DataOps) to support their work for the National Center for Biotechnology Information (NCBI) at the National Library of Medicine. The role involves developing and continuously improving a DataOps platform, maintaining tools and libraries, and ensuring compliance with federal application security regulations.
Responsibilities:
- Develops and continuously improves DataOps platform
- Develops and maintains common tools and libraries
- Evaluates new technologies and practices
- Helps NCBI developers with adoption of platform
- Ensures compliance with the Federal application security regulations and standards by providing automated solutions and compliance pipelines
- Embraces agile development and continuous improvement
- Encourages growth mindset and offers leadership opportunities at any level
Requirements:
- Strong coding skills in at least one programing language are required (Python, C++, ...)
- Kubernetes, containerization
- Google Cloud Platform (GCP), Amazon Web Services (AWS), Microsoft Azure or equivalent cloud services
- Apache Kafka, Google Cloud Pub/Sub or equivalent
- Apache AirFlow or equivalent
- Experience with data processing applications and modern cloud-based data processing infrastructure
- Linux command-line skills
- B.S. in a STEM field (Engineering, Computer Science, Mathematics, Physics) or equivalent industry experience in Systems Engineering
- Google Anthos
- Docker
- GitOps tools: ArgoCD or equivalent
- Infrastructure as code tools: Terraform or equivalent
- GitLab, GitHub, Bitbucket, Teamcity, Artifactory, or equivalent products for management of Git source control, CI/CD pipelines and artifact lifecycle management
- Modern observability and logging tools: Prometheus, EFK (ElasticSearch, fluentd, Kibana), TIGK (Telegraph, InfluxDB, Graphana, Kapacitor)
- DataDog, Sensu, Jaeger, Sentry, OpsGenie, PagerDuty, Splunk, or equivalent
- Secret Management tools such as Hashicorp Vault, CyberArk, Azure Key Vault, Google Cloud Secret Manager or equivalent
- Data transfer tools: AWS DataSync, Aspera, MinIO, CloudSoda or equivalent
- Apache Pulsar, RabbitMQ, Amazon Kinesis, Apache Flume, Apache Storm, Apache Spark Streaming, Google Cloud Pub/Sub
- Experience with best-practice design patterns in coding and architecture
- Experience working in Agile environment