Role Overview

Designs, builds, and oversees the deployment and operation of technology architecture, solutions, and software to capture, manage, store, and utilize structured and unstructured data from internal and external sources.
Establishes and builds processes and structures based on business and technical requirements to channel data from multiple inputs, route appropriately, and store using any combination of distributed (cloud) structures, local databases, and other applicable storage forms as required.
Develops technical tools and programming that leverage artificial intelligence, machine learning, and big-data techniques to cleanse, organize, and transform data and to maintain, defend, and update data structures and integrity on an automated basis.
Creates and establishes design standards and assurance processes for software, systems, and applications development to ensure compatibility and operability of data connections, flows, and storage requirements.
Reviews internal and external business and product requirements for data operations and activity and suggests changes and upgrades to systems and storage to accommodate ongoing needs.
May lead moderately complex projects from end to end.
Drives analyses from technical design through completion.
Understands and guides others on database/data lake sources and security requirements.
Ensures data is clean and in a usable and scalable format and helps others with data scrubbing and correlation methodologies.
Manages data validation and imputation techniques.
Analyzes complex datasets, unlocks patterns, and determines appropriate approaches to problem solving, reporting, and visualization.
Manages the development and scaling of data pipelines from multiple sources.
Develops and implements data architecture to store, process, analyze, and visualize large, high volume data sets.
Improves feature engineering to build innovative models to address a range of business problems.
Writes functional clean code and unit tests with no guidance.
Maintains and enhances solutions by fixing complicated errors, raising risks, and escalating issues where necessary.
Identifies immediate and larger scope problems through thorough code reviews.
Tests and reviews code to optimize scale, velocity, and reliability of deliverables.
Contributes to threat modeling of a feature.
Applies depth of knowledge of databases, relational databases, cloud services, and scripting languages.
Ensures data quality and creates and maintains an optimal data pipeline.
Assesses data compliance, fixes complex issues, and escalates multidimensional flags.
Monitors, tests, and improves metrics to optimize data pipeline quality and performance.
Partners with data scientist(s) and the business to understand, articulate, and resolve business problems and related security and privacy concerns.
Recommends optimal data sources, methodologies, analytics, and tools.
Designs and creates reporting and visualization for communicating data insights and links data to business outcomes.

Requirements

B.Tech, BE, ME, MTech, MCA or MS ( CS or IT or MIS ) or equivalent degrees
10+ years of professional experience designing and building software applications.
Understanding of Data Engineering, Master Data Management
Experience in RDBMS (Oracle), PL/SQL, Java and Python
Exposure to noSQL, Snowflake and big-data technologies
Exposure to Kafka, Pega and Solr
Knowledge of Agile and Waterfall development methodologies
Experience with infrastructure tools including Jira, Jenkins and Git.
Experience with automated testing.
Experience with CI/CD pipeline utilizing Jenkins automation a plus.
Good communication and organizational skills and can work successfully in a team environment.

Tech Stack

Cloud
Java
Jenkins
Kafka
NoSQL
Oracle
Python
RDBMS
SQL

Benefits

Flexible work arrangements
Professional development opportunities

Sr. Data Engineer

Key skills

About this role

Requirements: