Talent's Information
-
Location
Indore, India
-
Rate
$13.0 per Hour
-
Experience
7 Year
-
Languages Known
English,Hindi
Available for
Background Verification
40 Hr/Week
About Shailendra S
To be a key person in the professional and rewarding career in challenging environment where I can leverage my skills and acquire knowledge for the mutual development of organization, innovative and flexible. Experienced, result-oriented, resourceful and problem-solving Data engineer with leadership skills. Adapt and met challenges to tight release dates. Over 7+ years of diverse experience in Information Technology field, includes Development, and implementation of various applications in big data and Mainframe environments on cloud environments like AWS, Microsoft Azure.
Work Experience
Data Scintist
- January 2016 - June 2023 - 7 Year
- India
Projects
ETL And Data Warehousing Using NIFI
- February 2021 - May 2023 - 28 Months
Technologies
Role & Responsibility
ETL data pipeline for Data Quality, Data profiling and Data transformation using pyspark for data ingestion from multiple data sources like blob , s3 along with some data transformation and profiling rules. The processed data is then saved into the target layer which is Azure SQL.
Responsibilities:
Responsible for handling a team of 10-15 developers with multiple engagements of projects running in parallel along with ability to track daily project status on scrum agile approach
Involved in handling and solving technical problems of team by identifying roadblocks at hand at any stage of the project during POC, Dev, UAT or production and involving scaling up team resources so as to ensure smooth execution of project
Build Nifi pipeline optimization for large dataset ingestion (structured and unstructured) along with implementation of sequential job processing, Heap usage monitoring, database connection string auto-update with each job trigger
Building end to end big data ETL pipelines using Apache Nifi for creating data flows including Data Sourcing/ingestion for structured and unstructured data from AZURE BLOB storage ADLS Gen2.
Developing data flows for implementing data validation, data profiling along with data transformation rules integrated into Nifi flows invoked by front end once job is triggered.
Setup and implementation of pyspark script (for validation, profiling and transformation rules) using Apache spark 3.1 and integration of pyspark script via Nifi.
Implementation of Microservices API architecture using Django with seggregated architecture of Backend and Frontend for Master Data Management.
Microservice Integration
- June 2019 - January 2021 - 20 Months
Technologies
Role & Responsibility
-
It is a micro service based API. It is a developed to result a machine learning prediction from running python script as Lambda function and fetching data from bucket and final result of prediction is returned as a value from API endpoint
Responsibilities:
Handled the nested JSON structure of raw data and flattened it to consumable format to perform pyspark transformation.
Created workflow to automate the process of moving the incremental data from AWS S3 to redshift on an hourly basis using AWS Glue. Used job bookmarks and staging concept to populate incremental data in the existing table.
Using AWS Step function to orchestrate the workflow and scheduling it through AWS EventBridge.
Worked in agile methodology to define business goals into multiple sprints module and delivering each section of the requirement on sprint end basis.
BACKEND MDM Data
- May 2018 - May 2019 - 13 Months
Technologies
Role & Responsibility
-
Developing the backend of a web app using Django python and front end of the web app using Angular. The MDM pulls up data from multiple modules and then processes the same using python script. Our e-commerce web application offers a diverse range of products, competitive pricing, and personalized customer experience, making it a one-stop-shop for all your online shopping needs.
Responsibilities:
Responsible for creating a web framework in python using Django for Master Data Management (MDM) in python which would leverage the MDM data and will be displayed as web app.
Involved in requirement gathering, analysis, design, estimation with clients and testing of the assigned tasks in JIRA.
Responsible for reviewing python code for running the troubleshooting test-cases and bug issues.
Created database connection setup using pyodbc driver for mysql database module connector in Django.
Developed web applications in Django frameworks Model View Control (MVC) architecture.
Developed custom tags to simplify the template code. Designed UI screens using templates, Ajax, Html, and JSON and implemented javascript for client-side validations in forms and UI errors and success messages.
Involved in the analysis, design, and development and testing phases of software development life cycle (SDLC)
LAMBDA API ENDPOINT
- February 2016 - March 2018 - 26 Months
Technologies
Role & Responsibility
-
Creating a microservice in AWS using Lambda function for Machine learning prediction and displaying the end result in UI via javascipt event handler for a waste management use case of a particular housing segment.
Responsibilities:
Created a microservice API for a machine learning model that determines the prediction of recommended price and acceptance probability based on certain parameters of request input parameters.
Created a serverless yml template for cloud formation template for creating resource at runtime with predefined configuration of AWS parameters.
Implemented rest API on lambda function to determine recommended price based on certain given input feature of a location.
The response Json saved on s3 bucket with a unique transaction id along with environment details as a directory structure.
Monitoring and debugging of issues and events in cloudwatch logs.
Developed a UI using javascript for automating dataset upload to S3 bucket based on given input parameters from UI and also provided an option for fetching existing reports on dataset details for each lookup type.
Kafka Data Streaming
- January 2015 - February 2016 - 14 Months
Role & Responsibility
-
Real time kafka steaming and ingestion pipeline to stream and fetch data from different source as producer and provide it to end user as consumer.
Responsibilities:
Improved the performance and optimization of Hadoop program by 93% by using spark Context, Spark SQL spark Dataframes and PairRDD’s.
Developed a new method that moved data from multiple sources to HDFS via Apache/Confluent Kafka.
Created Azure Data pipelines for data ingestion, preparing and managing and serving data for visualization using PowerBi and several different machine learning use case
Implementation and ingestion along with setup of multiple data sources like Azure SQL, Kafka queue , Amazon S3 for real time data ingestion and processing the data using Azure Synapse and on the go data warehousing tool.
Soft Skills
Industry Expertise
Education
Computer Science in B.Tech
Indore University- June 2012 - June 2015
