User

Data Engineer

BMT Score
86
86%
  • Remote

Available for

About Saranya

A performance driven ambitious software developer with an extraordinary blend of and technical knowledge. Ability to communicate and motivate team members to enhance strategic goals and bottom line objectives. Creative problem solving and troubleshooting skills complemented by meticulous attention to details that will result in the success of an organization by developing new applications or improving existing ones.

Tech Stack Expertise

  • Tech Stack Expertise

    Go Lang

    Docker

    0 Years
  • Tech Stack Expertise

    Python

    Python,Spyder

    8 Years
  • Tech Stack Expertise

    AWS

    AWS

    0 Years
  • Tech Stack Expertise

    Azure

    Azure

    0 Years
  • Tech Stack Expertise

    Kotlin

    Spark

    4 Years

Work Experience

Images

Data Engineer

  • January 2017 - February 2023 - 6 Year
  • India

Projects

Images

ANAMAOLY DETECTION

  • June 2022 - February 2023 - 9 Months
Technologies
Role & Responsibility
    Anomaly Detection is used to detect the employees who does not follow the company rules and policies. The PySpark job runs on EMR cluster as Kubernets image to extract employee’s data from Snowflake using Snowflake Connector and the business logics are performed using Pandas and Spark-SQL. The transformed data is stored in Snowflake tables.
    RESPONSIBILITIES: 
    Developed PySpark and Python applications using Python (Pandas), Spark-SQL for data extraction, transformation, and aggregation from Snowflake Tables for Analysing & transforming the data to uncover insights into the customer usage patterns performing data transformations as per business requirement.
    Created DAG to schedule jobs in Airflow 
    Monitoring jobs and debugging and fixing errors, if there are any job failures.  
    Performing data cleansing and moving transformed data to Snowflake database for further process. 
     
...see less
Images

CREDIT SCORING

  • December 2021 - April 2022 - 5 Months
Technologies
Role & Responsibility

    Credit Scoring is used to analyse the customers transaction data from Fiserv, Equifax, Bestbuy and calculate a score. The scoring helps the bank to provide credit cards to customers who have a good transaction, and it helps to increase the credit card amount for the correct customers. The data from Fiserv, Equifax and Best buy will be ingested in Azure blob in Json format. Using Pyspark json format files are fetched from Azure and processed using Spark-SQL and the result data is stored in Hive

    RESPONSIBILITIES: 

    Developed PySpark applications using Spark-SQL for data extraction, transformation, and aggregation from multiple file formats for Analysing & transforming the data to uncover insights into the customer usage patterns performing data transformations as per business requirement.
    Involving on creating Table and then applied HiveQL on those tables for Data validation.
    Created shell scripts for automating daily tasks and for faster and parallel processing of queries.  
    Monitoring jobs and debugging and fixing errors, if there are any job failures. 
    Handled Hive Tables and loaded data from Hive for transformation using Spark-SQL. 
    Performing data cleansing and moving transformed data to SQL database for further process.
     
...see less
Images

BOT METRICS

  • July 2021 - December 2021 - 6 Months
Technologies
Role & Responsibility

    Bot Metrics is used to analyse the customers need and improve the performance of the BOT to answer almost all the questions asked by the customers and users. Pyspark is used to extract the json raw data and apply business logics to analyse the questions asked by the customers and to check whether it’s been answered by BOT or mapped to customer care which helps for the enhancement of BOT.


    RESPONSIBILITIES: 
    Design and develop ETL integration patterns using Python on Spark.
    Created Hive Tables for storing result data and extracted data from hive through spark.
    Created shell scripts for automating daily tasks and for faster and parallel processing of queries.  
    Scheduled jobs in Autosys.
    Monitoring jobs and debugging and fixing errors, if there are any job failures. 
    Handled Hive Tables and loaded data from Hive for transformation using Spark-SQL. 
    Performing data cleansing and moving transformed data to Hive tables for further process.

     
...see less
Images

ASSETINSIGHT

  • January 2017 - July 2021 - 55 Months
Technologies
Role & Responsibility
    Asset Insight is used to digitise assets using sensors on concrete pump, tower crane, forklift, excavator, tower crane, backhoe etc in DataBricks/HDInsight Cluster and some of the asset data is processed by Spark Streaming where the source is fetched from Kafka. Asset Insight uses predictive modelling to offer its clients a timely insight into the steps they need to take with respect to asset management in order to measure, monitor and predict the asset usage for construction. The data of each asset is fetched from the sensor and loaded into cloud using stream analytics. Using PySpark, source data from Azure Blob is extracted and Transformations are processed on it. After all processes are done output feeds are created and loaded into SQL database for developing predictive data models for business users as per requirement.
    RESPONSIBILITIES: 

    Developed PySpark applications using Spark-SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats for Analyzing& transforming the data to uncover insights into the customer usage patterns performing data transformations as per business requirement.
    Involving on creating Table and then applied HiveQL on those tables for Data validation.
    Created shell scripts for automating daily tasks and for faster and parallel processing of queries.  
    Monitoring jobs and debugging and fixing errors, if there are any job failures. 
    Handled Hive Tables and loaded data from Hive for transformation using Spark-SQL. 
    Performing data cleansing and moving transformed data to SQL database for further process.  
     
...see less

Industry Expertise

Education

Education

in BE

Tamil Nadu Unversity
  • June 2013 - June 2016

Our Suggestions