Talent's Information
-
Location
Bengaluru, India
-
Rate
$14.0 per Hour
-
Experience
10.2 Year
-
Languages Known
English,Hindi
Available for
Background Verification
40 Hr/Week
About Piyush
Experienced Data Architect well-versed in defining requirements, planning solutions and implementing structures at the enterprise level. Analytical problem-solver with a detail-oriented and methodical approach. Prepared to offer 10+ years of related experience to a dynamic new position with room for advancement.
Tech Stack Expertise
-
Azure
Azure data factory,Azure Data Bricks,Azure Data Lake
1 Years -
Kotlin
Spark
1 Years -
AWS
AWS EC2,AWS S3,AWS Lambda
2 Years -
Python
Python
1 Years
Work Experience
Data Engineer
- January 2013 - February 2023 - 10 Year
- India
Projects
Pricing Modele
- August 2021 - February 2022 - 7 Months
Technologies
Role & Responsibility
-
HealthCare Industry Insurance
Developed ingestion pipeline using Azure Data Factory (Azure Data Lake, Blob Storage), Databricks, Logic Apps and Azure SQL Database Successfully implementation of end-to-end project with designing the technical solution. Worked on the optimization of the databricks script via redesigning the strategy, databricks recommendation and principles of delta lake End to end installation of databricks setup such as installation of mount points, tokens, libraries and key vaults Setting up the cluster configuration while analyzing the data volume and client’s requirement Migration of data from on-premise SQL Server to Azure environment via ADF, ADLS and Azure SQL Database Auto trigger the email notifications using Azure Logic Apps and Azure triggers Creation of SQL stored procedure and use the same in data factory stored procedure activity Direct interaction with client to get the requirement and provide the estimation of delivery date Managing the team of 4 members and responsible for distribution of work in team Used Key vaults to store the credentials
Impetus
- January 2021 - August 2021 - 8 Months
Technologies
Role & Responsibility
-
It’s a financial project
Worked on AWS and Azure cloud providers with Big Data Technologies like Hadoop, Spark, Hive, Azure Data Bricks, EMR, EC2 instances etc Stored data on data lake storage like Azure data lake storage Gen2 and S3 bucket for AWS Experienced on handling billions of records and optimizing the same using hive partitions and cube aggregation strategy Creation of python script to increase the cardinality and volume of records in billions using python random libraries Determine the cost estimations of EC2, EMR machines, Azure databricks, VMs on the basis of data size volume, disk storage for local storage replica and number of instances of EC2 machines to optimize the query performance Worked on Snowflake data-warehouse to create the tables and view on top of data present in AWS S3 bucket Migration of AWS S3 Bucket to Azure blob storage using Azcopy utility Copy data from onpremise SQL Server environment to Azure data factory using powershell template and ARM template Transfer of local files to AWS S3 bucket using AWS S3 cp command Transfer local files to Hadoop environment using filezilla Knowledge of CI/CD pipelines to automatically transfer the Kyvos cube components to GIT hub and deploy the git hub files to target environment using jenkins and web-hook Developed the customer analytics and sales performance dashboards using Visualization tools like power bi and tableau Demoed the Kyvos product and end to end workflow to client.
Nsight Mortgage
- August 2019 - January 2021 - 18 Months
Technologies
Role & Responsibility
Banking Industry (Mortgage and Deposits).
Create python scripts to download data from different web data sources, analyzing AWS EC2 clusters and integrating with python script, create postgresql objects, building tableau reports and dashboards , airflow for monitoring or scheduling the jobs and use of AWS services like S3, EMR, lambda creation to automate the process. Designed internal process improvements to automate repetitive tasks, shortening data delivery times. Generated customized architecture specifically optimized for several database technologies, including SQL, python and AWS services. Gathered, defined and refined requirements, led project design and oversaw implementation. Gathered POC results on big technology stack i.e. Hive, Scoop, Spark, Kafka etc Mentored and coached other team members about domain and requirements. Code deployment through serverless framework. Create end user report or dashboard result - 1. Market Pulse - to get mortgage price loan rate related information in market so that consumers can get information of APR and invest accordingly. 2. Competitive Share - information for price across region and compare APR and other rates to analyze which region is winning or losing.
Big 3C
- September 2017 - June 2019 - 22 Months
Technologies
Role & Responsibility
Project Description HealthCare Industry- (Claims, Contract, Collection)
Developed ingestion pipeline using Azure Data Factory (Azure Data Lake, Blob Storage), Azure Analysis Services and Azure Workbook for scheduling the cube jobs Developed SQL stored procedures and views in Azure SQL database Automated overall process through powershell scripts Dropped unused pipelines in ADF via powershell scripts Experienced to create Azure tabular cube with more than 50 facts and 20 dimensional tables Improved SSAS modelling via changing the architecture of the cube Optimized stored procedures by partitioning the tables and indexes Updated statistics for incremental load in a week Improved dimensional tables logic to get rid of dirty read of data Provided solutions with BI tools and POC on Visualization tools i.e. Tableau and Power BI. Marked charts and graphs to record results and prepared and delivered presentations, training and briefings required by contract. Designed and prepared technical reports, studies and related documentation. Collaborated with senior management and business line management to identify and prioritize new concepts for development and launch. Designed and developed end to end solutions and analytical data structures.
Hyperwallet I
- June 2016 - October 2017 - 17 Months
Technologies
Role & Responsibility
-
Its a ecommerve site used for selling products-Retail
Developed SSIS packages from flat file, oledb data sources Developed Type 2 SCD concept in SQL server. Maintained indexes and optimized stored procedures. Validated and Analyzed data in facts and dimensions tables. Developed views on top of the staging tables. Maintained design documentation and SQL standards. Researched on integration of R scripts in tableau. Worked on BIDS, SSDT and SQL Server 2016. Developed Power BI and Tableau reports with all optimization approach. Optimized workflow by introducing the merge queries in SQL server instead of SSIS package. Developed SSAS solutions using multidimensional model Created agent jobs for automation of SSIS packages and SSAS solution
Ameriprise
- December 2015 - November 2016 - 12 Months
Technologies
Role & Responsibility
-
Finance and Banking Product
Provided solutions on integration services and SQL Optimization techniques. Managed customer relations and customer service through daily communication and interaction. Evaluated customer information to explore issues, develop potential solutions and maintain highquality service. Worked closely with customers, internal staff and other stakeholders to determine planning, implementation and integration of system-oriented projects. Created flowcharts, diagrams and other documentation
Eureka
- December 2012 - December 2015 - 37 Months
Technologies
Role & Responsibility
-
Product for Lifescience
Provided end to end solutions using MSBI tools Worked closely with customers, internal staff and other stakeholders to determine planning, implementation and integration of system-oriented projects. Conducted research to test and analyze feasibility, design, operation and performance of equipment, components and systems. Wrote and maintained custom scripts to increase system efficiency and performance time. Changed warehousing optimizing dimensions and hierarchies and added aggregations to cube. Configured, installed and upgraded databases. Provided day-to-day support to system users, educating employees on troubleshooting and problem-solving protocols. Installed, configured, tested and maintained operating systems, application software and system management toolsdocumentation
Soft Skills
Industry Expertise
Education
in MCA-
Karnataka Univeristy- June 2012 - June 2015
