Talent's Information
-
Location
Hyderabad, India
-
Rate
$14.0 per Hour
-
Experience
8 Year
-
Languages Known
English,Hindi
Available for
Background Verification
40 Hr/Week
About RajKumar
Tech Stack Expertise
-
AWS
AWS
1 Years
Work Experience
Kafka Engineer
- January 2015 - February 2023 - 8 Year
- India
Projects
Telefonica Fast_OSS
- February 2022 - February 2023 - 13 Months
Technologies
Role & Responsibility
-
Deploying Confluent Kafka and its components on Kubernetes
Deployed EKS clusters on AWS using terraform (Infrastructure as Code)
Deployed Confluent Platform on EKS k8s using helm charts.
Experience running Zookeeper less Kraft clusters in the development environment.
Exposing Kafka and other components outside of k8s using load balancers and ingress controllers.
Store the data beyond the pod lifecycle for stateful services like zookeeper and broker using statefulsets.
Responsibilities:
Using k8s native features like readiness and liveness probe to periodically check the health of the services.
Created DevOps pipeline for lifecycle management of Kafka topics, ACL’s, Avro Schema’s, Kafka connectors.
Filter, Aggregate, modify the data in real-time using KSQLDB and dump the data to Elasticsearch using Kafka connect framework.
Monitoring by Exposed JMX metrics from all the confluent platform components to Prometheus and visualize them in Grafana dashboards.
Alerting using Alert manager for service interruptions for all the confluent platform components
Experience deploying Elasticsearch, Kibana clusters on k8s using ELK stack operator.
Deployed Ngnix Ingress Controller and External DNS frameworks on EKS k8s clusters to expose all the services outside k8s and to create DNS records automatically based on K8S Ingress objects.
FreeNow Streaming Platform
- September 2019 - February 2022 - 30 Months
Technologies
Role & Responsibility
-
Design, Size and architect of Confluent Kafka platform and Client onboarding activities and Security features implementation.
Installing and configuring the Confluent Kafka on AWS EC2 instances and rest of the components such as schema registry, Kafka connect, KsqlDB and Control Center on k8s.
Deployment of Confluent platform with Security via Ansible playbook and Jenkins using Infrastructure as Code methodology.
Worked on GDPR Right to be forgotten project where we need to delete all the personal information stored in our data lake.
Implemented the project in Spark and Scala and ran the jobs on Databricks Platform and scheduled the jobs using airflow.
Written custom Databricks Airflow operator to submit the jobs to Databricks Platform from airflow.
Responsibilities:
DevOps pipeline for onboarding clients to Kafka platform including creating new use credentials, sending the credentials to the teams in an encrypted manner.
Written many automation scripts for lifecycle management of Kafka topics, acls, Schemas and Kafka connector and designed DevOps pipelines for the same.
Monitoring of Kafka platform using Prometheus and grafana in development environment and Confluent Control Center in production environment.
Used delta format for tables where we delete the PII information of customers using delta’s transactional capabilities
Scheduled other batch jobs on spark on k8s platform using google spark k8s operator.
Developed a Rest API to serve the customer data dump using Scala and Play framework using swagger API.
Written Multiple spark applications and airflow dags for scheduling based on stakeholder’s requests to access the data.
ITERGO Confluent Kafka Enterprise Platform
- May 2018 - August 2019 - 16 Months
Technologies
Role & Responsibility
Installing and configuring the Confluent Kafka enterprise platform and including services such as zookeeper, brokers, schema registry, rest proxy, Kafka connect, Ksql and Confluent Control Center on Virtual Machines
Involved in architectural decisions of the platform and Client onboarding activities and Security features implementation decisions.
Deployment of Confluent platform via Ansible playbook using methodology Infrastructure as Code.
Followed and implemented DevOps approach for the Deployment of clusters and other automation tasks by adopting GitHub, Jenkins, Ansible and Docker tools
Responsibilities:
Prepared the Ansible playbooks for auto deploying all the services and Implemented Jenkins pipeline for automation and operations.
Implementation of security features encryption and authentication with SSL and Authorization with ACLs in Kafka cluster
Configuring and running the schema registry and registering the several versions of Avro schemas and validating the incoming streams against the schemas.
Written the Avro schemas and for the business data and implemented the schema evolution concepts and validation.
Configuring and Running Kafka Connect with various connectors such as JMS and IBM MQ and JDBC integrating Kafka with external systems
Soft Skills
Industry Expertise
Education
in BE
Talangana University- June 2010 - June 2013
