About Me

• Demonstrated working knowledge of Azure cloud components, such as Databricks, Data Lake, Blob Storage, Data Factory, Storage Explorer, SQL DB and SQL DWH.
• Developed ETL pipelines in Ab Initio, optimizing data extraction, transformation, and loading processes for large-scale data integration.
• Implemented data movement and transformation solutions using Apache NiFi, ensuring efficient and reliable data workflows.
• Designed and optimized SQL queries to extract and process financial and healthcare datasets, improving data accessibility and reporting accuracy.
• Led the migration of legacy ETL processes to Snowflake, reducing query execution times by 40% through performance tuning and indexing.
• Automated data ingestion and validation using Azure Data Factory (ADF) and Synapse Analytics, improving efficiency in financial data processing.
• Utilized Databricks on Azure to analyze data from Azure data storages, leveraging Spark cluster capabilities to derive valuable insights and worked with Databricks lake house architeure and used the features such as Unity Catalog, DLT, DB-SQL and DBR-Workflows and Delta tables .
• Performed migrating SQL Database to Azure Data Lake, Azure data lake Analytics, Azure SQL Database, Databricks and Azure SQL Data warehouse and Controlling, granting database access and migrating on premise databases to Azure Data Lake store using Azure Data factory (ADF) and in Databricks notebooks.
• Maintain RDMS database, ensuring space needs are addressed in a timely manner and designed, programmed, implemented and monitored the master client Index system using RDMS and MQ to maintain a common client index for the legacy SAVERR system and the TIERS system. Also Provided database support for RDMS.
• Involved in creating End-to-End data pipeline within distributed environment using the big data tools, Spark framework and Tableau for data visualization. 
• Install and configure Apache Airflow for S3 bucket and Snowflake data warehouse and create dags to run the Airflow 
• Managed ingestion workflows into Azure Services (Azure Data Lake, Azure Storage, Azure SQL) and facilitated data processing in Azure Databricks, ensuring compliance with regulations and security standards.
• Use of Salesforce emailing APIs to automate email services.
• Implemented Jenkins jobs to create Azure infrastructure from GitHub repositories containing Terraform code and created on-premises active directory authentication using automation with ansible play books.
• Developed Spark applications using Spark-SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
• Developed dashboards and visualizations to help business users analyze data as well as providing data insight to upper management with a focus on Microsoft products like SQL Server Reporting Services (SSRS) and Power BI.
• Created Spark clusters and configured high concurrency clusters using Azure Databricks to speed up the preparation of high-quality data.
• Developed Spark Streaming applications to process real-time data from various sources, such as Kafka and Azure Event Hubs.
• Built streaming ETL pipelines using Spark Streaming to extract data from various sources, transform it in real-time, and load it into a data warehouse like Azure Synapse Analytics.
• Worked extensively with Azure BLOB and Data Lake storage, efficiently loading data into Azure SQL Synapse analytics (DW) and Good Knowledge of Data Build Tool (DBT) with Snowflake.

• Migrated batch processing jobs from Teradata to Snowflake, leveraging Snowflake’s time travel and zero-copy cloning features.

• Utilized tools like Azure Databricks or HDInsight to scale out the Spark Streaming cluster as per requirements.
• Developed Spark APIs to import data into HDFS from Teradata and created Hive tables.
• Designed and optimized Teradata SQL queries, reducing query execution time by 40% using indexing and partitioning strategies.
• Developed and managed ETL pipelines for processing large insurance datasets, utilizing Teradata BTEQ, FastLoad, and MultiLoad.
• Migrated data from Teradata to Snowflake, leveraging Teradata Parallel Transporter (TPT) and Snowpipe for efficient data movement.
• Created partitioned and bucketed Hive tables in Parquet File Formats with Snappy compression and Loaded data into Parquet Hive tables from Avro Hive tables.