Built ETL Pipeling using Azure cloud and Databricks

This project demonstrated a concrete Lakehouse architecture where all Data Engineers, Data Analysts and Data scientists could collaborate, Used Unity catalog demonstrating Fine grain access controls moreover Data Lineage which serves as the overview on the flow of data

🚀ETL Pipeline while creating a Medallion Architecture (BRONZE, SILVER AND GOLD) in Databricks
🚀Unity Catalog for Fine grain access controls and Data Lineage
🚀Orchestration using Databricks Workflows
🚀CI/CD using Azure DevOps

Benefits
✅Fine grain access controls
✅Data Lineage
✅Data Discoverability
✅Ready to use dataThis project demonstrated a concrete Lakehouse architecture where all Data Engineers, Data Analysts and Data scientists could collaborate, Used Unity catalog demonstrating Fine grain access controls moreover Data Lineage which serves as the overview on the flow of data 🚀ETL Pipeline while creating a Medallion Architecture (BRONZE, SILVER AND GOLD) in Databricks 🚀Unity Catalog for Fine grain access controls and Data Lineage 🚀Orchestration using Databricks Workflows 🚀CI/CD using Azure DevOps Benefits ✅Fine grain access controls ✅Data Lineage ✅Data Discoverability ✅Ready to use data

• Skills: Data Engineering · Apache Spark · PySpark · SQL · Azure Databricks · Microsoft Azure · Extract, Transform, Load (ETL) · Project Management · DevOps