Azure Databricks is an a combination of Spark, Microsoft and Databricks, that presents a just-in-time analytics platform, which empowers data personnel to easily build and deploy advanced data analytic solutions. The use of Azure Databricks by small, medium and large enterprises is gaining traction and relevance in the world of big data for many reasons. Databricks is an Apache Spark-based analytics platform. Microsoft has optimized Databricks for Azure cloud services platform. This service is available by the name of Azure Databricks. Databricks is integrated with Azure to provide one-click setup, streamlined workflows, and an interactive workspace. Azure Databricks enables collaboration between data scientists, data engineers, and business analysts. Python Programming and Fundamental SQL & databases are the prerequisites of Azure Databricks training. This Azure Databricks course starts with the concepts of the big data ecosystem and Azure Databricks.

Course Objectives:

• Big data ecosystem, also with Azure Databricks.
• Internal details of Spark
• RDD
• Data frames
• Workspace
• Jobs
• Kafka
• Streaming and other data sources for Azure Data bricks

Course content

Pyspark Introduction

• Pyspark Introduction
• Pyspark Components and Features

Spark Architecture and Internals

• Apache Spark Internal architecture
• jobs stages and tasks
• Spark Cluster Architecture Explained

Spark RDD

• Different Ways to create RDD in Databricks
• Spark Lazy Evaluation Internals & Word Count Program
• RDD Transformations in Databricks & coalesce vs repartition
• RDD Transformation and Use Cases

Spark SQL

• Spark SQL Introduction
• Different ways to create DataFrames

Spark SQL Intenals

• Catalyst Optimizer and Spark SQL Execution Plan
• Deep dive on Sparksession vs sparkcontext
• Spark SQL Basics
• RDD Transformation and Use Cases

Spark SQL Basics

• Spark SQL Basics
• Joins in Spark SQL

Spark SQL Functions and UDFs

• Spark SQL Functions
• Spark SQL UDFs
• Spark SQL Temp tables and Joins

Introduction to Azure Databricks

• Introduction to Databricks
• Azure Databricks Architecture
• Azure Databricks Main Concepts

Azure Databricks Account Creation

• Azure Free Account
• Free Subscription for Azure Databricks
• Create Databricks Community Edition Account

Databricks Cluster Types and Notebook Options

• Creating and configuring clusters
• Create Notebook
• Quick tour on notebook options

Databricks Utilities and Notebook Parameters

• Dbutils commands on files, directories
• Notebooks and libraries
• Databricks Variables
• Widget Types
• Databricks notebook parameters

Databricks CLI

• Azure Databricks CLI Installation
• Databricks CLI – DBFS, Libraries and Jobs

Databricks Integration with Azure Blob Storage

• Read data from Blob Storage and Creating Blob mount point

Dataframes in Azure Databricks

• What is a Dataframe?
• Using the Common Dataframe Methods
• Display Function

DataFrames Columns in Azure Databricks

• Column Class
• Working with the Column Expressions

Dataframes Advanced Methods in Azure Databricks

• Perform time and data manipulation
• Using Aggregate functions

Platform Architecture, Data protection in the Azure Databricks

• Azure Databricks platform architecture
• Perform data protection
• Security Scope of Azure Key Vault and Databricks
• Secure Access with the Azure Authentication and IAM
• Explain Security

Building and Querying a Data Lake

• Open Source Delta Lake
• How Azure Databricks manages Delta Lake

Process the Streaming Data with the Azure Databricks structured streaming

• Azure Databricks structured streaming
• Performing the Stream Processing through the structured streaming
• Working with the Time Windows
• Process the data from the Event Hubs with the structured streaming

Databricks Integration with Azure Data Lake Storage Gen1

• Reading Files from data lake storage Gen1

Databricks Integration with Azure Data Lake Storage Gen2

• Reading files from Azure Data Lake Storage Gen2

Reading and Writing CSV files in Databricks

• Read CSV Files
• Read TSV Files and PIPE Seperated CSV Files
• Read CSV Files with multiple delimiter in spark 2 and spark 3
• Reading different position Multidelimiter CSV files

Reading and Writing Parquet files in Databricks

• Read Parquet files from Data Lake Storage Gen2
• Reading and Creating Partition files in Spark

Parsing Complex Json FilesL

• Reading and Writing JSON Files
• Reading, Transforming and Writing Complex JSON files

Reading and Writing ORC and Avro Files

• Reading and Writing ORC and Avro Files

Databricks Integration with Azure Synapse

• Reading and Writing Azure Synapse data from Azure Databricks

Databricks Integration with Amazon Redshift (Redshift)

• Read and Write data from Redshift using databricks

Databricks Integration with Snowflake

• Reading and Writing data from Snowflake

Databricks Integration with CosmosDB SQL API

• Reading and Writing data from Azure CosmosDB Account

Databricks Integration with Azure Data Factory

• Azure Data Factory Integration with Azure Databricks

Databricks Streaming

• Delta Streaming in Azure Databricks
• Data Ingestion with Auto Loader in Azure Databricks

Implementing the CI/CD with the Azure DevOps

• What is CI/CD
• Creating the CI/CD process with the Azure DevOps

To see the full course content Download now