Big Data Engineer Master’s Program

Big Data Engineer Master’s Program makes you proficient in tools and systems used by Big Data experts. Big Data Master’s Program to professionals who seek to dependant on their knowledge in the field of Big Data. This Big Data Engineer Course includes training on Hadoop and Spark stack, Cassandra, Talend and Apache Kafka messaging system. DASVM’s comprehensive Big Data Engineer training course is designed by Industry top experts with real time project experience. It makes you proficient in tools and systems used by Big Data experts. It includes training on Hadoop and Spark, Java Essentials, and SQL. The program is customized based on current industry standards that comprise of major sub-modules as a part of the training process. This program is designed by the industry experts to provide hands-on training with tools that are used to speed up the training process.

img
request

Can’t find a batch you were looking for?

 

Big Data Engineer Master’s Program makes you proficient in tools and systems used by Big Data experts. Big Data Master’s Program to professionals who seek to dependant on their knowledge in the field of Big Data. This Big Data Engineer Course  includes training on Hadoop and Spark stack, Cassandra, Talend and Apache Kafka messaging system. DASVM's comprehensive Big Data Engineer training course is designed by Industry top experts with real time project experience. It makes you proficient in tools and systems used by Big Data experts. It includes training on Hadoop and Spark, Java Essentials, and SQL. The program is customized based on current industry standards that comprise of major sub-modules as a part of the training process. This program is designed by the industry experts to provide hands-on training with tools that are used to speed up the training process.

 
Learning Path:
  1. Java Essentials
  2. Big Data Hadoop
  3. Apache Cassandra
  4. Talend for Data Integration and Big Data
  5. Apache Spark and Scala
  6. Apache Kafka
 

Course content

 

Java Essentials

 

Introduction to Java
  • Introduction to Java
  • Class Files
  • Data types and Operations
  • Loops – for, while & do while
  • Bytecode Compilation Process If Conditions
Data Handling and Functions
  • Arrays – Single Dimensional and Multidimensional arrays
  • Functions with Arguments
  • Concept of Static Polymorphism
  • String buffer Classes
  • Functions
  • Function Overloading
  • If Conditions
  • String Handling -String
Object Oriented Programming in Java
  • Concept of Object Orientation
  • Classes and Objects
  • Methods and Constructors:
  • Default Constructors Inheritance
  • Final and Static
  • Attributes and Methods
  • Constructors with Arguments
  • Abstract
Data Handling and Functions
  • Packages and Interfaces
  • Package
  • Multi-threading
  • Access Specifiers
  • Exception Handling
Collections
  • Wrapper Classes and Inner Classes: Integer, Character, Boolean, Float etc.
  • Applet Programs: How to write UI programs with Applet, Java.lang, Java.io, Java.util.
  • Collections: ArrayList, Vector, HashSet, TreeSet, HashMap, HashTable.

 

 

Big Data & Hadoop

 

 

Understanding Big Data and Hadoop
  • Introduction to Big Data & Big Data Challenges
  • Limitations & Solutions of Big Data Architecture
  • Data types and Operations
  • Hadoop Processing: MapReduce Framework
  • Hadoop Storage: HDFS (Hadoop Distributed File System)
  • Hadoop & its Features
  • Hadoop Ecosystem
  • Hadoop 2.x Core Components
  • Different Hadoop Distributions
Hadoop Architecture and HDFS
  • Hadoop 2.x Cluster Architecture
  • Federation and High Availability Architecture
  • Typical Production Hadoop Cluster
  • Common Hadoop Shell Commands
  • Hadoop Cluster Modes
  • Hadoop 2.x Configuration Files
  • Single Node Cluster & Multi-Node Cluster set up
  • Basic Hadoop Administration
Hadoop MapReduce Framework
  • Traditional way vs MapReduce way
  • YARN Components
  • YARN Architecture
  • YARN Workflow
  • YARN MapReduce Application Execution Flow
  • Anatomy of MapReduce Program
  • Why MapReduce
  • MapReduce : Combiner & Partitioner
  • Input Splits, Relation between Input Splits and HDFS Blocks
  • Demo of Health Care Dataset
  • Demo of Weather Dataset
Advanced Hadoop MapReduce
  • Counters
  • MRunit
  • Custom Input Format
  • XML file Parsing using MapReduce
  • Distributed Cache
  • Reduce Join
  • Sequence Input Format
Apache Pig
  • Introduction to Apache Pig
  • Pig Components & Pig Execution
  • Pig Data Types & Data Models in Pig
  • Pig UDF & Pig Streaming
  • Aviation use-case in PIG
  • MapReduce vs Pig
  • Pig Latin Programs
  • Shell and Utility Commands
  • Testing Pig scripts with Punit
  • Pig Demo of Healthcare Dataset
Apache Hive
  • Introduction to Apache Hive
  • Hive Architecture and Components
  • Limitations of Hive
  • Comparison with Traditional Database
  • Hive Tables (Managed Tables & External Tables)
  • Importing Data
  • Querying Data & Managing Outputs
  • Hive Demo on Healthcare Dataset
  • Hive vs Pig
  • Hive Metastore
  • Hive Partition
  • Hive Data Types and Data Models
  • Hive Bucketing
  • Hive Script & Hive UDF
  • Retail use case in Hive
Advanced Apache Hive and HBase
  • Hive QL: Joining Tables, Dynamic Partitioning
  • Hive Indexes and views
  • Hive Thrift Server
  • HBase v/s RDBMS
  • HBase Architecture
  • HBase Configuration
  • Apache HBase: Introduction to NoSQL Databases and HBase
  • Custom MapReduce Scripts
  • Hive Query Optimizers
  • Hive UDF
  • HBase Components
  • HBase Run Modes
  • HBase Cluster Deployment
Advanced Apache HBase
  • HBase Data Model
  • HBase Client API
  • Apache Zookeeper Introduction
  • Zookeeper Service
  • Getting and Inserting Data
  • Hive Data Loading Techniques
  • ZooKeeper Data Model
  • HBase Bulk Loading
  • HBase Filters
Processing Distributed Data with Apache Spark
  • What is Spark
  • Spark Components
  • Why Scala
  • Spark RDD
  • Spark Ecosystem
  • What is Scala
  • SparkContext
Oozie and Hadoop Project
  • Oozie
  • Oozie Workflow
  • Scheduling Jobs with Oozie Scheduler
  • Oozie Commands
  • Oozie for MapReduce
  • Combining flow of MapReduce Jobs
  • Hadoop Talend Integration
  • Oozie Components
  • Demo of Oozie Workflow
  • Oozie Coordinator
  • Oozie Web Console
  • Hive in Oozie

 

 

Apache Cassandra

 

 

Introduction to Big Data and Cassandra
  • Introduction to Big Data and Problems caused by it
  • 5V – Volume, Variety, Velocity, Veracity and Value
  • Traditional Database Management System
  • Limitations of RDMS
  • NOSQL databases
  • Common characteristics of NoSQL databases
  • CAP theorem
  • How Cassandra solves the Limitations?
  • History of Cassandra
  • Features of Cassandra
Cassandra Data Model
  • Introduction to Database Model
  • Understand the analogy between RDBMS and Cassandra Data Model
  • Understand following Database Elements: Cluster, Keyspace, Column Family/Table, Cloumn Column Family Options
  • Columns
  • Wide Rows, Skinny Rows
  • Static and dynamic tables
Cassandra Architecture
  • Cassandra as a Distributed Database
  • Replication Factor
  • Data Replication in Cassandra
  • Gossip protocol – Detecting failures
  • Staged Event-Driven Architecture (SEDA)
  • Managers and Services
  • Consistency level
  • Repair
  • Key Cassandra Elements: Memtable, Commit log, SSTables
  • Gossip: Uses
  • Snitch: Uses Data Distribution
  • Virtual Nodes: Write path and Read path
  • Incremental repair
Deep Dive into Cassandra Database
  • Replication Factor
  • Defining columns and data types
  • Recognizing a partition key
  • Specifying a descending clustering order
  • Deleting data
  • Updating a TTL
  • Replication Strategy
  • Defining a partition key
  • Updating data
  • Tombstones
  • Using TTL
Node Operations in a Cluster
  • Cassandra nodes
  • Bootstrapping a node
  • Adding a node (Commissioning) in Cluste
  • Removing (Decommissioning) a node
  • What’s new in incremental repair
  • Cassandra and Spark Implementation
  • Specifying seed nodes
  • Defining a partition key
  • Removing a dead node
  • Repair
  • Read Repair
  • Run a Repair Operation
Managing and Monitoring the Cluster
  • Cassandra monitoring tools
  • Tailing
  • Using JConsole
  • Runtime Analysis Tools
  • Logging
  • Using Nodetool Utility
  • Learning about OpsCenter
Backup & Restore and Performance Tuning
  • Creating a Snapshot
  • Restoring from a Snapshot
  • RAM and CPU recommendations
  • Hardware choices
  • Cluster connectivity, security and the factors that affect distributed system performance
  • End-to-end performance tuning of Cassandra clusters against very large data sets
  • Selecting storage
  • Types of storage to Avoid
  • Load balance and streams
Hosting Cassandra Database on Cloud
  • Security
  • Ongoing Support of Cassandra Operational Data
  • Hosting a Cassandra Database on Cloud

 

 

Talend for Data Integration & Big data

 

 

Talend – A Revolution in Big Data
  • Working with ETL
  • Rise of Big Data
  • Role of Open Source ETL Technologies in Big Data
  • Comparison with other market leader tools in ETL domain
  • Importance of Talend (Why Talend)
  • Talend and its Products
  • Introduction of Talend Open Studio
  • TOS for Data Integration
  • GUI of TOS with Demo
Working with Talend Open Studio for DI
  • Launching Talend Studio
  • Working with projects
  • Working with different workspace directories
  • Creating and executing jobs
  • Connection types and triggers
  • Most frequently used Talend components [tJava, tLogRow, tMap]
  • Read & Write Various Types of Source/Target Systems
  • Working with files [CSV, XLS, XML, Positional]
  • Working with databases [MySQL DB]
  • Metadata management
Basic Transformations in Talend
  • Context Variables
  • tJoin
  • tSortRow
  • tReplicate
  • Lookup
  • Using Talend components
  • tFilter
  • tAggregateRow
  • tSplit
  • tRowGenerator
  • Accessing job level/ component level information within the job
  • SubJob (using tRunJob, tPreJob, tPostJob)
Advance Transformations & Executing Jobs remotely
  • Various components of file management (like tFileList, tFileAchive, tFileTouch, tFileDelete)
  • Error Handling [tWarn, tDie]
  • Type Casting (convert datatypes among source-target platforms)
  • Looping components (like tLoop, tForeach)
  • Using FTP components (like tFTPFileList, tFTPFileExists, tFTPGet, tFTPPut)
  • Exporting and Importing Talend jobs
  • How to schedule and run Talend DI jobs externally (using Command line)
  • Parameterizing a Talend job from command line
Big Data and Hadoop with Talend
  • Big Data and Hadoop
  • HDFS and MapReduce
  • Benefits of using Talend with Big Data
  • Integration of Talend with Big Data
  • HDFS commands Vs Talend HDFS utility
  • Big Data setup using Hortonworks Sandbox in your personal computer
  • Explaining the TOS for Big Data Environment
Hive in Talend
  • Hive and It’s Architecture
  • Connecting to Hive Shell
  • Set connection to Hive database using Talend
  • Create Hive Managed and external tables through Talend
  • Load and Process Hive data using Talend
  • Transform data from Hive using Talend
Pig and Kafka in Talend
  • Pig Environment in Talend
  • Pig Data Connectors
  • Integrate Personalized Pig Code into a Talend job
  • Apache Kafka
  • Kafka Components in TOS for Big data

 

 

Apache Spark & Scala

 

 

Introduction to Big Data Hadoop and Spark
  • What is Big Data?
  • Limitations and Solutions of Existing Data Analytics Architecture with Uber Use Case
  • What is Hadoop?
  • Big Data Customer Scenarios
  • How Hadoop Solves the Big Data Problem?
  • Hadoop’s Key Characteristics
  • Hadoop Ecosystem and HDFS
  • Rack Awareness and Block Replication
  • Hadoop Cluster and its Architecture
  • Big Data Analytics with Batch & Real-time Processing
  • What is Spark? Spark at Yahoo!
  • Hadoop Core Components
  • YARN and its Advantage
  • Hadoop: Different Cluster Modes
  • Why Spark is needed?
  • How Spark differs from other frameworks?
Introduction to Scala and Apache Spark
  • What is Scala?
  • Scala in other Frameworks
  • Basic Scala Operations Control Structures in Scala
  • Collections in Scala- Array
  • Why Scala for Spark?
  • Introduction to Scala REPL
  • Variable Types in Scala
  • Foreach loop, Functions and Procedures
  • ArrayBuffer, Map, Tuples, Lists, and more
Functional Programming and OOPs Concepts in Scala
  • Functional Programming
  • Anonymous Functions
  • Getters and Setters
  • Properties with only Getters
  • Singletons
  • Overriding Methods
  • Higher Order Functions
  • Class in Scala
  • Custom Getters and Setters
  • Auxiliary Constructor and Primary Constructor
  • Extending a Class
  • Traits as Interfaces and Layered Traits
Deep Dive into Apache Spark Framework
  • Spark’s Place in Hadoop Ecosystem
  • Spark Deployment Modes
  • Writing your first Spark Job Using SBT
  • Spark Web UI
  • Spark Components & its Architecture
  • Introduction to Spark Shell
  • Submitting Spark Job
  • Data Ingestion using Sqoop
Playing with Spark RDDs
  • Challenges in Existing Computing Methods
  • Probable Solution & How RDD Solves the Problem
  • What is RDD, It’s Functions, Transformations & Actions?
  • Data Loading and Saving Through RDDs
  • Key-Value Pair RDDs
  • Other Pair RDDs o RDD Lineage
  • RDD Lineage
  • RDD Persistence
  • WordCount Program Using RDD Concepts
  • RDD Partitioning & How It Helps Achieve Parallelization
  • Passing Functions to Spark
DataFrames and Spark SQL
  • Need for Spark SQL
  • Spark SQL Architecture
  • User Defined Functions
  • Interoperating with RDDs
  • What is Spark SQL?
  • SQL Context in Spark SQL
  • Data Frames & Datasets
  • JSON and Parquet File Formats
  • Spark – Hive Integration
  • Loading Data through Different Sources
Machine Learning using Spark MLlib
  • Why Machine Learning?
  • What is Machine Learning?
  • Where Machine Learning is Used?
  • Face Detection: USE CASE
  • Different Types of Machine Learning Techniques
  • Introduction to MLlib
  • Features of MLlib and MLlib Tools
  • Various ML algorithms supported by MLlib
Deep Dive into Spark MLlib
  • Supervised Learning – Linear Regression, Logistic Regression, DecisionmTree, Random Forest
  • Unsupervised Learning – K-Means Clustering & How It Works with MLlib
  • Analysis on US Election Data using MLlib (K-Means)
Understanding Apache Kafka & Apache Flume
  • Need for Kafka
  • Core Concepts of Kafka
  • Where is Kafka Used?
  • Configuring Kafka Cluster
  • What is Kafka?
  • Kafka Architecture
  • Understanding the Components of Kafka Cluster
  • Need of Apache Flume
  • What is Apache Flume?
  • Flume Sources
  • Flume Channels
  • Integrating Apache Flume and Apache Kafka
  • Basic Flume Architecture
  • Flume Sinks
  • Flume Configuration
Apache Spark Streaming- Processing Multiple Batches
  • Drawbacks in Existing Computing Methods
  • Why Streaming is Necessary?
  • What is Spark Streaming?
  • Spark Streaming Features
  • Spark Streaming Workflow
  • How Uber Uses Streaming Data
  • Streaming Context & DStreams
  • Transformations on DStreams
  • Describe Windowed Operators and Why it is Useful
  • Important Windowed Operators
  • Slice, Window and Reduce By Window Operators
  • Stateful Operators
Apache Spark Streaming- Data Sources
  • Apache Spark Streaming: Data Sources
  • Streaming Data Source Overview
  • Apache Flume and Apache Kafka Data Sources
  • Example: Using a Kafka Direct Data Source
  • Perform Twitter Sentimental Analysis Using Spark Streaming

 

 

Apache Kafka

 

 

Introduction to Big Data and Apache Kafka
  • Introduction to Big Data
  • Need for Kafka
  • Kafka Features
  • Kafka Architecture
  • ZooKeeper
  • Kafka Installation
  • Configuring Single Node Single Broker Cluster
  • Big Data Analytics
  • What is Kafka?
  • Kafka Concepts
  • Kafka Components
  • Where is Kafka Used?
  • Kafka Cluster
  • Types of Kafka Clusters
Kafka Producer
  • Configuring Single Node Multi Broker Cluster
  • Constructing a Kafka Producer
  • Sending a Message to Kafka
  • Producing Keyed and Non-Keyed Messages
  • Sending a Message Synchronously & Asynchronously
  • Configuring Producers
  • Serializers
  • Serializing Using Apache Avro
  • Partitions
Kafka Consumer
  • Consumers and Consumer Groups
  • Consumer Groups and Partition Rebalance
  • Subscribing to Topics
  • Configuring Consumers
  • Rebalance Listeners
  • Consuming Records with Specific Offsets
  • Standalone Consumer
  • Creating a Kafka Consumer
  • The Poll Loop
  • Commits and Offsets
  • Deserializers
Kafka Internals
  • Cluster Membership
  • Replication
  • Physical Storage
  • Broker Configuration
  • Using Producers in a Reliable System
  • Using Consumers in a Reliable System
  • The Controller
  • Request Processing
  • Reliability
  • Validating System Reliability
  • Performance Tuning in Kafka
Kafka Cluster Architectures & Administering Kafka
  • Use Cases – Cross-Cluster Mirroring
  • Other Cross-Cluster Mirroring Solutions
  • Topic Operations
  • Dynamic Configuration Changes
  • Consuming and Producing
  • Multi-Cluster Architectures
  • Apache Kafka’s MirrorMaker
  • Consumer Groups
  • Partition Management
  • Unsafe Operations
Kafka Monitoring and Kafka Connect
  • Considerations When Building Data Pipelines
  • Kafka Broker Metrics
  • Lag Monitoring
  • Kafka Connect
  • Kafka Connect Properties
  • Metric Basics
  • Client Monitoring
  • End-to-End Monitoring
  • When to Use Kafka Connect?
Kafka Stream Processing
  • Stream Processing
  • Stream-Processing Design Patterns
  • Kafka Streams: Architecture Overview
  • Stream-Processing Concepts
  • Kafka Streams by Example
Integration of Kafka With Hadoop, Storm and Spark
  • Apache Hadoop Basics
  • Kafka Integration with Hadoop
  • Configuration of Storm
  • Hadoop Configuration
  • Apache Storm Basics
  • Integration of Kafka with Storm
  • Apache Spark Basics
  • Kafka Integration with Spark
  • Spark Configuration
Integration of Kafka With Talend and Cassandra
  • Flume Basics
  • Integration of Kafka with Flume
  • Integration of Kafka with Cassandra
  • Talend Basics
  • Integration of Kafka with Talend
  • Cassandra Basics such as and KeySpace and Table Creation

 

To see the full course content Download now

Course Prerequisites

 
  • There are no such prerequisites for Big Data & Hadoop Course. However, prior knowledge of Core Java and SQL will be helpful but is not mandatory. Further, to brush up your skills, we offers a complimentary self-paced course on "Java essentials for Hadoop" when you enroll for the Big Data and Hadoop Course.
  • Whether you are an experienced professional working in the IT industry, or an aspirant planning to enter the data driven world of analytics, Masters Program is designed and developed to accommodate multitude professional backgrounds and allow its learners to have a successful journey in the Big Data industry.

Who can attend

 
  • Be a graduate (Engineering or Equivalent)

Number of Hours: 100hrs

Certification

None

Key features

  • One to One Training
  • Online Training
  • Fastrack & Normal Track
  • Resume Modification
  • Mock Interviews
  • Video Tutorials
  • Materials
  • Real Time Projects
  • Virtual Live Experience
  • Preparing for Certification

FAQs

DASVM Technologies offers 300+ IT training courses with 10+ years of Experienced Expert level Trainers.

  • One to One Training
  • Online Training
  • Fastrack & Normal Track
  • Resume Modification
  • Mock Interviews
  • Video Tutorials
  • Materials
  • Real Time Projects
  • Materials
  • Preparing for Certification

Call now: +91-99003 49889 and know the exciting offers available for you!

We working and coordinating with the companies exclusively to get placed. We have a placement cell focussing on training and placements in Bangalore. Our placement cell help more than 600+ students per year.

Learn from experts active in their field, not out-of-touch trainers. Leading practitioners who bring current best practices and case studies to sessions that fit into your work schedule. We have a pool of experts and trainers are composed with highly skilled and experienced in supporting you in specific tasks and provide professional support. 24x7 Learning support from mentors and a community of like-minded peers to resolve any conceptual doubts. Our trainers has contributed in the growth of our clients as well as professionals.

All of our highly qualified trainers are industry experts with at least 10-12 years of relevant teaching experience. Each of them has gone through a rigorous selection process which includes profile screening, technical evaluation, and a training demo before they are certified to train for us. We also ensure that only those trainers with a high alumni rating continue to train for us.

No worries. DASVM technologies assure that no one misses single lectures topics. We will reschedule the classes as per your convenience within the stipulated course duration with all such possibilities. If required you can even attend that topic with any other batches.

DASVM Technologies provides many suitable modes of training to the students like:

  • Classroom training
  • One to One training
  • Fast track training
  • Live Instructor LED Online training
  • Customized training

Yes, the access to the course material will be available for lifetime once you have enrolled into the course.

You will receive DASVM Technologies recognized course completion certification & we will help you to crack global certification with our training.

Yes, DASVM Technologies provides corporate trainings with Course Customization, Learning Analytics, Cloud Labs, Certifications, Real time Projects with 24x7 Support.

Yes, DASVM Technologies provides group discounts for its training programs. Depending on the group size, we offer discounts as per the terms and conditions.

We accept all major kinds of payment options. Cash, Card (Master, Visa, and Maestro, etc), Wallets, Net Banking, Cheques and etc.

DASVM Technologies has a no refund policy. Fees once paid will not be refunded. If the candidate is not able to attend a training batch, he/she is to reschedule for a future batch. Due Date for Balance should be cleared as per date given. If in case trainer got cancelled or unavailable to provide training DASVM will arrange training sessions with other backup trainer.

Your access to the Support Team is for lifetime and will be available 24/7. The team will help you in resolving queries, during and after the course.

Please Contact our course advisor +91-99003 49889. Or you can share your queries through info@dasvmtechnologies.com

like our courses