Scala Course Content

Module 01 - Introduction to Scala

• 1.1 Introducing Scala
• 1.2 Deployment of Scala for Big Data applications and Apache Spark analytics
• 1.3 Scala REPL, lazy values, and control structures in Scala
• 1.4 Directed Acyclic Graph (DAG)
• 1.5 First Spark application using SBT/Eclipse
• 1.6 Spark Web UI
• 1.7 Spark in the Hadoop ecosystem.

Module 02 - Pattern Matching

• 2.1 The importance of Scala
• 2.2 The concept of REPL (Read Evaluate Print Loop)
• 2.3 Deep dive into Scala pattern matching
• 2.4 Type interface, higher-order function, currying, traits, application space and Scala for data analysis

Module 03 - Executing the Scala Code

• 3.1 Learning about the Scala Interpreter
• 3.2 Static object timer in Scala and testing string equality in Scala
• 3.3 Implicit classes in Scala
• 3.4 The concept of currying in Scala
• 3.5 Various classes in Scala

Module 04 - Classes Concept in Scala

• 4.1 Learning about the Classes concept
• 4.2 Understanding the constructor overloading
• 4.3 Various abstract classes
• 4.4 The hierarchy types in Scala
• 4.5 The concept of object equality
• 4.6 The val and var methods in Scala

Module 05 - Case Classes and Pattern Matching

• 5.1 Understanding sealed traits, wild, constructor, tuple, variable pattern, and constant pattern

Module 06 - Concepts of Traits with Example

• 6.1 Understanding traits in Scala
• 6.2 The advantages of traits
• 6.3 Linearization of traits
• 6.4 The Java equivalent
• 6.5 Avoiding of boilerplate code

Module 07 - Scala–Java Interoperability

• 7.1 Implementation of traits in Scala and Java
• 7.2 Handling of multiple traits extending.

Module 08 - Scala Collections

• 8.1 Introduction to Scala collections
• 8.2 Classification of collections
• 8.3 The difference between iterator and iterable in Scala
• 8.4 Example of list sequence in Scala

Module 09 - Mutable Collections Vs. Immutable Collections

• 9.1 The two types of collections in Scala
• 9.2 Mutable and immutable collections
• 9.3 Understanding lists and arrays in Scala
• 9.4 The list buffer and array buffer
• 9.6 Queue in Scala
• 9.7 Double-ended queue Deque, Stacks, Sets, Maps, and Tuples in Scala

Module 10 - Use Case Bobsrockets Package

• 10.1 Introduction to Scala packages and imports
• 10.2 The selective imports
• 10.3 The Scala test classes
• 10.4 Introduction to JUnit test class
• 10.5 JUnit interface via JUnit 3 suite for Scala test
• 10.6 Packaging of Scala applications in the directory structure
• 10.7 Examples of Spark Split and Spark Scala

Spark Course Content

Module 11 - Introduction to Spark

• 11.1 Introduction to Spark
• 11.2 Spark overcomes the drawbacks of working on MapReduce
• 11.3 Understanding in-memory MapReduce
• 11.4 Interactive operations on MapReduce
• 11.5 Spark stack, fine vs. coarse-grained update, Spark stack, Spark Hadoop YARN, HDFS Revision, and YARN Revision
• 11.6 The overview of Spark and how it is better than Hadoop
• 11.7 Deploying Spark without Hadoop
• 11.8 Spark history server and Cloudera distribution

Module 12 - Spark Basics

• 12.1 Spark installation guide
• 12.2 Spark configuration
• 12.3 Memory management
• 12.4 Executor memory vs. driver memory
• 12.5 Working with Spark Shell
• 12.6 The concept of resilient distributed datasets (RDD)
• 12.7 Learning to do functional programming in Spark
• 12.8 The architecture of Spark

Module 13 - Working with RDDs in Spark

• 13.1 Spark RDD
• 13.2 Creating RDDs
• 13.3 RDD partitioning
• 13.4 Operations and transformation in RDD
• 13.5 Deep dive into Spark RDDs
• 13.6 The RDD general operations
• 13.7 Read-only partitioned collection of records
• 13.8 Using the concept of RDD for faster and efficient data processing
• 13.9 RDD action for the collect, count, collects map, save-as-text-files, and pair RDD functions

Module 14 - Aggregating Data with Pair RDDs

• 14.1 Understanding the concept of key-value pair in RDDs
• 14.2 Learning how Spark makes MapReduce operations faster
• 14.3 Various operations of RDD
• 14.4 MapReduce interactive operations
• 14.5 Fine and coarse-grained update
• 14.6 Spark stack

Module 15 - Writing and Deploying Spark Applications

• 15.1 Comparing the Spark applications with Spark Shell
• 15.2 Creating a Spark application using Scala or Java
• 15.3 Deploying a Spark application
• 15.4 Scala built application
• 15.5 Creation of the mutable list, set and set operations, list, tuple, and concatenating list
• 15.6 Creating an application using SBT
• 15.7 Deploying an application using Maven
• 15.8 The web user interface of Spark application
• 15.9 A real-world example of Spark
• 15.10 Configuring of Spark

Module 16 - Parallel Processing

• 16.1 Learning about Spark parallel processing
• 16.2 Deploying on a cluster
• 16.3 Introduction to Spark partitions
• 16.4 File-based partitioning of RDDs
• 16.5 Understanding of HDFS and data locality
• 16.6 Mastering the technique of parallel operations
• 16.7 Comparing repartition and coalesce
• 16.8 RDD actions

Module 17 - Spark RDD Persistence

• 17.1 The execution flow in Spark

• 17.2 Understanding the RDD persistence overview
• 17.3 Spark execution flow, and Spark terminology
• 17.4 Distribution shared memory vs. RDD
• 17.5 RDD limitations
• 17.6 Spark shell arguments
• 17.7 Distributed persistence
• 17.8 RDD lineage
• 17.9 Key-value pair for sorting implicit conversions like CountByKey, ReduceByKey, SortByKey, and AggregateByKey

Module 18 - Spark MLlib

• 18.1 Introduction to Machine Learning
• 18.2 Types of Machine Learning
• 18.3 Introduction to MLlib
• 18.4 Various ML algorithms supported by MLlib
• 18.5 Linear regression, logistic regression, decision tree, random forest, and K-means clustering techniques

Hands-on Exercise:

• 1. Building a Recommendation Engine

Module 19 - Integrating Apache Flume and Apache Kafka

• 19.1 Why Kafka and what is Kafka?
• 19.2 Kafka architecture
• 19.3 Kafka workflow
• 19.4 Configuring Kafka cluster
• 19.5 Operations
• 19.6 Kafka monitoring tools
• 19.7 Integrating Apache Flume and Apache Kafka

Hands-on Exercise:

1. Configuring Single Node Single Broker Cluster

2. Configuring Single Node Multi Broker Cluster

3. Producing and consuming messages

4. Integrating Apache Flume and Apache Kafka

Module 20 - Spark Streaming

• 20.1 Introduction to Spark Streaming
• 20.2 Features of Spark Streaming
• 20.3 Spark Streaming workflow
• 20.4 Initializing StreamingContext, discretized Streams (DStreams), input DStreams and Receivers
• 20.5 Transformations on DStreams, output operations on DStreams, windowed operators and why it is useful
• 20.6 Important windowed operators and stateful operators

Hands-on Exercise:

1. Twitter Sentiment analysis

2. Streaming using Netcat server

3. Kafka–Spark streaming

4. Spark–Flume streaming

Module 21 - Improving Spark Performance

• 21.1 Introduction to various variables in Spark like shared variables and broadcast variables
• 21.2 Learning about accumulators
• 21.3 The common performance issues
• 21.4 Troubleshooting the performance problems

Module 22 - Spark SQL and Data Frames

• 22.1 Learning about Spark SQL
• 22.2 The context of SQL in Spark for providing structured data processing
• 22.3 JSON support in Spark SQL
• 22.4 Working with XML data
• 22.5 Parquet files
• 22.6 Creating Hive context
• 22.7 Writing data frame to Hive
• 22.8 Reading JDBC files
• 22.9 Understanding the data frames in Spark
• 22.10 Creating Data Frames
• 22.11 Manual inferring of schema
• 22.12 Working with CSV files
• 22.13 Reading JDBC tables
• 22.14 Data frame to JDBC
• 22.15 User-defined functions in Spark SQL
• 22.16 Shared variables and accumulators
• 22.17 Learning to query and transform data in data frames
• 22.18 Data frame provides the benefit of both Spark RDD and Spark SQL
• 22.19 Deploying Hive on Spark as the execution engine

Module 23 - Scheduling/Partitioning

• 23.1 Learning about the scheduling and partitioning in Spark
• 23.2 Hash partition
• 23.3 Range partition
• 23.4 Scheduling within and around applications
• 23.5 Static partitioning, dynamic sharing, and fair scheduling
• 23.6 Map partition with index, the Zip, and GroupByKey
• 23.7 Spark master high availability, standby masters with ZooKeeper, single-node recovery with the local file system and high order functions

Spark and Scala Projects

• Movie Recommendation
• Deploy Apache Spark for a movie recommendation system. Through this project, you will be working with Spark MLlib, collaborative filtering, clustering, regression, and dimensionality reduction. By the completion of this project, you will be proficient in working with streaming data, sampling, testing, and statistics.
• Twitter API Integration for Tweet Analysis
• integrate Twitter API for analyzing tweets. You can use any of the scripting languages, like PHP, Ruby, or Python, for requesting the Twitter API and get the results in JSON format. You will have to perform aggregation, filtering, and parsing as per the requirement for the tweet analysis.
• Data Exploration Using Spark SQL – Wikipedia Data
• This project will allow you to work with Spark SQL and combine it with ETL applications, real-time analysis of data, performing batch analysis, deploying Machine Learning, creating visualizations, and processing of graphs.

Introduction to the Basics of Python

• Explaining Python and Highlighting Its Importance
• Setting up Python Environment and Discussing Flow Control
• Running Python Scripts and Exploring Python Editors and IDEs

Sequence and File Operations

• Defining Reserve Keywords and Command Line Arguments
• Describing Flow Control and Sequencing
• Indexing and Slicing
• Learning the xrange() Function
• Working Around Dictionaries and Sets
• Working with Files

Functions, Sorting, Errors and Exception, Regular Expressions, and Packages

• Explaining Functions and Various Forms of Function Arguments
• Learning Variable Scope, Function Parameters, and Lambda Functions
• Sorting Using Python
• Exception Handling
• Package Installation
• Regular Expressions

Python: An OOP Implementation

• Using Class, Objects, and Attributes
• Developing Applications Based on OOP
• Learning About Classes, Objects and How They Function Together
• Explaining OOPs Concepts Including Inheritance, Encapsulation, and Polymorphism, Among Others

Debugging and Databases

• Debugging Python Scripts Using pdb and IDE
• Classifying Errors and Developing Test Units
• Implementing Databases Using SQLite
• Performing CRUD Operations

Introduction to Big Data and Apache Spark

• What is Big Data?
• 5 V’s of Big Data
• Problems related to Big Data: Use Case
• What tools available for handling Big Data?
• What is Hadoop?
• Why do we need Hadoop?
• Key Characteristics of Hadoop
• Important Hadoop ecosystem concepts
• MapReduce and HDFS
• Introduction to Apache Spark
• What is Apache Spark?
• Why do we need Apache Spark?
• Who uses Spark in the industry?
• Apache Spark architecture
• Spark Vs. Hadoop
• Various Big data applications using Apache Spark

Python for Spark

• Introduction to PySpark
• Who uses PySpark?
• Why Python for Spark?
• Values, Types, Variables
• Operands and Expressions
• Conditional Statements
• Loops
• Numbers
• Python files I/O Functions
• Strings and associated operations
• Sets and associated operations
• Lists and associated operations
• Tuples and associated operations
• Dictionaries and associated operations

Hands-On:

• Demonstrating Loops and Conditional Statements
• Tuple – related operations, properties, list, etc.
• List – operations, related properties
• Set – properties, associated operations
• Dictionary – operations, related properties

Python for Spark: Functional and Object-Oriented Model

• Functions
• Lambda Functions
• Global Variables, its Scope, and Returning Values
• Standard Libraries
• Object-Oriented Concepts
• Modules Used in Python
• The Import Statements
• Module Search Path
• Package Installation Ways

Hands-On:

• Lambda – Features, Options, Syntax, Compared with the Functions
• Functions – Syntax, Return Values, Arguments, and Keyword Arguments
• Errors and Exceptions – Issue Types, Remediation
• Packages and Modules – Import Options, Modules, sys Path

Apache Spark Framework and RDDs

• Spark Components & its Architecture
• Spark Deployment Modes
• Spark Web UI
• Introduction to PySpark Shell
• Submitting PySpark Job
• Writing your first PySpark Job Using Jupyter Notebook
• What is Spark RDDs?
• Stopgaps in existing computing methodologies
• How RDD solve the problem?
• What are the ways to create RDD in PySpark?
• RDD persistence and caching
• General operations: Transformation, Actions, and Functions
• Concept of Key-Value pair in RDDs
• Other pair, two pair RDDs
• RDD Lineage
• RDD Persistence
• WordCount Program Using RDD Concepts
• RDD Partitioning & How it Helps Achieve Parallelization
• Passing Functions to Spark

Hands-On:

• Building and Running Spark Application
• Spark Application Web UI
• Loading data in RDDs
• Saving data through RDDs
• RDD Transformations
• RDD Actions and Functions
• RDD Partitions
• WordCount program using RDD’s in Python

PySpark SQL and Data Frames

• Need for Spark SQL
• What is Spark SQL
• Spark SQL Architecture
• SQL Context in Spark SQL
• User-Defined Functions
• Data Frames
• Interoperating with RDDs
• Loading Data through Different Sources
• Performance Tuning
• Spark-Hive Integration

Hands-On:

• Spark SQL – Creating data frames
• Loading and transforming data through different sources
• Spark-Hive Integration

Apache Kafka and Flume

• Why Kafka
• What is Kafka?
• Kafka Workflow
• Kafka Architecture
• Kafka Cluster Configuring
• Kafka Monitoring tools
• Basic operations
• What is Apache Flume?
• Integrating Apache Flume and Apache Kafka

Hands-On:

• Single Broker Kafka Cluster
• Multi-Broker Kafka Cluster
• Topic Operations
• Integrating Apache Flume and Apache Kafka

PySpark Streaming

• Introduction to Spark Streaming
• Features of Spark Streaming
• Spark Streaming Workflow
• StreamingContext Initializing
• Discretized Streams (DStreams)
• Input DStreams, Receivers
• Transformations on DStreams
• DStreams Output Operations
• Describe Windowed Operators and Why it is Useful
• Stateful Operators
• Vital Windowed Operators
• Twitter Sentiment Analysis
• Streaming using Netcat server
• WordCount program using Kafka-Spark Streaming

Hands-On:

• Twitter Sentiment Analysis
• Streaming using Netcat server
• WordCount program using Kafka-Spark Streaming
• Spark-flume Integration

Introduction to PySpark Machine Learning

• Introduction to Machine Learning- What, Why and Where?
• Use Case
• Types of Machine Learning Techniques
• Why use Machine Learning for Spark?
• Applications of Machine Learning (general)
• Applications of Machine Learning with Spark
• Introduction to MLlib
• Features of MLlib and MLlib Tools
• Various ML algorithms supported by MLlib
• Supervised Learning Algorithms
• Unsupervised Learning Algorithms
• ML workflow utilities

Hands-On:

• K- Means Clustering
• Linear Regression
• Logistic Regression
• Decision Tree
• Random Forest