Big Data Training

Hadoop is a Open Source from Apache, which provides reliable storage and faster process by using the Hadoop distibution file system and Map Reduce Program.

Introduction

• What is Cloud Computing
• What is Grid Computing
• What is Virtualization
• How above three are inter-related to each other
• What is Big Data
• Introduction to Analytics and the need for big data analytics
• Hadoop Solutions - Big Picture
• Hadoop distributions
• Comparing Hadoop Vs.Traditional systems
• Volunteer Computing
• Data Retrieval - Radom Access Vs. Sequential Access
• NoSQL Databases

HDFS (Hadoop Distributed File System)

• Blocks and Splits
• Input Splits
• HDFS Splits
• Data Replication
• Hadoop Rack Aware
• Data high availability
• Data Integrity
• Cluster architecture and block placement
• Accessing HDFS
• JAVA Approach
• CLI Approach

Hadoop Administrative Tasks

Setup Hadoop cluster of Apache, Cloudera and HortonWorks

• Install and configure Apache Hadoop
• Make a fully distributed Hadoop cluster on a single laptop/desktop (Psuedo Mode)
• Install and configure Cloudera Hadoop distribution in fully distributed mode
• Install and configure HortonWorks Hadoop Distribution in fully distributed mode
• Monitoring the cluster
• Getting used to management console of Cloudera and Horton Works
• Name Node in Safe mode
• Meta Data Backup
• Integrating Kerberos security in hadoop
• Ganglia and Nagios Cluster monitoring
• Benchmarking the Cluster
• Commissioning/Decommissioning Nodes

Hadoop Developer Tasks

Writing a MapReduce Program

• Examining a Sample MapReduce Program
• With Several Examples
• Basic API Concepts
• The Driver Code
• The Mapper
• The Reducer
• Hadoop's Streaming API

Common MapReduce Algorithms

• Sorting and Searching
• Indexing
• Classification/Machine Learning
• Term Frequency – Inverse Document Frequency
• Word Co-Occurrence
• Hands-On Exercise: Creating an Inverted Index
• Identify Mapper
• Identify Reducer
• Exploring well known problems using MapReduce applications

Advanced MapReduce Programming

• A Recap of the MapReduce Flow
• Custom Writables and WritableComparables
• The Secondary Sort
• Creating InputFormats and OutputFormats
• Pipelining Jobs With Oozie
• Map-Side Joins
• Reduce-Side Joins

Tuning for Performance

• Reducing network traffic with combiner
• Reducing the amount of input data
• Using Compression
• Running with speculative execution
• Refactoring code and rewriting algorithms Parameters affecting Performance
• Other Performance Aspects

Hadoop Ecosystem

Hive

• Hive concepts
• Hive architecture
• Install and configure hive on cluster
• Create database, access it console
• Buckets,Partitions
• Joins in Hive
• Inner joins
• Outer joins
• Hive UDF
• Hive UDAF
• Hive UDTF
• Develop and run sample applications in Java to access hive
• Load Data into Hive and process it using Hive

Sqoop

• Install and configure Sqoop on cluster
• Connecting to RDBMS
• Installing Mysql
• Import data from Oracle/Mysql to hive
• Export data to Oracle/Mysql
• Internal mechanism of import/export
• Import millions of records into HDFS from RDBMS using Sqoop

HBase

• HBase concepts
• HBase architecture
• Region server architecture
• File storage architecture
• HBase basics
• Cloumn access
• Scans
• HBase Use Cases
• Install and configure HBase on cluster

Cassandra

• Cassandra core concepts
• Install and configure Cassandra on cluster
• Create database, tables and access it console
• Developing applications to access data in Cassandra through Java
• Install and Configure OpsCenter to access Cassandra data using browser

Oozie

• Oozie architecture
• XML file specifications
• Install and configure Oozie on Cluster
• Specifying Work flow
• Action nodes
• Control nodes
• Oozie job coordinator
• Accessing Oozie jobs command line and using web console
• Create a sample workflows in oozie and run them on cluster
• Zookeeper, Flume, Chukwa,Avro, Scribe, Thrift, HCatalog
• Flume and Chukwa Concepts
• Use cases of Thrift ,Avro and scribe
• Install and Configure flume on cluster
• Create a sample application to capture logs from Apache using flume
• Analytics Basics
• Analytics and big data analytics
• Commonly used analytics algorithms
• Analytics tools like R and Weka
• R language basics
• Mahout
• CDH4 Enhancements
• Name Node High Availability
• Name Node federation
• Fencing
• YARN

As part of this Spark training classes the learners will be getting complete knowledge about these two advanced technologies and their concepts like RDD, Spark SQL, fundamentals of Scala programming and much more.

What You Will Learn In This Scala Certification Training?

• Basics of Apache Spark and Scala
• Learning to implement Spark on cluster
• In-depth study of Scala programming and its concepts
• Learning to write Spark applications in Python, Java and Scala
• Spark streaming
• Scala and Java interoperability
• Difference between Spark and Hadoop

Scala Course Content

Introduction of Scala

Introducing Scala and deployment of Scala for Big Data applications and Apache Spark analytics.

Pattern Matching

The importance of Scala, the concept of REPL (Read Evaluate Print Loop), deep dive into Scala pattern matching, type interface, higher order function, currying, traits, application space and Scala for data analysis.

Executing the Scala code

Learning about the Scala Interpreter, static object timer in Scala, testing String equality in Scala, Implicit classes in Scala, the concept of currying in Scala, various classes in Scala.

Classes concept in Scala

Learning about the Classes concept, understanding the constructor overloading, the various abstract classes, the hierarchy types in Scala, the concept of object equality, the val and var methods in Scala.

Case classes and pattern matching

Understanding Sealed traits, wild, constructor, tuple, variable pattern, and constant pattern.

Concepts of traits with example

Understanding traits in Scala, the advantages of traits, linearization of traits, the Java equivalent and avoiding of boilerplate code.

Scala java Interoperability

Implementation of traits in Scala and Java, handling of multiple traits extending.

Scala collections

Introduction to Scala collections, classification of collections, the difference between Iterator, and Iterable in Scala, example of list sequence in Scala.

Mutable collections vs. Immutable collections

The two types of collections in Scala, Mutable and Immutable collections, understanding lists and arrays in Scala, the list buffer and array buffer, Queue in Scala, double-ended queue Deque, Stacks, Sets, Maps, Tuples in Scala.

Use Case bobsrockets package

Introduction to Scala packages and imports, the selective imports, the Scala test classes, introduction to JUnit test class, JUnit interface via JUnit 3 suite for Scala test, packaging of Scala applications in Directory Structure, example of Spark Split and Spark Scala.

Spark Course Content

Introduction to Spark

Introduction to Spark, how Spark overcomes the drawbacks of working MapReduce, understanding in-memory MapReduce,interactive operations on MapReduce, Spark stack, fine vs. coarse grained update, Spark stack,Spark Hadoop YARN, HDFS Revision, YARN Revision, the overview of Spark and how it is better Hadoop, deploying Spark without Hadoop,Spark history server, Cloudera distribution.

Spark Basics

Spark installation guide,Spark configuration, memory management, executor memory vs. driver memory, working with Spark Shell, the concept of Resilient Distributed Datasets (RDD), learning to do functional programming in Spark, the architecture of Spark.

Working with RDDs in Spark

Spark RDD, creating RDDs, RDD partitioning, operations & transformation in RDD,Deep dive into Spark RDDs, the RDD general operations, a read-only partitioned collection of records, using the concept of RDD for faster and efficient data processing,RDD action for Collect, Count, Collectsmap, Saveastextfiles, pair RDD functions.

Aggregating Data with Pair RDDs

Understanding the concept of Key-Value pair in RDDs, learning how Spark makes MapReduce operations faster, various operations of RDD,MapReduce interactive operations, fine & coarse grained update, Spark stack.

Writing and Deploying Spark Applications

Comparing the Spark applications with Spark Shell, creating a Spark application using Scala or Java, deploying a Spark application,Scala built application,creation of mutable list, set & set operations, list, tuple, concatenating list, creating application using SBT,deploying application using Maven,the web user interface of Spark application, a real world example of Spark and configuring of Spark.

Parallel Processing

Learning about Spark parallel processing, deploying on a cluster, introduction to Spark partitions, file-based partitioning of RDDs, understanding of HDFS and data locality, mastering the technique of parallel operations,comparing repartition & coalesce, RDD actions.

Spark RDD Persistence

The execution flow in Spark, Understanding the RDD persistence overview,Spark execution flow & Spark terminology, distribution shared memory vs. RDD, RDD limitations, Spark shell arguments,distributed persistence, RDD lineage,Key/Value pair for sorting implicit conversion like CountByKey, ReduceByKey, SortByKey, AggregataeByKey

Spark Streaming & Mlib

Spark Streaming Architecture, Writing streaming programcoding, processing of spark stream,processing Spark Discretized Stream (DStream), the context of Spark Streaming, streaming transformation, Flume Spark streaming, request count and Dstream, multi batch operation, sliding window operations and advanced data sources. Different Algorithms, the concept of iterative algorithm in Spark, analyzing with Spark graph processing, introduction to K-Means and machine learning, various variables in Spark like shared variables, broadcast variables, learning about accumulators.

Improving Spark Performance

Introduction to various variables in Spark like shared variables, broadcast variables, learning about accumulators, the common performance issues and troubleshooting the performance problems.

Spark SQL and Data Frames

Learning about Spark SQL, the context of SQL in Spark for providing structured data processing, JSON support in Spark SQL, working with XML data, parquet files, creating HiveContext, writing Data Frame to Hive, reading JDBC files, understanding the Data Frames in Spark, creating Data Frames, manual inferring of schema, working with CSV files, reading JDBC tables, Data Frame to JDBC, user defined functions in Spark SQL, shared variable and accumulators, learning to query and transform data in Data Frames, how Data Frame provides the benefit of both Spark RDD and Spark SQL, deploying Hive on Spark as the execution engine.

Scheduling/ Partitioning

Learning about the scheduling and partitioning in Spark,hash partition, range partition, scheduling within and around applications, static partitioning, dynamic sharing, fair scheduling,Map partition with index, the Zip, GroupByKey, Spark master high availability, standby Masters with Zookeeper, Single Node Recovery With Local File System, High Order Functions.