Spark Basics

• Main Web Site
• Apache Spark on Wikipedia

Creating a SparkSession

Since Spark 2.x, SparkSession is the entry point for reading data and executing SQL queries over data.

Scala:

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("SparkBasics")
  .master("local[*]")
  .config("spark.some.config.option", "some-value")
  .getOrCreate()

Python (PySpark):

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("SparkBasics").master("local[*]").getOrCreate()

Useful Links

• Official Quick Start
• RDD Programming Guide
• Spark SQL, DataFrames and Datasets Guide
• Spark on AWS EMR
• Databricks Reference Apps

Spark Connectors

• MongoDB Spark connector
• Elasticsearch Hadoop (for Spark)

Ecosystem Integrations

• Graph Processing: Graphx programming guide
• Notebooks: Apache Zeppelin and JupyterLab are commonly used interactive environments for Spark. Zeppelin server is usually found at port 8890.
• Zeppelin