Question

Spark provides a much richer set of programming constructs and libraries that greatly simplify concurrent programming. In addition, because Spark data can be persistent over a session (unliike MapReduce which reads/writes data at each step in the job chain), it can be much faster for iteratvie programs and also enables interactive concurrent programming. See official documenttion for details, including setting up on Amazon . This article on how to set up Spark on EMR may also be helpful.

Very conceniently for learning, Spark provides an REPL shell where you can interactively type and run Spark programs. For example, this will open a Spark shell as an IPython Notebook (if spark is installed and pyspark is on your path):

IPYTHON_OPTS="notebook" pyspark

To whet your appetite, here is the stadnalone Spark version for the word count program.

%%file spark_count.py

from pyspark import SparkConf, SparkContext
conf = SparkConf().setMaster("local").setAppName("Word Count")
sc = SparkContext(conf = conf)

rdd = sc.textFile("<path_to_books>")
words = rdd.flatMap(lambda x: x.split())
result = words.countByValue()

Writing spark_count.py

And this is run by typing on the command line

bin/spark-submit spark_count.py

Of course, spark-submit has many options that can be provided to configure the job.