How do I set/get heap size for Spark (via Python notebook)

Question

I'm using Spark (1.5.1) from an IPython notebook on a macbook pro. After installing Spark and Anaconda, I start IPython from a terminal by executing: IPYTHON_OPTS="notebook" pyspark. This opens a webpage listing all my IPython notebooks. I can select one of them, opening it in a second webpage. SparkContext (sc) is available already, and my first command in the notebook is help(sc), which runs fine. The problem I'm having is that I am getting a Java heap space error that I don't know how to address. How do I view my current Java heap setting, and how do I increase it within the context of my setup. The error message I'm getting follows:

Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 19 in stage 247.0 failed 1 times, most recent failure: Lost task 19.0 in stage 247.0 (TID 953, localhost): java.lang.OutOfMemoryError: Java heap space

Create an RDD of LabeledPoint. It is not particularly huge, 100K observations x2K feature vector. — Kai, Oct 23 '15 at 17:06

score 20 · Answer 1 · answered Jun 14 '16 at 14:18

You can manage Spark memory limits programmatically (by the API).

As SparkContext is already available in your Notebook:

sc._conf.get('spark.driver.memory')

You can set as well, but you have to shutdown the existing SparkContext first:

conf = SparkConf().setAppName("App")
conf = (conf.setMaster('local[*]')
        .set('spark.executor.memory', '4G')
        .set('spark.driver.memory', '45G')
        .set('spark.driver.maxResultSize', '10G'))
sc = SparkContext(conf=conf)

If your workload is the same for all analysis, then editing spark-defaults.conf as cited above is the way to go.

score 7 · Answer 2 · edited Jan 09 '19 at 16:05

7

I solved it by creating a spark-defaults.conf file in apache-spark/1.5.1/libexec/conf/ and adding the following line to it: spark.driver.memory 14g

That solved my issue. But then I ran into another issue of exceeding max result size of 1024MB. The solution was to add another line in the file above: spark.driver.maxResultSize 2g

edited Jan 09 '19 at 16:05

Francesco Boi

125
5

answered Oct 23 '15 at 17:10

Kai

303
1
2
10

14g is not a lot??? It's not big data but it is actually a lot! – eliasah Oct 23 '15 at 18:26
Great answer and the only that worked for me. Thanks. – Francesco Boi Jan 09 '19 at 14:49

score 4 · Answer 3 · answered Jan 17 '19 at 10:12

4

Just use the config option when setting SparkSession (as of 2.4)

MAX_MEMORY = "5g"

spark = SparkSession \
    .builder \
    .appName("Foo") \
    .config("spark.executor.memory", MAX_MEMORY) \
    .config("spark.driver.memory", MAX_MEMORY) \
    .getOrCreate()

answered Jan 17 '19 at 10:12

LaSul

461
3
11

i get the error :This SparkContext may be an existing one. – Arash Jul 20 '19 at 13:13
Just replace you code by this one, instead of adding it. The message said that you already created one session, – LaSul Jul 21 '19 at 12:05
`spark.stop()` also useful if spark is running at the time – Chris Huang-Leaver Sep 08 '22 at 06:56

How do I set/get heap size for Spark (via Python notebook)

3 Answers3