Difference in resource requests via spark-llap

[John-yovoa]

Query: select DISTINCT student_id from mytestdb.mytesttable;

From TEZ mode only 2 reducers are required but with LLAP it requests 532 reducers

from TEZ:

---

VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED

Map 1 .......... container SUCCEEDED 95 95 0 0 0 0
Reducer 2 ...... container SUCCEEDED 2 2 0 0 0 0

VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 19.95 s

INFO : Query Execution Summary INFO : ---------------------------------------------------------------------------------------------- INFO : OPERATION DURATION INFO : ---------------------------------------------------------------------------------------------- INFO : Compile Query 0.72s INFO : Prepare Plan 7.20s INFO : Get Query Coordinator (AM) 0.00s INFO : Submit Plan 0.23s INFO : Start DAG 0.81s INFO : Run DAG 19.91s INFO : ---------------------------------------------------------------------------------------------- INFO : INFO : Task Execution Summary INFO : ---------------------------------------------------------------------------------------------- INFO : VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS OUTPUT_RECORDS INFO : ---------------------------------------------------------------------------------------------- INFO : Map 1 12119.00 1,251,490 37,190 819,833,947 15,665 INFO : Reducer 2 7747.00 7,150 620 15,665 0 INFO : ---------------------------------------------------------

From LLAP:

---

VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED

Map 1 .......... llap SUCCEEDED 96 96 0 0 0 0
Reducer 2 ...... llap SUCCEEDED 532 532 0 0 0 2

VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 29.83 s

`INFO : Query Execution Summary
INFO : ----------------------------------------------------------------------------------------------
INFO : OPERATION DURATION
INFO : ----------------------------------------------------------------------------------------------
INFO : Compile Query 11.87s
INFO : Prepare Plan 0.07s
INFO : Get Query Coordinator (AM) 0.00s
INFO : Submit Plan 0.03s
INFO : Start DAG 0.53s
INFO : Run DAG 29.77s
INFO : ----------------------------------------------------------------------------------------------
INFO :
INFO : Task Execution Summary
INFO : ----------------------------------------------------------------------------------------------
INFO : VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS OUTPUT_RECORDS
INFO : ----------------------------------------------------------------------------------------------
INFO : Map 1 27782.00 0 0 819,833,947 15,216
INFO : Reducer 2 24735.00 0 0 15,216 0
INFO : ----------------------------------------------------------------------------------------------
INFO :
INFO : LLAP IO Summary
INFO : ----------------------------------------------------------------------------------------------
INFO : VERTICES ROWGROUPS META_HIT META_MISS DATA_HIT DATA_MISS TOTAL_IO
INFO : ----------------------------------------------------------------------------------------------
INFO : Map 1 83209 4450 524 384.62MB 47.76MB 781.92s

INFO :
INFO : FileSystem Counters Summary
INFO :
INFO : Scheme: FILE
INFO : ----------------------------------------------------------------------------------------------
INFO : VERTICES BYTES_READ READ_OPS LARGE_READ_OPS BYTES_WRITTEN WRITE_OPS
INFO : ----------------------------------------------------------------------------------------------
INFO : Map 1 2.45MB 0 0 1.85MB 0
INFO : Reducer 2 0B 0 0 0B 0
INFO : ----------------------------------------------------------------------------------------------
INFO :
INFO : Scheme: HDFS
INFO : ----------------------------------------------------------------------------------------------
INFO : VERTICES BYTES_READ READ_OPS LARGE_READ_OPS BYTES_WRITTEN WRITE_OPS
INFO : ----------------------------------------------------------------------------------------------
INFO : Map 1 54.68MB 1048 0 0B 0
INFO : Reducer 2 0B 615 0 25.77KB 410
INFO : ----------------------------------------------------------------------------------------------
INFO :
INFO : LLAP WM Summary
INFO : ----------------------------------------------------------------------------------------------
INFO : VERTICES SPECULATIVE QUEUED SPECULATIVE RUNNING GUARANTEED QUEUED GUARANTEED RUNNING
INFO : ----------------------------------------------------------------------------------------------
INFO : Map 1 0.00s 0.00s 0.04s 1020.11s
INFO : Reducer 2 38.81s 43.56s 192.68s 9882.59s
INFO : ----------------------------------------------------------------------------------------------
INFO :
INFO :

`