| Query 1 (First Execution) | Query 1 (verify Caching) | Query 2 (Same Base Table) | |
| Impala | 5.84s | 0.15s | 0.44s |
| Hive on MR2 | 26.288s | 24.367s | 53.177s |
| Hive on SPark | 31.798s | 3.284s | 3.407s |
[donghua@cdh-vm ~]$ impala-shell -i cdh-vm.dbaglobe.com -d default
Starting Impala Shell without Kerberos authentication
Connected to cdh-vm.dbaglobe.com:21000
Server version: impalad version 2.10.0-cdh5.13.0 RELEASE (build 2511805f1eaa991df1460276c7e9f19d819cd4e4)
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.10.0-cdh5.13.0 (2511805) built on Wed Oct 4 10:55:37 PDT 2017)
To see more tips, run the TIP command.
***********************************************************************************
Query: use `default`
[cdh-vm.dbaglobe.com:21000] > show tables;
Query: show tables
Fetched 0 row(s) in 0.00s
[cdh-vm.dbaglobe.com:21000] > invalidate metadata;
Query: invalidate metadata
Query submitted at: 2017-12-04 21:47:45 (Coordinator: http://cdh-vm.dbaglobe.com:25000)
Query progress can be monitored at: http://cdh-vm.dbaglobe.com:25000/query_plan?query_id=b04a346400d38fed:21d3756a00000000
Fetched 0 row(s) in 3.64s
[cdh-vm.dbaglobe.com:21000] > show tables;
Query: show tables
+-------------+
| name |
+-------------+
| iotdatademo |
+-------------+
Fetched 1 row(s) in 0.00s
[cdh-vm.dbaglobe.com:21000] > select count(*) from iotdatademo;
Query: select count(*) from iotdatademo
Query submitted at: 2017-12-04 21:48:06 (Coordinator: http://cdh-vm.dbaglobe.com:25000)
Query progress can be monitored at: http://cdh-vm.dbaglobe.com:25000/query_plan?query_id=8c4202fcb2d7bfc4:891ac4a100000000
+----------+
| count(*) |
+----------+
| 864010 |
+----------+
Fetched 1 row(s) in 5.84s
[cdh-vm.dbaglobe.com:21000] > select count(*) from iotdatademo;
Query: select count(*) from iotdatademo
Query submitted at: 2017-12-04 21:48:14 (Coordinator: http://cdh-vm.dbaglobe.com:25000)
Query progress can be monitored at: http://cdh-vm.dbaglobe.com:25000/query_plan?query_id=d433089da3461f8:7b786b700000000
+----------+
| count(*) |
+----------+
| 864010 |
+----------+
Fetched 1 row(s) in 0.15s
[cdh-vm.dbaglobe.com:21000] > select eday, avg(speed) from iotdatademo group by eday order by eday;
Query: select eday, avg(speed) from iotdatademo group by eday order by eday
Query submitted at: 2017-12-04 21:48:41 (Coordinator: http://cdh-vm.dbaglobe.com:25000)
Query progress can be monitored at: http://cdh-vm.dbaglobe.com:25000/query_plan?query_id=7b4945e5560d6f73:cf15702700000000
+------+-------------------+
| eday | avg(speed) |
+------+-------------------+
| 0 | 80.42217204861018 |
| 1 | 80.4242077305864 |
| 2 | 80.516892013888 |
| 3 | 80.42997673611163 |
| 4 | 80.62740798611237 |
| 5 | 80.49621712963015 |
| 6 | 80.54539832175986 |
+------+-------------------+
Fetched 7 row(s) in 0.44s
[cdh-vm.dbaglobe.com:21000] > exit;
Goodbye donghua
[donghua@cdh-vm ~]$
[donghua@cdh-vm ~]$ beeline -n donghua -u jdbc:hive2://cdh-vm:10000
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
scan complete in 3ms
Connecting to jdbc:hive2://cdh-vm:10000
Connected to: Apache Hive (version 1.1.0-cdh5.13.0)
Driver: Hive JDBC (version 1.1.0-cdh5.13.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.1.0-cdh5.13.0 by Apache Hive
0: jdbc:hive2://cdh-vm:10000> select count(*) from iotdatademo;
INFO : Compiling command(queryId=hive_20171204220606_0f8b9bb5-5da0-415a-bf7b-d3e7068b031e): select count(*) from iotdatademo
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)
INFO : Completed compiling command(queryId=hive_20171204220606_0f8b9bb5-5da0-415a-bf7b-d3e7068b031e); Time taken: 0.178 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing command(queryId=hive_20171204220606_0f8b9bb5-5da0-415a-bf7b-d3e7068b031e): select count(*) from iotdatademo
INFO : Query ID = hive_20171204220606_0f8b9bb5-5da0-415a-bf7b-d3e7068b031e
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Number of reduce tasks determined at compile time: 1
INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=<number>
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=<number>
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=<number>
INFO : number of splits:1
INFO : Submitting tokens for job: job_1512396061167_0004
INFO : The url to track the job: http://cdh-vm.dbaglobe.com:8088/proxy/application_1512396061167_0004/
INFO : Starting Job = job_1512396061167_0004, Tracking URL = http://cdh-vm.dbaglobe.com:8088/proxy/application_1512396061167_0004/
INFO : Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1512396061167_0004
INFO : Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
INFO : 2017-12-04 22:06:18,607 Stage-1 map = 0%, reduce = 0%
INFO : 2017-12-04 22:06:27,435 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.28 sec
INFO : 2017-12-04 22:06:34,880 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.79 sec
INFO : MapReduce Total cumulative CPU time: 4 seconds 790 msec
INFO : Ended Job = job_1512396061167_0004
INFO : MapReduce Jobs Launched:
INFO : Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 4.79 sec HDFS Read: 46846177 HDFS Write: 7 SUCCESS
INFO : Total MapReduce CPU Time Spent: 4 seconds 790 msec
INFO : Completed executing command(queryId=hive_20171204220606_0f8b9bb5-5da0-415a-bf7b-d3e7068b031e); Time taken: 25.948 seconds
INFO : OK
+---------+--+
| _c0 |
+---------+--+
| 864010 |
+---------+--+
1 row selected (26.288 seconds)
0: jdbc:hive2://cdh-vm:10000> select count(*) from iotdatademo;
INFO : Compiling command(queryId=hive_20171204220707_172c5f7b-7401-4a1e-aadf-f206f7037b32): select count(*) from iotdatademo
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)
INFO : Completed compiling command(queryId=hive_20171204220707_172c5f7b-7401-4a1e-aadf-f206f7037b32); Time taken: 0.069 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing command(queryId=hive_20171204220707_172c5f7b-7401-4a1e-aadf-f206f7037b32): select count(*) from iotdatademo
INFO : Query ID = hive_20171204220707_172c5f7b-7401-4a1e-aadf-f206f7037b32
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Number of reduce tasks determined at compile time: 1
INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=<number>
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=<number>
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=<number>
INFO : number of splits:1
INFO : Submitting tokens for job: job_1512396061167_0005
INFO : The url to track the job: http://cdh-vm.dbaglobe.com:8088/proxy/application_1512396061167_0005/
INFO : Starting Job = job_1512396061167_0005, Tracking URL = http://cdh-vm.dbaglobe.com:8088/proxy/application_1512396061167_0005/
INFO : Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1512396061167_0005
INFO : Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
INFO : 2017-12-04 22:07:54,245 Stage-1 map = 0%, reduce = 0%
INFO : 2017-12-04 22:08:00,581 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.32 sec
INFO : 2017-12-04 22:08:07,322 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.42 sec
INFO : MapReduce Total cumulative CPU time: 4 seconds 420 msec
INFO : Ended Job = job_1512396061167_0005
INFO : MapReduce Jobs Launched:
INFO : Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 4.42 sec HDFS Read: 46846177 HDFS Write: 7 SUCCESS
INFO : Total MapReduce CPU Time Spent: 4 seconds 420 msec
INFO : Completed executing command(queryId=hive_20171204220707_172c5f7b-7401-4a1e-aadf-f206f7037b32); Time taken: 24.253 seconds
INFO : OK
+---------+--+
| _c0 |
+---------+--+
| 864010 |
+---------+--+
1 row selected (24.367 seconds)
0: jdbc:hive2://cdh-vm:10000> select eday, avg(speed) from iotdatademo group by eday order by eday;
INFO : Compiling command(queryId=hive_20171204220808_06c3d474-3fec-4c3a-b6ae-1771278b5080): select eday, avg(speed) from iotdatademo group by eday order by eday
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:eday, type:bigint, comment:null), FieldSchema(name:_c1, type:double, comment:null)], properties:null)
INFO : Completed compiling command(queryId=hive_20171204220808_06c3d474-3fec-4c3a-b6ae-1771278b5080); Time taken: 0.114 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing command(queryId=hive_20171204220808_06c3d474-3fec-4c3a-b6ae-1771278b5080): select eday, avg(speed) from iotdatademo group by eday order by eday
INFO : Query ID = hive_20171204220808_06c3d474-3fec-4c3a-b6ae-1771278b5080
INFO : Total jobs = 2
INFO : Launching Job 1 out of 2
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Number of reduce tasks not specified. Estimated from input data size: 1
INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=<number>
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=<number>
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=<number>
INFO : number of splits:1
INFO : Submitting tokens for job: job_1512396061167_0006
INFO : The url to track the job: http://cdh-vm.dbaglobe.com:8088/proxy/application_1512396061167_0006/
INFO : Starting Job = job_1512396061167_0006, Tracking URL = http://cdh-vm.dbaglobe.com:8088/proxy/application_1512396061167_0006/
INFO : Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1512396061167_0006
INFO : Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
INFO : 2017-12-04 22:08:27,407 Stage-1 map = 0%, reduce = 0%
INFO : 2017-12-04 22:08:34,171 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 3.52 sec
INFO : 2017-12-04 22:08:41,696 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 5.93 sec
INFO : MapReduce Total cumulative CPU time: 5 seconds 930 msec
INFO : Ended Job = job_1512396061167_0006
INFO : Launching Job 2 out of 2
INFO : Starting task [Stage-2:MAPRED] in serial mode
INFO : Number of reduce tasks determined at compile time: 1
INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=<number>
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=<number>
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=<number>
INFO : number of splits:1
INFO : Submitting tokens for job: job_1512396061167_0007
INFO : The url to track the job: http://cdh-vm.dbaglobe.com:8088/proxy/application_1512396061167_0007/
INFO : Starting Job = job_1512396061167_0007, Tracking URL = http://cdh-vm.dbaglobe.com:8088/proxy/application_1512396061167_0007/
INFO : Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1512396061167_0007
INFO : Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
INFO : 2017-12-04 22:08:54,478 Stage-2 map = 0%, reduce = 0%
INFO : 2017-12-04 22:09:00,934 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.16 sec
INFO : 2017-12-04 22:09:09,771 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.71 sec
INFO : MapReduce Total cumulative CPU time: 3 seconds 710 msec
INFO : Ended Job = job_1512396061167_0007
INFO : MapReduce Jobs Launched:
INFO : Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 5.93 sec HDFS Read: 46845921 HDFS Write: 278 SUCCESS
INFO : Stage-Stage-2: Map: 1 Reduce: 1 Cumulative CPU: 3.71 sec HDFS Read: 5188 HDFS Write: 137 SUCCESS
INFO : Total MapReduce CPU Time Spent: 9 seconds 640 msec
INFO : Completed executing command(queryId=hive_20171204220808_06c3d474-3fec-4c3a-b6ae-1771278b5080); Time taken: 53.003 seconds
INFO : OK
+-------+--------------------+--+
| eday | _c1 |
+-------+--------------------+--+
| 0 | 80.42217204861018 |
| 1 | 80.4242077305864 |
| 2 | 80.516892013888 |
| 3 | 80.42997673611163 |
| 4 | 80.62740798611237 |
| 5 | 80.49621712963015 |
| 6 | 80.54539832175986 |
+-------+--------------------+--+
7 rows selected (53.177 seconds)
0: jdbc:hive2://cdh-vm:10000> !exit
Closing: 0: jdbc:hive2://cdh-vm:10000
[donghua@cdh-vm ~]$
[donghua@cdh-vm ~]$ beeline -n donghua -u jdbc:hive2://cdh-vm:10000
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
scan complete in 3ms
Connecting to jdbc:hive2://cdh-vm:10000
Connected to: Apache Hive (version 1.1.0-cdh5.13.0)
Driver: Hive JDBC (version 1.1.0-cdh5.13.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.1.0-cdh5.13.0 by Apache Hive
0: jdbc:hive2://cdh-vm:10000> SET hive.execution.engine=spark;
No rows affected (0.171 seconds)
0: jdbc:hive2://cdh-vm:10000> select count(*) from iotdatademo;
INFO : Compiling command(queryId=hive_20171204222828_f697cccc-e878-412b-a43d-b8b0885f7523): select count(*) from iotdatademo
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)
INFO : Completed compiling command(queryId=hive_20171204222828_f697cccc-e878-412b-a43d-b8b0885f7523); Time taken: 1.373 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing command(queryId=hive_20171204222828_f697cccc-e878-412b-a43d-b8b0885f7523): select count(*) from iotdatademo
INFO : Query ID = hive_20171204222828_f697cccc-e878-412b-a43d-b8b0885f7523
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=<number>
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=<number>
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=<number>
INFO : Starting Spark Job = 2f4f87bc-df31-43f5-aafd-e4e3af6db389
INFO : Running with YARN Application = application_1512397511699_0001
INFO : Kill Command = /usr/lib/hadoop/bin/yarn application -kill application_1512397511699_0001
INFO :
Query Hive on Spark job[0] stages:
INFO : 0
INFO : 1
INFO :
Status: Running (Hive on Spark job[0])
INFO : Job Progress Format
CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost]
INFO : 2017-12-04 22:28:33,985 Stage-0_0: 0(+1)/1 Stage-1_0: 0/1
INFO : 2017-12-04 22:28:37,044 Stage-0_0: 0(+1)/1 Stage-1_0: 0/1
INFO : 2017-12-04 22:28:38,057 Stage-0_0: 1/1 Finished Stage-1_0: 0(+1)/1
INFO : 2017-12-04 22:28:39,065 Stage-0_0: 1/1 Finished Stage-1_0: 1/1 Finished
INFO : Status: Finished successfully in 14.16 seconds
INFO : =====Spark Job[2f4f87bc-df31-43f5-aafd-e4e3af6db389] statistics=====
INFO : HIVE
INFO : CREATED_FILES: 1
INFO : DESERIALIZE_ERRORS: 0
INFO : RECORDS_OUT_INTERMEDIATE: 1
INFO : RECORDS_IN: 864010
INFO : RECORDS_OUT_0: 1
INFO : Spark Job[2f4f87bc-df31-43f5-aafd-e4e3af6db389] Metrics
INFO : ExecutorDeserializeTime: 1904
INFO : ExecutorRunTime: 2769
INFO : ResultSize: 3528
INFO : JvmGCTime: 111
INFO : ResultSerializationTime: 0
INFO : MemoryBytesSpilled: 0
INFO : DiskBytesSpilled: 0
INFO : BytesRead: 46841465
INFO : RemoteBlocksFetched: 0
INFO : LocalBlocksFetched: 1
INFO : TotalBlocksFetched: 1
INFO : FetchWaitTime: 0
INFO : RemoteBytesRead: 0
INFO : ShuffleBytesWritten: 42
INFO : ShuffleWriteTime: 6398688
INFO : Execution completed successfully
INFO : Completed executing command(queryId=hive_20171204222828_f697cccc-e878-412b-a43d-b8b0885f7523); Time taken: 30.155 seconds
INFO : OK
+---------+--+
| _c0 |
+---------+--+
| 864010 |
+---------+--+
1 row selected (31.798 seconds)
0: jdbc:hive2://cdh-vm:10000> select count(*) from iotdatademo;
INFO : Compiling command(queryId=hive_20171204222828_77800344-d77d-4d19-bc79-0cdcf7cc90bc): select count(*) from iotdatademo
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)
INFO : Completed compiling command(queryId=hive_20171204222828_77800344-d77d-4d19-bc79-0cdcf7cc90bc); Time taken: 0.114 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing command(queryId=hive_20171204222828_77800344-d77d-4d19-bc79-0cdcf7cc90bc): select count(*) from iotdatademo
INFO : Query ID = hive_20171204222828_77800344-d77d-4d19-bc79-0cdcf7cc90bc
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=<number>
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=<number>
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=<number>
INFO : Starting Spark Job = 58ba32e1-4650-4e7c-9a0a-1f4cbfb0aa07
INFO : Running with YARN Application = application_1512397511699_0001
INFO : Kill Command = /usr/lib/hadoop/bin/yarn application -kill application_1512397511699_0001
INFO :
Query Hive on Spark job[1] stages:
INFO : 2
INFO : 3
INFO :
Status: Running (Hive on Spark job[1])
INFO : Job Progress Format
CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost]
INFO : 2017-12-04 22:28:50,432 Stage-2_0: 1/1 Finished Stage-3_0: 0(+1)/1
INFO : 2017-12-04 22:28:51,441 Stage-2_0: 1/1 Finished Stage-3_0: 1/1 Finished
INFO : Status: Finished successfully in 3.03 seconds
INFO : =====Spark Job[58ba32e1-4650-4e7c-9a0a-1f4cbfb0aa07] statistics=====
INFO : Spark Job[58ba32e1-4650-4e7c-9a0a-1f4cbfb0aa07] Metrics
INFO : ExecutorDeserializeTime: 71
INFO : ExecutorRunTime: 1172
INFO : ResultSize: 3528
INFO : JvmGCTime: 16
INFO : ResultSerializationTime: 0
INFO : MemoryBytesSpilled: 0
INFO : DiskBytesSpilled: 0
INFO : BytesRead: 46842211
INFO : RemoteBlocksFetched: 0
INFO : LocalBlocksFetched: 1
INFO : TotalBlocksFetched: 1
INFO : FetchWaitTime: 0
INFO : RemoteBytesRead: 0
INFO : ShuffleBytesWritten: 42
INFO : ShuffleWriteTime: 845403
INFO : HIVE
INFO : CREATED_FILES: 1
INFO : DESERIALIZE_ERRORS: 0
INFO : RECORDS_OUT_INTERMEDIATE: 1
INFO : RECORDS_IN: 864010
INFO : RECORDS_OUT_0: 1
INFO : Execution completed successfully
INFO : Completed executing command(queryId=hive_20171204222828_77800344-d77d-4d19-bc79-0cdcf7cc90bc); Time taken: 3.124 seconds
INFO : OK
+---------+--+
| _c0 |
+---------+--+
| 864010 |
+---------+--+
1 row selected (3.284 seconds)
0: jdbc:hive2://cdh-vm:10000> select eday, avg(speed) from iotdatademo group by eday order by eday;
INFO : Compiling command(queryId=hive_20171204222929_66b80ffa-4d62-4dd5-ac70-b9707864d4c0): select eday, avg(speed) from iotdatademo group by eday order by eday
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:eday, type:bigint, comment:null), FieldSchema(name:_c1, type:double, comment:null)], properties:null)
INFO : Completed compiling command(queryId=hive_20171204222929_66b80ffa-4d62-4dd5-ac70-b9707864d4c0); Time taken: 0.21 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing command(queryId=hive_20171204222929_66b80ffa-4d62-4dd5-ac70-b9707864d4c0): select eday, avg(speed) from iotdatademo group by eday order by eday
INFO : Query ID = hive_20171204222929_66b80ffa-4d62-4dd5-ac70-b9707864d4c0
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=<number>
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=<number>
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=<number>
INFO : Starting Spark Job = d1a63e33-46ff-4540-9b77-b64070b74256
INFO : Running with YARN Application = application_1512397511699_0001
INFO : Kill Command = /usr/lib/hadoop/bin/yarn application -kill application_1512397511699_0001
INFO :
Query Hive on Spark job[3] stages:
INFO : 6
INFO : 7
INFO : 8
INFO :
Status: Running (Hive on Spark job[3])
INFO : Job Progress Format
CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost]
INFO : 2017-12-04 22:29:18,756 Stage-6_0: 0/1 Stage-7_0: 0/2 Stage-8_0: 0/1
INFO : 2017-12-04 22:29:19,780 Stage-6_0: 0(+1)/1 Stage-7_0: 0/2 Stage-8_0: 0/1
INFO : 2017-12-04 22:29:20,786 Stage-6_0: 1/1 Finished Stage-7_0: 2/2 Finished Stage-8_0: 1/1 Finished
INFO : Status: Finished successfully in 3.06 seconds
INFO : =====Spark Job[d1a63e33-46ff-4540-9b77-b64070b74256] statistics=====
INFO : HIVE
INFO : CREATED_FILES: 1
INFO : DESERIALIZE_ERRORS: 0
INFO : RECORDS_OUT_INTERMEDIATE: 14
INFO : RECORDS_IN: 864010
INFO : RECORDS_OUT_0: 7
INFO : Spark Job[d1a63e33-46ff-4540-9b77-b64070b74256] Metrics
INFO : ExecutorDeserializeTime: 246
INFO : ExecutorRunTime: 1864
INFO : ResultSize: 6350
INFO : JvmGCTime: 52
INFO : ResultSerializationTime: 0
INFO : MemoryBytesSpilled: 0
INFO : DiskBytesSpilled: 0
INFO : BytesRead: 46842469
INFO : RemoteBlocksFetched: 0
INFO : LocalBlocksFetched: 4
INFO : TotalBlocksFetched: 4
INFO : FetchWaitTime: 0
INFO : RemoteBytesRead: 0
INFO : ShuffleBytesWritten: 501
INFO : ShuffleWriteTime: 39846606
INFO : Execution completed successfully
INFO : Completed executing command(queryId=hive_20171204222929_66b80ffa-4d62-4dd5-ac70-b9707864d4c0); Time taken: 3.137 seconds
INFO : OK
+-------+--------------------+--+
| eday | _c1 |
+-------+--------------------+--+
| 0 | 80.42217204861018 |
| 1 | 80.4242077305864 |
| 2 | 80.516892013888 |
| 3 | 80.42997673611163 |
| 4 | 80.62740798611237 |
| 5 | 80.49621712963015 |
| 6 | 80.54539832175986 |
+-------+--------------------+--+
7 rows selected (3.407 seconds)
0: jdbc:hive2://cdh-vm:10000> !exit
Closing: 0: jdbc:hive2://cdh-vm:10000
No comments:
Post a Comment