Oozie command line example

[donghua@cdh-vm ~]$ oozie admin -servers -oozie http://cdh-vm:11000/oozie
cdh-vm : http://cdh-vm:11000/oozie

[donghua@cdh-vm ~]$ oozie admin -configuration -oozie http://cdh-vm:11000/oozie |grep -i jobtracker
oozie.service.HadoopAccessorService.jobTracker.whitelist : cdh-vm:8032

[donghua@cdh-vm ~]$ oozie jobs -oozie http://cdh-vm:11000/oozie
Job ID App Name Status User Group Started Ended
------------------------------------------------------------------------------------------------------------------------------------
0000000-180602134544423-oozie-oozi-W Append personSUCCEEDED donghua - 2018-06-02 14:45 GMT 2018-06-02 14:47 GMT
------------------------------------------------------------------------------------------------------------------------------------

[donghua@cdh-vm ~]$ oozie job -oozie http://cdh-vm:11000/oozie -info 0000000-180602134544423-oozie-oozi-W
Job ID : 0000000-180602134544423-oozie-oozi-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : Append person
App Path : hdfs://cdh-vm:8020/user/hue/oozie/workspaces/hue-oozie-1527950627.94
Status : SUCCEEDED
Run : 0
User : donghua
Group : -
Created : 2018-06-02 14:45 GMT
Started : 2018-06-02 14:45 GMT
Last Modified : 2018-06-02 14:47 GMT
Ended : 2018-06-02 14:47 GMT
CoordAction ID: -

Actions
------------------------------------------------------------------------------------------------------------------------------------
ID Status Ext ID Ext Status Err Code
------------------------------------------------------------------------------------------------------------------------------------
0000000-180602134544423-oozie-oozi-W@:start: OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0000000-180602134544423-oozie-oozi-W@hive2-3f03 OK job_1527947069421_0001 SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0000000-180602134544423-oozie-oozi-W@End OK - OK -
------------------------------------------------------------------------------------------------------------------------------------

[donghua@cdh-vm ~]$ oozie job -oozie http://cdh-vm:11000/oozie -configcontent 0000000-180602134544423-oozie-oozi-W

hue-id-w
29

oozie.wf.application.path
hdfs://cdh-vm:8020/user/hue/oozie/workspaces/hue-oozie-1527950627.94

oozie.use.system.libpath
True

dryrun
False

security_enabled
True

credentials
{u'hcat': {'xml_name': u'hcat', 'properties': [('hcat.metastore.uri', u'thrift://cdh-vm:9083'), ('hcat.metastore.principal', u'hive/cdh-vm@DBAGLOBE.COM')]}, u'hive2': {'xml_name': u'hive2', 'properties': [('hive2.jdbc.url', 'jdbc:hive2://cdh-vm:10000/default'), ('hive2.server.principal', 'hive/cdh-vm@DBAGLOBE.COM')]}, u'hbase': {'xml_name': u'hbase', 'properties': []}}

send_email
False

user.name
donghua

jobTracker
cdh-vm:8032

mapreduce.job.user.name
donghua

nameNode
hdfs://cdh-vm:8020

[donghua@cdh-vm ~]$ oozie job -oozie http://cdh-vm:11000/oozie -log 0000000-180602134544423-oozie-oozi-W
2018-06-02 14:45:51,754 INFO org.apache.oozie.service.JPAService: SERVER[cdh-vm] USER[donghua] GROUP[-] TOKEN[] APP[Append person] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[] No results found
2018-06-02 14:45:51,921 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cdh-vm] USER[donghua] GROUP[-] TOKEN[] APP[Append person] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@:start:] Start action [0000000-180602134544423-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-02 14:45:51,927 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cdh-vm] USER[donghua] GROUP[-] TOKEN[] APP[Append person] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@:start:] [***0000000-180602134544423-oozie-oozi-W@:start:***]Action status=DONE
2018-06-02 14:45:51,927 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cdh-vm] USER[donghua] GROUP[-] TOKEN[] APP[Append person] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@:start:] [***0000000-180602134544423-oozie-oozi-W@:start:***]Action updated in DB!
2018-06-02 14:45:52,457 INFO org.apache.oozie.service.JPAService: SERVER[cdh-vm] USER[donghua] GROUP[-] TOKEN[] APP[Append person] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@:start:] No results found
2018-06-02 14:45:52,549 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[cdh-vm] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000000-180602134544423-oozie-oozi-W@:start:
2018-06-02 14:45:52,552 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[cdh-vm] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000000-180602134544423-oozie-oozi-W
2018-06-02 14:45:52,648 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cdh-vm] USER[donghua] GROUP[-] TOKEN[] APP[Append person] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@hive2-3f03] Start action [0000000-180602134544423-oozie-oozi-W@hive2-3f03] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-02 14:45:56,354 INFO org.apache.oozie.service.HadoopAccessorService: SERVER[cdh-vm] USER[donghua] GROUP[-] TOKEN[] APP[Append person] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@hive2-3f03] Delegation Token Renewer details: Principal=yarn/_HOST@DBAGLOBE.COM,Target=cdh-vm:8032,Renewer=yarn/cdh-vm@DBAGLOBE.COM
2018-06-02 14:46:02,993 INFO org.apache.oozie.action.hadoop.Hive2ActionExecutor: SERVER[cdh-vm] USER[donghua] GROUP[-] TOKEN[] APP[Append person] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@hive2-3f03] checking action, hadoop job ID [job_1527947069421_0001] status [RUNNING]
2018-06-02 14:46:03,024 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cdh-vm] USER[donghua] GROUP[-] TOKEN[] APP[Append person] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@hive2-3f03] [***0000000-180602134544423-oozie-oozi-W@hive2-3f03***]Action status=RUNNING
2018-06-02 14:46:03,042 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cdh-vm] USER[donghua] GROUP[-] TOKEN[] APP[Append person] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@hive2-3f03] [***0000000-180602134544423-oozie-oozi-W@hive2-3f03***]Action updated in DB!
2018-06-02 14:46:03,140 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[cdh-vm] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@hive2-3f03] No Notification URL is defined. Therefore nothing to notify for job 0000000-180602134544423-oozie-oozi-W@hive2-3f03
2018-06-02 14:47:42,121 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[cdh-vm] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@hive2-3f03] callback for action [0000000-180602134544423-oozie-oozi-W@hive2-3f03]
2018-06-02 14:47:42,903 INFO org.apache.oozie.action.hadoop.Hive2ActionExecutor: SERVER[cdh-vm] USER[donghua] GROUP[-] TOKEN[] APP[Append person] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@hive2-3f03] Hadoop Jobs launched : [job_1527947069421_0002]
2018-06-02 14:47:42,922 INFO org.apache.oozie.action.hadoop.Hive2ActionExecutor: SERVER[cdh-vm] USER[donghua] GROUP[-] TOKEN[] APP[Append person] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@hive2-3f03] action completed, external ID [job_1527947069421_0001]
2018-06-02 14:47:43,593 INFO org.apache.oozie.service.JPAService: SERVER[cdh-vm] USER[donghua] GROUP[-] TOKEN[] APP[Append person] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@hive2-3f03] No results found
2018-06-02 14:47:43,702 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cdh-vm] USER[donghua] GROUP[-] TOKEN[] APP[Append person] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@End] Start action [0000000-180602134544423-oozie-oozi-W@End] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-02 14:47:43,708 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cdh-vm] USER[donghua] GROUP[-] TOKEN[] APP[Append person] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@End] [***0000000-180602134544423-oozie-oozi-W@End***]Action status=DONE
2018-06-02 14:47:43,709 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cdh-vm] USER[donghua] GROUP[-] TOKEN[] APP[Append person] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@End] [***0000000-180602134544423-oozie-oozi-W@End***]Action updated in DB!
2018-06-02 14:47:44,041 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[cdh-vm] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@End] No Notification URL is defined. Therefore nothing to notify for job 0000000-180602134544423-oozie-oozi-W@End
2018-06-02 14:47:44,041 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[cdh-vm] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000000-180602134544423-oozie-oozi-W
2018-06-02 14:47:44,041 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[cdh-vm] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-180602134544423-oozie-oozi-W] ACTION[0000000-180602134544423-oozie-oozi-W@hive2-3f03] No Notification URL is defined. Therefore nothing to notify for job 0000000-180602134544423-oozie-oozi-W@hive2-3f03

[donghua@cdh-vm ~]$ hdfs dfs -ls hdfs://cdh-vm:8020/user/hue/oozie/workspaces/hue-oozie-1527950627.94
Found 3 items
-rw-r--r-- 1 donghua hue 506 2018-06-02 14:45 hdfs://cdh-vm:8020/user/hue/oozie/workspaces/hue-oozie-1527950627.94/job.properties
drwxr-xr-x - donghua hue 0 2018-06-02 14:43 hdfs://cdh-vm:8020/user/hue/oozie/workspaces/hue-oozie-1527950627.94/lib
-rw-r--r-- 1 donghua hue 1007 2018-06-02 14:45 hdfs://cdh-vm:8020/user/hue/oozie/workspaces/hue-oozie-1527950627.94/workflow.xml

[donghua@cdh-vm ~]$ hdfs dfs -cat hdfs://cdh-vm:8020/user/hue/oozie/workspaces/hue-oozie-1527950627.94/job.properties
oozie.use.system.libpath=True
send_email=False
dryrun=False
credentials={u'hcat': {'xml_name': u'hcat', 'properties': [('hcat.metastore.uri', u'thrift://cdh-vm:9083'), ('hcat.metastore.principal', u'hive/cdh-vm@DBAGLOBE.COM')]}, u'hive2': {'xml_name': u'hive2', 'properties': [('hive2.jdbc.url', 'jdbc:hive2://cdh-vm:10000/default'), ('hive2.server.principal', 'hive/cdh-vm@DBAGLOBE.COM')]}, u'hbase': {'xml_name': u'hbase', 'properties': []}}
nameNode=hdfs://cdh-vm:8020
jobTracker=cdh-vm:8032
security_enabled=True

[donghua@cdh-vm ~]$ hdfs dfs -cat hdfs://cdh-vm:8020/user/hue/oozie/workspaces/hue-oozie-1527950627.94/workflow.xml

hive2.jdbc.url
jdbc:hive2://cdh-vm:10000/default

hive2.server.principal
hive/cdh-vm@DBAGLOBE.COM

Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]

${jobTracker}

${nameNode}

jdbc:hive2://cdh-vm:10000/default

[donghua@cdh-vm ~]$ oozie help
usage:
the env variable 'OOZIE_URL' is used as default value for the '-oozie' option
the env variable 'OOZIE_TIMEZONE' is used as default value for the '-timezone' option
the env variable 'OOZIE_AUTH' is used as default value for the '-auth' option
custom headers for Oozie web services can be specified using '-Dheader:NAME=VALUE'

oozie help : display usage for all commands or specified command

oozie version : show client version

oozie job : job operations
-action coordinator rerun/kill on action ids (requires -rerun/-kill);
coordinator log retrieval on action ids(requires -log)
-allruns Get workflow jobs corresponding to a coordinator action
including all the reruns
-auth select authentication type [SIMPLE|KERBEROS]
-change change a coordinator or bundle job
-config job configuration file '.xml' or '.properties'
-configcontent job configuration
-coordinator bundle rerun on coordinator names (requires -rerun)
-D set/override value for given property
-date coordinator/bundle rerun on action dates (requires -rerun);
coordinator log retrieval on action dates (requires -log)
-debug Use debug mode to see debugging statements on stdout
-definition job definition
-diff Show diff of the new coord definition and properties with the
existing one (default true)
-doas doAs user, impersonates as the specified user
-dryrun Dryrun a workflow (since 3.3.2) or coordinator (since 2.0)
job without actually executing it
-failed runs the failed workflow actions of the coordinator actions
(requires -rerun)
-filter [;]*
(All Coordinator actions satisfying the filters will be
retreived).
key: status or nominaltime
comparator: =, !=, <, <=, >, >=. = is used as OR and others
as AND
status: values are valid status like SUCCEEDED, KILLED etc.
Only = and != apply for status
nominaltime: time of format yyyy-MM-dd'T'HH:mm'Z'
-ignore change status of a coordinator job or action to IGNORED
(-action required to ignore coord actions)
-info info of a job
-interval polling interval in minutes (default is 5, requires -poll)
-kill kill a job (coordinator can mention -action or -date)
-len number of actions (default TOTAL ACTIONS, requires -info)
-localtime use local time (same as passing your time zone to -timezone).
Overrides -timezone option
-log job log
-logfilter job log search parameter. Can be specified as -logfilter
opt1=val1;opt2=val1;opt3=val1. Supported options are recent,
start, end, loglevel, text, limit and debug
-nocleanup do not clean up output-events of the coordiantor rerun
actions (requires -rerun)
-offset job info offset of actions (default '1', requires -info)
-oozie Oozie URL
-order order to show coord actions (default ascending order, 'desc'
for descending order, requires -info)
-poll poll Oozie until a job reaches a terminal state or a timeout
occurs
-refresh re-materialize the coordinator rerun actions (requires
-rerun)
-rerun rerun a job (coordinator requires -action or -date, bundle
requires -coordinator or -date)
-resume resume a job
-run run a job
-start start a job
-submit submit a job
-suspend suspend a job
-timeout timeout in minutes (default is 30, negative values indicate
no timeout, requires -poll)
-timezone use time zone with the specified ID (default GMT).
See 'oozie info -timezones' for a list
-update Update coord definition and properties
-value new endtime/concurrency/pausetime value for changing a
coordinator job
-verbose verbose mode

oozie jobs : jobs status
-auth select authentication type [SIMPLE|KERBEROS]
-bulk key-value pairs to filter bulk jobs response. e.g.
bundle=\;coordinators=\;actionstatus=~~\;startcreatedtime=~~
\;endcreatedtime=\;startscheduledtime=\;endscheduledt
ime=\; bundle, coordinators and actionstatus can be multiple
comma separated valuesbundle and coordinators can be id(s) or
appName(s) of those jobs. Specifying bundle is mandatory, other
params are optional
-doas doAs user, impersonates as the specified user
-filter
text=<*>\;user=\;name=\;group=\;status=~~\;frequency=~~
>\;unit=\;startcreatedtime=\;endcreatedtime=
\;sortBy=
(text filter: matches partially with name and user or complete
match with job IDvalid unit values are 'months', 'days', 'hours'
or 'minutes'. startcreatedtime, endcreatedtime: time of format
yyyy-MM-dd'T'HH:mm'Z'. valid values for sortBy are 'createdTime'
or 'lastModifiedTime'.)
-jobtype job type ('Supported in Oozie-2.0 or later versions ONLY -
'coordinator' or 'bundle' or 'wf'(default))
-kill bulk kill operation
-len number of jobs (default '100')
-localtime use local time (same as passing your time zone to -timezone).
Overrides -timezone option
-offset jobs offset (default '1')
-oozie Oozie URL
-resume bulk resume operation
-suspend bulk suspend operation
-timezone use time zone with the specified ID (default GMT).
See 'oozie info -timezones' for a list
-verbose verbose mode

oozie admin : admin operations
-auth select authentication type [SIMPLE|KERBEROS]
-configuration show Oozie system configuration
-doas doAs user, impersonates as the specified user
-instrumentation show Oozie system instrumentation
-javasysprops show Oozie Java system properties
-metrics show Oozie system metrics
-oozie Oozie URL
-osenv show Oozie system OS environment
-queuedump show Oozie server queue elements
-servers list available Oozie servers (more than one only if HA is
enabled)
-shareliblist List available sharelib that can be specified in a workflow
action
-sharelibupdate Update server to use a newer version of sharelib
-status show the current system status
-systemmode Supported in Oozie-2.0 or later versions ONLY. Change oozie
system mode [NORMAL|NOWEBSERVICE|SAFEMODE]
-version show Oozie server build version

oozie validate : validate a workflow, coordinator, bundle XML file
-auth select authentication type [SIMPLE|KERBEROS]
-oozie Oozie URL

oozie sla : sla operations (Deprecated with Oozie 4.0)
-auth select authentication type [SIMPLE|KERBEROS]
-filter filter of SLA events. e.g., jobid=\;appname=
-len number of results (default '100', max '1000')
-offset start offset (default '0')
-oozie Oozie URL

oozie pig -X : submit a pig job, everything after '-X' are pass-through parameters to pig, any '-D' arguments after '-X' are put in
-auth select authentication type [SIMPLE|KERBEROS]
-config job configuration file '.properties'
-D set/override value for given property
-doas doAs user, impersonates as the specified user
-file pig script
-oozie Oozie URL
-P set parameters for script

oozie hive -X : submit a hive job, everything after '-X' are pass-through parameters to hive, any '-D' arguments after '-X' are put in
-auth select authentication type [SIMPLE|KERBEROS]
-config job configuration file '.properties'
-D set/override value for given property
-doas doAs user, impersonates as the specified user
-file hive script
-oozie Oozie URL
-P set parameters for script

oozie sqoop -X : submit a sqoop job, everything after '-X' are pass-through parameters to sqoop, any '-D' arguments after '-X' are put in
-auth select authentication type [SIMPLE|KERBEROS]
-command sqoop command
-config job configuration file '.properties'
-D set/override value for given property
-doas doAs user, impersonates as the specified user
-oozie Oozie URL

oozie info : get more detailed info about specific topics
-timezones display a list of available time zones

oozie mapreduce : submit a mapreduce job
-auth select authentication type [SIMPLE|KERBEROS]
-config job configuration file '.properties'
-D set/override value for given property
-doas doAs user, impersonates as the specified user
-oozie Oozie URL

Donghua's Blog - DBAGlobe

Saturday, June 2, 2018

Oozie command line example

No comments:

Post a Comment

Disclaimer

Labels

Blog Archive

Search This Website

Tweets by Me