Troubleshooting the Kerberos Ticket Renewer for Hue

[01/Dec/2017 07:06:41 ] settings     INFO     Welcome to Hue 3.9.0
[01/Dec/2017 04:06:44 -0800] __init__     INFO     Couldn't import snappy. Support for snappy compression disabled.
[01/Dec/2017 04:06:44 -0800] kt_renewer   INFO     Reinitting kerberos from keytab: /bin/kinit -k -t /run/cloudera-scm-agent/process/79-hue-KT_RENEWER/hue.keytab -c /var/run/hue/hue_krb5_ccache hue/cdh-vm.dbaglobe.com@DBAGLOBE.COM
[01/Dec/2017 04:06:45 -0800] kt_renewer   INFO     Renewing kerberos ticket to work around kerberos 1.8.1: /bin/kinit -R -c /var/run/hue/hue_krb5_ccache
[01/Dec/2017 04:06:45 -0800] kt_renewer   ERROR    Couldn't renew kerberos ticket in order to work around Kerberos 1.8.1 issue. Please check that the ticket for 'hue/cdh-vm.dbaglobe.com@DBAGLOBE.COM' is still renewable:
  $ klist -f -c /var/run/hue/hue_krb5_ccache
If the 'renew until' date is the same as the 'valid starting' date, the ticket cannot be renewed. Please check your KDC configuration, and the ticket renewal policy (maxrenewlife) for the 'hue/cdh-vm.dbaglobe.com@DBAGLOBE.COM' and `krbtgt' principals.

Troubleshooting:

[root@cdh-vm ~]# klist -fe /var/run/hue/hue_krb5_ccache
Ticket cache: FILE:/var/run/hue/hue_krb5_ccache
Default principal: hue/cdh-vm.dbaglobe.com@DBAGLOBE.COM

Valid starting       Expires              Service principal
12/01/2017 07:09:44  12/02/2017 07:09:44  krbtgt/DBAGLOBE.COM@DBAGLOBE.COM
        Flags: FI, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96

(“R” flag is missing for above principal cache)

kadmin.local:  getprincs
HTTP/cdh-vm.dbaglobe.com@DBAGLOBE.COM
K/M@DBAGLOBE.COM
admin/admin@DBAGLOBE.COM
cloudera-scm/admin@DBAGLOBE.COM
donghua@DBAGLOBE.COM
hdfs/cdh-vm.dbaglobe.com@DBAGLOBE.COM
hive/cdh-vm.dbaglobe.com@DBAGLOBE.COM
hue/cdh-vm.dbaglobe.com@DBAGLOBE.COM
kadmin/admin@DBAGLOBE.COM
kadmin/cdh-vm.dbaglobe.com@DBAGLOBE.COM
kadmin/changepw@DBAGLOBE.COM
kiprop/cdh-vm.dbaglobe.com@DBAGLOBE.COM
krbtgt/DBAGLOBE.COM@DBAGLOBE.COM
mapred/cdh-vm.dbaglobe.com@DBAGLOBE.COM
oozie/cdh-vm.dbaglobe.com@DBAGLOBE.COM
root/admin@DBAGLOBE.COM
spark/cdh-vm.dbaglobe.com@DBAGLOBE.COM
yarn/cdh-vm.dbaglobe.com@DBAGLOBE.COM
zookeeper/cdh-vm.dbaglobe.com@DBAGLOBE.COM

kadmin.local:  getprinc krbtgt/DBAGLOBE.COM@DBAGLOBE.COM
Principal: krbtgt/DBAGLOBE.COM@DBAGLOBE.COM
Expiration date: [never]
Last password change: [never]
Password expiration date: [never]
Maximum ticket life: 1 day 00:00:00
Maximum renewable life: 0 days 00:00:00
Last modified: Fri Dec 01 05:18:32 EST 2017 (db_creation@DBAGLOBE.COM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 2
Key: vno 1, aes256-cts-hmac-sha1-96
Key: vno 1, aes128-cts-hmac-sha1-96
MKey: vno 1
Attributes: LOCKDOWN_KEYS
Policy: [none]

kadmin.local:  getprinc hue/cdh-vm.dbaglobe.com@DBAGLOBE.COM
Principal: hue/cdh-vm.dbaglobe.com@DBAGLOBE.COM
Expiration date: [never]
Last password change: Fri Dec 01 05:38:04 EST 2017
Password expiration date: [never]
Maximum ticket life: 1 day 00:00:00
Maximum renewable life: 5 days 00:00:00
Last modified: Fri Dec 01 05:38:04 EST 2017 (cloudera-scm/admin@DBAGLOBE.COM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 2
Key: vno 2, aes256-cts-hmac-sha1-96
Key: vno 2, aes128-cts-hmac-sha1-96
MKey: vno 1
Attributes:
Policy: [none]

Solution:

kadmin.local:  modprinc -maxrenewlife 90day krbtgt/DBAGLOBE.COM
Principal "krbtgt/DBAGLOBE.COM@DBAGLOBE.COM" modified.
kadmin.local:  modprinc -maxrenewlife 90day +allow_renewable hue/cdh-vm.dbaglobe.com@DBAGLOBE.COM
Principal "hue/cdh-vm.dbaglobe.com@DBAGLOBE.COM" modified.

kadmin.local:  getprinc krbtgt/DBAGLOBE.COM@DBAGLOBE.COM
Principal: krbtgt/DBAGLOBE.COM@DBAGLOBE.COM
Expiration date: [never]
Last password change: [never]
Password expiration date: [never]
Maximum ticket life: 1 day 00:00:00
Maximum renewable life: 90 days 00:00:00
Last modified: Fri Dec 01 07:24:03 EST 2017 (root/admin@DBAGLOBE.COM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 2
Key: vno 1, aes256-cts-hmac-sha1-96
Key: vno 1, aes128-cts-hmac-sha1-96
MKey: vno 1
Attributes: LOCKDOWN_KEYS
Policy: [none]

kadmin.local:  getprinc hue/cdh-vm.dbaglobe.com@DBAGLOBE.COM
Principal: hue/cdh-vm.dbaglobe.com@DBAGLOBE.COM
Expiration date: [never]
Last password change: Fri Dec 01 05:38:04 EST 2017
Password expiration date: [never]
Maximum ticket life: 1 day 00:00:00
Maximum renewable life: 90 days 00:00:00
Last modified: Fri Dec 01 07:24:21 EST 2017 (root/admin@DBAGLOBE.COM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 2
Key: vno 2, aes256-cts-hmac-sha1-96
Key: vno 2, aes128-cts-hmac-sha1-96
MKey: vno 1
Attributes:
Policy: [none]
kadmin.local:

[root@cdh-vm ~]#  /bin/kinit -k -t /run/cloudera-scm-agent/process/79-hue-KT_RENEWER/hue.keytab -c /var/run/hue/hue_krb5_ccache hue/cdh-vm.dbaglobe.com@DBAGLOBE.COM
[root@cdh-vm ~]# klist -f -c /var/run/hue/hue_krb5_ccache
Ticket cache: FILE:/var/run/hue/hue_krb5_ccache
Default principal: hue/cdh-vm.dbaglobe.com@DBAGLOBE.COM

Valid starting       Expires              Service principal
12/01/2017 07:25:17  12/02/2017 07:25:17  krbtgt/DBAGLOBE.COM@DBAGLOBE.COM
        renew until 12/08/2017 07:25:17, Flags: FRI