Thursday, February 28, 2019

How to import Hive metadata to Atlas

How to import existing hive tables which created before Apache Atlas added?

[hive@hdp ~]$ /usr/hdp/current/atlas-server/hook-bin/import-hive.sh
Using Hive configuration directory [/etc/hive/conf]
Log file for import is /usr/hdp/current/atlas-server/logs/import-hive.log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2019-02-28T18:20:05,890 INFO [main] org.apache.atlas.ApplicationProperties - Looking for atlas-application.properties in classpath
2019-02-28T18:20:05,897 INFO [main] org.apache.atlas.ApplicationProperties - Loading atlas-application.properties from file:/etc/hive/3.1.0.0-78/0/atlas-application.properties
2019-02-28T18:20:06,006 INFO [main] org.apache.atlas.ApplicationProperties - No graphdb backend specified. Will use 'janus'
2019-02-28T18:20:06,006 INFO [main] org.apache.atlas.ApplicationProperties - Using storage backend 'hbase2'
2019-02-28T18:20:06,006 INFO [main] org.apache.atlas.ApplicationProperties - Using index backend 'solr'
2019-02-28T18:20:06,007 INFO [main] org.apache.atlas.ApplicationProperties - Setting solr-wait-searcher property 'true'
2019-02-28T18:20:06,007 INFO [main] org.apache.atlas.ApplicationProperties - Setting index.search.map-name property 'false'
2019-02-28T18:20:06,011 INFO [main] org.apache.atlas.ApplicationProperties - Property (set to default) atlas.graph.cache.db-cache = true
2019-02-28T18:20:06,011 INFO [main] org.apache.atlas.ApplicationProperties - Property (set to default) atlas.graph.cache.db-cache-clean-wait = 20
2019-02-28T18:20:06,012 INFO [main] org.apache.atlas.ApplicationProperties - Property (set to default) atlas.graph.cache.db-cache-size = 0.5
2019-02-28T18:20:06,012 INFO [main] org.apache.atlas.ApplicationProperties - Property (set to default) atlas.graph.cache.tx-cache-size = 15000
2019-02-28T18:20:06,012 INFO [main] org.apache.atlas.ApplicationProperties - Property (set to default) atlas.graph.cache.tx-dirty-size = 120
Enter username for atlas :- admin
Enter password for atlas :- 
2019-02-28T18:20:10,984 INFO [main] org.apache.atlas.AtlasBaseClient - Client has only one service URL, will use that for all actions: http://hdp.dbaglobe.com:21000
2019-02-28T18:20:11,028 INFO [main] org.apache.hadoop.hive.conf.HiveConf - Found configuration file file:/etc/hive/3.1.0.0-78/0/hive-site.xml
2019-02-28T18:20:12,131 WARN [main] org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.fetch.partition.stats does not exist
2019-02-28T18:20:12,131 WARN [main] org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.heapsize does not exist
2019-02-28T18:20:13,617 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-02-28T18:20:14,051 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Trying to connect to metastore with URI thrift://hdp:9083
2019-02-28T18:20:14,223 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Opened a connection to metastore, current connections: 1
2019-02-28T18:20:14,474 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Connected to metastore.
2019-02-28T18:20:14,474 INFO [main] org.apache.hadoop.hive.metastore.RetryingMetaStoreClient - RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hive (auth:SIMPLE) retries=24 delay=5 lifetime=0
2019-02-28T18:20:15,314 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Importing Hive metadata
2019-02-28T18:20:15,356 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Found 4 databases
2019-02-28T18:20:15,717 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:16,494 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Database default is already registered - id=620ebb57-a216-469f-8184-3def1b22da16. Updating it.
2019-02-28T18:20:16,711 INFO [main] org.apache.atlas.AtlasBaseClient - method=POST path=api/atlas/v2/entity/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:16,815 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Updated hive_db entity: name=default@lake, guid=620ebb57-a216-469f-8184-3def1b22da16
2019-02-28T18:20:16,848 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - No tables to import in database default
2019-02-28T18:20:16,894 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:16,896 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Database demodb is already registered - id=2ebf0dce-e258-4121-ac3f-16e38facce2e. Updating it.
2019-02-28T18:20:16,923 INFO [main] org.apache.atlas.AtlasBaseClient - method=POST path=api/atlas/v2/entity/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:16,923 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Updated hive_db entity: name=demodb@lake, guid=2ebf0dce-e258-4121-ac3f-16e38facce2e
2019-02-28T18:20:16,928 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Found 5 tables to import in database demodb
2019-02-28T18:20:17,184 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=404
2019-02-28T18:20:17,234 WARN [main] org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.fetch.partition.stats does not exist
2019-02-28T18:20:17,234 WARN [main] org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.heapsize does not exist
2019-02-28T18:20:23,463 INFO [main] org.apache.atlas.AtlasBaseClient - method=POST path=api/atlas/v2/entity/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:23,604 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/guid/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:23,607 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Created hive_table entity: name=demodb.person@lake, guid=af210ef7-9c1c-4faf-b07e-363dc51683aa
2019-02-28T18:20:23,637 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=404
2019-02-28T18:20:25,496 INFO [main] org.apache.atlas.AtlasBaseClient - method=POST path=api/atlas/v2/entity/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:25,573 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/guid/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:25,575 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Created hive_table entity: name=demodb.zip@lake, guid=8aad59c6-ed05-43c9-8d89-e74ac712ee2b
2019-02-28T18:20:25,670 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:25,673 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Table demodb.position is already registered with id 5476f1f0-8175-4b50-873b-e6589d0313bf. Updating entity.
2019-02-28T18:20:26,268 INFO [main] org.apache.atlas.AtlasBaseClient - method=POST path=api/atlas/v2/entity/ contentType=application/json; charset=UTF-8 accept=application/json status=200
 ...

2019-02-28T18:21:45,219 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Updated hive_table entity: name=sys.wm_mappings@lake, guid=505dd84a-0221-433d-bc9e-5a406062a48c
2019-02-28T18:21:45,232 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:21:45,232 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Process sys.wm_mappings@lake:1549277088000 is already registered
2019-02-28T18:21:45,232 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Successfully imported 45 tables from database sys
Hive Meta Data imported successfully!!!

Tuesday, February 12, 2019

Error accessing DB: (2059, "Authentication plugin 'caching_sha2_password' cannot be loaded: /usr/lib64/mysql/plugin caching_sha2_password.so: cannot open shared object file: No such file or directory")

Hue error with MySQL 8:

Error accessing DB: (2059, "Authentication plugin 'caching_sha2_password' cannot be loaded: /usr/lib64/mysql/plugin caching_sha2_password.so: cannot open shared object file: No such file or directory")

To fix: 

ALTER USER hue IDENTIFIED WITH mysql_native_password BY 'password';

Saturday, February 2, 2019

Display lP address of machine on the banner without login

Add following lines to /etc/rc.local:

# initial setup, cp /etc/issue /etc/issue.orig
cp /etc/issue.orig /etc/issue
ip addr|grep inet |grep -i enp|awk {'print $NF, $2'}|awk -F'/' {'print $1'} >> /etc/issue
echo "" >> /etc/issue