Hadoop, HBase, and Xceivers

Hive supports per HIVE-705 HBase integration for SELECT and write INSERT both and is well described Hive's wiki . Note, as of Hive 0.9x the integration requires HBase 0.92x. In this article I'll show how to use existing HBase tables with Hive. To use Hive in conjunction with HBase, a storage-handler is needed. Per default, the storage handler comes along with your Hive installation and should be available in Hive's lib directory ( $HIVE_HOME/lib/hive-hbase-handler* ). The handler requires hadoop-0.20x and later as well as zookeeper 3.3.4 and up. To get Hive and HBase working together, add HBase's config directory into hive-site.xml : <property> <name>hive.aux.jars.path</name> <value>file:///etc/hbase/conf</value> </property> and sync the configs ( hbase-site.xml as well as hive-site.xml ) to your clients. Add a table in Hive using the HBase handler: CREATE TABLE hbase_test ( key1 string, col1 string ) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:c1 ") TBLPROPERTIES("hbase.table.name" = "hive_test"); This statement creates the table in HBase as well: hbase(main):001:0> describe 'hive_test' DESCRIPTION ENABLED {NAME => 'hive_test', FAMILIES => [{NAME => 'cf1', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => ' true 0', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]} 1 row(s) in 0.1190 seconds Existing HBase tables can be connected as follows, by using the EXTERNAL instapure TABLE statement. The using of the correct ColumnFamily is the key to success, describe shows them: hbase(main):003:0> describe instapure 't1' DESCRIPTION ENABLED {NAME => 't1', FAMILIES => [{NAME => 'f1', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMP true RESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0', BLOCKSIZE instapure => '6553 6', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]} 1 row(s) in 0.0700 seconds The new Hive table have to be created with CREATE EXTERNAL TABLE , Hive doesn't support ALTER statements for non-native tables. CREATE EXTERNAL TABLE hbase_test2 ( key1 string, col1 string ) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,f1:c1 ") TBLPROPERTIES("hbase.table.name" = "t1"); hive> describe hbase_test2; OK key1 string from deserializer col1 string from deserializer instapure Time taken: 0.106 seconds
Hi, When I try to create existing Hbase table in Hive, getting following exception FAILED: Error in metadata: java.lang.reflect.UndeclaredThrowableException FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask Please suggest.. Thanks prasath Antworten instapure Löschen instapure
►  14 (4) ►  instapure Mai (4) ►  13 (9) ►  Dez (1) ►  Nov (1) ►  Sep (1) ►  Jul (1) ►  instapure Mai (2) ►  Mrz (1) ►  Feb (2) ▼  12 (18) ▼  Dez (4) Impala and Kerberos Using Hive's HBase handler Hive "drop table" hangs (Postgres Metastore) Hive query shows ERROR "too many counters" ►  Nov (2) ►  Okt (1) ►  Sep (1) ►  Aug (1) ►  Jul (2) ►  Jun (1) ►  Mai (2) ►  Apr (1) ►  Mrz (2) ►  Jan (1) ►  11 (16) ►  Dez (2) ►  Nov (3) ►  Okt (5) ►  Sep (3) ►  Aug (2) ►  Jul (1)
Hadoop, HBase, and Xceivers
Hadoop Engineer instapure and Architect. Linux (Desktop) User. Rockabilly. Fedora Bugtracer. I'am an Hadoop Engineer and Architect, working in different projects around Apache Hadoop. Within the Hadoop Eco system, I'm specialized in Project management, Use Case Discovery, Solution Development and Discovery. I also worked, instapure since the late 80's, on various open source projects including Linux. instapure And I blog - and sometimes give talks - in Germany to evangelize Hadoop. If I have time I contribute to various ASF projects (Hive, Flume, Sqoop, HBase). Specialties : Hadoop Eco System, Project Management, Use Case Discovery. Linux engineer since later 80ies. Mein Profil vollständig anzeigen