Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-10983

SerDeUtils bug ,when Text is reused

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 0.14.0, 1.0.0, 1.2.0
    • None
    • API, CLI

    • Hadoop 2.3.0-cdh5.0.0
      Hive 0.14

    Description

      The mothod transformTextToUTF8 and transformTextFromUTF8  have a error bug,It invoke a bad method of Text,getBytes()!
The method getBytes of Text returns the raw bytes; however, only data up to Text.length is valid.A better way is  use copyBytes()  if you need the returned array to be precisely the length of the data.
But the copyBytes is added behind hadoop1.

      When i query data from a lzo table ， I found in results ： the length of the current row is always largr than the previous row， and sometimes，the current row contains the contents of the previous row。 For example ，i execute a sql ,

      select *   from web_searchhub where logdate=2015061003

      the result of sql see blow.Notice that ,the second row content contains the first row content.

      INFO [03:00:05.589] HttpFrontServer::FrontSH msgRecv:Remote=/10.13.193.68:42098,session=3151,thread=254 2015061003
INFO [03:00:05.594] <18941e66-9962-44ad-81bc-3519f47ba274> session=901,thread=223ession=3151,thread=254 2015061003

      The content of origin lzo file content see below ,just 2 rows.

      INFO [03:00:05.635] <b88e0473-7530-494c-82d8-e2d2ebd2666c_forweb> session=3148,thread=285
INFO [03:00:05.635] HttpFrontServer::FrontSH msgRecv:Remote=/10.13.193.68:42095,session=3148,thread=285

      I think this error is caused by the Text reuse,and I found the solutions .

      Addicational, table create sql is : 

      CREATE EXTERNAL TABLE `web_searchhub`(
  `line` string)
PARTITIONED BY (
  `logdate` string)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '\\U0000'
WITH SERDEPROPERTIES (
  'serialization.encoding'='GBK')
STORED AS INPUTFORMAT  "com.hadoop.mapred.DeprecatedLzoTextInputFormat"
          OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat";

LOCATION
  'viewfs://nsX/user/hive/warehouse/raw.db/web/web_searchhub'

      Attachments

        1. HIVE-10983.1.patch.txt
          0.6 kB
          Xiaowei Wang
        2. HIVE-10983.2.patch.txt
          0.8 kB
          Xiaowei Wang
        3. HIVE-10983.3.patch.txt
          0.7 kB
          Xiaowei Wang
        4. HIVE-10983.4.patch.txt
          0.7 kB
          Xiaowei Wang
        5. HIVE-10983.5.patch.txt
          0.8 kB
          Xiaowei Wang

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. HIVE-11095 SerDeUtils another bug ,when Text is reused

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HIVE-11112 ISO-8859-1 text output has fragments of previous longer rows appended

          • Major - Major loss of function.
          • Closed
          incorporates

          Bug - A problem which impairs or prevents the functions of the product. HIVE-11112 ISO-8859-1 text output has fragments of previous longer rows appended

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-11095 SerDeUtils another bug ,when Text is reused

          • Major - Major loss of function.
          • Closed

          Activity

            People

              wisgood Xiaowei Wang
              wisgood Xiaowei Wang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: