Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-5288

Delta import is calling applyTranformer() during deltaQuery and causing ScriptException

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Won't Fix
    • 4.4
    • None
    • None

    Description

      While experimenting delta import, was getting Script Exception such as 'toString()' is not found on null.

      These are the queries that am using
      a) Query > SELECT PK_FIELD, JOIN_DATE, USER_NAME FROM USERS
      b) Delta Query > SELECY PK_FIELD FROM USERS WHERE LAST_MODIFIED_DATE > '${dih.last_index_time}'
      c) Delta Import Query > SELECT PK_FIELD, JOIN_DATE, USER_NAME FROM USERS WHERE PK_FIELD = '${dih.delta.PK_FIELD}'

      Have a script transformer as below

      function dynamicData(){
      var joinDt = row.get('JOIN_DATE');
      var dtDisplay = joinDt.toString(); //e.g to show that am not doing null check since join_date is a not null field
      ...........
      ...........
      return row;
      }

      <entity name="user" transformer="script:dynamicData" ...... >
      .......
      </entity>

      Problem: While performing delta import, was getting exception from Rhino engine on the script line 'joinDt.toString()'.

      The exception trace is as follows
      Caused by: javax.script.ScriptException: sun.org.mozilla.javascript.internal.EcmaError: TypeError: Cannot call method "t
      oString" of null (<Unknown source>#4) in <Unknown source> at line number 4
      at com.sun.script.javascript.RhinoScriptEngine.invoke(RhinoScriptEngine.java:300)
      at com.sun.script.javascript.RhinoScriptEngine.invokeFunction(RhinoScriptEngine.java:258)
      at org.apache.solr.handler.dataimport.ScriptTransformer.transformRow(ScriptTransformer.java:56)
      ... 8 more

      Root Cause: Since I know join_date can not be null, have explored the solr source code and noticed that applyTransformer() is called during deltaQuery and at that time join_date will not be available.

      Reference: EntityProcessorWrapper.nextModifiedRowKey()

      I think transformation is not required for deltaQuery since it is mainly designed to retrieving the primary keys of the modified rows. Further, the output of deltaQuery will be used only in another SQL.

      Work around:
      Just added a null check as a workaround as below

      function dynamicData(){
      var joinDt = row.get('JOIN_DATE');
      if(joinDt == null)

      { return row; }

      ...........
      ...........
      return row;
      }

      I don't have too much knowledge about Solr and hence my suggestion could be invalid while looking from main use cases.

      Please validate my comments once.

      Thanks
      Balaji

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              balajimanoharan Balaji Manoharan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: