Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-43583

When encryption is enabled on the External Shuffle Service, then processing of push meta requests throws NPE

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.4, 3.3.2, 3.4.0
    • 3.5.0
    • Shuffle
    • None

    Description

      After enabling support for over-the-wire encryption for spark shuffle services, the meta requests for push-merged blocks fail with this error:

      java.lang.RuntimeException: java.lang.NullPointerException
	at org.apache.spark.network.server.AbstractAuthRpcHandler.getMergedBlockMetaReqHandler(AbstractAuthRpcHandler.java:110)
	at org.apache.spark.network.crypto.AuthRpcHandler.getMergedBlockMetaReqHandler(AuthRpcHandler.java:144)
	at org.apache.spark.network.server.TransportRequestHandler.processMergedBlockMetaRequest(TransportRequestHandler.java:275)
	at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:117)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:140)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:53)
	at org.sparkproject.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at org.sparkproject.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)

      Attachments

        Issue Links

          relates to

          Sub-task - The sub-task of the issue SPARK-36744 Support IO encryption for push-based shuffle

          • Major - Major loss of function.
          • Open
          links to

          Pull request #41225 [Github] Pull Request #41225 (otterc)

          Activity

            People

              csingh Chandni Singh
              csingh Chandni Singh
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: