Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-1981 [Umbrella] Fix all flaky tests
  3. ZEPPELIN-1828

Flaky Test: RemoteInterpreterTest hanged in zeppelin-interpreter

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.8.0
    • None
    • None

    Description

      I saw this kind of hanged tests multiple times

      Running org.apache.zeppelin.interpreter.remote.RemoteInterpreterTest
      17:31:24,423  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 43625, -l, fakeRepo]
      17:31:24,926  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
      17:31:25,090  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterB
      17:31:25,124 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"id","messages":[]})
      java.lang.NullPointerException
      	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
      17:31:25,162  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
      17:31:26,126 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null
      17:31:27,274  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:131 - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
      17:31:27,276  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:164 - Interpreter process exited 0
      17:31:27,776  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823084409_-1062367753
      17:31:27,778  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 37763, -l, fakeRepo]
      17:31:28,281  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
      17:31:28,448  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job 0 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote220694569
      17:31:28,480 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"0","messages":[]})
      java.lang.NullPointerException
      	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
      17:31:28,520  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job 0 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote220694569
      17:31:28,521  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job 1 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote220694569
      17:31:28,523 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"1","messages":[]})
      java.lang.NullPointerException
      	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
      17:31:28,525  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job 1 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote220694569
      17:31:28,526  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job 2 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote220694569
      17:31:28,527 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"2","messages":[]})
      java.lang.NullPointerException
      	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
      17:31:28,530  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job 2 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote220694569
      17:31:28,535  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
      17:31:29,528 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null
      17:31:30,961  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:131 - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
      17:31:30,963  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:164 - Interpreter process exited 0
      17:31:31,463  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823087778_1506341451
      17:31:31,464  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 45441, -l, fakeRepo]
      17:31:31,967  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
      17:31:32,131  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job 0 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote959023117
      17:31:32,165 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"0","messages":[]})
      java.lang.NullPointerException
      	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
      17:31:32,631  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job 1 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote959023117
      17:31:32,634 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"1","messages":[]})
      java.lang.NullPointerException
      	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
      17:31:33,132  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job 2 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote959023117
      17:31:33,134 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"2","messages":[]})
      java.lang.NullPointerException
      	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
      17:31:33,200  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job 0 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote959023117
      17:31:33,636  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job 1 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote959023117
      17:31:33,637  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job 3 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote959023117
      17:31:33,639 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"3","messages":[]})
      java.lang.NullPointerException
      	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
      17:31:34,136  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job 2 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote959023117
      17:31:34,641  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job 3 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote959023117
      17:31:34,646  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
      17:31:35,641 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null
      17:31:37,072  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:131 - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
      17:31:37,074  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:164 - Interpreter process exited 0
      17:31:37,574  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823091464_1066082224
      17:31:37,576  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 37531, -l, fakeRepo]
      17:31:38,080  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
      17:31:38,249  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterB
      17:31:38,253  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job jobA started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote582688427
      17:31:38,288 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"jobA","messages":[]})
      java.lang.NullPointerException
      	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
      17:31:38,821  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job jobA finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote582688427
      17:31:38,822  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job jobB started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote582688427
      17:31:38,826 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"jobB","messages":[]})
      java.lang.NullPointerException
      	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
      17:31:39,828  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job jobB finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote582688427
      17:31:39,861  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
      17:31:40,828 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null
      17:31:42,289  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:131 - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
      17:31:42,290  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:164 - Interpreter process exited 0
      17:31:42,790  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823097576_2123324288
      17:31:42,792  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823102792_1551732183
      17:31:42,793  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 43603, -l, fakeRepo]
      17:31:43,296  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
      17:31:43,497 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"noteId","paragraphId":"id","messages":[]})
      java.lang.NullPointerException
      	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
      17:31:43,545  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823102793_1859137179
      17:31:43,552  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
      17:31:44,498 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null
      17:31:45,664  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:131 - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
      17:31:45,665  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:164 - Interpreter process exited 0
      17:31:46,267  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:532 - Push local angular object registry from ZeppelinServer to remote interpreter group groupId
      17:31:46,269  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823106166_-1057400618
      17:31:46,271  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 39299, -l, fakeRepo]
      17:31:46,774  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
      17:31:46,943  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterB
      17:31:46,977 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"id","messages":[]})
      java.lang.NullPointerException
      	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
      17:31:47,512 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"id","messages":[]})
      java.lang.NullPointerException
      	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
      17:31:48,519  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
      17:31:49,514 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null
      17:31:50,632  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:131 - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
      17:31:50,633  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:164 - Interpreter process exited 0
      17:31:51,133  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823106270_2108336050
      17:31:51,136  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 34103, -l, fakeRepo]
      17:31:51,638  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
      17:31:51,803  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job jobA started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote1184497162
      17:31:51,837 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"jobA","messages":[]})
      java.lang.NullPointerException
      	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
      17:31:51,911  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
      17:31:52,838 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null
      17:31:54,339  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:131 - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
      17:31:54,341  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:164 - Interpreter process exited 0
      17:31:54,841  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job jobA finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote1184497162
      17:31:54,841  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823111136_509381734
      17:31:54,843  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 42097, -l, fakeRepo]
      17:31:55,345  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
      17:31:55,520  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterB
      17:31:55,522  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823114842_-936486082
      17:31:55,529  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
      17:31:56,373 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null
      17:31:57,954  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:131 - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
      17:31:57,955  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:164 - Interpreter process exited 0
      17:31:58,464  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 35227, -l, fakeRepo]
      17:31:58,966  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterEnv
      17:31:59,223  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
      17:31:59,419 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"noteId","paragraphId":"id","messages":[]})
      java.lang.NullPointerException
      	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
      17:31:59,420 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null
      
      
      No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
      Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
      
      The build has been terminated
      

      Attachments

        Issue Links

          Activity

            People

              AlexanderShoshin Alexander Shoshin
              1ambda Hoon Park
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: