Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-2253

ALS fails if the data flow is split up and the input is non-deterministic

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Won't Do
    • None
    • None
    • Library / Machine Learning

    Description

      If the temporaryPath of Flink's ALS implementation is set, then the ALS job is executed one part after the other. This also includes the routing information calculation for the user and item blocks. If the input DataSet is calculated non-deterministically, e.g. if it's the result of a first operation, then the resulting routing information won't be consistent and ALS fails.

      A solution would be to pin the input DataSet so that it will only be executed once. Until we have this functionality, I propose to simply execute the user and item routing information at the same time.

      Attachments

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            trohrmann Till Rohrmann
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment