Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-5718

Notebook lost due to non-atomic file operations.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.9.0, 0.10.1
    • None
    • zeppelin-zengine
    • None

    Description

      Zeeplin nootbook file may become a xxx.zpln.tmp file and be lost when zeppelin restarts.

       
      org.apache.zeppelin.notebook.FileSystemStorage#writeFile

      The code shows that we need to change the .zpln file, we delete it first and then rename the
      .zpln.tmp file as .zpln file. This operation is non-atomic and may be cases where files have been deleted but not renamed.

        public void writeFile(final String content, final Path file, boolean writeTempFileFirst, Set<PosixFilePermission> permissions)
            throws IOException {
          FsPermission fsPermission;
          if (permissions == null || permissions.isEmpty()) {
            fsPermission = FsPermission.getFileDefault();
          } else {
            // FsPermission expects a 10-character string because of the leading
            // directory indicator, i.e. "drwx------". The JDK toString method returns
            // a 9-character string, so prepend a leading character.
            fsPermission = FsPermission.valueOf("-" + PosixFilePermissions.toString(permissions));
          }
          callHdfsOperation(new HdfsOperation<Void>() {
            @Override
            public Void call() throws IOException {
              InputStream in = new ByteArrayInputStream(content.getBytes(
                  zConf.getString(ZeppelinConfiguration.ConfVars.ZEPPELIN_ENCODING)));
              Path tmpFile = new Path(file.toString() + ".tmp");
              IOUtils.copyBytes(in, fs.create(tmpFile), hadoopConf);
              fs.setPermission(tmpFile, fsPermission);
              fs.delete(file, true);
              fs.rename(tmpFile, file);
              return null;
            }
          });
        }
      

       

      BTW VFSNotebookRepo has the same problem. For the processing of local files, are we considering not to use VFS2.

      org.apache.commons.vfs2.provider.AbstractFileObject#moveTo

      public void moveTo(final FileObject destFile) throws FileSystemException {
          if (canRenameTo(destFile)) {
              if (!getParent().isWriteable()) {
                  throw new FileSystemException("vfs.provider/rename-parent-read-only.error", getName(),
                          getParent().getName());
              }
          } else {
              if (!isWriteable()) {
                  throw new FileSystemException("vfs.provider/rename-read-only.error", getName());
              }
          }
      
          if (destFile.exists() && !isSameFile(destFile)) {
              destFile.deleteAll();
              // throw new FileSystemException("vfs.provider/rename-dest-exists.error", destFile.getName());
          }
      
          if (canRenameTo(destFile)) {
              // issue rename on same filesystem
              try {
                  attach();
                  // remember type to avoid attach
                  final FileType srcType = getType();
      
                  doRename(destFile);
      
                  FileObjectUtils.getAbstractFileObject(destFile).handleCreate(srcType);
                  destFile.close(); // now the destFile is no longer imaginary. force reattach.
      
                  handleDelete(); // fire delete-events. This file-object (src) is like deleted.
              } catch (final RuntimeException re) {
                  throw re;
              } catch (final Exception exc) {
                  throw new FileSystemException("vfs.provider/rename.error", exc, getName(), destFile.getName());
              }
          } else {
              // different fs - do the copy/delete stuff
      
              destFile.copyFrom(this, Selectors.SELECT_SELF);
      
              if ((destFile.getType().hasContent()
                      && destFile.getFileSystem().hasCapability(Capability.SET_LAST_MODIFIED_FILE)
                      || destFile.getType().hasChildren()
                              && destFile.getFileSystem().hasCapability(Capability.SET_LAST_MODIFIED_FOLDER))
                      && fileSystem.hasCapability(Capability.GET_LAST_MODIFIED)) {
                  destFile.getContent().setLastModifiedTime(this.getContent().getLastModifiedTime());
              }
      
              deleteSelf();
          }
      
      }
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            jiabao.sun jiabao.sun
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: