Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.9.0, 0.10.1
-
None
-
None
Description
Zeeplin nootbook file may become a xxx.zpln.tmp file and be lost when zeppelin restarts.
org.apache.zeppelin.notebook.FileSystemStorage#writeFile
The code shows that we need to change the .zpln file, we delete it first and then rename the
.zpln.tmp file as .zpln file. This operation is non-atomic and may be cases where files have been deleted but not renamed.
public void writeFile(final String content, final Path file, boolean writeTempFileFirst, Set<PosixFilePermission> permissions) throws IOException { FsPermission fsPermission; if (permissions == null || permissions.isEmpty()) { fsPermission = FsPermission.getFileDefault(); } else { // FsPermission expects a 10-character string because of the leading // directory indicator, i.e. "drwx------". The JDK toString method returns // a 9-character string, so prepend a leading character. fsPermission = FsPermission.valueOf("-" + PosixFilePermissions.toString(permissions)); } callHdfsOperation(new HdfsOperation<Void>() { @Override public Void call() throws IOException { InputStream in = new ByteArrayInputStream(content.getBytes( zConf.getString(ZeppelinConfiguration.ConfVars.ZEPPELIN_ENCODING))); Path tmpFile = new Path(file.toString() + ".tmp"); IOUtils.copyBytes(in, fs.create(tmpFile), hadoopConf); fs.setPermission(tmpFile, fsPermission); fs.delete(file, true); fs.rename(tmpFile, file); return null; } }); }
BTW VFSNotebookRepo has the same problem. For the processing of local files, are we considering not to use VFS2.
org.apache.commons.vfs2.provider.AbstractFileObject#moveTo
public void moveTo(final FileObject destFile) throws FileSystemException { if (canRenameTo(destFile)) { if (!getParent().isWriteable()) { throw new FileSystemException("vfs.provider/rename-parent-read-only.error", getName(), getParent().getName()); } } else { if (!isWriteable()) { throw new FileSystemException("vfs.provider/rename-read-only.error", getName()); } } if (destFile.exists() && !isSameFile(destFile)) { destFile.deleteAll(); // throw new FileSystemException("vfs.provider/rename-dest-exists.error", destFile.getName()); } if (canRenameTo(destFile)) { // issue rename on same filesystem try { attach(); // remember type to avoid attach final FileType srcType = getType(); doRename(destFile); FileObjectUtils.getAbstractFileObject(destFile).handleCreate(srcType); destFile.close(); // now the destFile is no longer imaginary. force reattach. handleDelete(); // fire delete-events. This file-object (src) is like deleted. } catch (final RuntimeException re) { throw re; } catch (final Exception exc) { throw new FileSystemException("vfs.provider/rename.error", exc, getName(), destFile.getName()); } } else { // different fs - do the copy/delete stuff destFile.copyFrom(this, Selectors.SELECT_SELF); if ((destFile.getType().hasContent() && destFile.getFileSystem().hasCapability(Capability.SET_LAST_MODIFIED_FILE) || destFile.getType().hasChildren() && destFile.getFileSystem().hasCapability(Capability.SET_LAST_MODIFIED_FOLDER)) && fileSystem.hasCapability(Capability.GET_LAST_MODIFIED)) { destFile.getContent().setLastModifiedTime(this.getContent().getLastModifiedTime()); } deleteSelf(); } }