Code Monkey home page Code Monkey logo

onedriver's Introduction

Run tests Coverage Status Copr build status

onedriver

onedriver is a native Linux filesystem for Microsoft OneDrive.

onedriver is a network filesystem that gives your computer direct access to your files on Microsoft OneDrive. This is not a sync client. Instead of syncing files, onedriver performs an on-demand download of files when your computer attempts to use them. onedriver allows you to use files on OneDrive as if they were files on your local computer.

onedriver is extremely straightforwards to use:

  • Install onedriver using your favorite installation method.
  • Click the "+" button in the app to setup one or more OneDrive accounts. (There's a command-line workflow for those who prefer doing things that way too!)
  • Just start using your files on OneDrive as if they were normal files.

I've spent a lot of time trying to make onedriver fast, convenient, and easy to use. Though you can use it on servers, the goal here is to make it easy to work with OneDrive files on your Linux desktop. This allows you to easily sync files between any number of Windows, Mac, and Linux computers. You can setup your phone to auto-upload photos to OneDrive and edit and view them on your Linux computer. You can switch between LibreOffice on your local computer and the Microsoft 365 online apps as needed when working. Want to migrate from Windows to Linux? Just throw all your Windows files into OneDrive, add your OneDrive account to Linux with onedriver, and call it a day.

Microsoft OneDrive works on Linux.

Getting started with your files on OneDrive is as easy as running: onedriver /path/to/mount/onedrive/at (there's also a helpful GUI!). To get a list of all the arguments onedriver can be run with you can read the manual page by typing man onedriver or get a quick summary with onedriver --help.

Key features

onedriver has several nice features that make it significantly more useful than other OneDrive clients:

  • Files are only downloaded when you use them. onedriver will only download a file if you (or a program on your computer) uses that file. You don't need to wait hours for a sync client to sync your entire OneDrive account to your local computer or try to guess which files and folders you might need later while setting up a "selective sync". onedriver gives you instant access to all of your files and only downloads the ones you use.

  • Bidirectional sync. Although onedriver doesn't actually "sync" any files, any changes that occur on OneDrive will be automatically reflected on your local machine. onedriver will only redownload a file when you access a file that has been changed remotely on OneDrive. If you somehow simultaneously modify a file both locally on your computer and also remotely on OneDrive, your local copy will always take priority (to avoid you losing any local work).

  • Can be used offline. Files you've opened previously will be available even if your computer has no access to the internet. The filesystem becomes read-only if you lose internet access, and automatically enables write access again when you reconnect to the internet.

  • Fast. Great care has been taken to ensure that onedriver never makes a network request unless it actually needs to. onedriver caches both filesystem metadata and file contents both in memory and on-disk. Accessing your OneDrive files will be fast and snappy even if you're engaged in a fight to the death for the last power outlet at a coffeeshop with bad wifi. (This has definitely never happened to me before, why do you ask?)

  • Has a user interface. You can add and remove your OneDrive accounts without ever using the command-line. Once you've added your OneDrive accounts, there's no special interface beyond your normal file browser.

  • Free and open-source. They're your files. Why should you have to pay to access them? onedriver is licensed under the GPLv3, which means you will always have access to use onedriver to access your files on OneDrive.

Quick start

Fedora/CentOS/RHEL

Users on Fedora/CentOS/RHEL systems are recommended to install onedriver from COPR. This will install the latest version of onedriver through your package manager and ensure it stays up-to-date with bugfixes and new features.

sudo dnf copr enable jstaf/onedriver
sudo dnf install onedriver

OpenSUSE

OpenSUSE users need to add the COPR repo either for Leap or Tumbleweed

# Leap 15.4
sudo zypper addrepo -g -r https://copr.fedorainfracloud.org/coprs/jstaf/onedriver/repo/opensuse-leap-15.4/jstaf-onedriver-opensuse-leap-15.4.repo onedriver
sudo zypper --gpg-auto-import-keys refresh
sudo zypper install onedriver

# Tumbleweed
sudo zypper addrepo -g -r https://copr.fedorainfracloud.org/coprs/jstaf/onedriver/repo/opensuse-tumbleweed/jstaf-onedriver-opensuse-tumbleweed.repo onedriver
sudo zypper --gpg-auto-import-keys refresh
sudo zypper install onedriver

Ubuntu/Pop!_OS/Debian

Ubuntu/Pop!_OS/Debian users can install onedriver from the OpenSUSE Build Service (despite the name, OBS also does a nice job of building packages for Debian). Like the COPR install, this will enable you to install onedriver through your package manager and install updates as they become available. If you previously installed onedriver via PPA, you can purge the old PPA from your system via: sudo add-apt-repository --remove ppa:jstaf/onedriver

Arch/Manjaro/EndeavourOS

Arch/Manjaro/EndeavourOS users can install onedriver from the AUR.

Post-installation, you can start onedriver either via the onedriver-launcher desktop app, or via the command line: onedriver /path/to/mount/onedrive/at/.

Gentoo

Gentoo users can install onedriver from this ebuild overlay provided by a user. If you don't want to add user-hosted overlays to your system you may copy the ebuild for the latest version to a local overlay, which can be created by following the instructions available in the Gentoo Wiki.

Make sure to carefully review the ebuild for the package before installing it

NixOS/NixPkgs

NixOS and Nix users can install onedriver from the unstable channel either by adding the package to their system's configuration (if they are using NixOS) or by installing it manually via nix-env -iA unstable.onedriver.

Multiple drives and starting OneDrive on login via systemd

Note: You can also set this up through the GUI via the onedriver-launcher desktop app installed via rpm/deb/make install. You can skip this section if you're using the GUI. It's honestly easier.

To start onedriver automatically and ensure you always have access to your files, you can start onedriver as a systemd user service. In this example, $MOUNTPOINT refers to where we want OneDrive to be mounted at (for instance, ~/OneDrive).

# create the mountpoint and determine the service name
mkdir -p $MOUNTPOINT
export SERVICE_NAME=$(systemd-escape --template [email protected] --path $MOUNTPOINT)

# mount onedrive
systemctl --user daemon-reload
systemctl --user start $SERVICE_NAME

# automatically mount onedrive when you login
systemctl --user enable $SERVICE_NAME

# check onedriver's logs for the current day
journalctl --user -u $SERVICE_NAME --since today

Building onedriver yourself

In addition to the traditional Go tooling, you will need a C compiler and development headers for webkit2gtk-4.0 and json-glib. On Fedora, these can be obtained with dnf install golang gcc pkg-config webkit2gtk3-devel json-glib-devel. On Ubuntu, these dependencies can be installed with apt install golang gcc pkg-config libwebkit2gtk-4.0-dev libjson-glib-dev.

# to build and run the binary
make
mkdir mount
./onedriver mount/

# in new window, check out the mounted filesystem
ls -l mount

# unmount the filesystem
fusermount3 -uz mount
# you can also just "ctrl-c" onedriver to unmount it

Running the tests

The tests will write and delete files/folders on your onedrive account at the path /onedriver_tests. Note that the offline test suite requires sudo to remove network access to simulate being offline.

# setup test tooling for first time run
make test-init

# actually run tests
make test

Installation from source

onedriver has multiple installation methods depending on your needs.

# install directly from source
make
sudo make install

# create an RPM for system-wide installation on RHEL/CentOS/Fedora using mock
sudo dnf install golang gcc webkit2gtk3-devel json-glib-devel pkg-config git \
    rsync rpmdevtools rpm-build mock
sudo usermod -aG mock $USER
newgrp mock
make rpm

# create a .deb for system-wide installation on Ubuntu/Debian using pbuilder
sudo apt update
sudo apt install golang gcc libwebkit2gtk-4.0-dev libjson-glib-dev pkg-config git \
    rsync devscripts debhelper build-essential pbuilder
sudo pbuilder create  # may need to add "--distribution focal" on ubuntu
make deb

Troubleshooting

During your OneDrive travels, you might hit a bug that I haven't squashed yet. Don't panic! In most cases, the filesystem will report what happened to whatever program you're using. (As an example, an error mentioning a "read-only filesystem" indicates that your computer is currently offline.)

If the filesystem appears to hang or "freeze" indefinitely, its possible the fileystem has crashed. To resolve this, just restart the program by unmounting and remounting things via the GUI or by running fusermount3 -uz $MOUNTPOINT on the command-line.

If you really want to go back to a clean slate, onedriver can be completely reset (delete all cached local data) by deleting mounts in the GUI or running onedriver -w.

If you encounter a bug or have a feature request, open an issue in the "Issues" tab here on GitHub. The two most informative things you can put in a bug report are the logs from the bug/just before encountering the bug (get logs via journalctl --user -u $SERVICE_NAME --since today ... see docs for correct value of $SERVICE_NAME) and/or instructions on how to reproduce the issue. Otherwise I have to guess what the problem is ๐Ÿ˜ž

Known issues & disclaimer

Many file browsers (like GNOME's Nautilus) will attempt to automatically download all files within a directory in order to create thumbnail images. This is somewhat annoying, but only needs to happen once - after the initial thumbnail images have been created, thumbnails will persist between filesystem restarts.

Microsoft does not support symbolic links (or anything remotely like them) on OneDrive. Attempting to create symbolic links within the filesystem returns ENOSYS (function not implemented) because the functionality hasn't been implemented... by Microsoft. Similarly, Microsoft does not expose the OneDrive Recycle Bin APIs - if you want to empty or restore the OneDrive Recycle Bin, you must do so through the OneDrive web UI (onedriver uses the native system trash/restore functionality independently of the OneDrive Recycle Bin).

onedriver loads files into memory when you access them. This makes things very fast, but obviously doesn't work very well if you have very large files. Use a sync client like rclone if you need the ability to copy multi-gigabyte files to OneDrive.

OneDrive is not a good place to backup files to. Use a tool like restic or borg if you're looking for a reliable encrypted backup tool. I know some of you want to "back up your files to OneDrive". Don't do it. Restic and Borg are better in every possible way than any OneDrive client ever will be when it comes to creating backups you can count on.

Finally, this project is still in active development and is provided AS IS. There are no guarantees. It might kill your cat.

onedriver's People

Contributors

dipunm avatar erik88 avatar fmoledina avatar foopsss avatar genericguy avatar jstaf avatar maartenplugwise avatar massix avatar msrn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

onedriver's Issues

Package for Raspberry Pi

With the advent of the RPi 4 with 8 GB ram, having Onedriver run on it is realistic.
Do you think you could build a package for it?

Thanks

Add file-backed cache

Some files are simply huge and are not good candidates to keep in memory.

  • Add a new cache backend that uses actual files instead of storing the data in memory
  • Integrate the file cache into the current memory cache backend so that big files are moved to the file cache once they become too large in memory.

Related to #4

Delay uploads to avoid frequent uploads after writes

Uploads should perhaps be on a short delay to prevent repeated uploads of the same item multiple times in a short period.

Example of what we need to fix (from the Truncate() test):

# more writes omitted
2019/05/05 15:34:50 - TRACE - 40:drive_item.go:265:Write() - /onedriver_tests/truncate.txt: 7 bytes at offset 0
2019/05/05 15:34:50 - TRACE - 39:drive_item.go:294:Flush() - /onedriver_tests/truncate.txt
2019/05/05 15:34:50 - INFO  - 122:upload.go:140:Upload() - /onedriver_tests/truncate.txt
2019/05/05 15:34:50 - TRACE - 51:fusefs.go:284:Open() - /onedriver_tests/truncate.txt
2019/05/05 15:34:50 - TRACE - 40:drive_item.go:256:Read() - /onedriver_tests/truncate.txt: 7 bytes at offset 0
2019/05/05 15:34:50 - TRACE - 51:drive_item.go:294:Flush() - /onedriver_tests/truncate.txt
2019/05/05 15:34:51 - TRACE - 120:upload.go:157:Upload() - Using simple upload for truncate.txt
2019/05/05 15:34:51 - TRACE - 121:upload.go:157:Upload() - Using simple upload for truncate.txt
2019/05/05 15:34:51 - TRACE - 122:upload.go:157:Upload() - Using simple upload for truncate.txt
2019/05/05 15:34:51 - TRACE - 104:upload.go:157:Upload() - Using simple upload for truncate.txt
2019/05/05 15:34:51 - TRACE - 119:upload.go:157:Upload() - Using simple upload for truncate.txt

onedriver-headless in debian Sid

Hi,
i'm trying to use onedriver-headless v0.8.0 into a debian:sid container
I'm able to login, but then i get this error, what am i missing ?

/tmp/onedriver-headless -d /mnt/OneDriveValerio/

-- login dance --

INFO[2020-04-08T06:14:05]main.go:103 000001:main.main() onedriver v0.8.0 a895b06
FATA[2020-04-08T06:14:05]cache.go:45 000001:fs.NewCache() Could not open DB err="open /root/.cache/onedriver/onedriver.db: no such file or directory"

image

Thanks

Metadata updates

Currently only new files and file contents are uploaded to the server. Metadata updates need to be included as well.

  • Updating file metadata locally should update the metadata on the server.

Provide user-friendly GUI / .desktop launcher

  • Create a nice GUI setup tool that lets you do the following without knowing anything about the command-line:
    • Choose a mount location
    • Start/stop onedriver and configure it to run on login
    • Get the onedriver logs out of systemd
  • Create a .desktop launcher for the setup tool.
  • Create some user-friendly docs pages (so you don't have to be a developer to get it working)

Problem with Office 365 (?)

Hi there, and many thanks for this. It works pretty well for a simple Microsoft account, but when I try to connect to a 365 account, I get:

(process:6962): libsoup-WARNING **: 11:01:47.797: gssapi step failed: Unspecified GSS failure.  Minor code may provide more information: SPNEGO cannot find mechanisms to negotiate

and then

2019/02/10 11:02:10 OpenDir("/")
2019/02/10 11:02:10 Auth tokens expired, attempting renewal...

Any ideas?

Implement upload sessions

File uploads are currently capped to 4MB until upload sessions are implemented.

  • Implement upload sessions
  • Upload sessions should snapshot the file in its current state (if the content changes mid-upload, they will fail).
  • Wait 15-30s before uploading to catch multiple upload attempts for same file in short period, and ensure only a single upload session is running at any one time.
  • Write tests to monitor success of upload sessions

Investigate GVFS/GOA integration

It'd be awesome if onedriver could integrate with GNOME Online Accounts and GVFS so that users could setup Onedrive and authenticate from the Online Accounts widget. The GVFS integration would let the Onedriver backend popup in Nautilus, similar to how existing functionality works for Google Drive.

  • ????
  • ????
  • Profit.

Variable log levels

Currently the log is incredibly spammy OR doesn't trace application execution enough (a lot of calls like Utimens or Truncate are simply left out of the logs).

  • Make log level configurable
  • Log all FUSE syscalls at TRACE log level

Files are locked after restart?

Trying to open a powerpoint stored on onedrive with wps office. The first time it is opened it works fine, but after a reboot when I try to open it a message appears "Document has been locked by username, do you want to open as read only".

image

I am online and installed from the releases page. I have not got it launching on startup yet, rather just open the app first.

Any help with how to fix this would be appreciated!

Linux CLI-only setup

Hi there!

I'd like to leave an "anti-request" here, since I would like to be able to use the Linux client without a GUI (or X, or anything else) and right now authorisation is hard-coded with an #ifdef(__linux__) to pop up a browser.

Could there be an option (build or run time) not to use WebKitGTK at all?

Periodically prune disk and memory caches

Currently onedriver's on disk cache and memory cache have no upper bound on how big they can get. We should periodically prune the onedriver db and in-memory caches to keep our resource footprint small.

Horrible death results in unusable mount point.

I don't know if I am doing something wrong.

OneDriver mounted to "OneDrive" mount point in my home works great until I suspend
the machine for a certain length of time.

Upon recovery from suspend, the onedriver process seems to die a horrible horrible death
and leaves a weird ???? ???? OneDrive mount point in my directory.
I cannot delete this directory and it keeps telling me that the "Transport has no end point"
The only way to recover is by restarting(rebooting) the computer.
I tried looking for the onedriver process or something else that I could kill to allow deletion of the
mountpoint but to no avail.

Implement mv

  • Implement Rename() call to support renames/moves
  • Tests...

OneDrive Business some files are not shown in the directory

I noticed that some files from OneDrive Business folder with plain name like "Thursday Notes.txt" are not shown in the directory.

I tried to modify file over MS OneDrive webpage and suddenly the file appeared in the listing.

For file listing there is no spectacular in logs like:

DEBU[2020-03-12T20:51:54]inode.go:244 001047:graph.(*Inode).Readdir()

I compiled the most recent version of onedriver:

onedriver v0.7.2 237b1613

fusermount version: 2.9.7

Create a GUI for managing multiple drives

Though multiple drives can be mounted via systemd, it's still a bit of a pain to use, esp. for users who may not like using the command line. This issue tracks creating a GUI application that can be used to:

  • Create and setup multiple mountpoints
  • Indicate which account is linked to each mountpoint
  • Open the associated drive in the user's default file browser via xdg-open
  • Provide a checkbox to start the associated drives on login

High memory usage

During normal usage, the onedriver process' memory usage can get rather high. It's worth doing an optimization pass post-initial release to see if we can get it to be less of a memory hog.

Fix systemd user service race condition on boot

It looks like there's a race condition on startup that prevents onedriver from reading the children of the root item correctly (either in the go-fuse cache, kernel cache, or onedriver itself).

To reproduce:

  • Shut down the systemd user service
  • Expire auth tokens (make expire_now)
  • Reboot, and login as fast as possible

The filesystem should show up as blank post login. Currently working around this with just sleeping on systemd unit startup, but it would be worthwhile to dig through the FUSE logs and see where the empty root item inode is coming from.

Cache memory management

It's possible to run out of memory by reading too many large files (since onedriver is currently storing everything in memory). Add a feature to:

  • Purge the file content for a file based on size and last access
  • Limit the cache to a maximum size
  • Make maximum cache size a configurable command line parameter

Integration tests

Until recently, it hasn't been possible to write integration tests for onedriver (since stuff was not being uploaded back to the server). Write a set of integration tests that tests functionality for:

  • touch
    • Empty file creation
    • File mtime and atime (local-only, not supported by server) updates
  • mv
    • Move
    • Rename
  • cp
  • Truncate and append via: echo something > file
  • Append via echo something >> file
  • Write to middle of file
  • Write to end of file
  • Read from file at various positions
  • mkdir
  • rmdir
  • mkdir -> rmdir -> mkdir -> rmdir
  • ls
  • stat (stuff that hasn't been explicitly tested elsewhere)

Rewrite using the new go-fuse/fs API

Sadly, it looks like go-fuse has deprecated the pathfs API used by this project. This isn't necessarily a bad thing (the cache ID changes resulted in an internal change in the filesystem structure to be much more in line with the nodefs/fs APIs), but will require some time to rewrite things to use the new go-fuse/fs API.

Create a homebrew, flatpak or docker, for easier adoption.

The are more benefits than downsides.

Most people don't want to install anything extra on their systems development dependencies. The average user doesn't even know how to compile, so by providing a precompiled binary, project growth will rise. And there are many upsides of a growing the user base, donations more developers etc. Simple equation.

Trash handling in Nautilus

When I delete a file from the mounted directory e.g. using the rm command it is successfully transferred to the OneDrive Trash. Trying to delete a file using the Nautilis filemanager results in a warning message like: trash folder cannot be found or created. Still the file is moved to the OneDrive trash with an additional folder .Trash-1000.

Additionally it would be nice to be able to access the OneDrive Trashbin using the onedriver application.

Duplicate deadlock in Flush

DEBU[2019-08-19T20:53:02]graph/drive_item.go:282 0069:graph.(*DriveItem).Flush()                                               path="/Documents/modeling%20backup/weapons/tracer_ii.kra.wRLcsY"
POTENTIAL DEADLOCK: Duplicate locking, saw callers this locks in one goroutine:
current goroutine 69 lock &{{{1 0} 0 0 -1073741824 0}} 
all callers to this lock in the goroutine
graph/drive_item.go:283 graph.(*DriveItem).Flush {  } <<<<<
~/go/pkg/mod/github.com/hanwen/[email protected]/fuse/nodefs/fsops.go:490 nodefs.(*rawBridge).Flush { return opened.WithFlags.File.Flush() }
~/go/pkg/mod/github.com/hanwen/[email protected]/fuse/opcode.go:373 fuse.doFlush { req.status = server.fileSystem.Flush((*FlushIn)(req.inData)) }
~/go/pkg/mod/github.com/hanwen/[email protected]/fuse/server.go:428 fuse.(*Server).handleRequest { req.handler.Func(ms, req) }
~/go/pkg/mod/github.com/hanwen/[email protected]/fuse/server.go:400 fuse.(*Server).loop { ms.handleRequest(req) }

graph/drive_item.go:135 graph.DriveItem.ID { func (d DriveItem) ParentID() string { } <<<<<
graph/drive_item.go:291 graph.(*DriveItem).Flush { // ensureID() is no longer used here to make upload dispatch even faster }
~/go/pkg/mod/github.com/hanwen/[email protected]/fuse/nodefs/fsops.go:490 nodefs.(*rawBridge).Flush { return opened.WithFlags.File.Flush() }
~/go/pkg/mod/github.com/hanwen/[email protected]/fuse/opcode.go:373 fuse.doFlush { req.status = server.fileSystem.Flush((*FlushIn)(req.inData)) }
~/go/pkg/mod/github.com/hanwen/[email protected]/fuse/server.go:428 fuse.(*Server).handleRequest { req.handler.Func(ms, req) }
~/go/pkg/mod/github.com/hanwen/[email protected]/fuse/server.go:400 fuse.(*Server).loop { ms.handleRequest(req) }

Add support for working with shared items

Started working on this in the shared-items branch. So far it seems like shared items are identical to normal items, with the addition of a remoteItem facet (nothing special aside from the fact that the parentReference of the remoteItem has a different drive id). Support can be handled easily with two caveats:

  • We need to track the drive ID of all items and change everything to start making requests to /drives/<drive-id>/items/<item-id> instead of just /me/drive/items/<item-id>
  • The remote ID of the item can change when it's moved to or from a share. So we should check for the ID changing on a rename op (or the parent drive ID changing).

Make tests runnable via Travis

Currently we can't run on Travis because Travis doesn't have any way of getting through OAuth2. Find a way of getting it to store auth tokens between sessions to fix this.

  • Create an additional Onedrive account for Travis
  • Upload a set of auth tokens for Travis and somehow keep these secret.
  • Make sure auth tokens are carried over between builds and limit the number of concurrent builds to 1

onedriver deletes edited file remotely and cannot upload a new one

Hi Jeff,

I tried now the latest version of onedriver [1] and noticed a strange behavior. I edited little text file (2 lines) and saved it after adding one line. onedriver effectively deleted it remotely, while preserving local version. Here are my logs:

Initial edit (file was still on onedrive):

DEBU[2020-04-19T09:31:36]inode.go:784 003297:fs.(*Inode).Open() Opening file for I/O.                         id=01USGCVPY3EP3PJ76SZFGYGAPVBRBGE424 path="/SAPDocuments/_work_notes/How to squash commits to keep repo concise and cle.txt"
INFO[2020-04-19T09:31:36]inode.go:825 003297:fs.(*Inode).Open() Found content in cache.                       id=01USGCVPY3EP3PJ76SZFGYGAPVBRBGE424 path="/SAPDocuments/_work_notes/How to squash commits to keep repo concise and cle.txt"
DEBU[2020-04-19T09:31:36]inode.go:612 000042:fs.(*Inode).Create()                                               id=01USGCVP733ZIJ4JG3B5EL4HK7V5BYTD5L mode=100600 name=".How to squash commits to keep repo concise and cle.txt.swp" path=/SAPDocuments/_work_notes
DEBU[2020-04-19T09:31:36]inode.go:612 000042:fs.(*Inode).Create()                                               id=01USGCVP733ZIJ4JG3B5EL4HK7V5BYTD5L mode=100600 name=".How to squash commits to keep repo concise and cle.txt.swx" path=/SAPDocuments/_work_notes
DEBU[2020-04-19T09:31:36]inode.go:461 002474:fs.(*Inode).Flush()                                               id=local-jARQT2RVhVrt13P6i5xC path="/SAPDocuments/_work_notes/.How to squash commits to keep repo concise and cle.txt.swx"
DEBU[2020-04-19T09:31:36]inode.go:428 002474:fs.(*Inode).Fsync()                                               id=local-jARQT2RVhVrt13P6i5xC path="/SAPDocuments/_work_notes/.How to squash commits to keep repo concise and cle.txt.swx"
DEBU[2020-04-19T09:31:36]inode.go:660 003297:fs.(*Inode).Unlink() Unlinking inode.                              id=01USGCVP733ZIJ4JG3B5EL4HK7V5BYTD5L name=".How to squash commits to keep repo concise and cle.txt.swx" path=/SAPDocuments/_work_notes
DEBU[2020-04-19T09:31:36]inode.go:461 002474:fs.(*Inode).Flush()                                               id=local-PYK7vmub2X8uXIu6FNcJ path="/SAPDocuments/_work_notes/.How to squash commits to keep repo concise and cle.txt.swp"
DEBU[2020-04-19T09:31:36]inode.go:428 002474:fs.(*Inode).Fsync()                                               id=local-PYK7vmub2X8uXIu6FNcJ path="/SAPDocuments/_work_notes/.How to squash commits to keep repo concise and cle.txt.swp"
DEBU[2020-04-19T09:31:36]inode.go:660 000042:fs.(*Inode).Unlink() Unlinking inode.                              id=01USGCVP733ZIJ4JG3B5EL4HK7V5BYTD5L name=".How to squash commits to keep repo concise and cle.txt.swp" path=/SAPDocuments/_work_notes
DEBU[2020-04-19T09:31:36]inode.go:612 000042:fs.(*Inode).Create()                                               id=01USGCVP733ZIJ4JG3B5EL4HK7V5BYTD5L mode=100600 name=".How to squash commits to keep repo concise and cle.txt.swp" path=/SAPDocuments/_work_notes
DEBU[2020-04-19T09:31:36]inode.go:461 000042:fs.(*Inode).Flush()                                               id=01USGCVPY3EP3PJ76SZFGYGAPVBRBGE424 path="/SAPDocuments/_work_notes/How to squash commits to keep repo concise and cle.txt"
DEBU[2020-04-19T09:31:36]inode.go:428 000042:fs.(*Inode).Fsync()                                               id=01USGCVPY3EP3PJ76SZFGYGAPVBRBGE424 path="/SAPDocuments/_work_notes/How to squash commits to keep repo concise and cle.txt"
DEBU[2020-04-19T09:31:36]inode.go:784 002474:fs.(*Inode).Open() Opening file for I/O.                         id=01USGCVPY3EP3PJ76SZFGYGAPVBRBGE424 path="/SAPDocuments/_work_notes/How to squash commits to keep repo concise and cle.txt"
INFO[2020-04-19T09:31:36]inode.go:825 002474:fs.(*Inode).Open() Found content in cache.                       id=01USGCVPY3EP3PJ76SZFGYGAPVBRBGE424 path="/SAPDocuments/_work_notes/How to squash commits to keep repo concise and cle.txt"
DEBU[2020-04-19T09:31:36]inode.go:461 003297:fs.(*Inode).Flush()                                               id=01USGCVPY3EP3PJ76SZFGYGAPVBRBGE424 path="/SAPDocuments/_work_notes/How to squash commits to keep repo concise and cle.txt"
DEBU[2020-04-19T09:31:36]inode.go:428 003297:fs.(*Inode).Fsync()                                               id=01USGCVPY3EP3PJ76SZFGYGAPVBRBGE424 path="/SAPDocuments/_work_notes/How to squash commits to keep repo concise and cle.txt"
DEBU[2020-04-19T09:31:50]delta.go:21 000022:fs.(*Cache).DeltaLoop() Fetching deltas from server.                 
INFO[2020-04-19T09:31:50]delta.go:44 000022:fs.(*Cache).DeltaLoop() Fetched 3 deltas.                            
DEBU[2020-04-19T09:31:50]delta.go:122 000022:fs.(*Cache).applyDelta() Applying delta                                id=01USGCVP6NSQBTPVVNLFE2VP6CCTQQDNKF name=
INFO[2020-04-19T09:31:50]delta.go:149 000022:fs.(*Cache).applyDelta() Applying server-side deletion of item.        delta=delete id=01USGCVP6NSQBTPVVNLFE2VP6CCTQQDNKF name=
DEBU[2020-04-19T09:31:50]delta.go:122 000022:fs.(*Cache).applyDelta() Applying delta                                id=01USGCVP56Y2GOVW7725BZO354PWSELRRZ name=root
DEBU[2020-04-19T09:31:50]delta.go:122 000022:fs.(*Cache).applyDelta() Applying delta                                id=01USGCVP72RGN57VEFC5HKCE6QECKAZZAA name=
INFO[2020-04-19T09:31:50]delta.go:149 000022:fs.(*Cache).applyDelta() Applying server-side deletion of item.        delta=delete id=01USGCVP72RGN57VEFC5HKCE6QECKAZZAA name=
INFO[2020-04-19T09:31:50]cache.go:558 000022:fs.(*Cache).SerializeAll() Serializing cache metadata to disk.          
DEBU[2020-04-19T09:31:57]inode.go:461 000042:fs.(*Inode).Flush()                                               id=local-rL5Fc3GcuHC03kaunATU path="/SAPDocuments/_work_notes/.How to squash commits to keep repo concise and cle.txt.swp"
DEBU[2020-04-19T09:31:57]inode.go:428 000042:fs.(*Inode).Fsync()                                               id=local-rL5Fc3GcuHC03kaunATU path="/SAPDocuments/_work_notes/.How to squash commits to keep repo concise and cle.txt.swp"
DEBU[2020-04-19T09:31:57]inode.go:660 003297:fs.(*Inode).Unlink() Unlinking inode.                              id=01USGCVP733ZIJ4JG3B5EL4HK7V5BYTD5L name=".How to squash commits to keep repo concise and cle.txt.swp" path=/SAPDocuments/_work_notes
DEBU[2020-04-19T09:31:59]upload_session.go:191 004515:fs.(*UploadSession).Upload() Uploading file.                               id=01USGCVP6RPID7ILVPOJFIPOFYRMEMDLYV
ERRO[2020-04-19T09:31:59]upload_session.go:216 004515:fs.(*UploadSession).Upload() Error during small file upload.               err="HTTP 404 - itemNotFound: The resource could not be found." id=01USGCVP6RPID7ILVPOJFIPOFYRMEMDLYV response=
ERRO[2020-04-19T09:32:01]upload_manager.go:46 000055:fs.(*UploadManager).uploadLoop() Upload failed.                                id=01USGCVP6RPID7ILVPOJFIPOFYRMEMDLYV

Second attempt to upload file after another local fails too:

DEBU[2020-04-19T09:26:24]inode.go:660 000027:fs.(*Inode).Unlink() Unlinking inode.                              id=01USGCVP733ZIJ4JG3B5EL4HK7V5BYTD5L name="How to squash commits to keep repo concise and cle.txt~" path=/SAPDocuments/_work_notes
DEBU[2020-04-19T09:26:24]inode.go:461 000028:fs.(*Inode).Flush()                                               id=local-VbhV3vC5AWX39IVUWSP2 path="/SAPDocuments/_work_notes/.How to squash commits to keep repo concise and cle.txt.swp"
DEBU[2020-04-19T09:26:24]inode.go:428 000028:fs.(*Inode).Fsync()                                               id=local-VbhV3vC5AWX39IVUWSP2 path="/SAPDocuments/_work_notes/.How to squash commits to keep repo concise and cle.txt.swp"
DEBU[2020-04-19T09:26:25]inode.go:660 000042:fs.(*Inode).Unlink() Unlinking inode.                              id=01USGCVP733ZIJ4JG3B5EL4HK7V5BYTD5L name=".How to squash commits to keep repo concise and cle.txt.swp" path=/SAPDocuments/_work_notes
DEBU[2020-04-19T09:26:25]upload_session.go:191 000475:fs.(*UploadSession).Upload() Uploading file.                               id=01USGCVP4CXU2CXPSNE5AIA7IH4XD6HXPP
DEBU[2020-04-19T09:26:25]upload_session.go:191 000474:fs.(*UploadSession).Upload() Uploading file.                               id=01USGCVPY3EP3PJ76SZFGYGAPVBRBGE424
ERRO[2020-04-19T09:26:25]upload_session.go:216 000475:fs.(*UploadSession).Upload() Error during small file upload.               err="HTTP 404 - itemNotFound: The resource could not be found." id=01USGCVP4CXU2CXPSNE5AIA7IH4XD6HXPP response=
ERRO[2020-04-19T09:26:27]upload_manager.go:46 000055:fs.(*UploadManager).uploadLoop() Upload failed.                                id=01USGCVP4CXU2CXPSNE5AIA7IH4XD6HXPP
DEBU[2020-04-19T09:26:35]delta.go:21 000022:fs.(*Cache).DeltaLoop() Fetching deltas from server.                 
INFO[2020-04-19T09:26:35]delta.go:44 000022:fs.(*Cache).DeltaLoop() Fetched 6 deltas.                            
DEBU[2020-04-19T09:26:35]delta.go:122 000022:fs.(*Cache).applyDelta() Applying delta                                id=01USGCVPY3EP3PJ76SZFGYGAPVBRBGE424 name="How to squash commits to keep repo concise and cle.txt"
INFO[2020-04-19T09:26:35]delta.go:210 000022:fs.(*Cache).applyDelta() Overwriting local item, no local changes to preserve.  delta=overwrite id=01USGCVPY3EP3PJ76SZFGYGAPVBRBGE424 name="How to squash commits to keep repo concise and cle.txt"
DEBU[2020-04-19T09:26:35]delta.go:122 000022:fs.(*Cache).applyDelta() Applying delta                                id=01USGCVP56Y2GOVW7725BZO354PWSELRRZ name=root
DEBU[2020-04-19T09:26:35]delta.go:122 000022:fs.(*Cache).applyDelta() Applying delta                                id=01USGCVP2JU76MZZJGCZG36HLCOEJG3O2J name=SAPDocuments
INFO[2020-04-19T09:26:35]delta.go:225 000022:fs.(*Cache).applyDelta() Skipping, no changes relative to local state.  delta=skip id=01USGCVP2JU76MZZJGCZG36HLCOEJG3O2J name=SAPDocuments

[1] 49991e7

Update cache based on server deltas

Onedriver currently does not check for server-side updates after fetching the metadata for an item the first time.

  • The cache should have a dedicated update thread that polls the delta endpoint at a regular interval and applies the changes to the cache
  • Relevant integration tests

Create a donate option

Well, I read all onedrives projetcs for linux and your project is the best by far, but I can not recommend it to my friends until it have a gui. And as you are the only developer (I am a developer too, but I don't know to to help in this project), I though that would be cool to have a donation option, then me and the other people who can not help programming can at least pay a pizza or a coffee to you haha.

Suggestion: Daemon Architecture for Background Syncs

At first I need to admit I really like the concept of using OneDrive as a Drive Mount in you file system.
Yet I think a pure online usage of OneDrive is not always possible and some offline caching mechanisms should be implemented.
During my tests I encountered some problems with the file system encountering crashes or beeing quite unresponsive due to nonexistent or slow network connections.
Resulting these crashes, I often needed to restart the application and "resync" my OneDrive files.
During these rebuilds of the mount structure, no cache was used to speed up the process.

Therefore I would suggest a new architectural approach to the application.
In order to enable consistent caching as well as autostart possibilities to the application you should consider splitting up the application into two seperate processes.

Process 1 handles all user and UI related tasks and provide the desired configuration.
Process 2 (the daemon) handles the synchronisation and caching of the onedrive mount.

The daemon process could be initialized and controlled by the OSs Init-System (like systemd) and provide convenient and fault tolerant access to the onedrive mount.

I'm looking forward to hear you opinion on my suggestion.

Transition between online and offline use gracefully

Currently the filesystem can be used either online or offline depending on if it's able to contact the network on startup.

Change things so that if the delta thread succeeds, the fs is online, and if it fails, it is offline. We will need to store the delta token/position on disk to achieve this between startups (starts offline, goes online).

Implement statfs

  • Implement the statfs fuse syscall so that users can check their quotas and other stuff like that.
  • Mitigate the macOS issue where statfs is called continuously

cannot upload/make folders

Hi, I don't understand why I seem to be the only one with this problem, but I can't figure out what I did incorrectly either in the installation or usage.
Other than in the home directory of my mounted OneDrive folder, every time I mkdir/vim a new folder/file or even try to edit a file like "libreoffice xxxx" or "vim xxxx" (read-only "less xxxx" works), onedriver immediately crashes and stops with:
"- FATAL - 1:drive_item.go:70:NewDriveItem() - Parent or parent cache cannot be nil:"
last message before that is the creation of the files/folders, e.g.:
"- TRACE - 1:fusefs.go:316:Create() - "

Deleting files/folders seem to be fine though. Also, everything seems to work perfectly fine in the root directory of my OneDrive.

Fix race conditions in filesystem

Sync every FS operation is called asynchronously as a goroutine by go-fuse, we'll need to wrap sensitive filesystem/drive item resources to prevent race conditions.

  • Add race condition tests to test suite
  • Eliminate all race conditions in current codebase

Sometimes some images gets corrupted

Yesterday I've uploaded a folder with some wallpapers in it, today, when I opened that folder I've noticed some files are corrupted.
This is a normal file:
image
This is a corrupted file (look at the size and at the file format):
image

While I was writing this up, I've noticed a bunch of corrupted files I've lost on another folder.
image

I don't know the cause of the problem.

My system:
image

Disable thumbnailing for mount

Upon mounting onedriver and viewing the mounted folder in Nautilus, Nautilus/GNOME will attempt to download every single file sequentially in order to thumbnail them. After some research it appears impossible to tell GNOME to ignore thumbnailing the mount (meaning that Nautilus will try to download everything on every mount, which is a huge bandwidth suck). However, if GNOME fails to create a thumbnail, it'll create a failed thumbnail and never retry. Perhaps we can create a failed thumbnail for every item in the drive on populating the cache to prevent Nautilus from downloading things when we don't want it to.

Ref: https://developer.gnome.org/gnome-desktop/stable/GnomeDesktopThumbnailFactory.html

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.