Code Monkey home page Code Monkey logo

cloudfuse's Introduction

Cloudfuse is a FUSE application which provides access to Rackspace's
Cloud Files (or any installation of Swift).

Cloud Files is a remote storage system which is similar in principle to
Amazon S3.  It provides a simple RESTful interface to storing and retrieving
objects.

    http://www.rackspace.com/cloud/files/

Swift, the software behind Cloud Files, has been open-sourced as part of the
OpenStack project.

    http://swift.openstack.org/


BUILDING:

    You'll need libcurl, fuse, libssl, and libxml2 (and probably their dev
    packages) installed to build it.  From a base Debian or Ubuntu install,
    this should get you to a point you can build and run it:
        apt-get install build-essential libcurl4-openssl-dev libxml2-dev \
             libssl-dev libfuse-dev libjson-c-dev

    For CentOS or similar,
        yum install gcc make fuse-devel curl-devel libxml2-devel openssl-devel

    Cloudfuse is built and installed like any other autoconf configured code.
    Normally,
        ./configure
        make
        sudo make install

    But I'm no autoconf wizard, and there may be dragons lurking there.


USE:

    You'll need to install fuse to use this application. It may have already
    been installed as a dependency if you followed the "BUILDING" instructions
    above.
    
    On Debian:
       apt-get install fuse
       
    On CentOS or similar:
       yum install fuse

    The following settings can be defined in the file ~/.cloudfuse:
        username=[Account username for authentication, required]
        api_key=[API key for authentication with Rackspace]
        tenant=[Tenant name for authentication with Openstack]
        password=[Authentication password with Openstack]
        authurl=[Authentication url, defaults to Rackspace's cloud]
        region=[Regional endpoint to use]
        use_snet=[True to use Rackspace ServiceNet for connections]
        cache_timeout=[Seconds for directory caching, default 600]
        verify_ssl=[False to disable SSL cert verification]

    For authenticating with Rackspace's cloud, at minimum "username" and
    "api_key" must be set.

    For authenticating with Keystone, "username", "password", "tenant", and
    "authurl" should probably be defined.

    These settings can also be specified as mount options on the command line:
        cloudfuse -o username=redbo,api_key=713aa... mountpoint/

    Or as mount options in /etc/fstab:
        cloudfuse /mnt/cloudfiles fuse username=redbo,api_key=713aa...,user 0 0

    It also inherits a number of command-line arguments and mount options from
    the Fuse framework.  The "-h" argument should provide a summary.


EXAMPLE:

    A typical ~/.cloudfuse configuration file for use with Rackspace:
        username=demo
        api_key=643afce8b5187d40ba15e4827384fc5b
        
        # if no region is selected, it will use your default region.
        #   region=DFW
        
        # if connecting within a Rackspace datacenter, ServiceNet can be
        # used to avoid bandwidth charges.
        #   use_snet=true

    A typical ~/.cloudfuse configuration file for use with OpenStack,
    noting that "tenant" should be the tenant name, rather than the ID:
        username=demo
        tenant=demo
        password=supersecret
        authurl=http://10.10.0.1:5000/v2.0

DEBUGGING:
    Cloudfuse may demonstrate the following symptoms:
        - console locks up for minutes
        - error message "Link has been severed"
        
    You can debug these situations in two ways:
        (1) Look at /var/log/syslog or /var/log/messages on your swift server
        (2) Use the -d, -f, or debug flag in your cloudfuse config and watch
            the logs or console for errors.
        
    Frequently, these are network or certificate errors.  Swift may not log 
    certificate failures, so the debug switch is important if you're using
    cloudfuse for the first time, particularly in a non-Rackspace environment.
    
    e.g. * Peer certificate cannot be authenticated with given CA certificates
         * About to connect() to 192.168.0.2 port 8080 (#0)
         *   Trying 192.168.0.2... * connected
         * successfully set certificate verify locations:
         *   CAfile: none
           CApath: /etc/ssl/certs
         * SSL certificate problem, verify that the CA cert is OK. Details:
         error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
         * Closing connection #0


BUGS/SHORTCOMINGS:

    * rename() doesn't work on directories (and probably never will).
    * When reading and writing files, it buffers them in a local temp file.
    * It keeps an in-memory cache of the directory structure, so it may not be
      usable for large file systems.  Also, files added by other applications
      will not show up until the cache expires.
    * The root directory can only contain directories, as these are mapped to
      containers in cloudfiles.
    * Directory entries are created as empty files with the content-type
      "application/directory".
    * Cloud Files limits container and object listings to 10,000 items.
      cloudfuse won't list more than that many files in a single directory.


AWESOME CONTRIBUTORS:

    * Tim Dysinger                                 https://github.com/dysinger
    * Chris Wedgwood                               https://github.com/cwedgwood
    * Nick Craig-Wood                              https://github.com/ncw
    * Dillon Amburgey                              https://github.com/dillona
    * Manfred Touron                               https://github.com/moul
    * David Brownlee                               https://github.com/abs0
    * Mike Lundy                                   https://github.com/novas0x2a
    * justinb                                      https://github.com/justinsb


Thanks, and I hope you find it useful.

Michael Barton
<[email protected]>

cloudfuse's People

Contributors

abs0 avatar clayg avatar cwedgwood avatar davidjb avatar dillona avatar dysinger avatar fordodone avatar gaul avatar h6w avatar justinsb avatar lucasvr avatar maxstepanov avatar moul avatar ncw avatar novas0x2a avatar pataquets avatar redbo avatar stevenshiau avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cloudfuse's Issues

Appending to files is broken

To reproduce:

$ echo "foo" > /path/to/cloudfuse/bucket/filename
$ echo "bar" >> /path/to/cloudfuse/bucket/filename
$ cat /path/to/cloudfuse/bucket/filename
bar

omitting directory ‘cloudfuse’

hi all, when i tried to install cloudfuse
i have this error:
/usr/bin/install: omitting directory ‘cloudfuse’
make: *** [install] Error 1

i did follow all the instructions but not work, anyone who can help me?

Thanks
Will

libxml2 requested during ./configure

Hello Cloudfuse ;-)

Seem unable to satisfy ./configure that libxml2 is present / available on centos 5.5.

Any suggestions ?

== copied from console ==

Installed:
libxml2.i386 0:2.6.26-2.1.2.8.el5_5.1

Updated:
libxml2.x86_64 0:2.6.26-2.1.2.8.el5_5.1

Complete!
[root@rackspace-01 redbo-cloudfuse-0c63cc1]# ./configure
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for a BSD-compatible install... /usr/bin/install -c
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for XML... no
configure: error: 'Unable to find libxml2. Please make sure library and header files are installed.'

== thanks ==

Attempting to touch (update access/modification time) wipes file contents in object store

If I run the touch command on an existing file on a cloudfuse directory (update access / modification times), the contents are wiped, including on the server's object store.

[ec2-user@production backup]$ ls -l
-rw-rw-rw- 1 root root 2000000000 Feb 27 06:56 backup.tar.aa
[ec2-user@production backup]$ touch backup.tar.aa
touch: setting times of `backup.tar.aa': Function not implemented
[ec2-user@production backup]$ ls -l
-rw-rw-rw- 1 root root          0 Feb 27 06:56 backup.tar.aa

Attempting to download/inspect the file via the OpenStack dashboard reveals the file is empty. I'm not worried about touch not working (access/modification times don't concern me) - but as the attempt to do this equals wiping the file, this could easily lead to inadvertent data loss.

mmap fails when using MAP_SHARED option

strace shows:
mmap(NULL, 400, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = -1 ENODEV (No such device)

Although MAP_PRIVATE seems to work without issue, but not useful...

can cloudfuse work with memory mapped files?

dir double list

root@Com3:/cloud/build# df -h | grep cloudfuse
cloudfuse 8.0T 0 8.0T 0% /cloud
root@Com3:/cloud/build# ls -l
total 4
-rw-rw-rw- 1 root root 185 Dec 8 23:56 pip-delete-this-directory.txt
drwxr-xr-x 2 root root 0 Dec 9 09:10 python-swiftclient
drwxr-xr-x 2 root root 0 Dec 9 09:10 python-swiftclient

the dir "python-swiftclinet" has been list double

can you help me?

Problems with cloudfuse, using rsync and files over 40MB

I have approximately ~550 files in a cloudfiles container, with mostly 1-10MB files. I have a few 40-140MB files. I'm using rsync to copy those files to disk (actually I'm using rsync to copy them to S3, but the problem presents itself even when copying to local disk, and I've been copying to local disk for debugging).

When copying the larger files, rsync reports the job being completed, but only lists ~10KB transferred. The resulting file is a 0 (zero) byte file. If I run rsync again, it will of course try to sync the file again, but usually just produce the same zero byte file with an updated timestamp. Running rsync in the middle of the night (PST) tends to produce better results, and the file will transfer successfully ~1 in 10 times it is attempted. During the day, less - which leads me to believe it might be related to rackspace API or cloudfiles load?

Copying from cloudfiles using standard "cp" unix command works every time.

Running cloudfuse in debug mode shows a 200 HTTP response code from rackspace, and looks the same when rsync runs as it does when unix cp runs.

I'm mounting cloudfiles using:
cloudfuse -f -o ro,allow_other /mnt/cloudfiles-prod/
(I run into the same problem when mounting it read/write).

rsync:
[root@admin3-prod s3-prod-backup]# rsync -hivrd --temp-dir=/tmp/test/ --inplace --size-only --whole-file --stats /mnt/cloudfiles-prod/prod-tma/ /tmp/s3-prod-backup
building file list ... done

f+++++++ c3029_p697_d2129_f1227

Number of files: 536
Number of files transferred: 1
Total file size: 797.49M bytes
Total transferred file size: 140.75M bytes
Total bytes sent: 10.61K
Total bytes received: 42

sent 10.61K bytes received 42 bytes 1.25K bytes/sec
total size is 797.49M speedup is 74839.34

change /tmp location

I would like to cache files anywhere else but /tmp
I'm speaking of course about the temporary space cloudfuse uses to store files before the actual transfer.
Since im using "secure temp" on my cpnael system it is limited in size, in this case 2 GB, so im limited to storing files no bigger than 2GB which is a big limitation.

To solve this I've tried changing the tmpdir= in install-sh
to this:
tmpdir=/fusetmp/ins$RANDOM-$$

but still, nothing was changed.

is there a configuration option im missing?

Is adding a user to the fuse group necessary?

After installing fuse, a fuse group was not created. I'm wondering if adding a user to the fuse group ( sudo usermod -a -G fuse [username]) is really necessary? It doesn't appear to make a difference.

Copying large files get stuck (Debian)

Hey,
When trying to copy cloudserver image (~1GB) from mounted container, the copy gets stuck.
I'm using iftop to monitor network, and while the copy starts with reasonable transfer rate, it decreases with progress, and at some point it gets stuck.
When trying to kill cp / cloudfuse processes, the container hangs and ls -la gives:
d????????? 0 user user 4096 Sep 19 22:36 mountpoint

How to make mount point accessible to php?

Hi

Not sure if this the right place for this, but here goes.

I've got a mount point successfully setup and working (i.e. I can drag files into containers). I now want a PHP application to be able to write to it, but the php function is_dir() is returning false on the containers within. This means I can't really do anything with it.

The mount command I use is:

cloudfuse -o umask=0777,username=x,api_key=y,authurl=https://lon.auth.api.rackspacecloud.com/v1.0 /media/thecloud/

I don't think it's a question of permissions, as PHP will return a correct result for is_dir() even if it cannot write to it.

anyone got any ideas? many thanks

/etc/fstab not working

This mounts and works fine for me. Now I need to get cloudfuse automounting with /etc/fstab working. It won’t auto-mount at boot. If anyone can help, I am using the following line in that file:
cloudfuse /home/user/cloud fuse defaults,allow_other,gid=1002,umask=007,username=cloudusername,api_key=XXXXXXXXX 0 0

Also, if I have that in fstab and I try mount -a I get this output:
mount: wrong fs type, bad option, bad superblock on cloudfuse,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount. helper program)
In some cases useful info is found in syslog – try
dmesg | tail or so

I think a clue is that after a reboot, if I manually try to mount it with the cloudfuse command, I initially get this message:
fuse: device not found, try ‘modprobe fuse’ first

and if I enter ‘modprobe fuse’ and then try to manually mount, it works.

Can anyone tell me what I need to do to make this work with fstab?

Mounted Folders Don't Display to FTP Users

After mounting the folder and accessing the server with FTP, the mounted folder does not seem to exist anywhere, and can not be navigated to manually.

This is a pain for some programs that need FTP for backup purposes.

Files not showing / displaying / listing for .CDN_ACCESS_LOGS/ RackSpace log directory.

RackSpace Support Suggested I post this issue here:

I have a mounted CDN parent directory with several subdirectories.

meuser@mydomain: ~$ cdn
total 4.0K
d-------wx 2 root root 0 Dec 31 1996 ./
drwxr-xr-x 3 root root 4.0K Feb 11 2012 ../
d-------wx 2 root root 0 Sep 11 20:01 .CDN_ACCESS_LOGS/
d-------wx 2 root root 0 Sep 11 20:01 cloudservers/
d-------wx 2 root root 0 Sep 11 20:01 mymainbackups/
d-------wx 2 root root 0 Sep 11 20:01 mymainfiles/

I automount at boot using /etc/crontab. Everything works great except that I can
not see the files in the .CDN_ACCESS_LOGS/ directory.

I can read/write/delete etc all the files in all of the subdirectories except for .CDN_ACCESS_LOGS/

In actuality, there are 4 log files in .CDN_ACCESS_LOGS/ which I can see through the RackSpace control panel.

meuser/2012/07/30/19/317ea70e8f004ac386cfffbf51adac9.log.gz
meuser/2012/07/30/20/241f5a989cb44896b916f2aaf134f6c.log.gz
meuser/2012/08/11/20/749ce9176fa64bb9babe9b3e65ea7c9.log.gz
meuser/2012/08/12/15/5cb28ac4f00747a0b7cd0835f7823e6.log.gz

I can open the .CDN_ACCESS_LOGS/ directory on the server, but it appears empty.

The files are not hidden, they just don't show up.

Becoming root does not help.

I have restarted and remounted using the following (which results in 003 and 644 permissions, respectively):
modprobe fuse
cloudfuse /media/cloudfiles -o allow_other,umask=774,nonempty (and umask=133)

Also tried the following (which results in 755 permissions):
modprobe fuse
cloudfuse /media/cloudfiles -o allow_other,nonempty

But to no avail.

I can not rationalize the behavior of this one CDN mount subdirectory using CloudFuse. Neither can RackSpace Support.

cloudfuse can't connect on CentOs 5.8

I try compile cloudfuse on Debian Testing. It's compile fine. All works, all connect.

But if compiling cloudfuse in CentOs 5.8, then all compiling fine, but dont connect with same option in Debian.

I diff ./configure output in Debian and in CentOs - its identical:

checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for a BSD-compatible install... /usr/bin/install -c
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for XML... yes
checking for CURL... yes
checking for FUSE... yes
checking for OPENSSL... yes
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for size_t... yes
checking for working alloca.h... yes
checking for alloca... yes
checking for ANSI C header files... (cached) yes
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking for stdint.h... (cached) yes
checking stddef.h usability... yes
checking stddef.h presence... yes
checking for stddef.h... yes
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking for strings.h... (cached) yes
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking for unistd.h... (cached) yes
checking pthread.h usability... yes
checking pthread.h presence... yes
checking for pthread.h... yes
checking fuse.h usability... no
checking fuse.h presence... no
checking for fuse.h... no
checking curl/curl.h usability... yes
checking curl/curl.h presence... yes
checking for curl/curl.h... yes
checking libxml/tree.h usability... no
checking libxml/tree.h presence... no
checking for libxml/tree.h... no
checking openssl/crypto.h usability... yes
checking openssl/crypto.h presence... yes
checking for openssl/crypto.h... yes
checking for an ANSI C-conforming const... yes
checking for uid_t in sys/types.h... yes
checking for mode_t... yes
checking for off_t... yes
checking for size_t... (cached) yes
checking whether struct tm is in sys/time.h or time.h... time.h
checking for struct stat.st_blocks... yes
checking for stdlib.h... (cached) yes
checking for GNU libc compatible malloc... yes
checking whether time.h and sys/time.h may both be included... yes
checking for sys/time.h... (cached) yes
checking for unistd.h... (cached) yes
checking for alarm... yes
checking for working mktime... yes
checking return type of signal handlers... void
checking for vprintf... yes
checking for _doprnt... no
checking for ftruncate... yes
checking for memmove... yes
checking for strcasecmp... yes
checking for strchr... yes
checking for strdup... yes
checking for strncasecmp... yes
checking for strrchr... yes
checking for strstr... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating config.h

But in CentOs cloudfuse don't connected (use same options in Debian):

$ cloudfuse -o username=AAA,api_key=XXX,authurl=https://auth.selcdn.ru /mnt/storage
<html>
 <head>
  <title>403 Forbidden</title>
 </head>
 <body>
  <h1>403 Forbidden</h1>
  Access was denied to this resource.<br /><br />
 </body>
Unable to authenticate.

Why? How to correct compile cloudfuse in CentOs 5.8?

just a ? not bug: use_snet=true

is there a way to confirm that use_snet=true actually uses servicenet. this question may sound ridiculous but I want to avoid a spike charge in my nightly processes. I am already using Jungle Disc on the same server taking advantage of servicenet but I wanted to confirm this utility as well.

thank you for your help,

chris

"Link has been severed" issue on RHEL/CentOS when use_snet=true

This is related to #38 and #50.

The "Link has been severed" issue still appears when cloudfuse is compiled on RHEL/CentOS systems, but only if use_snet=true. If use_snet=false, everything seems to work just fine.

Compiling with a statically linked libcurl appears to fix the issue, but this is not ideal for most systems.

Torrenting

I'm unable to torrent by porting through cloudfuse to Cloud Files, unless I'm doing things wrong here. I followed these instructions, hoping that it would resolve any conflicts, but I'm getting a Storage error while using rtorrent.

cloudfuse over private net

Hello, sorry for my question, but i'm new to rackspace and i need to learn a lot.

I'm testing cloudfuse to do incremental backups from my cloud servers to cloud files.

questions:

  1. i have to pay for bandwidth used from and to cloud files and cloud servers? Tech sopport said that i can use a private net to comunicate with cloud files from my cloud servers with the API... Is cloudfuse using this private net?

  2. i think i'll use rdiff-backup over cloud fuse... do you think this is a good solution? or do you have others solutions to suggest me?

Sorry for my english and thank you very much!!

rdiff-backup doesn't work

I'm trying to use rdiff-backup over cloudfuse, but il doesn't work.

this is a simpe test and the output.... any help?

root@backup:~# rdiff-backup /etc/ /cloudfiles/testbackup/
Warning: hard linking not supported by filesystem at /cloudfiles/testbackup/rdiff-backup-data
Exception '[Errno 38] Function not implemented: '/cloudfiles/testbackup/rdiff-backup.tmp.1'' raised of class '<type 'exceptions.OSError'>':
File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 304, in error_check_Main
try: Main(arglist)
File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 324, in Main
take_action(rps)
File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 280, in take_action
elif action == "backup": Backup(rps[0], rps[1])
File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 346, in Backup
backup.Mirror(rpin, rpout)
File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line 38, in Mirror
DestS.patch(dest_rpath, source_diffiter)
File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line 232, in patch
ITR(diff.index, diff)
File "/usr/lib/pymodules/python2.6/rdiff_backup/rorpiter.py", line 281, in call
last_branch.fast_process(*args)
File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line 529, in fast_process
if self.patch_to_temp(mirror_rp, diff_rorp, tf):
File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line 559, in patch_to_temp
rpath.copy_attribs(diff_rorp, new)
File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line 191, in copy_attribs
if not rpin.isdev(): rpout.setmtime(rpin.getmtime())
File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line 959, in setmtime
try: self.conn.os.utime(self.path, (long(time.time()), modtime))

Traceback (most recent call last):
File "/usr/bin/rdiff-backup", line 30, in
rdiff_backup.Main.error_check_Main(sys.argv[1:])
File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 304, in error_check_Main
try: Main(arglist)
File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 324, in Main
take_action(rps)
File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 280, in take_action
elif action == "backup": Backup(rps[0], rps[1])
File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 346, in Backup
backup.Mirror(rpin, rpout)
File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line 38, in Mirror
DestS.patch(dest_rpath, source_diffiter)
File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line 232, in patch
ITR(diff.index, diff)
File "/usr/lib/pymodules/python2.6/rdiff_backup/rorpiter.py", line 281, in call
last_branch.fast_process(*args)
File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line 529, in fast_process
if self.patch_to_temp(mirror_rp, diff_rorp, tf):
File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line 559, in patch_to_temp
rpath.copy_attribs(diff_rorp, new)
File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line 191, in copy_attribs
if not rpin.isdev(): rpout.setmtime(rpin.getmtime())
File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line 959, in setmtime
try: self.conn.os.utime(self.path, (long(time.time()), modtime))
OSError: [Errno 38] Function not implemented: '/cloudfiles/testbackup/rdiff-backup.tmp.1'

Buffer overflow with PKI tokens

Cloudfuse uses a 4KB buffer to build request headers.

Starting with the Grizzly version, OpenStack can be configured to use PKI tokens, which are significantly larger than UUIDs (~ 4700 bytes in our tests). The buffer overflows, the end of the token is truncated and authentication fails.

Quick fix: raise the max header size in cloudfsapi.h.

    #define MAX_HEADER_SIZE 8192

df: `/media/cloudfiles': Transport endpoint is not connected

I keep getting this error periodically on my installation of cloud fuse and it occurs both on ubuntu 10.04 as well as 10.10. I am using rackspace cloud servers

when i type a df -h i see this error
df: `/media/cloudfiles': Transport endpoint is not connected

and in the /var/log/syslog i see this error
kernel: [249662.999667] cloudfuse[2459]: segfault at 1000000d7 ip 00007f8da9424095 sp 00007f8da460a000 error 4 in libxml2.so.2.7.7[7f8da93ec000+144000]

it has been really difficult for me to scale due to this issue occurring on my servers at randoom time intervals. My configuration is such that i mount all my media files to the cloud files and then all my servers can access the same files

The second issue is that if the media file is uploaded from one server it takes about 5 to 10 mins for the other server to see it

This is the command i use for mounting cloudfiles

sudo cloudfuse /media/cloudfiles -o allow_other,nonempty

Please let me know whats missing here?

Add header to the object

Hi all!

is there any way put the custom header over the object with Cloudfiles?

I am trying to put the header "Access-Control-Allow-Origin" on all obeject uploaded.

Thanks guys!

subfolders don't work

My cloud files container has subfolders, but they don't show up in my mounted cloud fuse dir. Is there something special that has to be done to make this work?

Not sure why.

I'm unable to transfer any files larger than around 256mb. Any ideas?

Two attempted transfers below:

[Did Transfer]

PUT /v1//backups%5Falpha/256 HTTP/1.1
User-Agent: CloudFuse
Host: snet-storage4.clouddrive.com
Accept: /
X-Auth-Token:
Content-Type: application/octet-stream
Content-Length: 268435456

[Did not transfer]

PUT /v1//backups%5Falpha/1024 HTTP/1.1
User-Agent: CloudFuse
Host: snet-storage4.clouddrive.com
Accept: /
X-Auth-Token:
Content-Type: application/octet-stream
Content-Length: 1073741824

Cause of Transport end point is not connected

Hi,
first of all I wan't to thank you A LOT for this great app. It makes all cloudfiles operations of my website a lot easier and FAST. :)

But there is one issue I'm facing that can´t provide the reliability I need for put it on productions servers: I'm using cloudfuse to commit uploaded files from my website to cloudfiles, when I do that, I create/del/copy dirs and files. Your app take cares of it really good and fast using (mkdir, unlink, copy in PHP). THE THING IS THAT sometimes, without a debug output, after a few operations an error is triggered and when I try to access cloudfiles mount point again I get this message Transport end point is not connected. Can you do any sort of redundancy (If it fails, try to connect and send the file again, but leave mount point intact)? And about the 10,000 file listing limits, can you also do a recursive function for listing that catch all files? I just did that in php api and works well.

I'm on Debian Lenny. PHP 5.3, Apache 2.2. In my settings I use apache GID and UID, async_read, and snet.

Please can you help me or fix this issue?
Thank you very mutch, I'll appreciate any help.

subdir module and symlinks

I am using this great tool you made and I can link into the cloudfuse mount but not back outside of it with symlinks I seenthe subdir module and rellinks option and wonder if that would do the trick? I just cant seem to figure out the sequence for the module option i have tried module=subdir[rellinks] module=subdir:rellinks module=subdir,rellinks , nothing gives any help would be appreciated.

install fails if /usr/local/bin does not exist

# make install
/usr/bin/install -c cloudfuse /usr/local/bin/cloudfuse
/usr/bin/install: cannot create regular file `/usr/local/bin/cloudfuse': No such file or directory
make: *** [install] Error 1

Reliability or Config Issues?

I'm curious about the reliability of cloudfuse, or if there is any other documentation which can be referenced to understand the implication of various options. I'm wondering if there are things you can or can't do with it. I'm trying to get this working, but am having some odd behavior.

First, I can only run in dev mode (with the -d option). This allows me to create directories, move and copy files. When running without dev mode, doing anything in the mounted directory (including ls) seems to lock up the console session. mv produces errors like "preserving times for XYZ: function not implemented, but the mv seems to work.

I'm also curious if anyone has gotten this to work as a file system with ftp. I'm getting frequent timeouts when trying to list contents or do uploads, but sometimes it works okay. Not sure if this is due to dev mode or not. Again, running without -d mode, nothing seems to happen, and I usually need to kill everything and remount.

Thanks.

rsync issues

I am trying to implement a backup solution from a local Linux server to cloud files. The local system has 300G in 600,000+ files. A very small percentage of files changes daily, which would be an ideal situation for rsync. Cloudfuse is used on the system being backed up, and thus has a local mount point. rsync has been very problematic, and I have yet to get the filesystem completely synced for the first time (after trying for 2 weeks). Using rsync to sync to and from clould servers on the other hand (Linux to Linux) is very reliable, so I don't think there are any network type issues.

rsync will always fail at some point. Sometimes it fails in seconds, sometimes minutes, and sometimes hours. Typically, the process just hangs with no error message. But sometimes it dies outright, with error messages. Typically, the mountpoint is fubar and cloudfuse has to be killed manually, to get back to a sane starting point.

Below is a list of various error messages.

If anyone is using rsync successfully in any similar situation, I'd appreciate knowing what options are being invoked with rsync as I cannot find a successful combination. I wonder if this is even feasible and whether I am wasting my life trying :/

Error potpourri:

rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe (32)
rsync: close failed on "/mnt/rackspace/backups/rsync/titan/raid/profiles/": No such file or directory (2)
rsync: connection unexpectedly closed (1283 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(632) [sender=3.0.4]

rsync: recv_generator: mkdir "/mnt/rackspace/backups/rsync/titan/raid/profiles/hanna" failed: Transport endpoint is not connected (107)
*** Skipping any contents from this failed directory ***
(NOTE: This is bogus, the directory in question does indeed exist and nothing unusual about it).
1.
io timeout after 900 seconds -- exiting
rsync error: timeout in data send/receive (code 30) at io.c(237) [sender=3.0.4]

rsync: connection unexpectedly closed (4873 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(632) [sender=3.0.4]

fuse: bad mount point `/mnt/rackspace': Transport endpoint is not connected

Can't find where cloudfuse cache files...

Hi,
I'm looking to parse temp files downloaded by cloudfuse, but I can't find where.

I was thinking that it was in TMP dir but there is nothing !

Do you know where cloudfuse store temp files ?

Thanks for your help !!

Unable to build, Ubuntu 9.10, AMD64

./configure runs ok, but make fails. I do have curl, libxml2, libxml2-dev, libfuse, libfuse-dev.

The first few lines of the make output are:
gcc -D_FILE_OFFSET_BITS=64 -I. -I/usr/include/libxml2 -g -O2 -o cloudfuse cloudfsapi.c cloudfuse.c -lxml2 -lxml2
In file included from cloudfsapi.c:12:
cloudfsapi.h:4:23: error: curl/curl.h: No such file or directory
cloudfsapi.h:5:24: error: curl/types.h: No such file or directory
cloudfsapi.h:6:23: error: curl/easy.h: No such file or directory

OUTPUT OF CONFIGURE

********************************************

$ ./configure
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for a BSD-compatible install... /usr/bin/install -c
checking for main in -lcurl... no
checking for main in -lfuse... no
checking for main in -lmagic... no
checking for main in -lxml2... yes
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for working alloca.h... yes
checking for alloca... yes
checking for ANSI C header files... (cached) yes
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking for unistd.h... (cached) yes
checking for an ANSI C-conforming const... yes
checking for uid_t in sys/types.h... yes
checking for mode_t... yes
checking for off_t... yes
checking for size_t... yes
checking whether struct tm is in sys/time.h or time.h... time.h
checking for stdlib.h... (cached) yes
checking for GNU libc compatible malloc... yes
checking whether time.h and sys/time.h may both be included... yes
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking for unistd.h... (cached) yes
checking for alarm... yes
checking for working mktime... yes
checking for stdlib.h... (cached) yes
checking for GNU libc compatible realloc... yes
checking for ftruncate... yes
checking for memmove... yes
checking for strcasecmp... yes
checking for strchr... yes
checking for strdup... yes
checking for strncasecmp... yes
checking for strrchr... yes
checking for strstr... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating config.h

OUTPUT OF MAKE

********************************************

steve@digi05:~/Downloads/redbo-cloudfuse-7643a7f$ make
gcc -D_FILE_OFFSET_BITS=64 -I. -I/usr/include/libxml2 -g -O2 -o cloudfuse cloudfsapi.c cloudfuse.c -lxml2 -lxml2
In file included from cloudfsapi.c:12:
cloudfsapi.h:4:23: error: curl/curl.h: No such file or directory
cloudfsapi.h:5:24: error: curl/types.h: No such file or directory
cloudfsapi.h:6:23: error: curl/easy.h: No such file or directory
cloudfsapi.c: In function ‘dispatch_clear’:
cloudfsapi.c:144: warning: ignoring return value of ‘ftruncate’, declared with attribute warn_unused_result
cloudfsapi.c: In function ‘send_request’:
cloudfsapi.c:175: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘’ token
cloudfsapi.c:175: error: ‘curl_pool’ undeclared (first use in this function)
cloudfsapi.c:175: error: (Each undeclared identifier is reported only once
cloudfsapi.c:175: error: for each function it appears in.)
cloudfsapi.c:201: warning: assignment makes pointer from integer without a cast
cloudfsapi.c:205: error: ‘CURL’ undeclared (first use in this function)
cloudfsapi.c:205: error: ‘curl’ undeclared (first use in this function)
cloudfsapi.c:210: error: ‘CURLOPT_UPLOAD’ undeclared (first use in this function)
cloudfsapi.c:211: error: ‘CURLOPT_INFILESIZE’ undeclared (first use in this function)
cloudfsapi.c:212: error: ‘CURLOPT_READDATA’ undeclared (first use in this function)
cloudfsapi.c:213: warning: assignment makes pointer from integer without a cast
cloudfsapi.c:223: warning: assignment makes pointer from integer without a cast
cloudfsapi.c:227: error: ‘CURLOPT_NOBODY’ undeclared (first use in this function)
cloudfsapi.c:228: error: ‘CURLOPT_CUSTOMREQUEST’ undeclared (first use in this function)
cloudfsapi.c:234: warning: assignment makes pointer from integer without a cast
cloudfsapi.c:235: error: ‘CURLOPT_VERBOSE’ undeclared (first use in this function)
cloudfsapi.c:236: error: ‘CURLOPT_WRITEHEADER’ undeclared (first use in this function)
cloudfsapi.c:237: error: ‘CURLOPT_HEADER’ undeclared (first use in this function)
cloudfsapi.c:238: error: ‘CURLOPT_FOLLOWLOCATION’ undeclared (first use in this function)
cloudfsapi.c:239: error: ‘CURLOPT_MAXREDIRS’ undeclared (first use in this function)
cloudfsapi.c:240: error: ‘CURLOPT_NOSIGNAL’ undeclared (first use in this function)
cloudfsapi.c:241: error: ‘CURLOPT_TIMEOUT’ undeclared (first use in this function)
cloudfsapi.c:242: error: ‘CURLOPT_CONNECTTIMEOUT’ undeclared (first use in this function)
cloudfsapi.c:243: error: ‘CURLOPT_HTTPHEADER’ undeclared (first use in this function)
cloudfsapi.c:244: error: ‘CURLOPT_USERAGENT’ undeclared (first use in this function)
cloudfsapi.c:245: error: ‘CURLOPT_URL’ undeclared (first use in this function)
cloudfsapi.c:246: error: ‘CURLOPT_HEADERFUNCTION’ undeclared (first use in this function)
cloudfsapi.c:248: error: ‘CURLOPT_WRITEDATA’ undeclared (first use in this function)
cloudfsapi.c:252: error: ‘CURLOPT_WRITEFUNCTION’ undeclared (first use in this function)
cloudfsapi.c:254: error: ‘CURLOPT_SSL_VERIFYPEER’ undeclared (first use in this function)
cloudfsapi.c:256: error: ‘CURLINFO_RESPONSE_CODE’ undeclared (first use in this function)
cloudfsapi.c: In function ‘object_read_from’:
cloudfsapi.c:286: warning: initialization makes pointer from integer without a cast
cloudfsapi.c: In function ‘object_write_to’:
cloudfsapi.c:297: warning: initialization makes pointer from integer without a cast
cloudfsapi.c:304: warning: ignoring return value of ‘ftruncate’, declared with attribute warn_unused_result
cloudfsapi.c: In function ‘object_truncate’:
cloudfsapi.c:313: warning: initialization makes pointer from integer without a cast
cloudfsapi.c: In function ‘list_directory’:
cloudfsapi.c:339: warning: initialization makes pointer from integer without a cast
cloudfsapi.c:340: warning: initialization makes pointer from integer without a cast
cloudfsapi.c: In function ‘delete_object’:
cloudfsapi.c:418: warning: initialization makes pointer from integer without a cast
cloudfsapi.c: In function ‘create_directory’:
cloudfsapi.c:426: warning: initialization makes pointer from integer without a cast
cloudfsapi.c: In function ‘cloudfs_connect’:
cloudfsapi.c:447: error: ‘CURL_GLOBAL_ALL’ undeclared (first use in this function)
cloudfsapi.c:464: warning: assignment makes pointer from integer without a cast
cloudfsapi.c:466: warning: assignment makes pointer from integer without a cast
cloudfuse.c:2:18: error: fuse.h: No such file or directory
In file included from cloudfuse.c:12:
cloudfsapi.h:4:23: error: curl/curl.h: No such file or directory
cloudfsapi.h:5:24: error: curl/types.h: No such file or directory
cloudfsapi.h:6:23: error: curl/easy.h: No such file or directory
cloudfuse.c:212: warning: ‘struct fuse_file_info’ declared inside parameter list
cloudfuse.c:212: warning: its scope is only this definition or declaration, which is probably not what you want
cloudfuse.c: In function ‘cfs_fgetattr’:
cloudfuse.c:214: error: dereferencing pointer to incomplete type
cloudfuse.c:216: error: dereferencing pointer to incomplete type
cloudfuse.c: At top level:
cloudfuse.c:224: error: expected declaration specifiers or ‘...’ before ‘fuse_fill_dir_t’
cloudfuse.c:224: warning: ‘struct fuse_file_info’ declared inside parameter list
cloudfuse.c:246: warning: ‘struct fuse_file_info’ declared inside parameter list
cloudfuse.c: In function ‘cfs_create’:
cloudfuse.c:249: error: dereferencing pointer to incomplete type
cloudfuse.c:252: error: dereferencing pointer to incomplete type
cloudfuse.c: At top level:
cloudfuse.c:256: warning: ‘struct fuse_file_info’ declared inside parameter list
cloudfuse.c: In function ‘cfs_open’:
cloudfuse.c:259: error: dereferencing pointer to incomplete type
cloudfuse.c:259: error: ‘O_WRONLY’ undeclared (first use in this function)
cloudfuse.c:259: error: (Each undeclared identifier is reported only once
cloudfuse.c:259: error: for each function it appears in.)
cloudfuse.c:268: error: dereferencing pointer to incomplete type
cloudfuse.c:269: error: dereferencing pointer to incomplete type
cloudfuse.c: At top level:
cloudfuse.c:274: warning: ‘struct fuse_file_info’ declared inside parameter list
cloudfuse.c: In function ‘cfs_read’:
cloudfuse.c:276: error: dereferencing pointer to incomplete type
cloudfuse.c:277: error: dereferencing pointer to incomplete type
cloudfuse.c: At top level:
cloudfuse.c:280: warning: ‘struct fuse_file_info’ declared inside parameter list
cloudfuse.c: In function ‘cfs_release’:
cloudfuse.c:282: error: dereferencing pointer to incomplete type
cloudfuse.c:283: error: dereferencing pointer to incomplete type
cloudfuse.c:283: error: ‘O_RDWR’ undeclared (first use in this function)
cloudfuse.c:283: error: dereferencing pointer to incomplete type
cloudfuse.c:283: error: ‘O_WRONLY’ undeclared (first use in this function)
cloudfuse.c:285: error: dereferencing pointer to incomplete type
cloudfuse.c:295: error: dereferencing pointer to incomplete type
cloudfuse.c: At top level:
cloudfuse.c:309: warning: ‘struct fuse_file_info’ declared inside parameter list
cloudfuse.c: In function ‘cfs_ftruncate’:
cloudfuse.c:311: error: dereferencing pointer to incomplete type
cloudfuse.c:312: error: dereferencing pointer to incomplete type
cloudfuse.c: At top level:
cloudfuse.c:317: warning: ‘struct fuse_file_info’ declared inside parameter list
cloudfuse.c: In function ‘cfs_write’:
cloudfuse.c:319: error: dereferencing pointer to incomplete type
cloudfuse.c:321: error: dereferencing pointer to incomplete type
cloudfuse.c: At top level:
cloudfuse.c:334: warning: ‘struct fuse_file_info’ declared inside parameter list
cloudfuse.c:345: warning: ‘struct statvfs’ declared inside parameter list
cloudfuse.c: In function ‘cfs_statfs’:
cloudfuse.c:347: error: dereferencing pointer to incomplete type
cloudfuse.c:348: error: dereferencing pointer to incomplete type
cloudfuse.c:349: error: dereferencing pointer to incomplete type
cloudfuse.c:349: error: ‘INT_MAX’ undeclared (first use in this function)
cloudfuse.c:350: error: dereferencing pointer to incomplete type
cloudfuse.c:350: error: dereferencing pointer to incomplete type
cloudfuse.c:351: error: dereferencing pointer to incomplete type
cloudfuse.c:351: error: dereferencing pointer to incomplete type
cloudfuse.c:352: error: dereferencing pointer to incomplete type
cloudfuse.c:353: error: dereferencing pointer to incomplete type
cloudfuse.c:354: error: dereferencing pointer to incomplete type
cloudfuse.c:355: error: dereferencing pointer to incomplete type
cloudfuse.c: In function ‘main’:
cloudfuse.c:389: error: variable ‘args’ has initializer but incomplete type
cloudfuse.c:389: error: storage size of ‘args’ isn’t known
cloudfuse.c:432: error: variable ‘cfs_oper’ has initializer but incomplete type
cloudfuse.c:433: error: unknown field ‘readdir’ specified in initializer
cloudfuse.c:433: warning: excess elements in struct initializer
cloudfuse.c:433: warning: (near initialization for ‘cfs_oper’)
cloudfuse.c:434: error: unknown field ‘mkdir’ specified in initializer
cloudfuse.c:434: warning: excess elements in struct initializer
cloudfuse.c:434: warning: (near initialization for ‘cfs_oper’)
cloudfuse.c:435: error: unknown field ‘read’ specified in initializer
cloudfuse.c:435: warning: excess elements in struct initializer
cloudfuse.c:435: warning: (near initialization for ‘cfs_oper’)
cloudfuse.c:436: error: unknown field ‘create’ specified in initializer
cloudfuse.c:436: warning: excess elements in struct initializer
cloudfuse.c:436: warning: (near initialization for ‘cfs_oper’)
cloudfuse.c:437: error: unknown field ‘open’ specified in initializer
cloudfuse.c:437: warning: excess elements in struct initializer
cloudfuse.c:437: warning: (near initialization for ‘cfs_oper’)
cloudfuse.c:438: error: unknown field ‘fgetattr’ specified in initializer
cloudfuse.c:438: warning: excess elements in struct initializer
cloudfuse.c:438: warning: (near initialization for ‘cfs_oper’)
cloudfuse.c:439: error: unknown field ‘getattr’ specified in initializer
cloudfuse.c:439: warning: excess elements in struct initializer
cloudfuse.c:439: warning: (near initialization for ‘cfs_oper’)
cloudfuse.c:440: error: unknown field ‘release’ specified in initializer
cloudfuse.c:440: warning: excess elements in struct initializer
cloudfuse.c:440: warning: (near initialization for ‘cfs_oper’)
cloudfuse.c:441: error: unknown field ‘rmdir’ specified in initializer
cloudfuse.c:441: warning: excess elements in struct initializer
cloudfuse.c:441: warning: (near initialization for ‘cfs_oper’)
cloudfuse.c:442: error: unknown field ‘ftruncate’ specified in initializer
cloudfuse.c:442: warning: excess elements in struct initializer
cloudfuse.c:442: warning: (near initialization for ‘cfs_oper’)
cloudfuse.c:443: error: unknown field ‘truncate’ specified in initializer
cloudfuse.c:443: warning: excess elements in struct initializer
cloudfuse.c:443: warning: (near initialization for ‘cfs_oper’)
cloudfuse.c:444: error: unknown field ‘write’ specified in initializer
cloudfuse.c:444: warning: excess elements in struct initializer
cloudfuse.c:444: warning: (near initialization for ‘cfs_oper’)
cloudfuse.c:445: error: unknown field ‘unlink’ specified in initializer
cloudfuse.c:445: warning: excess elements in struct initializer
cloudfuse.c:445: warning: (near initialization for ‘cfs_oper’)
cloudfuse.c:446: error: unknown field ‘fsync’ specified in initializer
cloudfuse.c:446: warning: excess elements in struct initializer
cloudfuse.c:446: warning: (near initialization for ‘cfs_oper’)
cloudfuse.c:447: error: unknown field ‘statfs’ specified in initializer
cloudfuse.c:447: warning: excess elements in struct initializer
cloudfuse.c:447: warning: (near initialization for ‘cfs_oper’)
cloudfuse.c:448: error: unknown field ‘chmod’ specified in initializer
cloudfuse.c:448: warning: excess elements in struct initializer
cloudfuse.c:448: warning: (near initialization for ‘cfs_oper’)
cloudfuse.c:449: error: unknown field ‘chown’ specified in initializer
cloudfuse.c:449: warning: excess elements in struct initializer
cloudfuse.c:449: warning: (near initialization for ‘cfs_oper’)
cloudfuse.c:432: error: storage size of ‘cfs_oper’ isn’t known
make: *
* [cloudfuse] Error 1

Incorrect MIME type

It looks like cloudfuse sets the content type for every file to application/octet-stream. This can cause problems when using RSCF as a CDN. For example, Chrome and Firefox will reject CSS files served as application/octet-stream.

cloudfuse ls || mv || cp just hangs

I'm on CentOS 6.2 and the mount and auth are working great!

I'm trying to list the files/containers in my cloudfuse mount and nothing is showing up. My suspicion is it has something to do with my container sizes, as there are several containers that have 20+ GB worth of stuff, namely our cloudserver images.

Could this be an issue? Is there any way to post/watch debugging for list/move/copy operations like there is with the -d switch for Authentication?

UPDATE: it finally timed out

[%]$ ls -la
ls: reading directory .: Link has been severed
total 0

Screen Shot 2013-01-04 at 5 30 29 PM

Spec file

If you're installing this on multiple RHEL-based boxes, you might find building an RPM worth while. I've stuck the spec file I used here:
https://gist.github.com/2960984

I wasn't sure what version number to call the RPM, so I used the date of the latest commit. You will need to update the name of the tarball and %setup -n line to reflect the commit you download.

Hope it saves someone some time.

Configurable settings_filename

Presently cloudfuse only allows command-line options or a hard-coded file at $HOME/.cloudfuse. It should also allow a user-specified file for users with multiple configurations.

cd into subdirectory with many siblings

I understand that cloudfuse cannot list more than 10,000 subdirectories, but can it also not "cd" into a subdirectory if it's it has more than 10,000 siblings? It seems like only the 10,000 that make it into the "ls" are able to entered into with "cd". I am getting the following error:

bash: cd: /path/to/my/file/1/0/281: No such file or directory

Is there a way around this? This is holding us back from finishing something we've been working on, so any help would be appreciated.

Thank you,

Richard

many small files causes for slow copy

I'm using rsync to copy many small files. They're about 1-2K. Using rsync to transfer 200K worth of files takes about 1.5-2 minutes. Similarly, simply deleting the files with rm -rf also took about 1.5-2 minutes.

Creating a 25mb file using: time dd if=/dev/zero of=blah bs=1024 count=25600 also took 1.5-2 minutes.

is this a caching issue?

When mounted at cloudfuse, authority can not be granted.

Hi.

objectstorage to the server in the following "/etc/fstab" settings, it was cloudfuse mount.
~~~/etc/fstab~~
cloudfuse /objectstorage/infra fuse username=*****:*** _,api_key=_,authurl=**__/******_/_/,allow_other,uid=0,gid=12300,umask=127 0 0
~~~~~~~~~~

All the user has (allow_other) I is allowed .
Please tell me how to set the "uid = 0, gid = 12300, umask = 127".

Link has been severed

Running Fedora release 15 (Lovelock)

When i mount a folder to cloudfiles i get an error after trying to list the buckets under the mount point.

cloudfuse /mnt/cloudfiles/

[root@ul966js ~]# :Server )ls -l /mnt/cloudfiles/
ls: reading directory /mnt/cloudfiles/: Link has been severed
total 0

When i run the mount command with the -f parameter everything mounts and works as intended.

cloudfuse -f /mnt/cloudfiles/

I've repeated the exact same process i used to compile/configure cloudfuse on a Debian machine and it works perfectly without the -f flag.

I found the -f flag fix from this URL: http://makewhatis.com/2011/05/mounting-your-rackspace-cloudfiles-in-centosrhel-5-55-6-via-cloudfuse so it looks like this issue affects Centos/RHEL 5.5/5.6 as well.

Curl timeout is too short for some connections

When running cloudefuse I would receive a "Unable to authenticate." error.
Using the same authentication url and credentials, I was able to authenticate.

The root cause is that my connection was timing out. Please increase the default or add an option to set the timeout.

I will post a patch that adds a connect_timeout option.

stability

running a centos 5.8 system, cloudfuse SOMETIMES hangs on heavy usage (mainly my daily backup routine - to the fuse).
Sometimes I have to boot this production server to make fuse work again. Undesirable.
Im cron-remounting every day, and still no go.

using cloudfuse-0.1

newer ver? its not a bug its a feature? HELP!

Unable to connect to cloud files

Hi There,

I have just completed an install of cloudfuse on a Rackspace cloud server to allow us to gain access to cloud files.

As per you READ ME after the installation I created the file .cloudfuse in the $HOME directory with the following content:

username=[our username]
api_key=[our api key]

When I run then run cloudfuse /usr/cloud/ i get:

Unable to authenticate.
Bad username or password

I have spoken with Rackspace who have stated that the user name and API key that I am using are correct and everything is fine from there prospective and have advised that I log an issue.

Thanks

Read only for everyone but root?

This works great for me, but I can not get it to allow anyone but root to write to this location. I even tried mounting it inside the user's home directory, but as soon as it's mounted it reverts back to root:root with 755 permissions. Is there a way to mount it read-write for a non-root user? The mount command output claims it is mounted rw, but it won’t let other users cd to the directory or copy files there. I did create a fuse group, chgrp'd /dev/fuse to the fuse group (it was root:root) and made my normal user a member of the fuse group and remounted but it still doesn't allow them to cd to it or even view it properly with an ls.

For my non-root user, an ls -la of the folder I have mounted looks like this:
d????????? ? ? ? ? ? cloud

for root, it looks like this:
drwxr-xr-x 2 root root 0 Dec 31 1969 cloud

If I try to mount it as the regular user, the same command that works as root:
cloudfuse -o username=mycloudusername,api_key=75847... /home/user/cloud
produces this output:
fuse: failed to exec fusermount: No such file or directory

Am I missing something stupid here?

Mounting just one container

Is it possible to mount just one container. By default, cloudfuse mounts all available containers under the given account. Is it possible to specify the container name to mount as a command line parameter?

Compiling on Ubuntu 10.04

Hey, when I run make on ubuntu 10.04 I get the follow error. First is the configure output

checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for a BSD-compatible install... /usr/bin/install -c
checking for curl_easy_init in -lcurl... yes
checking for fuse_main in -lfuse... yes
checking for xmlCreatePushParserCtxt in -lxml2... yes
checking for magic_buffer in -lmagic... no
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for working alloca.h... yes
checking for alloca... yes
checking for ANSI C header files... (cached) yes
checking curl/curl.h usability... yes
checking curl/curl.h presence... yes
checking for curl/curl.h... yes
checking for fuse.h... yes
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking for unistd.h... (cached) yes
checking for strings.h... (cached) yes
checking pwd.h usability... yes
checking pwd.h presence... yes
checking for pwd.h... yes
checking alloca.h usability... yes
checking alloca.h presence... yes
checking for alloca.h... yes
checking pthread.h usability... yes
checking pthread.h presence... yes
checking for pthread.h... yes
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for XML... yes
checking for an ANSI C-conforming const... yes
checking for uid_t in sys/types.h... yes
checking for mode_t... yes
checking for off_t... yes
checking for size_t... yes
checking whether struct tm is in sys/time.h or time.h... time.h
checking for stdlib.h... (cached) yes
checking for GNU libc compatible malloc... yes
checking whether time.h and sys/time.h may both be included... yes
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking for unistd.h... (cached) yes
checking for alarm... yes
checking for working mktime... yes
checking for stdlib.h... (cached) yes
checking for GNU libc compatible realloc... yes
checking for ftruncate... yes
checking for memmove... yes
checking for strcasecmp... yes
checking for strchr... yes
checking for strdup... yes
checking for strncasecmp... yes
checking for strrchr... yes
checking for strstr... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating config.h

run make:
gcc -g -O2 -I/usr/include/libxml2 -o cloudfuse cloudfsapi.c cloudfuse.c -lxml2 -lfuse -lcurl -lxml2
cloudfsapi.c: In function ‘object_write_fp’:
cloudfsapi.c:240: warning: ignoring return value of ‘ftruncate’, declared with attribute warn_unused_result
cloudfuse.c: In function ‘cfs_fgetattr’:
cloudfuse.c:218: warning: cast to pointer from integer of different size
cloudfuse.c: In function ‘cfs_read’:
cloudfuse.c:284: warning: cast to pointer from integer of different size
cloudfuse.c: In function ‘cfs_flush’:
cloudfuse.c:289: warning: cast to pointer from integer of different size
cloudfuse.c: In function ‘cfs_release’:
cloudfuse.c:310: warning: cast to pointer from integer of different size
cloudfuse.c: In function ‘cfs_ftruncate’:
cloudfuse.c:326: warning: cast to pointer from integer of different size
cloudfuse.c:326: warning: ignoring return value of ‘ftruncate’, declared with attribute warn_unused_result
cloudfuse.c: In function ‘cfs_write’:
cloudfuse.c:335: warning: cast to pointer from integer of different size

I've tried both libcurl libraries that apt-get has. The configure doesnt seem to find anything wrong with the libraries though.

Cheers

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.