Kia ora e te tangata, whakawhetai mo to s3fslite tuatahi, ka whakatauhia e au te raru o nga tono e rapu ana ahau.
Engari he take: I eke ahau ki te s3 ki te wordpress, he pai te mahi me nga whakaahua panui i tukuna paitia e au e s3fs i mua i roto i te wordpress(Ko taku tikanga he pai te mahi a s3fs ki te tuku me te panui), heoi, kaore i taea te tuku whakaahua ki s3 ma s3fslite, ka tio te pae tukanga tukuake ki te 100% karekau e taea te whakaoti pai.

dear russ,

maybe this is not intended at all, but i wanted to give it a try. i installed all dependencies using macports, but s3fslite still won't compile. i have absolutely no clue about c-coding.
(s3fs compiles, but doesn't work, so i came here. plus i mainly want to use it with rsync anyway)

this is the make output:

g++ -Wall -Wextra -Wno-unused-parameter -Werror -Os -D__FreeBSD__=10 -D_FILE_OFFSET_BITS=64 -I/opt/local/include/fuse -I/opt/local/include -I/opt/local/include -I/opt/local/include/libxml2 -c -o common.o common.cpp
g++ -Wall -Wextra -Wno-unused-parameter -Werror -Os -D__FreeBSD__=10 -D_FILE_OFFSET_BITS=64 -I/opt/local/include/fuse -I/opt/local/include -I/opt/local/include -I/opt/local/include/libxml2 -c -o fileinfo.o fileinfo.cpp
fileinfo.cpp: In constructor ‘Fileinfo::Fileinfo(std::string, stat_)’:
fileinfo.cpp:15: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:15: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:15: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:15: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:16: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp: In member function ‘void Fileinfo::toStat(stat_)’:
fileinfo.cpp:31: error: invalid application of ‘sizeof’ to incomplete type ‘stat’
fileinfo.cpp:32: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:33: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:34: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:35: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:36: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:37: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:38: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:39: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:40: error: ‘S_ISREG’ was not declared in this scope
fileinfo.cpp:41: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:41: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
make: *** [fileinfo.o] Error 1

any idea?
cheers, yanone

If I create a folder I also get an empty file with the mime type 'application/x-directory'. If I create a file in this folder then I see this file both in the folder and in the root. This behavior confuses rsync a lot!

I have this problem on Ubuntu Lucid Lynx, but also on the Freetz platform.

Issued commands:

mkdir folder
ls -> "folder"
cd folder
touch file
cd ..
ls -Ral

.:
drwxr-xr-x 1 root root 0 May 5 12:48 .
drwxr-xr-x 4 bittorre users 4096 May 6 21:02 ..
-rw-r--r-- 1 root root 0 May 6 21:01 file
drwxr-xr-x 1 root root 0 May 6 21:01 folder

./folder:
drwxr-xr-x 1 root root 0 May 6 21:01 .
drwxr-xr-x 1 root root 0 May 5 12:48 ..
-rw-r--r-- 1 root root 0 May 6 21:01 file

Syslog:

May 6 21:01:45 fritz user.info syslog: getattr[/folder]
May 6 21:01:45 fritz user.info syslog: Filecache::new[/folder]
May 6 21:01:45 fritz user.info syslog: S3request::get_fileinfo[/folder]
May 6 21:01:45 fritz user.info syslog: getattr[/folder]: File not found
May 6 21:01:45 fritz user.info syslog: mkdir[/folder] mode[0755]
May 6 21:01:45 fritz user.info syslog: getattr[/folder]
May 6 21:01:46 fritz user.info syslog: Filecache::from_queue resurrecting file[/folder]
May 6 21:01:47 fritz user.info syslog: getattr[/]
May 6 21:01:47 fritz user.info syslog: Filecache::new[/]
May 6 21:01:47 fritz user.info syslog: getattr[/]
May 6 21:01:47 fritz user.info syslog: getattr[/]
May 6 21:01:47 fritz user.info syslog: readdir[/] offset[0]
May 6 21:01:47 fritz user.info syslog: Filecache::sync all
May 6 21:01:47 fritz user.info syslog: Filecache::fsync[/]
May 6 21:01:47 fritz user.info syslog: Filecache::fsync[/folder]
May 6 21:01:47 fritz user.info syslog: fsync: !exists || dirty_data
May 6 21:01:47 fritz user.info syslog: S3request::put_file[/folder]
May 6 21:01:47 fritz user.info syslog: S3request::get_directory[/] marker[]
May 6 21:01:47 fritz user.info syslog: getattr[/folder]
May 6 21:01:48 fritz user.info syslog: Filecache::from_queue resurrecting file[/folder]
May 6 21:01:48 fritz user.info syslog: Filecache::from_queue resurrecting file[/]
May 6 21:01:50 fritz user.info syslog: getattr[/folder]
May 6 21:01:53 fritz user.info syslog: Filecache::from_queue expiring[/]
May 6 21:01:53 fritz user.info syslog: Filecache::fsync[/]
May 6 21:01:53 fritz user.info syslog: Filecache::delete[/]
May 6 21:01:53 fritz user.info syslog: Filecache::from_queue resurrecting file[/folder]
May 6 21:01:55 fritz user.info syslog: getattr[/folder/file]
May 6 21:01:55 fritz user.info syslog: Filecache::new[/folder/file]
May 6 21:01:55 fritz user.info syslog: S3request::get_fileinfo[/folder/file]
May 6 21:01:56 fritz user.info syslog: getattr[/folder/file]: File not found
May 6 21:01:56 fritz user.info syslog: getattr[/folder/file]
May 6 21:01:56 fritz user.info syslog: getattr[/folder/file]: File not found
May 6 21:01:56 fritz user.info syslog: mknod[/folder/file] mode[0100644]
May 6 21:01:56 fritz user.info syslog: getattr[/folder/file]
May 6 21:01:56 fritz user.info syslog: open[/folder/file] flags[020002]
May 6 21:01:56 fritz user.info syslog: release[/folder/file]
May 6 21:01:58 fritz user.info syslog: Filecache::from_queue expiring[/folder]
May 6 21:01:58 fritz user.info syslog: Filecache::fsync[/folder]
May 6 21:01:58 fritz user.info syslog: Filecache::delete[/folder]
May 6 21:01:58 fritz user.info syslog: Filecache::from_queue resurrecting file[/folder/file]
May 6 21:02:02 fritz user.info syslog: getattr[/]
May 6 21:02:02 fritz user.info syslog: Filecache::new[/]
May 6 21:02:02 fritz user.info syslog: getattr[/]
May 6 21:02:02 fritz user.info syslog: getattr[/]
May 6 21:02:02 fritz user.info syslog: readdir[/] offset[0]
May 6 21:02:02 fritz user.info syslog: Filecache::sync all
May 6 21:02:02 fritz user.info syslog: Filecache::fsync[/]
May 6 21:02:02 fritz user.info syslog: Filecache::fsync[/folder/file]
May 6 21:02:02 fritz user.info syslog: fsync: !exists || dirty_data
May 6 21:02:02 fritz user.info syslog: S3request::put_file[/folder/file]
May 6 21:02:03 fritz user.info syslog: S3request::get_directory[/] marker[]
May 6 21:02:03 fritz user.info syslog: Filecache::from_queue expiring[/folder/file]
May 6 21:02:03 fritz user.info syslog: Filecache::fsync[/folder/file]
May 6 21:02:03 fritz user.info syslog: Filecache::delete[/folder/file]
May 6 21:02:03 fritz user.info syslog: Filecache::from_queue resurrecting file[/]
May 6 21:02:03 fritz user.info syslog: getattr[/]
May 6 21:02:03 fritz user.info syslog: getattr[/folder]
May 6 21:02:03 fritz user.info syslog: Filecache::new[/folder]
May 6 21:02:03 fritz user.info syslog: getattr[/folder/file]
May 6 21:02:03 fritz user.info syslog: Filecache::new[/folder/file]
May 6 21:02:03 fritz user.info syslog: readdir[/folder] offset[0]
May 6 21:02:03 fritz user.info syslog: Filecache::sync all
May 6 21:02:03 fritz user.info syslog: Filecache::fsync[/]
May 6 21:02:03 fritz user.info syslog: Filecache::fsync[/folder]
May 6 21:02:03 fritz user.info syslog: Filecache::fsync[/folder/file]
May 6 21:02:03 fritz user.info syslog: S3request::get_directory[/folder] marker[]
May 6 21:02:04 fritz user.info syslog: getattr[/folder]
May 6 21:02:04 fritz user.info syslog: getattr[/]
May 6 21:02:04 fritz user.info syslog: Filecache::from_queue resurrecting file[/folder]
May 6 21:02:05 fritz user.info syslog: Filecache::from_queue resurrecting file[/]
May 6 21:02:08 fritz user.info syslog: Filecache::from_queue expiring[/folder/file]
May 6 21:02:08 fritz user.info syslog: Filecache::fsync[/folder/file]
May 6 21:02:08 fritz user.info syslog: Filecache::delete[/folder/file]
May 6 21:02:09 fritz user.info syslog: Filecache::from_queue expiring[/folder]
May 6 21:02:09 fritz user.info syslog: Filecache::fsync[/folder]
May 6 21:02:09 fritz user.info syslog: Filecache::delete[/folder]
May 6 21:02:10 fritz user.info syslog: Filecache::from_queue expiring[/]
May 6 21:02:10 fritz user.info syslog: Filecache::fsync[/]
May 6 21:02:10 fritz user.info syslog: Filecache::delete[/]
May 6 21:02:17 fritz user.info syslog: statfs[/]

If I delete the file with the same name as the directory (using a Firefox plugin), everything is ok again:

ls -Ral

.:
drwxr-xr-x 1 root root 0 May 5 12:48 .
drwxr-xr-x 4 bittorre users 4096 May 6 21:02 ..
-rw-r--r-- 1 root root 0 May 6 21:01 file

Patch for trunk 20100518 version:

--- s3fs_org.cpp    2010-06-08 08:52:30.950200000 +0200
+++ s3fs.cpp    2010-06-08 13:32:55.357720457 +0200
@@ -999,7 +999,7 @@ int main(int argc, char *argv[]) {
     if (AWSSecretAccessKey.size() == 0) {
         std::string line;
         std::ifstream passwd("/etc/passwd-s3fs");
-        while (getline(passwd, line)) {
+        while (!getline(passwd, line).eof()) {
             if (line[0]=='#')
                 continue;
             size_t pos = line.find(':');
@@ -1055,7 +1055,7 @@ int main(int argc, char *argv[]) {
     // load the list of mime types
     std::string line;
     std::ifstream passwd("/etc/mime.types");
-    while (getline(passwd, line)) {
+    while (!getline(passwd, line).eof()) {
         if (line[0] == '#')
             continue;
         std::stringstream tmp(line);

Patch for trunk 20100518 version:

--- s3request_org.cpp   2010-06-08 08:52:30.950200000 +0200
+++ s3request.cpp   2010-06-08 08:57:38.687515801 +0200
@@ -681,6 +681,9 @@ bool S3request::get_directory(std::strin
     bool moretocome = false;
     marker = "";
 
+    if (path != "/")
+        prefix = path.substr(1) + "/";
+
     // parse the response
     xmlDocPtr doc =
         xmlReadMemory(response.c_str(), response.size(), "", NULL, 0);

Hi Russ!

as per title, if /etc.mime.types does not exist then the parser in s3fs.cpp ('// load the list of mime types') goes into an endless loop as eof never goes to true and the process then uses 100% cpu.

quick fix was to create an empty mime.types.

regards
charlie

Patch for trunk 20100518 version:

--- s3request_org.cpp   2010-06-09 07:23:25.812378444 +0200
+++ s3request.cpp   2010-06-09 07:27:06.173402565 +0200
@@ -166,7 +166,7 @@ size_t uploadCallback(void *data, size_t
 {
     int *fd = static_cast(userPtr);
     size_t len = read(*fd, data, blockSize * numBlocks);
-#ifdef DEBUG_WIRE
+#if defined DEBUG && defined DEBUG_WIRE
     if (len != blockSize * numBlocks) {
         syslog(LOG_INFO, "uploadCallback: short read [%u] != [%u]",
                 (unsigned) len, (unsigned) (blockSize * numBlocks));

Hi there

tcpdump shows a mass of DNS lookups (over 50/sec) for the s3.amazonaws.com bucket name when rsync is running against it. Would there be a performance benefit in s3fs caching the returned IP addresses for (say) 10sec? (shouldn't be too long as I suspect those can change) - might speed up the associated 50 HTTP requests/sec that occur next ;-)

Jason

With the help of some others I have cross-compiled s3fslite v1.0 for Freetz (http://trac.freetz.org/wiki/WikiStart.en). See here for a description of my attempts: http://trac.freetz.org/ticket/796.

I have two little requests: can bset be replaced by memset and can unistd.h be included in common.cpp, so that these not have to be patched for Freetz.

Listing files from a bucket works, however creating files results in "Software caused connection abort". What could be the cause of this? The syslog is not showing any error messages:

Apr 29 12:53:13 fritz user.info syslog: init[0814-6721-4383-test-us]
Apr 29 12:53:15 fritz user.info syslog: getattr[/s3fs.txt]
Apr 29 12:53:15 fritz user.info syslog: Filecache::new[/s3fs.txt]
Apr 29 12:53:15 fritz user.info syslog: S3request::get_fileinfo[/s3fs.txt]

make

g++ -Wall -Wextra -Wno-unused-parameter -Werror -Os -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -c -o common.o common.cpp
g++ -Wall -Wextra -Wno-unused-parameter -Werror -Os -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -c -o fileinfo.o fileinfo.cpp
g++ -Wall -Wextra -Wno-unused-parameter -Werror -Os -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -c -o attrcache.o attrcache.cpp
g++ -Wall -Wextra -Wno-unused-parameter -Werror -Os -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -c -o filecache.o filecache.cpp
g++ -Wall -Wextra -Wno-unused-parameter -Werror -Os -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -c -o s3request.o s3request.cpp
s3request.cpp: In static member function ‘static void S3request::put_file(Fileinfo_, int)’:
s3request.cpp:590: error: ‘CURLOPT_SEEKDATA’ was not declared in this scope
s3request.cpp:591: error: ‘CURLOPT_SEEKFUNCTION’ was not declared in this scope
make: *_* [s3request.o] Error 1