oniony / tmsu Goto Github PK

I walked through the quick tutorial at http://tmsu.org. Everything worked great until I got to the virtual file system. I'm on OS X 10.10.1, with the latest release plus the fix mentioned at #9.

ashley@sidekick:/Volumes$ tmsu mount mp
tmsu: invalid command 'mount'.
ashley@sidekick:/Volumes$

Daan Bakker said:

I'm working on a script to automatically tag all my files, but I noticed it is very slow and requires a ton of IO (to save the db) to call "tmsu (un)tag [file] [tags]" for every file. (In my situation I do need to give every file different tags, so I cannot simply use the -r flag on the directory)
The solution I propose is to add an option that makes tmsu read commands from STDIN (instead of using parameters). Tmsu could then commit only after the last command was received.

I manually compile the i686 binary on my own laptop and am now able to use the Codeship build server for an x86-64 binary. However it would be useful to have binaries available for a range of architectures.

Need to find a way of automatically producing builds for different architectures, ideally on a continuous integration basis.

Possibly a minor issue, but I just spotted this interesting discrepancy:

time sqlite3 .tmsu/db 'select count(*) from (select 1 from file_tag where tag_id = (select id from tag where name = "tagname") group by file_id)'`

420825
sqlite3 .tmsu/db   0.68s user 0.03s system 99% cpu 0.712 total

time tmsu -D .tmsu/db files --count tagname
420825
tmsu -D .tmsu/db files --count tagname  14.22s user 1.15s system 100% cpu 15.365 total

Now, unless I misunderstand what TMSU is supposed to be doing, my query above should always achieve the same result. And it's 22 times faster, which leads me to think that something TMSU is doing is rather inefficient for this case.

More seriously, the analogous non --count command is 11 times faster than tmsu files:

time tmsu -D .tmsu/db files tagname > /dev/null
tmsu -D .tmsu/db files tagname > /dev/null  16.10s user 1.45s system 100% cpu 17.547 total

time sqlite3 -separator / .tmsu/db 'select F.directory,F.name from file_tag as FT join file as F on F.id=FT.file_id where FT.tag_id = (select id from tag where name = "tagname") group by file_id' > /dev/null
sqlite3 -separator / .tmsu/db  > /dev/null  1.44s user 0.15s system 99% cpu 1.591 total

(all timings are done from a warm start -- that is, running the command twice consecutively and taking the second timing. Cold starts add up to 2s to TMSU timings, up to 1s to sqlite3 timings.)

I understand why more generalized code would be much slower -- have to do grouping so that you get only files for which all the criteria are satisfied, rather than files for which one of the criteria are satisfied, but IME, the single positive tag case is common enough to deserve its own fast code path.

I have a couple of pools of files that I'd like to tag using TMSU separately. I would prefer not to have to remember to alias tmsu -D each time I want to work on a specific pool. It would be awesome if tmsu ascended (or could be configured to ascend) from the current path looking for .tmsu directories like git does with .git. Would this be something you would consider including?

If we tag three files:

$ tmsu tag file1 year=2015
$ tmsu tag file2 banana
$ tmsu tag file3 year=2016

And then run a negative query using the not operator on the year we get the results we would expect:

$ tmsu files not year == 2015
./file2
./file3

However, if we use the != operator (or its alias ne) we get different behaviour:

$ tmsu files "year != 2015"
./file3

The file file2 is omitted because it does not have a value for the year tag. I would have expected it to have been reported though.

$ tmsu tag /home/paul/somefile test
$ mkdir /home/paul/mp
$ tmsu mount /home/paul/mp
$ ls /home/paul/mp/tags/test
somefile.1
$ tmsu tag /home/paul/mp/tags/test/somefile.1 banana

The last action above adds the path /home/paul/mp/tags/test/somefile.1 to the database and tags it banana. I think it should, instead, add the banana tag to the already tagged file: /home/paul/somefile.

Hello,

since the current stable release does not work for me, I wanted to confirm the error by compiling it myself before creating an issue for it. Unfortunately, I can't seem to get that to work either. While it compiles fine, it gives an error during the unit tests (and also when using tmsu mount):

go test github.com/oniony/TMSU/...
?       github.com/oniony/TMSU  [no test files]
2015/10/21 09:26:34 Could not find fusermount binary: exec: "/bin/bin/fusermount": stat /bin/bin/fusermount: no such file or directory
FAIL    github.com/oniony/TMSU/cli      0.016s

Output of whereis fusermount:

fusermount: /usr/bin/fusermount /usr/share/man/man1/fusermount.1.bz2

I believe that it is a problem with my system / environment or with Go (I have never worked with or compiled anything with Go before). printenv does not show anything pointing to /bin/bin, though. Does Go use a different environment variable with a strange default value or something? The problem exists on both the master branch and the v0.5 branch. My apologies if this is not actually a TMSU error but an error on my system, feel free to close the issue if that is the case.

Thanks in advance.

Edit: go env output:

GOARCH="amd64"
GOBIN=""
GOCHAR="6"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/saviola/gocode"
GORACE=""
GOROOT="/usr/lib/go"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
CC="x86_64-pc-linux-gnu-gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

go version: go version go1.4.2 linux/amd64

Regression: this used to be possible.

Attempting to tag a file when the VFS is mounted causes a pause of several seconds and then the tool exits without error but does not create the tag.

Can you make TMSU merge tags of identical files when performing the 'repair' operation?
Here is a test case:

### Prepare data: Duplicate file.txt to 2 different directories.
mkdir test/
mkdir another/

echo "Test Repair Merge Tags" > file.txt
cp file.txt test/
mv file.txt another/

### Tag each file with diffferent tag
tmsu tag --tags "repair" test/file.txt
tmsu tag --tags "merge" another/file.txt

tmsu files repair
tmsu files merge

### Show duplicate files are identical(time and hash).
ls -al test/file.txt
ls -al another/file.txt
md5sum test/file.txt
md5sum another/file.txt


### Realize there is duplicate file. Therefore, delete 1 of them.
rm -f another/file.txt

### Repair and expect merge of tags(i.e. test/file.txt should be tagged with 'repair' & 'merge' but it is not the case)
tmsu repair . .
tmsu tags test/file.txt

Creating a tag value without tagging a file doesn't work (i.e., tmsu tag --create author=bob) because it's interpreted as a tag with a comparison operator ('='). This is particularly janky since tmsu help tag includes it as a use case.

Tag values also can't be renamed, merged, part of implications, etc. since the commands interpret author=bob as a whole tag. I see some of this in your TODO, though.

For example, I tend to use tagvalues to define sets, using the tag { -- or occasionally the tag storyboard, etc, according to exactly what it is. The existing 'tmsu values $TAGNAME' interface supports this, so it would be nice to be able to type tmsu files {=<tab and get a list of sets (the output of tmsu values "{", approximately)

The current behaviour is that no options are offered when Tab is pressed on tmsu files sometag=.

(btw (offtopic): integration test is currently failing on master, with lots of error messages referring to RECURSIVE, like this: ./merge/merge_two_tags_nonexistant_source_tag [PASS] 4d3 < tmsu: could not query files: near "RECURSIVE": syntax error ./merge/merge_two_values [FAIL])

Currently, files such as src/tmsu/cli/cli.go contain:

import (
    "fmt"
    "os"
    "os/user"
    "path/filepath"
    "tmsu/common/log"
    _path "tmsu/common/path"
)

This prevents me from building TMSU by using go get -u github.com/oniony/TMSU/src/tmsu/cli (which I realize is not what you are recommending), and makes packaging your software (e.g. for Debian) unnecessarily hard.

Could you please switch to import paths such as github.com/oniony/TMSU/src/tmsu/common/log, as is customary when writing and publishing Go code?

Thank you for considering.

Is there any plan to provide executable binary for MS Windows platform?
I wrote a set of Lua scripts to extend MPV media player. It uses TSMU as a backend. I would like to be able to use my scripts in Windows too.

thx!

This is my first time trying out TMSU.

I created a local database (tmsu init), tagged a file with a couple of tags, mounted the VFS, and browsed it in the terminal. All went well thus far.

Then, I browsed the VFS in my graphical file browser (Thunar). Once I browsed a tag directory to see files tagged with that tag in Thunar, Thunar showed an error message:

Failed to open directory "<my-tag-name>".
Software caused connection abort.

So, I went back to the terminal, and now browsing any directory under the VFS in the terminal gives the error:

ls: cannot access <any-path-under-VFS>: Transport endpoint is not connected

While looking for similar reports on the web, I saw the suggestion to mount the VFS in the foreground using a command such as tmsu vfs .tmsu/db tagfs (tagfs is my mountpoint). However, this command never succeeds, even before any problems take place. Issuing this command results in:

/bin/fusermount: mountpoint is not empty
/bin/fusermount: if you are sure this is safe, use the 'nonempty' mount option
tmsu: could not mount virtual filesystem at '.tmsu/db': could not mount virtual filesystem at '.tmsu/db': fusermount exited with code 256

I assure you, the tagfs mountpoint is absolutely empty! Or at least as far as ls -a can tell.

Please help! This has not been a pleasant experience so far :(.

I'm not sure what is going on here yet, but this just occurred with tags having
738 and 758 taggings respectively, when merging the smaller into the larger.

The result is that all of the files tagged A get tagged B, and all of the files tagged A are no longer tagged A, but the tag A is not deleted. You can however delete it manually.

I've created a test script which successfully reproduces the problem:

#!/bin/sh
mkdir -p /tmp/tmsumerge
cd /tmp/tmsumerge
tmsu -v init
for V in $(seq 1 1000); do echo $V > $V; tmsu -v tag --tags=taga $V ; done
for V in $(seq 1001 1750); do echo $V > $V; tmsu -v tag --tags=tagb $V ; done
echo
echo
echo
# will fail
tmsu -v merge tagb taga
echo
echo taga count
tmsu -v files --count taga
echo tagb count
tmsu -v files --count tagb

TMSU version: git 62adf28 (performance improvements to VFS)

In an email, Victoria Stuart asked:

Does tmsu support wildcard expressions? E.g., for tagging? searches?

To which I had to reply 'no'. From what I gathered Victoria wanted to be able to do:

$ tmsu files she*

To list files that match any of the tags that match the pattern, e.g. 'sheep', 'shenanigans', 'shellac'. Or, when tagging a file:

$ tmsu tag somefile car*

To tag a file 'carpet', 'cardigan' and 'carpel-tunnel-syndrome'.

This suggests she is trying to build some sort of structure with the tags so perhaps tag hierarchies need to be considered or thought about. Personally I've always felt that it would be better to use tag intersections to do this. For example, instead of defining the following tags:

* car-hatchback-mercedes
* car-saloon-audi
* van-volkswagen

It would be better to define:

* car
* van
* hatchback
* saloon
* make=mercedes
* make=audi
* make=volkswagen

And to apply the tags as appropriate. Finding the appropriate items is then a case of identifying the files with the intersection of traits:

$ tmsu files car saloon make=audi

TMSU use to store directory fingerprints by recursively enumerating the directory, calculating fingerprints for every file within it, building a report of these fingerprints and then creating a further fingerprint of this report. For large directories this was particularly slow and not very intuitive to the user when a tagging operation was suddenly inexplicably slow.

However, I've since realised it might be sufficient to build a directory fingerprint based only on the sizes of the files within it. This should be pretty fast and provide a largely useful fingerprint for identifying duplicate or moved directories.

Hello,

I'm a huge fan of the VFS features. It used to work well in my previous explorer, but now I'm used ranger as a file explorer and it does not work anymore.

I can navigate through the tags without any problem, but as soon as I try to open a file, it gets disconnected, I have the error "Transport endpoint is connected".

Can you think of any reason for this issue ?

Thanks

To be quite clear, this issue has nothing to do with the upcoming release of 0.5.0. I was thinking about implementing this myself as an extension, and then started thinking about whether it might belong in core TMSU.

The basic idea is that many kinds of files which may be tagged can usefully be said to have 'groups' of tags.

For example, supposing you have a photo collection including, among other things, photos of Alice and Bob. Alice and Bob are often seen together in these photos. Alice doesn't wear hats as a rule, but Bob does. Being able to group tags, like {alice hat} and do group queries '{alice hat}`, allows you to find photos where -Alice- is wearing a hat, as opposed to (a few pictures with Alice wearing a hat, hidden within hundreds with Alice and Bob being both in the picture, and Bob's usual hat being in the picture); The logical grouping {alice hat} is more useful in this case than the undifferentiated 'alice bob hat'

Hopefully, the mechanism described above is quite simple and obvious: each file has an arbitrary number of groups, identified only by ID number, and each group has an arbitrary number of tags. One might propose that 0 is a valid group id, which can be logically thought of as 'anything else' (that is not otherwise in a logical grouping. For example, the background in photos is usually not going to be worth grouping tags in, it might reasonably be seen as an undifferentiated set of tags like 'beach sand palm_tree tree water sea plane'.

One complication that may not immediately be obvious with the above is that you may want a tagging to participate in multiple groups -- certainly, there is no guarantee that only one person in a photo is going to wear a hat.

The obvious implementation, AFAICS, would expand the file_tag table to have four fields and use all four as a compound PRIMARY KEY, similar to the current implementation with tag values.

The non-obvious implementation might separate tag groups into a separate (group_id, tag_id, value_id) table, with each unique set of tag=values occurring exactly once, and then have file_tag be keyed only by (file_id, group_id). This is rather more radical, I mention it mainly because I think it has better performance characteristics for querying.

The most complicated part would be querying; specifically, querying at an acceptable speed. The mentioned table refactoring is the only way I have seen so far to minimize use of recursive SQL queries -- it would allow finding the groups which match the query (recursively) in the most likely much smaller (group_id, tag_id, value_id) table, and then non-recursively finding the (file_id, group_id) tuples which reference one of those group_ids. However, this would make normal ungrouped queries more complex to perform. (It also probably wouldn't be that fast for cases where many groups match the query)

There are significant technical challenges involved, however I felt I should propose the idea so that it can at least be considered.

Kitlau said:

.. by 'complex' I mean '251 tags connected by 250 "or" operators'. I have a tag grouping system. >This tag group is probably too large -- the average group is 30 or less tags ;)
Steps to reproduce (no, my actual tags are not just numbers ;):
tmsu -D /tmp/test.db tag --create $(seq 0 255)
tmsu -D /tmp/test.db files --count $(seq 0 255 |tr \n ' ' | sed -Ee 's/ $//' -e 's/ / or /g')
Comments:
This is obviously an extreme corner case, and normally a query that large would be a sign of something insane occurring. So from my perspective, simply documenting the parser limit would be enough to address this issue. It appears to be 67 operators (allowing for an expression that has 68 tags and 67 'or's). You can check this by comparing the output of
tmsu -D /tmp/test.db files --count $(seq 0 67 |tr \n ' ' | sed -Ee 's/ $//' -e 's/ / or /g')
with
tmsu -D /tmp/test.db files --count $(seq 0 68 |tr \n ' ' | sed -Ee 's/ $//' -e 's/ / or /g')
(unless the exact number is somehow directly dependent on go internals, I guess. For me, the former returns 0, while the latter returns 'parser stack overflow' )

For users who store their files in a distributed fashion (say, WebDAV, git-annex, etc.), it might be useful if TMSU included a sort of «merge driver» to unify two TMSU databases that have diverged. This way, one wouldn’t have to necessarily make sure only one client updates tags at a time.

I guess the problem would be tag deletions. A union merge strategy would bring back deleted tags. So this would entail a three-way merge instead. Not sure if it’s the most pressing issue, but having support for decentralized tagging would be awesome.

The only choice was to build from source for my platform, since a binary distribution is not available (yet?).

I'm stuck at the following point after issuing make:

go clean tmsu
rm -Rf bin
rm -Rf tmsu-x86_64-0.5.0
rm -f tmsu-x86_64-0.5.0.tgz
go build -o bin/tmsu tmsu
# tmsu/common/terminal
src/tmsu/common/terminal/print.go:29: undefined: Width
src/tmsu/common/terminal/print.go:108: undefined: Width
make: *** [compile] Error 2

The offending lines are:

line #29: PrintColumnsWidth(items, Width())

line #108: PrintWrappedWidth(text, Width())

So, what should be the definition of Width() then?

Thanks

Would it be possible to add full text tags? For things like summary of a TV episode, song's lyrics, etc.

The full text values would be different from the existing values in that they're arbitrary long, without character restrictions and usually unique for each file & tag.

It would probably require a new command line option:

$ tmsu tag summer.mp3 music big-jazz mp3 --text "comments=Best. Song. Ever."

which would save the value Best. Song. Ever. for file summer.mp3 and tag comments.

I was trying to set up TMSU for use within a git-annex repository, and was delighted to see the symlinkTargetName{,NoExt} option exist. However, it seems that when a regular (non-symlinked) file is tagged, tmsu will still use the name as the fingerprint (despite what the option name indicates). Would it be reasonable to fall back to a regular hash for non-symlinked files? Perhaps by specifying the desired digest in a setting, such as symlinkTargetNameNoExt,dynamic:SHA1?

In the best case, tmsu could also fall-back to its regular behavior when the file is a symlink, but the target name is not “special”. If the config option is meant to target git-annex interoperability, git-annexes known key formats could be used as a pattern against which to match; alternatively, it could be checked whether the symlink goes to […]/.git/annex. Or, for a more generic solution, maybe a config option specifying a regular expression might be viable?

Thanks for making TMSU!

Hello, is it possible to replace tags in one command, similar to:

$ tmsu tags ./file
foo bar baz and other tags and=values
$ tmsu tag --replace --tags="foo new" ./file
$ tmsu tags ./file
foo new

I wrote ui script tmsu-tagd , it edits tags of one file. In implementation it stores list of tags and for retagging it does:

$ tmsu untag --all ./file
$ tmsu tag --tags "foo new" ./file || tmsu tag --tags "foo old" ./file

So it always untags all and then tags again, which isn't very nice.
Is replacing tags in one command already available or will it be in future?

Since tmsu doesn't explicitly monitor filesystem changes, it'd be nice if there were an analogue of git mv - a move command that also adds the move to the staging area - for tmsu, that would move the file while updating the database so that no tag info is lost.

Thomas Gläßle said:

I have the following suggestion:
When tagging a file with author=me, I'd say it fits my expectations much more nicely to tag the file with me and make author a meta-tag of me, i.e. tag the object me with author.
That way, the folder mp/tags/author wouldn't show all the files that have an author, but rather only the author names, also you could query for the actual author name (me) even if you forgot what the meta-tag (was it author or interpret?) was called. This would also come in handy in the situation where you first tagged all your files with the name of their author, and later want to put some meta-information about these names: you'd just have to add a meta-tag for all the names.
I haven't totally worked out all the details, so I don't know whether it is easy to do or will have some bad connotations (like many tag values ~ many tags ~ chaos?). Just saying, this seems more intuitive to me:).
Best regards, Thomas
EDIT: Just having written this, I realize the situation is not as easy, as the meta-tag might change for individual files. E.g. an email might have recipient=me or sender=me. Maybe, I'll come back with a cleaner idea later.

Just encountered this message when trying to look up help for 'repair' (tmsu help repair) while 'tmsu repair -u' was running in another tab.

This is a minor bug, I guess, but it doesn't seem that extraordinary that the user might want to look up help while TMSU is running, and I don't see any reason that the help command should actually need a db lock.

I've used tmsu mount to get my tags etc, however, I'd like to see an 'untagged' tag so that I can do:

ashley@sidekick:~/mt/queries$ ls untagged
file1 file2 file3

now:

$ tmsu imply new_tag some_tag
tmsu: no such tag 'new_tag'
$ tmsu imply existing_tag new_tag
tmsu: no such tag 'new_tag'

Multiple identical directories are being created for tag values, test case:

> tmsu tags
Genre
> tmsu values
SciFi
> tmsu files
first_contact.avi the_wrath_of_khan.avi
> tmsu mount /media/test
> tree /media/test/tags/Genre
/media/test/tags/Genre/
|-- first_contact.1.avi
|-- =SciFi
|    |-- first_contact.1.avi
|    |-- the_wrath_of_khan.2.avi
|-- =SciFi
|    |-- first_contact.1.avi
|    |-- the_wrath_of_khan.2.avi
|-- the_wrath_of_khan.2.avi

Each additional file tagged with Genre=SciFi will add an additional (identical) =SciFi directory.

Expectation would be a single =SciFi directory.

> tmsu version
TMSU 0.6.0
....

Please let me know what additional information would be helpful.

I tried looking into the src code a bit, assuming I just need to change something along the lines of

show directory

into

if not directory.exists then
    show directory

But don't know Go or VFS arch. Will try to look into more later. Maybe a book-keeping dictionary?

visited = {
    'Genre': ['SciFi'],
    ....
} 

...

if directory not in visited:
    show directory
    visited[tag] += directory

If the name of a directory is numeric the file with the this fileId is shown in the directory as seen in the following example.

mkdir data
cd data
tmsu init
mkdir 1 2
echo a > 1/a
echo b > 1/b
echo c > 2/c
echo d > 2/d
tmsu tag --tags foo -r 1
tmsu tag --tags bar -r 2
mkdir ../mnt
tmsu mount ../mnt
cd ..
ls -al mnt/tags/bar

ls shows both files tagged with 'bar' but also file 'b' which doesn't have this tag. This happens because its fileId is '2' which is also the tagId of 'bar'.

@shabbyrobe mentioned in issue #8 that they would like to have relative rather than absolute paths stored in the database:

TMSU stores the full path to the file in the database, but this limits my ability to either share the database with others or to move it around on my own machine. It makes more sense with my desired workflow to store paths relative to the .tmsu folder, but I understand this limits the existing ability of TMSU to tag files outside a user's home directory.

One of the main use cases I have for portable, database relative tagging is for the various bands I play or have played in. We share huge collections of snippets and ideas and each band has a distinct set of members so I need one .tmsu folder per band. Being able to tag our snippets in a portable manner using TMSU is the killer workflow upgrade we've been desperate for. I would also like to use it for some photo collections, one private, one to share with my family.

Currently TMSU makes every path absolute before storing them in the database. This has some benefits (e.g. the '..' path element is resolved) but has some limitations:

• External storage mounted under different mount points will break the paths in the database
• The TMSU database is not portable between machines unless identically set up

To solve this problem I think TMSU should not make the paths absolute but should instead resolve paths against a fixed filesystem point. For now, it would make sense to use the parent directory of the .tmsu directory of the active database as that fixed point (In the future there TMSU could offer configuration so that this can be changed).

The following operation:

/home/fred/food $ tmsu tag italian/pizza yum

Would currently store the pizza file as /home/fred/food/italian/pizza. With this proposed change, using the default database (/home/fred/.tmsu/default.db) would store the file as food/italian/pizza, (i.e. paths would be relative to the user's home directory by default by virtue of the fact that that's where the .tmsu directory is).

If we created a local database at /home/fred/food/.tmsu/db (see issue #8), then the path stored would be italian/pizza. This would allow the food directory to be moved, shared or archived whilst preserving the TMSU database.

But what if the database has been specified with --database or TMSU_DB? We cannot assume that the specified database is in a .tmsu directory. We could either store the paths relative to the database file's directory or we could preserve the existing behaviour of using absolute paths when the database is specified explicitly.

Finally, existing databases will have absolute paths stored so repair will need to be upgraded to fix these.

I can't create/access a new default database when running 0.5, but works fine with 0.4.3. The messages I get when I run any tmsu command:

tmsu: creating database at '/home/[username]/.tmsu/default.db'.
tmsu: could not open storage: could not open database at '/home/[username]/.tmsu/default.db': database transaction error: unable database file

Creating the database with 0.4.3 and then manually "repair" (modify to relative paths) in 0.5 works fine without problems.

After trying to create a new database with 0.5 I can't find any ".tmsu" directory (or a default.db file) in $HOME.

Can you make query tag case insensitive.
For example:

# Should also list files tagged as 'Mp3'.
$ tmsu files mp3

I'm using TMSU 0.4.3.

Currently the option parsing facility cannot capture option relationships which means invalid option combinations are not detected generally: instead these have to be checked for explicitly by each command.

There needs to be a way to specifiy:

• An option that is only valid in the presence of other specific options.
• An option that is not valid in the presence of other specific options.

A few times now, e.g. in issue #11 and #20, people have requested the ability to move files on disk or merge files whilst preserving the tag information. I've been reluctant to add such functionality because I think the purity of having TMSU not touch the managed files quite an asset: it gives a lot of confidence to new users having its modus operandi cleary defined.

However it can also be quite limiting and the commands that must be issued to perform these simple operations are not always altogether obvious.

As a compromise I've decided to provide these operations as simple shell scripts/functions included in the TMSU distributable. This feels the more Unixy way of doing things, reduces the complexity of TMSU and reduces the opportunity for bugs.

As a start these will include:

• mv: move a file from one place on disk to another whilst preserving tags
• merge: apply the tags from one file to a second and then delete the first (useful for duplicate file management)
• rm: remove a file and delete its tag information

The exact names are to be decided.

Hello,

here's a minimal shell session to reproduce and illustrate the problem (on my machine) using the latest stable release:

saviola@trinity ~/tmp/test $ tmsu init
tmsu: creating database at '/home/saviola/tmp/test/.tmsu/db'.
saviola@trinity ~/tmp/test $ mkdir tmsu            
saviola@trinity ~/tmp/test $ tmsu -D .tmsu/db mount tmsu
saviola@trinity ~/tmp/test $ touch file
saviola@trinity ~/tmp/test $ tmsu -D .tmsu/db tag file tag1
tmsu: New tag 'tag1'.
saviola@trinity ~/tmp/test $ cd tmsu/tags/tag1
/usr/bin/python3.4: error while loading shared libraries: libpython3.4.so.1.0: file too short
saviola@trinity ~/tmp/test/tmsu/tags/tag1 $ ls -lA
total 0
lrwxr-xr-x 1 saviola saviola 0 Oct 21 14:32 file.1 -> /home/saviola/tmp/test/file
saviola@trinity ~/tmp/test/tmsu/tags/tag1 $ ....
saviola@trinity ~/tmp/test $ mv file file1
saviola@trinity ~/tmp/test $ cd tmsu/tags/tag1 
saviola@trinity ~/tmp/test/tmsu/tags/tag1 $ ls -lA
total 0
lrwxr-xr-x 1 saviola saviola 0 Jan  1  0001 file.1 -> /home/saviola/tmp/test/file

So in short: When browsing the vfs, I get errors only when the linked file actually exists, otherwise it works fine. The error appears after every change of directory below the "tags" directory (and when entering directories in the queries directory) in the vfs. It may be worth noting that with the current stable release, I get an additional error when trying to list the files ("error while loading shared libraries: libacl.1.so: file too short" or something very similar), this seems to be fixed in the current master, at least making TMSU usable for me. Also, I cannot reproduce the issue on my notebook with the current stable release. Since both systems are similar, I will try to look for version differences in the installed software to identify the culprit, is there some easy way to track down all dependencies of TMSU (and their dependencies, and so on) to narrow things down a bit?

As I said, this does not break TMSU for me, but it may be worth looking into, if and when you have some time (may be another upstream issue, I don't know). Thanks, and keep up the good work.

KDE software typically store tags in the extended attributes of the file under the key "user.xdg.tags". It would be amazing if TMSU could also do that / read tags from there.

I added locking (semaphores) to fix issue #28 but this is merely patches up the underlying problem. The database layer uses connections and transactions poorly. If each concurrent access used its own connection (pooled) then the locking would be unnecessary as the database transactionality would handle the concurrency issues.

Currently there are a number of restrictions on which characters are allowed to be used in tag and value names. These are present for several reasons:

1. To make parsing file queries easier.
2. To avoid invalid names when mounting the virtual filesystem.

Category 2 consists only of the slash characters, the backslash being illegal on Windows and the forwardslash on most other platforms.

Items that fall into category 1 should be relaxed where possible.

• Changes to the scanner should make it possible to use backslash escaping for problematic characters thus allowing their use within tag and value names.
• More difficult to remove would be restrictions on keywords that are used as operators in the query language, e.g. 'and', 'or', 'not', 'lt', &c..

TODOs

• subcommand files to render = delimiter in colour (when colour turned on)
• facility for customizing the TMSU colours (à la LS_COLORS).
• fix bug with tag --tags not handling whitespace (and presumably literal =).
• test coverage for all problematic characters.
• Zsh completion to correctly escape characters that are used by the shell.
• update wiki on names and explain how to handle both shell and TMSU escapes for problematic characters.

Testing off commit 36de52a

tsmu-fs-mv curdir/filename newdir

curdir/filename moves to newdir/filename
tags move from curdir/filename to newdir

Getting an 'unsorted' list (which is effectively, actually sorted by how recently a tag was applied on the file) is much more useful for some uses.
I've run some tests and it looks like you can get sorting-by-recency (oldest -> newest tagging) just by leaving out the 'order by' clause from the SQL.

--recency would be a decent option name, I guess.

The result of the following:

cd /tmp
tmsu init
tmsu files " -foo"

is tmsu: no such tag '-foo'.

Whereas tmsu files -foo correctly says

tmsu: invalid option '-foo'

and echo foo> foo; tmsu tag foo ' -foo' correctly says

tmsu: could not create tag ' -foo': tag names cannot contain space or tab.

IMO the correct error for this situation is probably analogous to the tmsu tag output:

tmsu: query contains invalid tag '-foo': tag names cannot begin with '-'.

As far as I can tell this bug is not exploitable, just misleading.

Calinou said:

Hi, it could be useful to have translation support for the program and maybe for the site.

Commands like tmsu fix -R or tmsu query -f complain about invalid options. repair and files work properly though.

Consider the following:

$ tmsu imply apple banana
$ tmsu imply banana apple
$ tmsu tag somefile apple
$ tmsu tags --count somefile
0

Neither tag is applied because TMSU has logic to not explicitly apply implied tags. However the logic is flawed as when there is a circular dependency it is not clever enough to realise it needs to break the circle and apply the first tag.

This problem also exists with larger chains:

$ tmsu imply aubergine broccoli
$ tmsu imply broccoli cauliflower
$ tmsu imply cauliflower aubergine
$ tmsu tag somefile aubergine
$ tmsu tags --count somefile
0

When I specify custom database path using -D, paths stored in the DB are absolute. Would it be possible to allow using relative paths? Having absolute paths in a DB stored stored on a portable storage device sort of defeats the purpose of using -D.