No more image cover after 2000 books sent to Kobo about calibre-kobo-driver HOT 25 CLOSED

As per http://www.mobileread.com/forums/showpost.php?p=2521110&postcount=217 it seems that the Kobo devices use FAT32, not exFAT as I'd been assuming, so this is a device limitation. Nothing I can do. I would suggest contacting Kobo and asking them to send in a feature request to support exFAT (or even better, ext2 through ext4) as options for the user data partition. Be prepared for a very frustrating experience, my experience with Kobo is that if you ask anything not in their standard question list they either get hopelessly confused or tell you it's not possible.

from calibre-kobo-driver.

Hi jgoguen,

As I said in the forum, not sure the type of partition is the cause.
With previous versions of the plugin I sent about 4200 books. They all had their cover images.
This bug appeared when trying ton send the same books (same names) with the last version of the plugin built with files on GitHub..
I send my books with Calibre, without sending covers, only the books.
Covers images are built when books are prompted the first time in the library list by the Kobo.
Each time I send books (200 at a time), I list 40 pages in library (40*5=200), just to make images built.

Not sure my english is as good as necessary to correctely explain. I hope you'll understand me.

from calibre-kobo-driver.

Did you not send cover images with previous versions? I don't actually touch images at all, I only set the ImageID field in the database. That could be causing some issue, I've discovered that I'm actually setting the ImageID even if covers aren't being sent. If that doesn't fix it, I'm putting my money on a device issue with the file system.

from calibre-kobo-driver.

I never sent cover images with any version of the plugin.
Do you want me to test with this option or do I wait for an update of the driver ?
If I have to wait for un update, will I have to retest from crash (remove all books) or not ?

Thanks.

from calibre-kobo-driver.

You can test as soon as the code is pushed if you want, but you'll need to test with books that are not currently there and probably have to look in the database to verify that the cover images are not set to the values calibre will use. The simplest way to do this would be to remove all books and move at least enough to reproduce the issue (so for you, at least 2100 books), unplug the device, plug it back in, and once calibre has finished updating metadata query the database. For Windows, I would recommend the SQLite extension for Firefox (I've used https://addons.mozilla.org/en-us/firefox/addon/sqlite-manager/). You will also need to make sure that your .kobo/images/ directory is empty

The specific query that would need to be run, which I can't even say what to expect for results quite yet, is:

SELECT * FROM content WHERE ContentType = 6 AND ImageID IS NOT NULL AND ImageID != '' AND ImageID NOT LIKE 'file:%'

That will give all rows for the main book entry that has an image ID that does not start with the standard calibre ImageID prefix.

from calibre-kobo-driver.

Ok jgoguen.

I'll wait for the new version of the plugin and will do the test on my Mini after a factory reset.
I'll give you an answer as soon as possible, Maybe not before tomorrow evening (in 24h, it's near 10PM in France)

from calibre-kobo-driver.

jgoguen,

I've got the new driver.py and factory reseted my Kobo Mini.
To be sure I understood the test to do. Do I active the transfer of cover images form Calibre or not ?
Waiting for your answer to start.

Thanks

EDIT : To see if ImageId from Calibre is in database, I suppose (If I understood your explanations) the request has to be :
SELECT * FROM content WHERE ContentType = 6 AND ImageID IS NOT NULL AND ImageID != '' AND ImageID LIKE 'file:%'

I did it, and no row returned.

With NOT like, 82 rows, all with ContentURL = http://www.kobobooks.fr/ebook/...

from calibre-kobo-driver.

You should do the test without sending cover images, just like you were doing before.

from calibre-kobo-driver.

Ok, I begin, but I won't finish tonight (1AM here)
I'll continue tomorrow.

from calibre-kobo-driver.

There's a change to the query, I forgot that symbols are replaced in the ImageID set from calibre. The query isn't much different though, just dropping the colon after 'file':

SELECT * FROM content WHERE ContentType = 6 AND ImageID IS NOT NULL AND ImageID != '' AND ImageID LIKE 'file%'

from calibre-kobo-driver.

Hi jgoguen,

Test is not finished, but partial feedback can be given.
As I said before, I send books 200 at a time.
After each transfer, I list books in library to let the Kobo build cover images for the 200 books.
After the transfer of 1200 books, and before ejecting the Kobo, I transfer the KoboReader.sqlite on my PC.
So, to be clear,1200 books on the Kobo, 1000 parsed by listing the library.
The sql request returns 1000 rows.

from calibre-kobo-driver.

Interesting. Can you run this query and post the results?

SELECT ContentID, ImageID, Title FROM content WHERE ContentType = 6 AND ImageID IS NOT NULL AND ImageID != '' AND ImageID LIKE 'file%' LIMIT 3;

from calibre-kobo-driver.

Here is the result :

(if you prefer a file to download, just ask me)

"file:///mnt/onboard/48, Nexus/Maitre Philippe De Lyon - Nexus 48.kepub.epub","file____mnt_onboard_48,Nexus_Maitre_Philippe_De_Lyon-_Nexus_48_kepub_epub","Maître Philippe De Lyon"
"file:///mnt/onboard/Abbott, Edwin Abbott/Flatland - Edwin Abbott Abbott.kepub.epub","file____mnt_onboard_Abbott,Edwin_Abbott_Flatland-_Edwin_Abbott_Abbott_kepub_epub","Flatland"
"file:///mnt/onboard/Abbott, Jeff/Faux-semblants - Jeff Abbott.kepub.epub","file____mnt_onboard_Abbott,Jeff_Faux-semblants-_Jeff_Abbott_kepub_epub","Faux-semblants"

from calibre-kobo-driver.

Bad news ;(

2000 ebooks sent to the Kobo.
Listing the last 200 in the library, 12 have no cover images.

I got the KoboReader.sqlite AFTER the parse of all books in the library.
The request you gave me
SELECT * FROM content WHERE ContentType = 6 AND ImageID IS NOT NULL AND ImageID != '' AND ImageID LIKE 'file%'
returns 2000 rows, the exact number of books sent.

A request on the Content table returning few books with no cover images and few books with cover images doesn't show significant differences.
select *
from Content
where ContentType = 6
and ImageID is not null
and ImageID != ''
and ImageID like 'file%'
and ( Title like '%Hornblower%' -- Books without cover images
or Attribution = 'Edmond About' -- Books with cover images
)

I have read the post on MobileRead from Davidfor (http://www.mobileread.com/forums/showpost.php?p=2521110&postcount=217)
6266 files in the .kobo\images directory.
If I calculate the sum files names length, I get 733470.
The total size of these 6266 files is 455 Mo, 508 Mo on disk.

If you need more, I'll reply you with pleasure.

from calibre-kobo-driver.

As I understand it, this would mean that for those 6266 files you're consuming at least 31826 directory entries (6266 for the standard 8.3 filename support plus 28211 more directory entries for the long file names) and possibly actually 32549 or more directory entries Add to that directory entries for the files already on the system and directory entries for the books and it's quite conceivable that you're hitting the upper limit of directory entries permitted on a FAT32 file system just as davidfor suggested.

The only bug here that I can address is that the driver is setting the ImageID even though you have chosen not to send cover images. To help with that, could you restart in debug mode, send a few books, disconnect your device, plug it back in, let calibre do its processing, then post that debug log?

from calibre-kobo-driver.

Something I forget in my previous post, no file founded in the images folder for the books without cover images.

Now, the test :
I sent 8 books, no cover image after refresh process.
The log is long ! 1208 lines, I try to copy it here ...

GitHub send me an error message : There was an error posting your comment: Body is too long (maximum is 65535 characters)

You can find the log file here : http://depositfiles.com/files/3cbtmi0k1

from calibre-kobo-driver.

It's good that there's no image found, maybe. The calibre driver deletes images from the device if covers are not being uploaded, but it could also mean that the database ImageID is being set anyway. Nothing I saw in your log suggests that, so I'm assuming it means that images are not uploaded and this is actually fixed.

If you're near the limit of FAT directory entries, the behaviour could be unexpected. I'd recommend keeping only a relativly few books (for you, maybe about 1000 at a time) on the device to make sure there's always lots of space for new directory entries.

from calibre-kobo-driver.

I'd like you to try one more query with these 8 books you uploaded still on your device:

SELECT ImageID, Title FROM content WHERE ImageID LIKE 'file____mnt_onboard_%Histoire_2e_Guerre_Mondiale%_kepub_epub'

I'm expecting this to return 5 results, but it's possible that even if results come back it's OK so I'll need to see the query results before I can really say if everything is working as expected.

from calibre-kobo-driver.

Thanks jgoguen, for your job and the disponibility.

Sad conclusion for me. My wish was to have my full library on the Kobo. It seems it's not possible.
Hope the Davidfor suggest will happen a day, changing the partition type.

Thanks again.

from calibre-kobo-driver.

Result for the sql request you asked :
5 rows returned

"file____mnt_onboard_Gallo,Max[Histoire_2e_Guerre_Mondiale-1]1940-De_l'abime_a_l'esperance-_Max_Gallo_kepub_epub","[Histoire 2e Guerre Mondiale-1] 1940-De l'abîme a l'espérance"
"file____mnt_onboard_Gallo,Max[Histoire_2e_Guerre_Mondiale-2]1941-Le_monde_prend_feu-_Max_Gallo_kepub_epub","[Histoire 2e Guerre Mondiale-2] 1941-Le monde prend feu"
"file____mnt_onboard_Gallo,Max[Histoire_2e_Guerre_Mondiale-3]1942-Le_jour_se_leve-_Max_Gallo_kepub_epub","[Histoire 2e Guerre Mondiale-3] 1942-Le jour se lève"
"file____mnt_onboard_Gallo,Max[Histoire_2e_Guerre_Mondiale-4]1943-Le_souffle_de_la_victoire-_Max_Gallo_kepub_epub","[Histoire 2e Guerre Mondiale-4] 1943-Le souffle de la victoire"
"file____mnt_onboard_Gallo,Max[Histoire_2e_Guerre_Mondiale-5]1944-1945-Le_triomphe_de_la_liberte-_Max_Gallo_kepub_epub","[Histoire 2e Guerre Mondiale-5] 1944-1945-Le triomphe de la liberte"

from calibre-kobo-driver.

The partition change does fix the issue, ext4 is a more modern file system while FAT32 is from the days where it was acceptable to say that all files had to have names no more than 8 characters long (plus 3 for the extension), but it's not easy to make this work properly on Mac and even harder on Windows. Neither provide native read/write ext2/3/4 support, requiring third-party tools to read the partition, and I believe you still need Linux just to create the file system. Kobo can't provide this by default since they can't require Windows and Mac users to install this other software. The best we could expect them to do is to change the internal scripts to allow it to "Just Work" for those of us who are able to change the partition file system.

There's some potential consequences to this too, especially for people who download some books from Kobo and sideload some books. The Kobo software (I assume, it's the only explanation that makes sense) runs as "root", so books put on the device by the Kobo downloader would be owned by root while sideloaded books would be owned by your user. This would make it necessary to remember to re-apply file system permission changes every time a device sync is done.

from calibre-kobo-driver.

I just noticed you updated with the SQL results. This indicates to me that something is setting the ImageID to the calibre standard. All I can really say, based on the debug log provided, is that it doesn't seem to be me. You could ask davidfor if it's something in his driver but I'm confident that it isn't me.

from calibre-kobo-driver.

I'll test when I'll have some times, after a factory reset, without your plugin.
But, I now better understand and suppose too that the plugin is not the cause.
I think I will suggest Davidfor to add an option to transfer books not with their original calibres's names but in a short format. The problem, if it is possible is to ensure the synchronisation with Calibre if names of files are different betwwen Kobo and Calibre library.

from calibre-kobo-driver.

It's already possible to keep track of files despite different file names on the device. If you download a book from Kobo it gets on the device with a UUID-style file name that calibre never uses but calibre still notices if it's a book that you have in your library. I'm not certain, but I believe calibre looks at the author and title to determine if a book is in your library.

One thing for you to take more of that I just realized, there are three options for sending covers; if any one of them is enabled, cover images are sent to the device, but only two of them are used to consider whether to delete the cover image files from the device. So when you try this next, make sure all three cover image options are disabled if you haven't been doing that already.

from calibre-kobo-driver.

Hello !

I found a solution to increase the capacity at more than 3500 books.
I've build a new template for the folders and files names based on the internal Id in Calibre.
Not sure to have the capacity to go up to 4500 as I want, but it's much better than before.

Thanks for your help and the help of Davidfor.

from calibre-kobo-driver.