VImage.write in Python seems to take a lot of time about libvips HOT 14 CLOSED

That's how vips works --- it delays all processing until the final .write(), then runs all the queued-up operations in parallel.

Perhaps there's some operation earlier in the pipeline that's taking lots of time? Some file formats (png is one of the worst) can be very slow to write as well. Can you post a complete program that shows the problem?

from libvips.

Hmm... I did only open/affinei and write operation on jpeg image about 1-2MB in size. Let me try it on different platform and profile it.

from libvips.

That could be about right, depending on your processor. I get:

$ time vips affine x.jpg x2.jpg "0.9 0 0 0.9"
real 0m0.287s
user 0m0.516s
sys 0m0.080s
$ header x.jpg
x.jpg: 2477x4096 uchar, 3 bands, srgb, partial VipsImage (0xc47060)
$ ls -l x.jpg
-rw-rw-r-- 1 john john 983202 Feb 19 08:45 x.jpg

ie. about 300ms for a 1MB jpg file.

ImageMagick is a bit slower:

$ time convert x.jpg -resize 90% x2.jpg
real 0m0.813s
user 0m2.384s
sys 0m0.156s

from libvips.

I compared it with PIL and PIL seems to be a lot faster. The file can be downloaded from http://www.unearthedoutdoors.net/global_data/true_marble/download

import os, sys
import Image
im = Image.open("/tmp/TrueMarble.250m.21600x21600.B4.png")
size = 1024, 1024
im.thumbnail(size, Image.ANTIALIAS)
im2 = im.convert("RGBA")
im2.save("/tmp/test2.jpg","JPEG")

% time python /tmp/test.py
python /tmp/test.py  0.57s user 0.26s system 99% cpu 0.831 total
% time vipsthumbnail /tmp/TrueMarble.250m.21600x21600.B4.png -s 1024
vipsthumbnail /tmp/TrueMarble.250m.21600x21600.B4.png -s 1024  4.13s user 0.08s system 62% cpu 6.753 total

from libvips.

That's a very unusual png though. It's a 1.3gb image with every pixel having the same dark blue value, compressed down to 50kb. PIL is somehow cleverly spotting this (I wonder how? interesting) and short-circuiting the load process. vipsthumbnail really is loading and averaging all 450m pixels.

If you try on a more typical image, PIL is slower. For example, "wtc.png" is a 10,000 x 10,000 pixel aerial photograph:

$ time vipsthumbnail wtc.png -s 1024
real    0m3.346s
user    0m3.352s
sys 0m0.044s
$ time ./pil.py wtc.png x.jpg
real    0m9.195s
user    0m9.005s
sys 0m0.172s

vipsthumbnail needs 40mb of memory, PIL 400mb.

from libvips.

When I tried it, I don't see much differences:

% time python test2.py :tmp:Wtc-photo.jpg                           
python test2.py :tmp:Wtc-photo.jpg  2.27s user 0.37s system 50% cpu 5.191 total
% time python test.py :tmp:Wtc-photo.jpg                                                
python test.py :tmp:Wtc-photo.jpg  0.50s user 0.03s system 99% cpu 0.534 total

## test2.py ##
import sys
from vipsCC import *

try:
    fred = VImage.VImage (sys.argv[1])
    bob = fred.shrink(8,8)
    jim = bob.affine(1024.0/9372*8,0,0,1024.0/9372*8,0,0,0,0,1024,1024)
    jim.write ("/tmp/test.jpg")
except VError.VError, e:
    e.perror (sys.argv[0])

On the side note, I'm trying to emulate vipsthumbnail in Python. I couldn't find a way to do vips_foreign_load to perform shrink operation up front when I load an image. Did I use the wrong shrink API?

from libvips.

What's test.py? Is that the PIL one? It might not be doing the block-shrink-before-affine thing, perhaps? You should check the result for aliasing artifacts.

pyvips uses the vips7 API, so downsample on load is done via the filename. You append ":2" to the jpeg filename you pass to the constructor to get the x2 downsampled version.

http://www.vips.ecs.soton.ac.uk/supported/7.24/doc/html/libvips/VipsFormat.html#im-jpeg2vips

There will be a new Python binding based around the much better vips8 API at some point.

from libvips.

Sorry for not being clear. test.py uses the PIL script in the previous comment. I checked the result for both of them. They look quite similar with very small differences. Will the new vips8 api still be based on SWIG?

If I use :8, it seems to perform much better.

import sys
from vipsCC import *

try:
    fred = VImage.VImage (sys.argv[1] + ":8")
#    bob = fred.shrink(8,8)
    jim = fred.affine(1024.0/9372*8,0,0,1024.0/9372*8,0,0,0,0,1024,1024)
    jim.write ("/tmp/test.jpg")
except VError.VError, e:
    e.perror (sys.argv[0])

python /tmp/test2.py /tmp/Wtc.jpg  0.41s user 0.05s system 76% cpu 0.601 total

from libvips.

The new one uses gobject-introspection. The whole vips8 API is discoverable at runtime. The Python binding uses getattr to map member access to vips8 API calls. The whole binding is done in Python, and is only 160 lines of code.

You can test-drive it here:

https://github.com/jcupitt/libvips/tree/master/python

it's missing array constants still (must fix this), but it does work. Example code:

https://github.com/jcupitt/libvips/blob/master/python/try4.py

from libvips.

Ah, I hadn't noticed you'd switched to jpg images. Yes, PIL does the doansample-on-load trick for you automatically, I think.

Here are two shrinkers:

#!/usr/bin/python

import sys
from vipsCC import *

try:
    fred = VImage.VImage (sys.argv[1])
    bob = fred.shrink(8,8)
    jim = bob.affine(1024.0/9372*8,0,0,1024.0/9372*8,0,0,0,0,1024,1024)
    jim.write (sys.argv[2])
except VError.VError, e:
    e.perror (sys.argv[0])

#!/usr/bin/python

import os, sys
import Image

im = Image.open(sys.argv[1])
size = 1024, 1024
im.thumbnail(size, Image.ANTIALIAS)
im2 = im.convert("RGBA")
im2.save(sys.argv[2], "JPEG")

If I run them on png, I see a large speed difference:

$ time ./vips.py wtc.png vips.jpg
real    0m4.914s
user    0m4.716s
sys 0m0.932s
$ time ./pil.py wtc.png pil.jpg
real    0m14.576s
user    0m13.809s
sys 0m0.472s

If I try jpg, then PIL is faster, since it's doing downsample-on-load automatically:

$ time ./pil.py ../wtc.jpg pil.jpg
real    0m0.756s
user    0m0.716s
sys 0m0.036s
$ time ./vips.py ../wtc.jpg vips.jpg
real    0m2.092s
user    0m2.204s
sys 0m0.804s

If I run vips with downsample-on-load, it's faster:

$ time ./vips.py ../wtc.jpg:8 vips.jpg
real    0m0.409s
user    0m0.392s
sys 0m0.036s

(I changed the vips.py source in this case to remove the .shrink(), since that's being done by libjpeg)

from libvips.

I ran some tests on larger set of files. I think VIPS is a bit slower when running under concurrent workload i.e. trying to generate thumbnails in multiple process at the same time. Have you ever looked at EPEG? It seems to be a lot faster.

from libvips.

Yes, vips tries to use all your CPUs, so if you run lots of them at the same time, it'll slow down.

I'll have a look at EPEG.

from libvips.

You're right, epeg is very quick:

$ time vipsthumbnail -n -d wtc.jpg
real    0m0.247s
user    0m0.236s
sys 0m0.008s
$ time epeg -m 128 wtc.jpg x.jpg
real    0m0.217s
user    0m0.212s
sys 0m0.004s

So epeg is about 10 - 15% faster. The speed difference is less on a really large file:

$ time vipsthumbnail -n -d huge.jpg
real    0m2.461s
user    0m2.416s
sys 0m0.040s
$ time epeg -m 128 huge.jpg x.jpg
real    0m2.288s
user    0m2.240s
sys 0m0.040s

epeg is about 5 - 10% faster there.

vipsthumbnail will do many other formats quickly, such as png and tiff, so it has that in it's favour. vipsthumbnail can also do colour management and is much better at preserving image metadata.

from libvips.

I'll close this issue since it seems to be done. Please reopen if you wish.

from libvips.