Code Monkey home page Code Monkey logo

Comments (9)

aleju avatar aleju commented on May 22, 2024 1

I just tried the above script with similar image sizes and keypoints as in your example. Still seems to generate reasonable outputs (see below). Do you maybe happen to resize the images before or after the augmentation? It is possible that you are working with the wrong image shape in your keypoints, leading to them being rotated on a larger image than the one that you augment. In the below script, you can uncomment #kps = [ia.KeypointsOnImage(kps, shape=(img.shape[0]+50, img.shape[1]+50))] (and comment the other kps=... line) to see the effect.

from __future__ import print_function, division
import imgaug as ia
from imgaug import augmenters as iaa
from scipy import misc, ndimage
import numpy as np

ia.seed(57)

def drawpoint(img, kp, size=4):
    h, w = img.shape[0:2]
    x = kp.x
    y = kp.y
    x1 = x - size
    x2 = x
    y1 = y - size
    y2 = y
    if 0 <= x1 < w and 0 <= x2 < w and 0 <= y1 < h and 0 <= y2 < h:
        img[y1:y2, x1:x2, 0] = 255
    return img

def drawpoints(img, kps, size=4):
    for kp in kps:
        drawpoint(img, kp, size=size)
    return img

sw = 256  # size of the image
sh = 305  # size of the image
pd = 25  # positions of each corner pixel (e.g. at (25, 25), (-25, 25), ...)
ps = 6  # size of each square to draw
img = np.zeros((sh, sw, 3), dtype=np.uint8)  # black example image
# add white squares in the corners
img[pd - ps:pd, pd - ps:pd, :] = 255
img[-pd - ps:-pd, pd - ps:pd, :] = 128
img[pd - ps:pd, -pd - ps:-pd, :] = 64
img[-pd - ps:-pd, -pd - ps:-pd, :] = 255
img[0, :, :] = 20
img[-1, :, :] = 20
img[:, 0, :] = 20
img[:, -1, :] = 20

# define one test keypoint at (100, 100)
kps_coords = [(50, 75), (150, 75), (100, 120), (60, 160), (120, 160)]
kps = []
img_with_kp = np.copy(img)
for kp_x, kp_y in kps_coords:
    img_with_kp[kp_y - ps:kp_y, kp_x - ps:kp_x, 2] = 255
    kps.append(ia.Keypoint(kp_x, kp_y))
kps = [ia.KeypointsOnImage(kps, shape=img.shape)]
#kps = [ia.KeypointsOnImage(kps, shape=(img.shape[0]+50, img.shape[1]+50))]
img_org2 = kps[0].draw_on_image(np.copy(img_with_kp))

nb_augs = 1000  # how often to augment here
aug_seq = iaa.Sequential({
    iaa.Affine(rotate=(-60, 60), shear=(-20, 20))  # rotate -40 to 40 deg
})
aug_aff = iaa.Affine(rotate=(-60, 60), shear=(-20, 20))

###############
# make deterministic
###############
aug_seq = aug_seq.to_deterministic()
aug_aff = aug_aff.to_deterministic()
###############

# augment images
imgs_seq_aug = aug_seq.augment_image(img_with_kp)
imgs_aff_aug = aug_aff.augment_image(img_with_kp)

# augment keypoints
kps_seq_aug = []
kps_aff_aug = []
kps_seq_aug = aug_seq.augment_keypoints(kps)
kps_aff_aug = aug_aff.augment_keypoints(kps)
#seq_x, seq_y = kps_seq_aug[0].keypoints[0].x, kps_seq_aug[0].keypoints[0].y
#aff_x, aff_y = kps_aff_aug[0].keypoints[0].x, kps_aff_aug[0].keypoints[0].y

# code to test draw_on_image
img_org = kps[0].draw_on_image(np.copy(img_with_kp))
img_seq_aug = kps_seq_aug[0].draw_on_image(np.copy(imgs_seq_aug))
img_aff_aug = kps_aff_aug[0].draw_on_image(np.copy(imgs_aff_aug))
img_seq_aug2 = drawpoints(np.copy(img_seq_aug), kps_seq_aug[0].keypoints)
img_aff_aug2 = drawpoints(np.copy(img_aff_aug), kps_aff_aug[0].keypoints)
# misc.imshow(np.hstack([img_seq_aug, img_aff_aug, img_org.astype(np.uint8)]))

# plot
misc.imsave("test_image2.png", np.hstack([img_seq_aug2, img_aff_aug2, img_org]))

With correct image shape:
test_image2

With incorrect image shape:
test_image2

from imgaug.

aleju avatar aleju commented on May 22, 2024

I just checked that and everything seemed correct to me.
The below script creates a black test image of size 150x150 with white squares in the corners. It uses one keypoint at (100, 100). It runs 1000 augmentations of both image and keypoint with rotations in range (-15, 15) and averages the result images.

from __future__ import print_function, division
import imgaug as ia
from imgaug import augmenters as iaa
from scipy import misc
import numpy as np

s = 150 # size of the image
pd = 25 # positions of each corner pixel (e.g. at (25, 25), (-25, 25), ...)
ps = 4 # size of each square to draw
img = np.zeros((s, s, 3), dtype=np.uint8) # black example image

# add white squares in the corners
img[ pd-ps:pd,   pd-ps:pd,  :] = 255
img[-pd-ps:-pd,  pd-ps:pd,  :] = 255
img[ pd-ps:pd,  -pd-ps:-pd, :] = 255
img[-pd-ps:-pd, -pd-ps:-pd, :] = 255

# define one test keypoint at (100, 100)
kp_x = 100
kp_y = 100
kps = [ia.KeypointsOnImage([ia.Keypoint(kp_x, kp_y)], shape=img.shape)]
img_with_kp = np.copy(img)
img_with_kp[kp_y-ps:kp_y, kp_x-ps:kp_x, 1] = 255

nb_augs = 1000 # how often to augment here
aug = iaa.Affine(rotate=(-15, 15)) # rotate -15 to 15 deg

# augment the image N times
imgs_aug = aug.augment_images(
    np.tile(img[np.newaxis, ...], (nb_augs, 1, 1, 1))
)

# augment the keypoints N times
kps_aug = []
for i in xrange(nb_augs):
    kps_aug = aug.augment_keypoints(kps)
    x, y = kps_aug[0].keypoints[0].x, kps_aug[0].keypoints[0].y
    
    # code to test draw_on_image
    #img2 = kps_aug[0].draw_on_image(np.copy(img))
    #misc.imshow(img2)

    # draw each keypoint as a (ps x ps) square at its new (augmented) coords
    # only draw if the kp is still inside the image
    x1 = x - ps
    x2 = x
    y1 = y - ps
    y2 = y
    if 0 <= x1 < s and 0 <= x2 < s and 0 <= y1 < s and 0 <= y2 < s:
        imgs_aug[i, y1:y2, x1:x2, 1] = 255

# plot
imgs_aug_avg = np.average(imgs_aug, axis=0)
#misc.imshow(np.hstack([img_with_kp, imgs_aug_avg.astype(np.uint8)]))
misc.imsave("test_image.png", np.hstack([img_with_kp, imgs_aug_avg.astype(np.uint8)]))

Result image:
test_image

Left: Unaugmented image
Right: Average over 1000 augmented/rotated versions. White are the corner pixels (part of the image), green is the keypoint.

from imgaug.

araghava92 avatar araghava92 commented on May 22, 2024

I just tried out your code. It looks great.

I changed it to match mine. I first used affine directly as you did and I think it worked but then added sequential affine to compare. This is the result I see:

test_image

Heres my code: (Hope I didn't commit some very basic mistake)

from __future__ import print_function, division
import imgaug as ia
from imgaug import augmenters as iaa
from scipy import misc, ndimage
import numpy as np

s = 150  # size of the image
pd = 25  # positions of each corner pixel (e.g. at (25, 25), (-25, 25), ...)
ps = 4  # size of each square to draw
img = np.zeros((s, s, 3), dtype=np.uint8)  # black example image
# add white squares in the corners
img[pd - ps:pd, pd - ps:pd, :] = 255
img[-pd - ps:-pd, pd - ps:pd, :] = 255
img[pd - ps:pd, -pd - ps:-pd, :] = 255
img[-pd - ps:-pd, -pd - ps:-pd, :] = 255

# define one test keypoint at (100, 100)
kp_x = 100
kp_y = 100
kps = [ia.KeypointsOnImage([ia.Keypoint(kp_x, kp_y)], shape=img.shape)]
img_with_kp = np.copy(img)
img_with_kp[kp_y - ps:kp_y, kp_x - ps:kp_x, 1] = 255

nb_augs = 1000  # how often to augment here
aug_seq = iaa.Sequential({
    iaa.Affine(rotate=(-40, 40))  # rotate -40 to 40 deg
})
aug_aff = iaa.Affine(rotate=(-40, 40))

imgs_seq_aug = aug_seq.augment_image(img)
imgs_aff_aug = aug_aff.augment_image(img)

# augment keypoints 
kps_seq_aug = []
kps_aff_aug = []
kps_seq_aug = aug_seq.augment_keypoints(kps)
kps_aff_aug = aug_aff.augment_keypoints(kps)
seq_x, seq_y = kps_seq_aug[0].keypoints[0].x, kps_seq_aug[0].keypoints[0].y
aff_x, aff_y = kps_aff_aug[0].keypoints[0].x, kps_aff_aug[0].keypoints[0].y

# code to test draw_on_image
img_org = kps[0].draw_on_image(np.copy(img_with_kp))
img_seq_aug = kps_seq_aug[0].draw_on_image(np.copy(imgs_seq_aug))
img_aff_aug = kps_aff_aug[0].draw_on_image(np.copy(imgs_aff_aug))
# misc.imshow(np.hstack([img_seq_aug, img_aff_aug, img_org.astype(np.uint8)]))

# plot
misc.imsave("test_image.png", np.hstack([img_seq_aug, img_aff_aug, img_org]))

from imgaug.

aleju avatar aleju commented on May 22, 2024

Seems OK to me at first glance. What do you expect to see instead? The slight difference of the points in the third image is expected as the drawing routine of draw_on_image() is a bit different compared to the one in this script.

Btw, Sequential() usually expects to get a list instead of dictionary, i.e. aug_seq = iaa.Sequential([iaa.Affine(rotate=(-40, 40))]) instead of aug_seq = iaa.Sequential({iaa.Affine(rotate=(-40, 40))}). Shouldn't cause an issue here, but when there are multiple augmenters in Sequential it might lead to them being executed in an unexpected order as standard dictionaries to not have a clearly determined order.

from imgaug.

araghava92 avatar araghava92 commented on May 22, 2024

Sorry, I should have mentioned the order. The third is the original image and the first two are the translated.
misc.imsave("test_image.png", np.hstack([img_seq_aug, img_aff_aug, img_org]))

In the third image when you consider the green point and the white points to its right as a triangle you will notice its scalar(none of the distances between the points seem equal).
But when you do the same with the first two images, it looks almost isosceles.

What I expect here is for the distances between the ints to remain the same even after the transformation.

It is kind of hard to spot the difference here, but when I am using face landmarks, the diff is noticeable. A point I marked above an eye was inside the eye after the translation.
(The image below I generated without Sequential().

In the image below, the first is the original and the third and the forth are rotate and shear affines respectively.

image

P.S. Thanks a lot for the pointer with the param type dict that I got wrong

from imgaug.

aleju avatar aleju commented on May 22, 2024

Are you using determinism in your facial landmark script? You have to call aug_det = aug.to_deterministic() before augmenting the image and landmarks. Otherwise, a different rotation might be applied to image and landmarks.
Below is (mostly) the script from above, but modified with determinism:

from __future__ import print_function, division
import imgaug as ia
from imgaug import augmenters as iaa
from scipy import misc, ndimage
import numpy as np

ia.seed(57)

# simple function to draw a keypoint on an image
def drawpoint(img, kp, size=4):
    h, w = img.shape[0:2]
    x = kp.x
    y = kp.y
    x1 = x - size
    x2 = x
    y1 = y - size
    y2 = y
    if 0 <= x1 < w and 0 <= x2 < w and 0 <= y1 < h and 0 <= y2 < h:
        img[y1:y2, x1:x2, 0] = 255
    return img

s = 150  # size of the image
pd = 25  # positions of each corner pixel (e.g. at (25, 25), (-25, 25), ...)
ps = 6  # size of each square to draw
img = np.zeros((s, s, 3), dtype=np.uint8)  # black example image
# add white squares in the corners
img[pd - ps:pd, pd - ps:pd, :] = 255
img[-pd - ps:-pd, pd - ps:pd, :] = 128
img[pd - ps:pd, -pd - ps:-pd, :] = 64
img[-pd - ps:-pd, -pd - ps:-pd, :] = 255

# define one test keypoint at (100, 100)
kp_x = 100
kp_y = 100
kps = [ia.KeypointsOnImage([ia.Keypoint(kp_x, kp_y)], shape=img.shape)]
img_with_kp = np.copy(img)
img_with_kp[kp_y - ps:kp_y, kp_x - ps:kp_x, 2] = 255
img_org2 = kps[0].draw_on_image(np.copy(img_with_kp))

nb_augs = 1000  # how often to augment here
aug_seq = iaa.Sequential({
    iaa.Affine(rotate=(-60, 60))  # rotate -40 to 40 deg
})
aug_aff = iaa.Affine(rotate=(-60, 60))

###############
# make deterministic
###############
aug_seq = aug_seq.to_deterministic()
aug_aff = aug_aff.to_deterministic()
###############

# augment images
imgs_seq_aug = aug_seq.augment_image(img_with_kp)
imgs_aff_aug = aug_aff.augment_image(img_with_kp)

# augment keypoints
kps_seq_aug = []
kps_aff_aug = []
kps_seq_aug = aug_seq.augment_keypoints(kps)
kps_aff_aug = aug_aff.augment_keypoints(kps)
seq_x, seq_y = kps_seq_aug[0].keypoints[0].x, kps_seq_aug[0].keypoints[0].y
aff_x, aff_y = kps_aff_aug[0].keypoints[0].x, kps_aff_aug[0].keypoints[0].y

# code to test draw_on_image
img_org = kps[0].draw_on_image(np.copy(img_with_kp))
img_seq_aug = kps_seq_aug[0].draw_on_image(np.copy(imgs_seq_aug))
img_aff_aug = kps_aff_aug[0].draw_on_image(np.copy(imgs_aff_aug))
img_seq_aug2 = drawpoint(np.copy(img_seq_aug), kps_seq_aug[0].keypoints[0])
img_aff_aug2 = drawpoint(np.copy(img_aff_aug), kps_aff_aug[0].keypoints[0])
# misc.imshow(np.hstack([img_seq_aug, img_aff_aug, img_org.astype(np.uint8)]))

# plot
misc.imsave("test_image2.png", np.hstack([img_seq_aug2, img_aff_aug2, img_org, img_org2]))

test_image2

Images are the same as before (ignore the fourth one).
Blue = keypoint, but drawn on the image before augmentation. So it is modified by augment_image() instead of augment_keypoints()
Green = keypoint after augmentation, drawn by draw_on_image()
Red = keypoint after augmentation, drawn by drawkeypoint(), which uses a similar drawing technique as the blue keypoint.

from imgaug.

araghava92 avatar araghava92 commented on May 22, 2024

I tried with to_deterministic too. Here is the result

image

Shear works well. with deterministic but the rotate is still off.

from imgaug.

araghava92 avatar araghava92 commented on May 22, 2024

You were right man. The issue had something to do with the shape.
And it was occurring in the test code because of the deterministic function that I missed.
Between the two of them, I was a little confused about the error.
Sorry for wasting your time by raising an issue here.
Thanks a lot for all your help and such prompt responses.

from imgaug.

qingmeizhujiu avatar qingmeizhujiu commented on May 22, 2024

I alse meet the error. And, when I use "deterministic", the error is fixed.

from imgaug.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.