blue channel missing, and red and green channels reversed, when decompressing 3DC compressed normal maps about nvidia-texture-tools HOT 14 OPEN

amorilia,

thanks for the detailed bug report. 

nvdecompress uses a flag in the DDS header to determine whether the ATI2 blocks 
are
used for normal map compression or not. If the file was generated by nvtt the 
flag
should be set, if it was generated by another tool, it's probably not.

The code that handles that is in nvtt/DirectDrawSurface.cpp:903

// If normal flag set, convert to normal.
if (header.pf.flags & DDPF_NORMAL)
{
    if (header.pf.fourcc == FOURCC_ATI2)
    {
        for (int i = 0; i < 16; i++)
        {
            Color32 & c = rgba->color(i);
            c = buildNormal(c.r, c.g);
        }
    }
    ...

ATI2 files can be used for other purposes other than storing normals, so by 
default
the XY blocks are decompressed and stored in the RG channels, just like the 
hardware
does. When the normal flag is present, the normal is reconstructed from the RG 
channels.

It might be interesting to add a command line option to indicate that the input 
is
supposed to be a normal. I think the current behavior should be preserved, 
unless the
general consensus differed. I'll keep the bug open for a while, in case anybody 
wants
to add their 2 cents. Let me know if you have other thoughts.

Yes, I hadn't noticed the normal flag check in the code! Indeed it is missing 
on the
particular files I was testing.

So I tested this again by forcing the normal flag on read for ATI2 files:

--- src/nvimage/DirectDrawSurface.cpp   (revision 493)
+++ src/nvimage/DirectDrawSurface.cpp   (working copy)
@@ -604,6 +604,7 @@
        {
                (*stream) << header;
        }
+       if (header.pf.fourcc == FOURCC_ATI2) header.setNormalFlag(true); 
 }

Now running nvdecompress on the file again, I do get a blue channel. I guess it 
does
make sense to assume a normal map by default for ATI2 files even if their flag 
isn't
set, since that's what they have been designed for after all?

Still, one issue left: red and green channels seem swapped in the screenshots 
from
WTV and ddsview. So I downloaded ATI's compressonator, and guess what (see 
attached
screenshot)? :-) Seems like WTV and ddsview don't agree with the 
compressonator. That
leaves me wondering whether it is possible that some ATI2 files have X and Y 
channels
swapped.

Oops, scrap that last line, the shot is from test.dds which was one of my tests 
with
nvcompress. Actually compressonator, WTV, and ddsview all agree on the file!!!
Attached a correct screenshot. So it seems that red and green channels must be
swapped after all (how weird nothing of this is mentioned in the specs...).

Here's a quick patch for nvdecompress input options, including forcing of the 
normal
map flag.

I still haven't identified the cause of the red and green channels being 
swapped on
3DC. Any progress on that front?

Sorry for the delay, I'll integrate this ASAP. I thought that in comment 3 you
retired what you said about about the swapped channels. Let me check that out.

I've integrated your patch. I also looked at the problem of the swapped 
coordinates.
If you look at the DX10 block compression format spec:

http://msdn2.microsoft.com/en-us/library/bb694531.aspx

you will see that BC5 is supposed to have two blocks, the first one encodes the 
R and
the second one encodes the G channels. That's what I implement in the texture 
tools.
However, ATI's 3Dc format uses the opposite convention, G and then R.

Unfortunately, most viewers out there assume the 3Dc convention. On the other 
side,
the compressonator uses the bitcount field of the DDS header to indicate how the
components are swizzled. However, I haven't been able to figure out how's that 
done.
Here are the bitcount values for some of their formats:

ATI2 YX:            0 (0x00000000)
ATI2 XY:   1498952257 (0x59583241) (BC5)
DXT5 ATI2:  893661761 (0x35443241)
DXT5 xGBR: 1380075384 (0x52424778)
DXT5 RxBG: 1195538514 (0x47427852)
DXT5 RGxB: 1199063634 (0x47784252)
DXT5 xRBG: 1195528824 (0x47425278)
DXT5 RGxB: 1115178834 (0x42784752)
DXT5 xGxR: 1383614328 (0x52784778) (DXT5n)

Any idea how the swizzle is encoded?

Thanks for integrating the patch!

I'm not sure what your question is about the swizzle. You mean to ask whether 
there's
a pattern linking the bitcount field to the swizzle in some logical way? The 
bitcount
field is simply used as a string as far as I can see. For instance, the first is

0x59583241 = 0x41 0x32 0x58 0x59 = "A2XY"

and the last is

0x52784778 = 0x78 0x47 0x78 0x52 "xGxR"

and so on (the second expression is how it's stored in the memory: the byte 
order
must be swapped to read the integer as a 4-byte string).

Doh! that was so obvious. Those are simply FOURCC codes. I'll change the 
compressor
to output the files with the appropriate swizzle code, and improve the 
decompressor
to handle most of them.

Brilliant! I'll be looking forward to the implementation, as it would be nice to
support all those formats with Blender too. Aside, for your information, your 
DDS
decompression code, which I have ported to Blender, will likely make it 
(finally!)
into the next release, Blender 2.46. :-)

That is awesome!

ATI2 files are not output with the appropriate swizzle code, so they should be
displayed correctly in third party tools.

I'll add support for reading the swizzle codes next.

I check out the latest version from svn, and "red and green channels reversed" 
the problem still exists.

how to work around this?

finally, manually swap the channel fixed this issue. 

ugly solution, hope for a better one.