intelxed / xed Goto Github PK

Sorry for opening another issue, but I might have found a bug regarding MVEX conversion functionality. The following instruction should be invalid:

62 f1 78 59 59 00 | vmulps zmm0, k1, zmm0, xmmword ptr [rax]{sint8}

Table of valid MVEX.SSS values:

S2S1S0 | Function                  | Usage
-----------------------------------------------------
000    | no conversion             | [rax]
001    | broadcast 1 element (x16) | [rax] {1to16}
010    | broadcast 4 elements (x4) | [rax] {4to16}
011    | float16 to float32        | [rax] {float16}
100    | uint8 to float32          | [rax] {uint8}
101    | reserved                  | N/A
110    | uint16 to float32         | [rax] {uint16}
111    | sint16 to float32         | [rax] {sint16}

I did not test all of the other classes, but at least for Si32 (e.g. vpaddsetsd) XED correctly detects MVEX.SSS = 011 as invalid.

Some of the AMD XOP instructions seem to have incorrect operand definitions. BLCFILL for example uses the GPRy_N() register-class, which is declared as:

xed_reg_enum_t GPRy_N()::
EASZ=3 | OUTREG=VGPR64_N() 
EASZ=2 | OUTREG=VGPR32_N()
EASZ=1 | OUTREG=VGPR32_N()

The register-width should scale by the effective operand-size instead of the address-size, like it is described in the documentation:

In 64-bit mode, the operand size is determined by the value of XOP.W. If XOP.W is 1, the operand size
is 64-bit; if XOP.W is 0, the operand size is 32-bit. In 32-bit mode, XOP.W is ignored. 16-bit operands
are not supported.

Besides that, XOP.W = 0 (W0) should be removed from the pattern.

It seems like (at least) all definitions defined in 'amd-xop-isa.txt' below BEXTR (including this one) are affected, but there might be more instructions with an incorrect W0 pattern.

consider the following instruction
48 2B E0 sub rsp, rax
it has REX=48, opcode=2B and modrm=E0 bytes

xed_decoded_inst_s::has_modrm = 1
xed_decoded_inst_s::modrm_byte = E0
xed_decoded_inst_s::modrm = 0

along with has_modrm and modrm_byte fields the structure has modrm field which is not filled in

xed_operand_values_has_modrm_byte which (according to its name) is supposed to return has_modrm=1 field, but returns modrm=0 field instead

FIX should be in change of line 237 in xed-operand-values-interface.c from
return xed3_operand_get_modrm(p);
to
return xed3_operand_get_has_modrm(p);

what is the meaning of field modrm? maybe it should be deleted or deprcated if not used?

When running on a Linux 64 machine, our fuzzing tool generated a series of test cases that we think result from two similar root causes: in xed-disas-elf.c, during process_elf64 and symbols_elf64, the actual parameter offset of lookup64 seems to be calculated wrongly.

For process_elf64, some test cases cause segfaults with the regular compilation options, others can trigger crashes with help of address sanitizer.

• crash site: xed-disas-elf.c:446
  direct segfault
  w01_000001,sig:6,Splice:48:102,src:w00_000000.gdb.txt
  w01_000001,sig:6,Splice:48:102,src:w00_000000.input.txt
  crash with asan
  w01_000003,sig:6,Flip1:496:53984,src:w01_000822.err.txt
  w01_000003,sig:6,Flip1:496:53984,src:w01_000822.input.txt

For symbols_elf64, the crash we found can be triggered with address sanitizer.

• crash site: xed-disas-elf.c:557:53
  w02_000000,sig:6,Flip1:496:37632,src:w01_000000.err.txt
  w02_000000,sig:6,Flip1:496:37632,src:w01_000000.input.txt

We are not aware whether 32bit elf disassembly is affected since our fuzzer hasn't covered the corresponding functions yet.

I may have totally missed it, but it would be super nice to have some information on how those files should be parsed.

The idea is that one could reuse this data set, parse those files and generate their own tables to support encoders/decoders written in different languages. I've noticed that other people have had that idea as well, in this thread, Russ Cox, a golang developer, mentions it for the Go compiler.

Of course, it's possible to look - or even modify - the python parser, but having a document with at least the overview of the process would help a lot.

I'm seeing a problem building XED with Python 3.x, depending on where
I put the mbuild directory.

This is with the latest xed (bd73bdb from Jan 29) and latest mbuild
(bb9123152a from Nov 27). I clone xed, then cd into xed and clone
mbuild, so mbuild is a subdir of xed.

With python 2.7, this works (with mbuild a subdir of xed), but with
python 3.6.3, I get:

$ ./mfile.py
module 'mbuild' has no attribute 'check_python_version'

But if I move the xed and mbuild directories to be side by side, then
it works with 2.7 and 3.6.3.

Strangely, with python 3.1.5, it fails in a different way with both
directory layouts (fails as subdir and side-by-side).

$ ./mfile.py
[INVOKED] ./mfile.py
...
[PYTHON VERSION] 3.1.5
'_io.FileIO' object has no attribute 'read1'

So, first let me check, is mbuild as a subidr of xed supposed to be
supported? Is python 3.x supported?

The reason I ask is that I've written a spack recipe for xed. (Are
you familiar with the spack package manager from Livermore?) Spack is
sometimes a bit rigid in their way of doing things. In this case,
they don't handle the case of multiple tar files or multiple git
clones very easily.

With spack, I can easily clone mbuild inside xed, but putting them
side-by-side is tougher. Maybe I just got lucky that it worked as a
subdir with 2.7.

So, I'm looking for guidance ... is it easier to fix xed to work with
Python 3.x as a subdir, or is it easier to push on spack to fix their
directory structure.

I'm hoping this will be a short fix for xed, but if I'm pushing the
bounds of what mbuild supports, then I apologize. :-)

And btw, whatever the answer for the directory layout is, thanks very,
very much for making XED available on github!! This simplifies my
work life somewhat.

Thanks,

--Mark Krentel

Hi there,

not sure about this one as the SDM is rather vague about it, but for the MULTISOURCE4 registers it says:

the notation of “+3” is used to denote that the instruction accesses 4 source registers based on
that operand; sources are consecutive, start in a multiple of 4 boundary,

Does that mean the instruction #UDs / is invalid, if the register operand does not refer to a "multiple of 4"? E.g. if ZMM31 is used as the source register.

At the moment, XED allows all possible register values ZMM0..ZMM31.

XED issues XED_REG_EIP as a base register for LEA instead of XED_REG_RIP in 64bit mode for XED_OPERAND_AGEN operand. for example (vm: hex-code -> op-code):

1000194ca: 488D3D 8F5A5900 -> lea rdi, 0x1005aef60 (EIP)

vs.

1001b4d36: 488D05 23A23F00 -> lea rax, 0x1005aef60 (RIP)
1001f1cbd: 4C8D2D 9CD23B00 -> lea r13, 0x1005aef60 (RIP)

Would you like to replace any double quotes by angle brackets around file names for include statements?

The function system_install() in xed_mbuild.py uses _set_perm() to set
the permissions of shared libraries to 755 (among other uses). But
when copying shared libraries (with --prefix), the call is misspelled
with 2 underscores on line 1818.

xed_mbuild.py:
1808 # copy the libraries
1809 libs = _gen_lib_names(env)
1810 if len(libs) == 0:
1811 xbc.cdie("No libraries found for install")
1812 for f in libs:
1813 mbuild.msgb("COPY", "{} <- {}".format(lib,f))
1814 if do_system_copy:
1815 mbuild.copy_file(f, lib)
1816 fn = mbuild.join(lib,os.path.basename(f))
1817 if env['shared']:
1818 __set_perm(fn)
1819 else:
1820 mbuild.make_read_only(fn)

As a result, './mfile.py --prefix=DIR --shared' fails with:

[STRIPPING] obj/libxed.so
[STRIP CMD] strip -x obj/libxed.so
[Making install dirs (if they do not exist)]
[MKDIR] /home/krentel/xed/prefix/include/xed
[MKDIR] /home/krentel/xed/prefix/lib
[COPY] /home/krentel/xed/prefix/lib <- obj/libxed.so
[COPY] /home/krentel/xed/prefix/lib <- obj/libxed.so
global name '__set_perm' is not defined

This code path only occurs when using both --prefix and --shared.
Curiously, this doesn't happen with 'install' target to make a kit.
Must be a different call to _set_perm().

Btw, both the caller and the callee of mbuild.copy_file() write a COPY
message, so all the COPY messages are written twice. You might want
to settle on one or the other.

--Mark

ICLASS:      VEXTRACTPS
CPL:         3
CATEGORY:    AVX512
EXTENSION:   AVX512EVEX
ISA_SET:     AVX512F_128N
EXCEPTIONS:     AVX512-E9NF
REAL_OPCODE: Y
PATTERN:    EVV 0x17 V66 V0F3A MOD[0b11] MOD=3 BCRC=0 REG[rrr] RM[nnn]  VL128    NOEVSR  ZEROING=0 MASK=0 UIMM8()
OPERANDS:    REG0=GPR32_B():w:d:f32 REG1=XMM_R3():r:dq:f32 IMM0:r:b
IFORM:       VEXTRACTPS_GPR32f32_XMMf32_IMM8_AVX512

By document:

In 64-bit mode, destination register operand has default operand size of 64 bits. 
The upper 32-bits of the register are filled with zero. REX.W is ignored.

128-bit Legacy SSE version: 
When a REX.W prefix is used in 64-bit mode with a general purpose register (GPR) 
as a destination operand, the packed single quantity is zero extended to 64 bits.

VEX.128 and EVEX encoded version: 
When VEX.W1 or EVEX.W1 form is used in 64-bit mode with a general purpose register (GPR) 
as a destination operand, the packed single quantity is zero extended to 64 bits.

So the WIG notation is wrong, and GPR32 should be GPRy.

in the script file xed_mbuild.py:1473

    if ['encoder'] and env['decoder']:
         nongen_lib_sources.extend(_get_src(env,'encdec'))

the env[] missing. I suppose it should be:

    if env['encoder'] and env['decoder']:
         nongen_lib_sources.extend(_get_src(env,'encdec'))

Hi there, it looks like some (all?) instructions with E9NF exception-class are missing a VL128 in the pattern. According to the intel SDM (page 120) EVEX.LL != 00b (VL=128) should #UD.

There are several insts, f.e.

{
ICLASS: VPCOMW
CPL: 3
CATEGORY: XOP
ISA_SET: XOP
EXTENSION: XOP
ATTRIBUTES: AMDONLY

PATTERN: XOPV 0xCD VNP W0 VL128  XMAP8 MOD[mm] MOD!=3 REG[rrr] RM[nnn] MODRM() UIMM8()
OPERANDS: REG0=XMM_R():w:dq:i16 REG1=XMM_N():r:dq:i16 MEM0:r:dq:i16 IMM0:r:b:i16

PATTERN: XOPV 0xCD VNP W0 VL128  XMAP8 MOD[0b11] MOD=3 REG[rrr] RM[nnn] UIMM8()
OPERANDS: REG0=XMM_R():w:dq:i16 REG1=XMM_N():r:dq:i16 REG2=XMM_B():r:dq:i16 IMM0:r:b:i16
}

The last IMM0:r: b:i16 is UIMM8 by document. Seems it's a copy paste action.

Hi there,

I think I found one more issue. XED disassembles C4 E2 BD 90 5C DD A2 as vpgatherdq ymm3, qword ptr [rbp+xmm3*8-0x5e], ymm8.

The SDM says:

If any pair of the index, mask, or destination registers are the same, this instruction results a UD fault.

Obviously XMM3 is not the same register as YMM3, but as they are sharing the lower 128-bits, this combination should #UD as well. The SDM is not very clear about this, but fter looking at the pseudo-code and the implications, I'm pretty sure the #UD condition refers to identical register-ids (instead of actually identical registers).

ARM includes a full machine readable description of the semantics of each instruction in their xml files: https://developer.arm.com/products/architecture/a-profile/exploration-tools https://alastairreid.github.io/alastairreid.github.io/ARM-v8a-xml-release/

I'm not sure if it's practical to get to that level of accuracy/completeness but the more information available the better.

It would be nice to be able to build and use XED on other architectures, even if the feature set is reduced (i.e. not being able to query the current CPUID state).

I found quite a collection of crashes with some simple AFL fuzzing of the 'xed' example binary with elf file inputs.

Here is the output from AFL after 30 minutes.

Many of those 'unique crashes' are duplicates but I have seen crashes in the elf parser as well as in the instruction decoder as well.

Below are my reproduction steps on Linux:

# grab afl and build it
wget http://lcamtuf.coredump.cx/afl/releases/afl-latest.tgz
tar -xvf ./afl-latest.tgz
cd afl-2.35b

# I prefer to use the afl-clang-fast compiler
cd llvm_mode
make
cd ../../

# build xed with the afl compiler
git clone https://github.com/intelxed/xed.git xed
git clone https://github.com/intelxed/mbuild.git mbuild
cd xed
./mfile.py --cc=../afl-2.35b/afl-clang-fast --cxx=../afl-2.35b/afl-clang-fast++ --build-dir=./lib/
cd examples
./mfile.py --cc=../../afl-2.35b/afl-clang-fast --cxx=../../afl-2.35b/afl-clang-fast++ --inc-dir=../lib/
cd ../../afl-2.35b
mkdir -p ./output/xed/

# start the fuzzer
./afl-fuzz -i ./testcases/others/elf -o ./output/xed -f /tmp/xed_fuzz.elf -- ../xed/examples/obj/examples/xed -i @@

I included a .so in the testcases/other/elf/ (/lib/LLVMHello.so), but crashes should start imminently after starting the fuzzer with a single elf file.

I am happy to send a tgz of all the crash samples I have collected if issues arise with reproducing the fuzzer itself.

Hi, is there a reason, XED checks for invalid control-registers (e.g. CR1), but not for invalid debug-registers (DR8-DR15 and maybe DR4, DR5)?

Regarding to the SDM these ones will cause #UD:

Use of the REX.R prefix causes an invalid-opcode exception.

and

When the DE flag in CR4 is set, attempts to reference DR4 and DR5 result in an
undefined opcode (#UD) exception.

I guess introducing a new mode for DR4/DR5 would be overkill, but the REX.R thing should be easy to implement.

We found several illegal memory reads bugs as to xed decoder with our fuzzing tool FOT (we plan to open source in future). The bug can be replayed with the following compilation options.

../xed/mfile.py install --cc=clang --cxx=clang++ --extra-flags="-fsanitize=address" --extra-linkflags="-fsanitize=address" --opt=0 --debug --install-dir=$PWD/asan
cd asan/examples/
./mfile.py --cc=clang --cxx=clang++ --extra-flags="-fsanitize=address" --extra-linkflags="-fsanitize=address" --debug --opt=0
cd obj/examples

and run

./xed -i <input_file>

clang version is 64bit 4.0.0-1ubuntu1~16.04.2 (tags/RELEASE_400/rc1).

We attach each kind of crashes separately based on the crash site. The attached files are into 2 categories,

1. *.input.txt - input files
2. *.err.txt -- stderr output with Address sanitizer

The build documentation is right now fairly complicated.

It would be good if there was a simple paragraph in the main README that describes how to build and install.

Hi,

PATTERN   : 0x9C DF64() EOSZ=2 not64
PATTERN   : 0x9D DF64() EOSZ=2 not64

Seems DF64() should be removed

Seems typos:
xed-reg-tables.txt:551

# MOV to CR cannot load CS 
xed_reg_enum_t SEG_MOV()::

should be "MOV to SEG"

xed-state-bits.txt:76

############################################################3333
f2_prefix              REP=2  # REPNZ, REPNE
f3_prefix              REP=3  # REPZ,  REPNZ

3333 typo, because shift key missed
"REPNZ" should be "REPE, REP"

Hey there,
Is there any interest/ongoing work for having XED build with CMake? mbuild.py makes it a bit harder to integrate with other toolchains.

Cheers,
Christian

ICLASS    : XCHG
CPL       : 3
CATEGORY  : DATAXFER
EXTENSION : BASE
ISA_SET   : I86
ATTRIBUTES : LOCKED HLE_ACQ_ABLE HLE_REL_ABLE
PATTERN   : 0x87 MOD[mm] MOD!=3 REG[rrr] RM[nnn] MODRM() lock_prefix
OPERANDS  : MEM0:rw:v REG0=GPRv_R():rw
PATTERN   : 0x87 MOD[mm] MOD!=3 REG[rrr] RM[nnn] MODRM() nolock_prefix
OPERANDS  : MEM0:rw:v REG0=GPRv_R():rw

By convention, XED split it to 2 individual entries, LOCKED=>lock_prefix, LOCKABLE=>nolock_prefix. f.e.

{
ICLASS    : ADD_LOCK
DISASM    : add
CPL       : 3
CATEGORY  : BINARY
EXTENSION : BASE
ISA_SET   : I86
ATTRIBUTES : BYTEOP LOCKED HLE_ACQ_ABLE HLE_REL_ABLE
FLAGS     : MUST [ of-mod sf-mod zf-mod af-mod pf-mod cf-mod ]

PATTERN   : 0x80 MOD[mm] MOD!=3 REG[0b000] RM[nnn] MODRM() SIMM8() lock_prefix
OPERANDS  : MEM0:rw:b IMM0:r:b:i8
IFORM     : ADD_LOCK_MEMb_IMMb_80r0
}
{
ICLASS    : ADD
CPL       : 3
CATEGORY  : BINARY
EXTENSION : BASE
ISA_SET   : I86
ATTRIBUTES : BYTEOP LOCKABLE
FLAGS     : MUST [ of-mod sf-mod zf-mod af-mod pf-mod cf-mod ]

PATTERN   : 0x80 MOD[mm] MOD!=3 REG[0b000] RM[nnn] MODRM() SIMM8() nolock_prefix
OPERANDS  : MEM0:rw:b IMM0:r:b:i8
IFORM     : ADD_MEMb_IMMb_80r0
}

I only focus on the data file and know nothing about the processing code. I guess If possible, the nolock_prefix and lock_prefix flag in pattern can be removed.

The xed-ex1 example fails at sign extending immediates, seemingly always printing 0. I'd guess this might be an issue with xed_sign_extend_arbitrary_to_64.

➜  xed git:(master) ✗ obj/examples/xed-ex1 -64 c7 45 2c 11 00 01 00
Attempting to decode: c7 45 2c 11 00 01 00 
iclass MOV	category DATAXFER	ISA-extension BASE	ISA-set I86
instruction-length 7
operand-width 32
effective-operand-width 32
effective-address-width 64
stack-address-width 64
iform-enum-name MOV_MEMv_IMMz
iform-enum-name-dispatch (zero based) 16
iclass-max-iform-dispatch 22
Operands
#   TYPE               DETAILS        VIS  RW       OC2 BITS BYTES NELEM ELEMSZ   ELEMTYPE   REGCLASS
#   ====               =======        ===  ==       === ==== ===== ===== ======   ========   ========
0   MEM0           (see below)   EXPLICIT   W         V   32     4     1     32        INT    INVALID
1   IMM0              0x0(32b)   EXPLICIT   R         Z   32     4     1     32        INT    INVALID
Memory Operands
  0 written BASE= RBP/GPR DISPLACEMENT_BYTES= 1 0x000000000000002c base10=44 ASZ0=64
  MemopBytes = 4
ATTRIBUTES: HLE_REL_ABLE SCALABLE 
ISA SET: [I86]

Disassembly:

mov dword ptr [rbp+0x2c], 0x10011

I had a quick look at how the formatter does the printing for immediates and patched the example accordingly, seemingly fixing the issue.

diff --git a/examples/xed-ex1.c b/examples/xed-ex1.c
index c959499..52d71cf 100644
--- a/examples/xed-ex1.c
+++ b/examples/xed-ex1.c
@@ -309,14 +309,12 @@ void print_operands(xed_decoded_inst_t* xedd) {
               ibits = xed_decoded_inst_get_immediate_width_bits(xedd);
               if (xed_decoded_inst_get_immediate_is_signed(xedd)) {
                   xed_uint_t rbits = ibits?ibits:8;
-                  xed_int32_t x = xed_decoded_inst_get_signed_immediate(xedd);
-                  xed_uint64_t y = xed_sign_extend_arbitrary_to_64(
-                                              (xed_uint64_t)x, ibits);
-                  xed_itoa_hex_ul(buf, y, rbits, no_leading_zeros, 64, lowercase);
+                  xed_int32_t x = xed_operand_values_get_immediate_int64(xedd);
+                  xed_itoa_hex_ul(buf, x, rbits, no_leading_zeros, 64, lowercase);
               }
               else {
-                  xed_uint64_t x =xed_decoded_inst_get_unsigned_immediate(xedd);
                   xed_uint_t rbits = ibits?ibits:16;
+                  xed_uint64_t x = xed_operand_values_get_immediate_uint64(xedd);
                   xed_itoa_hex_ul(buf, x, rbits, no_leading_zeros, 64, lowercase);
               }
 #if defined (_WIN32) && !defined(PIN_CRT)

The py3 branch in https://github.com/rscohn2/mbuild works with python2 and python3

The py3 branch in https://github.com/rscohn2/xed builds the tests with python2 and python3

The tests pass with python2 and fail with python3. Here is an example:

py3mbuild) rscohn1@ubuntu-xenial:~/workspaces$ xed-ref/build/examples/xed-ex3 -64 vaddps ymm3 k1 ymm1 ymm2
Encode request:
VADDPS MODE:2, REG0:YMM3, REG1:K1, REG2:YMM1, REG3:YMM2, SMODE:2
OPERAND ORDER: REG0 REG1 REG2 REG3 

Encodable! 62F1742958DA
(py3mbuild) rscohn1@ubuntu-xenial:~/workspaces$ xed/build/examples/xed-ex3 -64 vaddps ymm3 k1 ymm1 ymm2
Encode request:
VADDPS MODE_FIRST_PREFIX, REG0:YMM3, REG1:K1, REG2:YMM1, REG3:YMM2, SMODE:2
OPERAND ORDER: REG0 REG1 REG2 REG3 

Encodable! 62F1742958DA
(py3mbuild) 

I tried inspecting the output of the generator, but python2 and python3 emit the functions in different order, so diff does not help. It may be beneficial to sort the functions before emitting so it is not sensitive to python versions. Spot checking shows no differences.

@markcharney: Can you look at this? I think we are close.

python3 always rebuilds the encoder because of a sig mismatch on prep-inputs.
Contents changes because order of input file changes with dict iteration changes.

With instructions that have a 67 address size override prefix, XED sets the base register to RIP rather than EIP even though it correctly detects the EASZ as 32 bits.

➜  xed git:(master) ✗ obj/examples/xed-ex1 -64 67 13 2d ee fb 15 35                                                                                 
Attempting to decode: 67 13 2d ee fb 15 35 
iclass ADC	category BINARY	ISA-extension BASE	ISA-set I86
instruction-length 7
operand-width 32
effective-operand-width 32
effective-address-width 32
stack-address-width 64
iform-enum-name ADC_GPRv_MEMv
iform-enum-name-dispatch (zero based) 10
iclass-max-iform-dispatch 18
Operands
#   TYPE               DETAILS        VIS  RW       OC2 BITS BYTES NELEM ELEMSZ   ELEMTYPE   REGCLASS
#   ====               =======        ===  ==       === ==== ===== ===== ======   ========   ========
0   REG0              REG0=EBP   EXPLICIT  RW         V   32     4     1     32        INT        GPR
1   MEM0           (see below)   EXPLICIT   R         V   32     4     1     32        INT    INVALID
2   REG1           REG1=RFLAGS SUPPRESSED  RW         Y   32     4     1     32        INT      FLAGS
Memory Operands
  0    read BASE= RIP/ IP DISPLACEMENT_BYTES= 4 0x000000003515fbee base10=890633198 ASZ0=32
  MemopBytes = 4
FLAGS:
   reads-rflags of-mod sf-mod zf-mod af-mod pf-mod cf-tst cf-mod 
       read:                            cf  mask=0x1
    written:             of sf zf af pf cf  mask=0x8d5
  undefined:                                mask=0x0
ATTRIBUTES: SCALABLE 
67 PREFIX
Number of legacy prefixes: 1 
ISA SET: [I86]

Intel SDM:

RIP-relative addressing is enabled by 64-bit mode, not by a 64-bit address-size. The use of the address-size prefix does not disable RIP-relative addressing. The effect of the address-size prefix is to truncate and zero-extend the computed effective address to 32 bits.

Hi,

On Debian 9, when building processor-trace (https://github.com/01org/processor-trace), I get an error caused by xed-iform-map.h:

[ 90%] Building C object ptxed/CMakeFiles/ptxed.dir/src/ptxed.c.o                                                                  
cd /home/vext01/research/hardware_tracer/deps/processor-trace/ptxed && /usr/bin/cc  -DFEATURE_ELF -DFEATURE_SIDEBAND -DFEATURE_THRE
ADS -DPT_VERSION_BUILD=0 -DPT_VERSION_EXT=\"\" -DPT_VERSION_MAJOR=1 -DPT_VERSION_MINOR=6 -D_POSIX_C_SOURCE=200809L -I/home/vext01/r
esearch/hardware_tracer/deps/processor-trace/include -I/home/vext01/research/hardware_tracer/deps/processor-trace/libipt/include -I
/home/vext01/research/hardware_tracer/deps/processor-trace/sideband/include -I/home/vext01/research/hardware_tracer/deps/processor-
trace/include/posix -I/home/vext01/research/hardware_tracer/deps/processor-trace/ptxed/include -I/home/vext01/research/hardware_tra
cer/deps/processor-trace/ptxed/../libipt/internal/include  -I/home/vext01/research/hardware_tracer/deps/xed/include/public/xed -I/h
ome/vext01/research/hardware_tracer/deps/xed/obj -Wno-error -g -std=c99 -fvisibility=hidden -Wall -Wextra -pedantic -Werror   -o CM
akeFiles/ptxed.dir/src/ptxed.c.o   -c /home/vext01/research/hardware_tracer/deps/processor-trace/ptxed/src/ptxed.c                 
In file included from /home/vext01/research/hardware_tracer/deps/xed/include/public/xed/xed-inst.h:41:0,                           
                 from /home/vext01/research/hardware_tracer/deps/xed/include/public/xed/xed-decoded-inst.h:28,                     
                 from /home/vext01/research/hardware_tracer/deps/xed/include/public/xed/xed-decode.h:24,                           
                 from /home/vext01/research/hardware_tracer/deps/xed/include/public/xed/xed-interface.h:40,                        
                 from /home/vext01/research/hardware_tracer/deps/processor-trace/ptxed/src/ptxed.c:47:                             
/home/vext01/research/hardware_tracer/deps/xed/include/public/xed/xed-iform-map.h:74:1: error: ‘inline’ is not at beginning of decl
aration [-Werror=old-style-declaration]                                                                                             xed_iclass_enum_t XED_INLINE xed_iform_to_iclass(xed_iform_enum_t iform) {                                                        
 ^~~~~~~~~~~~~~~~~                                                                                                                 cc1: all warnings being treated as errors                                                                                          

I've fixed it locally by removing XED_INLINE but it could also be fixed by moving XED_INLINE to the beginning of the line.

Thanks

I couldn't access this from outside of Intel

http://mjc.intel.com

Permissions?

Bob Davies

How can I find out wether a branch/call is direct or indirect? There seems to be no API nor ICLASS which lets us do it.

Hi!

There are several instructions (like VCVTUSI2SS), for which SDM says: "EVEX.W in non-64 bit is ignored; the instructions behaves as if the W0 version is used", however xed doesn't allow W=1 in 32 bit mode:

./obj/examples/xed -32 -d 62 f1 86 08 7b 00
62F186087B00
ERROR: GENERAL_ERROR Could not decode at offset: 0x0 PC: 0x0: [62F186087B00000000000000000000]

With W=0 everything is fine as expected:

./obj/examples/xed -32 -d 62 f1 06 08 7b 00
62F106087B00
ICLASS: VCVTUSI2SS CATEGORY: CONVERT EXTENSION: AVX512EVEX IFORM: VCVTUSI2SS_XMMf32_XMMf32_MEMu32_AVX512 ISA_SET: AVX512F_SCALAR
SHORT: vcvtusi2ss xmm0, xmm7, dword ptr [eax]

{
ICLASS:      VCMPPD
CPL:         3
CATEGORY:    AVX512
EXTENSION:   AVX512EVEX
ISA_SET:     AVX512F_512
EXCEPTIONS:     AVX512-E2
REAL_OPCODE: Y
ATTRIBUTES:  MASKOP_EVEX MXCSR
PATTERN:    EVV 0xC2 V66 V0F MOD[0b11] MOD=3 BCRC=1 REG[rrr] RM[nnn] FIX_ROUND_LEN512() SAE()  W1    ZEROING=0 UIMM8()
OPERANDS:    REG0=MASK_R():w:mskw:TXT=SAESTR REG1=MASK1():r:mskw REG2=ZMM_N3():r:zf64 REG3=ZMM_B3():r:zf64 IMM0:r:b
IFORM:       VCMPPD_MASKmskw_MASKmskw_ZMMf64_ZMMf64_IMM8_AVX512
}

here:
REG0=MASK_R():w:mskw:TXT=SAESTR

Seems the TXT suffix should be at REG3.

And there are 3 other insts contain mask register with sae flag

In order to build the examples directory I had to run:

./mfile.py --build-dir=./lib/
cd examples
./mfile.py --inc-dir=../lib/

The linker fails to find the libxed.a because it always looked in ../lib/ for me, no matter how many linker flags I tweaked in the mfile.py config.
The compiler would fail looking for xed-build-defines.h in the ../includes/* directory without adding the --inc-dir= flag.

This could be just on my machine but was unable to correctly build the examples without these flag modifications.

The first operand of the PEXT instruction (in hsw-bmi-vex-isa.xed.txt) should have the type REG0=VGPR32_R():w:d instead of REG0=VGPR32_R():rw:d as it is never read (according to the instruction set reference).

Hey there,

sorry for bothering you again. I tried to decode a random byte sequence and XED reported this:

Attempting to decode: 62 22 f9 85 a2 66 78 5e 24 04 cf 7e 23 d1 17
iclass VSCATTERDPD      category KNC    ISA-extension KNCE      ISA-set KNCE
instruction-length 7
operand-width 64
effective-operand-width 64
effective-address-width 64
stack-address-width 64
iform-enum-name VSCATTERDPD_MEMzv_MASK1mskw_ZMMzf64
iform-enum-name-dispatch (zero based) 0
iclass-max-iform-dispatch 1
Operands
#   TYPE               DETAILS        VIS  RW       OC2 BITS BYTES NELEM ELEMSZ   ELEMTYPE   REGCLASS
#   ====               =======        ===  ==       === ==== ===== ===== ======   ========   ========
0   MEM0           (see below)   EXPLICIT  CW        ZV   64     8     1     64     DOUBLE    INVALID
1   REG1               REG1=K5   EXPLICIT  RW      MSKW   16     2    16      1        INT       MASK
2   REG0            REG0=ZMM28   EXPLICIT   R      ZF64  512    64     8     64     DOUBLE        ZMM
Memory Operands
  0 written BASE= RAX/GPR INDEX= ZMM24/ZMM SCALE= 1 DISPLACEMENT_BYTES= 2 0x00000000000003c0 base10=960 ASZ0=64
  MemopBytes = 8
ATTRIBUTES: KNC_F64 KNC_SCATTER MASKOP_EVEX REQUIRES_ALIGNMENT SCATTER SPECIAL_AGEN_REQUIRED
WRITE-MASKING
ISA SET: [KNCE]

I noticed the modrm.rm field is not 4 (= no SIB/VSIB byte should be present), but XED behaves like there is a valid VSIB. Shouldn't this instruction just #UD?

The correct(?) form disassembles as well, but with different length, mem.index, mem.scale and mem.disp:

Attempting to decode: 62 22 f9 85 a2 64 78 5e 24 04 cf 7e 23 d1 17
iclass VSCATTERDPD      category KNC    ISA-extension KNCE      ISA-set KNCE
instruction-length 8
operand-width 64
effective-operand-width 64
effective-address-width 64
stack-address-width 64
iform-enum-name VSCATTERDPD_MEMzv_MASK1mskw_ZMMzf64
iform-enum-name-dispatch (zero based) 0
iclass-max-iform-dispatch 1
Operands
#   TYPE               DETAILS        VIS  RW       OC2 BITS BYTES NELEM ELEMSZ   ELEMTYPE   REGCLASS
#   ====               =======        ===  ==       === ==== ===== ===== ======   ========   ========
0   MEM0           (see below)   EXPLICIT  CW        ZV   64     8     1     64     DOUBLE    INVALID
1   REG1               REG1=K5   EXPLICIT  RW      MSKW   16     2    16      1        INT       MASK
2   REG0            REG0=ZMM28   EXPLICIT   R      ZF64  512    64     8     64     DOUBLE        ZMM
Memory Operands
  0 written BASE= RAX/GPR INDEX= ZMM31/ZMM SCALE= 2 DISPLACEMENT_BYTES= 2 0x00000000000002f0 base10=752 ASZ0=64
  MemopBytes = 8
ATTRIBUTES: KNC_F64 KNC_SCATTER MASKOP_EVEX REQUIRES_ALIGNMENT SCATTER SPECIAL_AGEN_REQUIRED
WRITE-MASKING
ISA SET: [KNCE]

I only tested the EVEX and MVEX form of VSCATTERDPD and VGATHERDPD, but I'm pretty sure this happens for every instruction that uses the VSIB encoding.

Another (minor) thing I was wondering about: Are KNC instructions only valid in 64-bit mode? Because in the manual it says:

Real-Address Mode and Virtual-8086
#UD Instruction not available in these modes
Protected and Compatibility Mode
#UD Instruction not available in these modes

for every single KNC VEX/MVEX instruction.

Thanks in advance!

Hey there,

it seems like one of the latest changes broke the build with --knc switch.

[WARNING] isa_set referenced by chip model hierarchy,but not used by any instructions: FAT_NOP
[WARNING] isa_set referenced by chip model hierarchy,but not used by any instructions: PREFETCHW
[WARNING] bad isa_set referenced cpuid file: XED_ISA_SET_CLWB
[ERROR] Found bad isa_sets in cpuid input files.
  File "./pysrc/generator.py", line 6445, in <module>
    main()
  File "./pysrc/generator.py", line 6429, in main
    gen_cpuid_map(agi)
  File "./pysrc/generator.py", line 5848, in gen_cpuid_map
    make_cpuid_mappings(agi, mappings)
  File "./pysrc/generator.py", line 5814, in make_cpuid_mappings
    die("Found bad isa_sets in cpuid input files.")
  File "./pysrc/genutil.py", line 83, in die
    traceback.print_stack()

Not sure how to fix this.

I would like to point out that an identifier like "_XED_INTERFACE_H_" does eventually not fit to the expected naming convention of the C language standard.
Would you like to adjust your selection for unique names?

Hi!

Currently xed allows to change the address size of bndmov by 0x67 prefix:

./obj/examples/xed -mpx -64 -d 66 0f 1b 08
660F1B08
ICLASS: BNDMOV CATEGORY: MPX EXTENSION: MPX IFORM: BNDMOV_MEMdq_BND ISA_SET: MPX
SHORT: bndmov xmmword ptr [rax], bnd1

./obj/examples/xed -mpx -64 -d 67 66 0f 1b 08
67660F1B08
ICLASS: BNDMOV CATEGORY: MPX EXTENSION: MPX IFORM: BNDMOV_MEMdq_BND ISA_SET: MPX
SHORT: bndmov xmmword ptr [eax], bnd1

However objdump ignores it. I didn't find anything regarding this in SDM, so not 100% sure which behaviour is correct.

This does not decode:
c5 92 12 00 vmovsldup xmm0,XMMWORD PTR [rax]

Test setup uses examples/xed-dec-print.c, but with lines 38-45 changed to:

    // example instructions
    xed_uint_t   bytes = 4;
    xed_uint8_t  itext[XED_MAX_INSTRUCTION_BYTES] = { 0xc5, 0x92, 0x12, 0x00 };

    xed_tables_init();
    
    mmode=XED_MACHINE_MODE_LONG_64;
    stack_addr_width=XED_ADDRESS_WIDTH_64b;

Hi there,

while using some random data, I noticed XED beeing unable to decode certain instructions if one of the following conditions are met (64-bit mode):

EVEX
1. modrm.reg encodes a MASK register and EVEX.R   !=    0 (correct behavior)
2. modrm.reg encodes a MASK register and EVEX.R'  !=    0 (correct behavior)
3. modrm.rm  encodes a MASK register and EVEX.B   !=    0 (should be ignored)

VEX
4. VEX.vvvv  encodes a MASK register and VEX.vvvv != 1xxx
5. modrm.reg encodes a MASK register and VEX.R    !=    0 (might be correct)
6. modrm.rm  encodes a MASK register and VEX.B    !=    0

I did not find VEX-specific information about mask-register encoding, but according to table 2.33 (page 99) and table 2.39 (page 102/103) (Intel® 64 and IA-32 Architectures Software Developer’s Manual, March 2017) case 3. should not #UD for EVEX-instructions (EVEX.B is ignored instead).

Are these tables valid for VEX and MVEX instructions as well or do these ones follow different reg-encoding/#UD rules?

If they are valid, case 6. should be handled in the same way as case 3. and for case 4. only bits [2:0] are used, so I guess ignoring the most-significant bit of .vvvv should be the correct way of handling this one.

In addition EVEX.R' should always be ignored in non 64-bit modes and not cause a decoding-error.

For instance:

{
ICLASS    : JMP
CPL       : 3
CATEGORY  : UNCOND_BR
EXTENSION : BASE
ISA_SET   : I86
ATTRIBUTES: MPX_PREFIX_ABLE
PATTERN   : 0xE9 not64 BRDISPz()
OPERANDS  : RELBR:r:z REG0=XED_REG_EIP:rw:SUPP
PATTERN   : 0xE9 mode64 FORCE64() BRDISP32()
OPERANDS  : RELBR:r:d REG0=XED_REG_RIP:rw:SUPP
}

I didn't test it, but I don't see clear indication in data files that case is handled differently between AMD64 and EMT64.

I believe AMD64 and EMT64 don't handle the near JMP the same way:

• AMD document says 16-bit displacement (prefix 0x66 not ignored), so total length is 4 bytes.
• Intel document says 32-bit displacement (prefix 0x66 ignored), so total length is 6 bytes.

Also, www.sandpile.org says so;

.Df64 | defaults to O64 in PM64; 66h results in O16 in AMD64 but is ignored in EM64T (near branches)

The issue applies for every instruction branching with a word displacement.

Similar to xed_agen, I think it would be useful to have an api that uses the xed_register_callback_fn_t callback to definitely determine when a conditional operand is actually read/written.

I think the operand types for VANDPD, VANDNPD, VORPD, and VXORPD in avx-isa.txt should be f64 instead of u64.

000000B8  8FEA7012CA03000000  lwpval ecx,edx,dword 0x3
000000C1  8FEAF012CA03000000  lwpval rcx,edx,dword 0x3

XDIS b8: XOP       XOP        8FEA7012CA03000000       lwpval ecx, edx, 0x3
ERROR: GENERAL_ERROR Could not decode at offset: 0xc1 PC: 0xc1: [8FEAF012CA03000000]

In XED data file:

PATTERN: XOPV 0x12 VNP W0 VL128  XMAPA MOD[0b11] MOD=3 REG[0b001] RM[nnn] UIMM32()
OPERANDS: REG0=VGPRy_N():w:y REG1=GPRv_B():r:d IMM0:r:d

same as

LWPVAL reg32.vvvv, reg/mem32, imm32 8F RXB.0A 0.src1.0.00 12 /1 /imm32
LWPVAL reg64.vvvv, reg/mem32, imm32 8F RXB.0A 1.src1.0.00 12 /1 /imm32

Seems to be an error action?

The bytes 48 8c 20 decode to a MOV_MEMw_SEG, operating on a 64-bit memory operand. It seems like this should really be named as a MOV_MEMv_SEG and marked with the SCALABLE attribute.

I'm looking at adding XED support to some other open source projects.

However the building is fairly complicated because XED doesn't seem to have an install target for Linux to install it into the standard /usr/local/{include,lib64} locations (unless I'm missing something). This requires specifying the (non standard) xed kit name everywhere in the other projects, which makes it difficult to have a build environment which works on a wide range of systems.

Would it be possible to add a standard install target?

I'm not sure if this is an actual issue, but I played with the JMP (0xE9) instruction and got the following results for the suppressed instruction-pointer operand in 32-bit mode:

1. xed-ex1 e900000000 -> REG0=EIP SUPPRESSED RW V 32
2. xed-ex1 66e900000000 -> REG0=EIP SUPPRESSED RW V 16
3. xed-ex1 67e900000000 -> REG0=IP SUPPRESSED RW V 32
4. xed-ex1 6766e900000000 -> REG0=IP SUPPRESSED RW V 16

Especially the third one looks fishy, because the operand-size is 32-bit, but the register (IP) itself is only 16-bits wide.

The Intel doc contains the following pseudo code, where EIP is always accessed as a full register (32-bit) regardless of the effective operand-size:

IF OperandSize = 32
  THEN
    EIP ← tempEIP;
  ELSE
    IF OperandSize = 16
      THEN (* OperandSize = 16 *)
        EIP ← tempEIP AND 0000FFFFH;
      ELSE (* OperandSize = 64)
        RIP ← tempRIP;
    FI;
FI;

Therefore I think the actual XED-operand-size for IP/EIP/RIP should always be the total width of the used register regardless of the effective-operand-size.

Howdy!

I am looking for the best way to hook XED into the new Studio IDE for analyzing machine code generated by RaptorJIT. I am writing for help to find the right initial approach. I hope you will indulge a little thinking out aloud...

The problem I want to solve is to convert a binary blob of machine code into a higher-level representation. I will use the high-level representation both for presenting to the user, either as textual disassembly or visual dependency graphs (etc), and also for automatic cross-referencing, e.g. with JIT IR code and PEBS data and so on.

I would like to make this machine-code-to-abstract-structure conversion using a command-line tool similar to xed. This would take binary object code for input and generate all the information that I need in easy-to-parse file formats (e.g. XML/JSON/CSV/msgpack/etc.) I will be operating on small amounts of code at any one time: about one thousand instructions or less.

The information that I would like to get for each instruction is:

• Unique ID e.g. starting address of the instruction.
• Textual disassembly (Intel syntax.)
• Register dependencies on other instructions (set of instruction IDs for each operand.)
• Flags dependencies on other instructions.
• Other dependencies e.g. with serializing instructions like CPUID or RDTSCP etc.

Just poking around it seems like I may be able to get all of this from the xed command. It seems like I could do that with three invocations: xed -i x to get a one-liner disassembly of each instruction; xed -xml -i x to get a more structured view of the same thing; and xed -dot -i x to get the dependency information (if that dependency info is complete enough?)

I would probably want to post-process this to put all of the information in one place, e.g. have one XML representation that also includes the textual disassembly and dependency information. This could get messy (e.g. parsing dot files) and so it could make more sense to modify xed to emit the format that I want, or write a new decoder in C, or write a new decoder in some higher-level language like Python that had a xed binding.

So! I'd really appreciate some tips. Are there any off-the-shelf programs that can give me what I want already? Or are there suitable xed bindings to high level languages that could be recommended to write this quickly? Or does it make sense to parse and combine the xed output? Or should I extend xed or write a new decoder in C?

(Thanks for reading!)