The upstream sleigh source code builds with C++11 standard:

https://github.com/NationalSecurityAgency/ghidra/blob/b6181be1e44cd2de24f1493cb45148fc873e8487/Ghidra/Features/Decompiler/src/decompile/cpp/Makefile#L47

We should also build the libraries with that standard.

The other tools can be built with higher standards, but we should try to stick as close as possible to upstream's build settings and options.

There seems to be a regression at 387f9b0 which has broken disassembly. I'll eyeball the SLEIGH changes that went into that release and see if I submit a patch to Ghidra.

Before:

tetsuo@Alexs-MacBook-Pro install % ./bin/sleigh-lift disassemble x86-64.sla 4881ecc00f0000
0x00000000: SUB RSP,0xfc0

After:

tetsuo@Alexs-MacBook-Pro install % ./bin/sleigh-lift disassemble x86-64.sla 4881ecc00f0000
0x00000000: DEC AX
0x00000001: DEC AX
0x00000002: DEC AX
0x00000003: DEC AX
0x00000004: DEC AX
0x00000005: DEC AX
0x00000006: DEC AX

In remill builds on latest sleigh (74838a3) the value for the sleighspecinstalldir is static const char *kSleighSpecInstallDir = "share/sleigh/specfiles";. This value causes searching for sla files to fail when the build directory has been deleted, which occurs in the Dockerfile for remill

In #85, I disabled support for building documentation on Windows because I couldn't figure out how to set up Doxygen properly.

If someone is interested in having documentation building supported in Windows, then a PR would be much appreciated.

Hi team! Thanks for your work, it's really nice! Hope the Ghidra team accept your request and maintain themself a standalone sleigh version.

However, for the moment i'm trying to build it using your repo but without success on windows. When i run the command cmake --install build --prefix ./install it result in:

file INSTALL cannot find
"C:/Users/s0urc3/sleigh/build/Release/sla.lib": File exists"

Any idea wassup?

When new files are added to Sleigh, sometimes our weekly sync with Ghidra continues working fine and we don't notice. This leads us to having missing headers which we have to fix later down the track (#107).

We should write a script as part of our weekly sync that identifies new files and either adds them to the PR or fails loudly so we can manually fix it.

As per @pgoodman's comment at #9 (comment), we should look into a way of exposing a helper that finds SLA files in the SLEIGH installation that it belongs to. Ideally it should just look like this:

/*
* Returns a string containing the path to the .sla file for a given architecture.
* @param arch: The architecture for which we want to find a .sla file
* @return: An absolute path to the .sla file
* @throw std::invalid_argument: An unknown arch is provided.
*/
std::string FindSlaFile(const std::string& arch);

In #158 the auto-update bot force-pushed over changes I made.

The bot should not do anything if a branch of the same name already exists.

Funny enough, it looks like my Windows build system had bfd and libiberty headers and library dependencies in C:\Strawberry\c\include and C:\Strawberry\c\lib\ so these pre-requisites can be satisfied by installing Strawberry Perl.

Developers can choco install the packages for git cmake ninja doxygen.install graphviz etc and get everything else needed.

When building, though, we need to set the _WINDOWS and _WIN64 compile-time definitions, so that all the conditional compilation preprocessor statements in the Ghidra headers will work.

We should aim to publish a release of pre-built binaries whenever Ghidra publishes a new stable release and whenever we feel like larger changes have happened in the packaging logic...

I'm not sure if there's a good way to keep some sort of semantic versioning scheme with both the packaging changes and the actual Ghidra changes, then again I'm not sure if upstream even keeps track of breaking changes in the C++ code

CMake Presets are a good way to encapsulate and reproduce CMake configuration, build, and test settings in a single file that can be used for CI and as a starting point for development builds.

CMake Presets also allow users to easily override compiler settings and flags without modification to the CMake file(s), which is useful if you want to enable/disable -Werror.

A good example of a CMake presets file can be found here https://github.com/friendlyanon/cmake-init-shared-static/blob/master/CMakePresets.json.

While presets files require a CMake version greater than 3.19, users and developers are encouraged to use the latest version of CMake for development and/or local builds, even if this project states CMake version 3.15 as the minimum supported.

CMake doesn't exactly have an official convention for variable names or capitalization outside of what the docs say you will get, but there are auto-generated variable names that come from the same capitalization as used in the project(<name> ...) declaration, like _SOURCE_DIR, CMAKE_DISABLE_FIND_PACKAGE_, and others.

While the project is still young, I think we should refactor all cases of variables to use the same capitalization as used in the project(sleigh ...) declaration.

We should probably check the length of the lifted instruction to make sure we arent reading bytes from the 0 fill

ie. ./extra-tools/sleigh-lift/sleigh-lift disassemble ARM7_le.sla 002C -p ./specfiles should not print:
0x00000000: andeq r2,r0,r0, lsl #0x18

and should probably present an error like "not enough bytes"

For users who are not aware of how fills work on the backend/context registers etc the current behavior is non-obvious

The binaries (sleigh-lift) should take a --version argument that outputs relevant version information; ideally we want:

• Git commit hash of the build
• Any ghidra component versions used

The way the Makefiles in upstream are constructed require us to rebuild the same source files for each component with slightly different compile definitions. This isn't a problem per se, but it can be time consuming if another project wants to use CMake's ExternalProject command (rather than add_subdirectory, because ExternalProject gives a lot more control over the build environment for separate settings than the main project, etc.).

Professional CMake A Practical Guide, 10th ed. gives a good summary:

The ExternalProject module’s main purpose is to enable downloading and building external projects that cannot be easily made part of the main project directly. The external project is added as its own separate child build, effectively isolated from the main project and treated more or less as a black box. This means it can be used to build projects for a different architecture, different build settings or even to build a project with a build system other than CMake. It can also be used to handle a project that defines targets or install components that clash with those of the main project.

I think the default, top-level build (cmake -B build -S .) should include all of the same targets we do now, but there should be options that have default values dependent on CMake's PROJECT_IS_TOP_LEVEL variable. All target options are ON when the project is top-level, but it should be possible to turn them off as well. The following are a list of variables that could be used for target groups (discussion is appreciated for use-cases I might not have thought about):

• sleigh_BUILD_SLA_FILES - Toggle for compiling sleigh files (implies the compiler is built)
  • Use-case: Someone doesn't want to use these sla files
• sleigh_BUILD_SLA_COMPONENTS - Toggle for all slaspec components (compiler, library, slafiles); implies sleigh_BUILD_SLA_FILES
  • Use-case: Someone wants to compile their own slaspec files and also use the library to do stuff with them (could turn off sleigh_BUILD_SLA_FILES manually if they don't want/need our sla files
• sleigh_BUILD_DECOMPILER_COMPONENTS - Toggle for building all decompiler components (decompiler, library)
• sleigh_BUILD_EXTRA_TOOLS - Toggle for building and including all other tools, like the lifter

The above variables will also affect how we handle the installation procedure. It would also make sense to use CMake's install COMPONENTS to let users know which components are installed or not for a configured distribution.

Final note, the releases on this repo will be fully-featured. This issue is to mainly help people who want/need to build from source and prevent time-consuming target building for targets they don't need.

There is a new stable release https://github.com/NationalSecurityAgency/ghidra/releases/tag/Ghidra_10.1_build

After cloning Ghidra and checking out the 10.1 release tag, we can run

git diff --compact-summary Ghidra_10.0.4_build -- Ghidra/Features/Decompiler/src/decompile

to see a summary of changes to the sleigh source code---mostly to check whether a source file was added or removed.

System: Endeavour os, kernel 5.18.11-arch1-1
Cmake version: 3.23.2
Commit: 5ee2f2c

Following steps at Build section

git clone https://github.com/lifting-bits/sleigh.git
cd sleigh
# Configure CMake
cmake -B build -S .
# Build SLEIGH
cmake --build build -j

results in compilation failure

Expected behavior:
Tests passed without error messages and failures
Actual behavior:
Tests failing with the following message:

Consolidate compiler generated dependencies of target sla
[ 41%] Built target sla
/usr/include/c++/12.1.0/bits/stl_vector.h:1206: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::front() [with _Tp = cmGccStyleDependency; _Alloc = std::allocator<cmGccStyleDependency>; reference = cmGccStyleDependency&]: Assertion 'this->empty()' failed.
make[2]: *** [CMakeFiles/decomp.dir/build.make:1421: CMakeFiles/decomp.dir/depend] Aborted (core dumped)
make[1]: *** [CMakeFiles/Makefile2:1217: CMakeFiles/decomp.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 41%] Linking CXX executable ghidra_test_dbg
[ 55%] Built target ghidra_test_dbg
make: *** [Makefile:146: all] Error 2

However, I am not expirienced user of cmake, so maybe there is some error from my side

This project should expose and install a function to compile an arbitrary slaspec file.

This would be similar in spirit to how the FindProtobuf.cmake file exposes their codegen capabilities in CMake.

This is an essential feature for downstream users to compile their own custom/patched slaspec files.

Fortunately, there is already some rough function-like capability to compile a given sleigh file here

I think a standalone function should live in a cmake/utils.cmake or something (as long as it's separate from a CMakeLists.txt), which is then installed into the directory alongside the project config file.

I use sleigh-lift disassemble to run the testsuite of my VAX processor implementation.

For longer instructions (more than 15 bytes), sleigh-lift reports

Too many bytes specified to bytes arg, would result in a 32-bit overflow.

because ParseHexBytes() confuses bits with bytes

It would be nice to publish the doxygen-built documentation to somewhere public like GitHub Pages using the gh-pages branch. This should initially happen for at least every stable release.

The ghidra source code is riddled with using namespace std in various header files. This is bad practice and it would be nice if this could be patched away. It causes all kinds of confusing issues (one notable one being with std::byte and byte from some Windows COM headers).

Ideally these occurrences would be patched out, but obviously the cost is high because you'd have to maintain an extensive patch set...

Now that upstream has added namespace ghidra to the source files, it probably makes more sense to place the header files in a layout like include/ghidra/*.hh.

This layout is still unofficial because upstream does not separate the headers.

This would be a breaking change because downstream users would be required to change the header include paths from include <sleigh/header.hh> to include <ghidra/header.hh>.

A patch was generated (#143) to remove the using namespace std from the sleigh headers due to issues with compilation on Windows (#139).

This repo should include a test that will detect (in some way) whether the rebased patch continues to work.

Compiling the project and running the tests is a good indication that the patch continues to work, but if a new header file is added and includes a using namespace std; that we forget to remove, then we could have regressions.

It's a bit unclear to me what the test would look like, but maybe somebody has a better idea.

At the moment, we're only building on Ubuntu in our CI. We should add macOS builds once the BFD and Iberty dependencies have been removed.

At the moment, loading spec files on different threads is unsafe because of how the XML parsing is implemented. We should consider adding a wrapper API in the support lib that protects this behind some kind of locking mechanism.

In #85 support for Windows was added to CI, however there are failing tests that I was unable to figure out. I'm not sure if it's a Windows environment thing when running the tests or if there's a bug with the Windows version of sleigh.

Anyone who wants to take a deeper dive into the failing tests is encouraged to do so and either write up their findings on this issue or create a PR with any fixes!

In the meantime, I am going to gate PRs around requiring the Mac and Linux CI to fully pass before merging, which should help to keep track of the tests we expect to pass while ignoring the Windows tests (although the Windows build step will have to be checked manually 😞 due to GitHub Actions not supporting known failure checks).

It would be nice to support not only the latest stable version of Ghidra but also secondary support for building Ghidra from the latest-ish commit on their master branch. This could be implemented through a CMake cache variable to set a stable or HEAD Ghidra release type (i.e. sleigh_GHIDRA_RELEASE_TYPE).

While this would add a few conditional constructs in the CMake and potentially make it harder to read, I think it would be worth it to provide support for these two Ghidra release types. The conditionals would check the value of sleigh_GHIDRA_RELEASE_TYPE and do the right thing, with HEAD value as the moving target.

I can see this being implemented using CMake's FetchContent module. The FetchContent module provides a nice way to pull the Ghidra repo, check out a commit, apply optional patches, and even use your own local copy of a Ghidra source repo for full control over what you are building. This allows for more flexible development of patches for upstream contribution. You can use your own Ghidra source directory during CMake configuration (with a clean build directory) by setting both

-Dsleigh_GHIDRA_RELEASE_TYPE=HEAD -DFETCHCONTENT_SOURCE_DIR_GHIDRASOURCE=<path to your>/ghidra

I've taken a stab at implementing the source code fetching in a cmake/setup-ghidra-source.cmake file, which can be called in the top-level CMakeLists.txt.

I am happy to hook this all up with what exists right now, if you think this is a good approach

CMakeLists.txt

cmake_minimum_required(VERSION 3.21)

# Sets "library_root" variable for decompiler cpp root directory
# Use "ghidrasource_SOURCE_DIR" variable for Ghidra repo root
include(cmake/setup-ghidra-source.cmake)

project(
    sleigh
    VERSION ${sleigh_GHIDRA_VERSION}
    DESCRIPTION "Unofficial CMake build for Ghidra SLEIGH Decompiler"
    HOMEPAGE_URL "https://github.com/lifting-bits/sleigh"
    LANGUAGES CXX
)
...

cmake/setup-ghidra-source.cmake

# ---- Setup Ghidra Source code ----

# Set up Ghidra repo human-readable version settings
set(sleigh_GHIDRA_RELEASE_TYPE "stable" CACHE
  STRING "Ghidra release type to use. Make sure sleigh_GHIDRA_COMMIT is the correct corresponding commit. 'HEAD' is used for active development purposes."
)
# This is just helper for CMake UIs. CMake does not enforce that the value matches one of those listed.
set_property(CACHE sleigh_GHIDRA_RELEASE_TYPE PROPERTY STRINGS "stable" "HEAD")

# **** Setup pinned git info ****

# Ghidra pinned stable version commit
set(ghidra_version "10.0.4")
set(ghidra_git_tag "5b07797cb859d8801ca3a1e08bda1321ca3ce002")
# pinned stable patches list
set(ghidra_patches "")

# Ghidra pinned commits used for pinning last known working HEAD commit
if("${sleigh_GHIDRA_RELEASE_TYPE}" STREQUAL HEAD)
  # TODO: Try to remember to look at Ghidra/application.properties
  # TODO: CMake only likes numeric characters in the version string....
  set(ghidra_version "10.1")
  set(ghidra_git_tag "55b8fcf7d4aeaca37ffb5c947340915d69c84224")
  set(ghidra_patches
    PATCH_COMMAND git am --ignore-space-change --ignore-whitespace --no-gpg-sign
      "${CMAKE_CURRENT_SOURCE_DIR}/patches/HEAD/0001-Fix-arg-parsing-in-sleigh-C-test-runner.patch"
  )
endif()

# For use in the CMake `project` command to set version
set(sleigh_GHIDRA_VERSION "${ghidra_version}" CACHE
  STRING "Numeric Ghidra version corresponding to sleigh_GHIDRA_RELEASE_TYPE and sleigh_GHIDRA_COMMIT. This is used during packaging"
)

set(sleigh_GHIDRA_COMMIT "${ghidra_git_tag}" CACHE
  STRING "Ghidra repo commit to use/checkout. Ensure this correct with respect to sleigh_GHIDRA_VERSION and sleigh_GHIDRA_RELEASE_TYPE."
)
string(SUBSTRING "${ghidra_git_tag}" 0 7 ghidra_short_commit)

message(STATUS "Using Ghidra version ${sleigh_GHIDRA_VERSION} at commit ${ghidra_short_commit}")

include(FetchContent)
# Write out source directory with identifiable version info
FetchContent_Declare(GhidraSource
    GIT_REPOSITORY https://github.com/NationalSecurityAgency/ghidra
    GIT_TAG "${sleigh_GHIDRA_COMMIT}"
    GIT_PROGRESS TRUE
    SOURCE_DIR "${CMAKE_SOURCE_DIR}/source_${sleigh_GHIDRA_VERSION}_${ghidra_short_commit}"
    ${ghidra_patches}
)
FetchContent_MakeAvailable(GhidraSource)

message(STATUS "Ghidra source located at '${ghidrasource_SOURCE_DIR}'")

# Sanity check on Ghidra source code checkout
set(library_root "${ghidrasource_SOURCE_DIR}/Ghidra/Features/Decompiler/src/decompile/cpp")
if(NOT EXISTS "${library_root}/sleigh.hh")
  message(FATAL_ERROR "The Ghidra source directory has not been initialized correctly. Could not find '${library_root}'")
endif()

Related to changes made in #15

I think some sort of bot or GitHub Action would be the best way to keep track of new commits made to Ghidra's default branch (master). Sort of like Dependabot but we would have to script it ourselves. I see the workflow like:

1. Create a weekly GitHub Action cron job
2. Grab the latest commit from default development Ghidra branch (master)
3. Run some checks for updates and make the necessary source code changes (see below for what should be included)
4. If changes, create new branch, commit and push as new PR to GitHub
5. Let CI run on created PR
6. Wait for manual approval/changes

Checks for "updates":

• New commit to HEAD of Ghidra master branch and update the src/setup-ghidra-source.cmake file/line
• Automatically update the listing of slaspec files with some find . -name "*.slaspec" magic
• Check and report in PR details if any files have been added to some monitored directories (removed files will almost certainly cause compilation errors and be caught by CI)

The cron-job action should ideally be runnable locally as well.

Originally discussed by @ekilmer in #15 (comment)

https://github.com/NationalSecurityAgency/ghidra/releases/tag/Ghidra_10.1.2_build

At the moment, this CMake wrapper only builds the core part of SLEIGH but omits some optional components. However, the installation includes headers for those optional parts.

So if an application using SLEIGH tries to use an optional component, it will compile fine since the header exists in the installation but will hit an error at linking time. To solve this, we should only install the headers that we can use.

The relevant part of the CMake configuration is here:

Add an executable under example/ showing how to get up and running with SLEIGH's API. Ideally we'd have something that takes a binary and has an option to print either Ghidra P-Code or the disassembled instructions.