Allow users to add a manually created mountpoint to the thumbtack database.

Some volumes throw errors during the mount attempt that are not checked within imagemounter, and are raised within thumbtack. Thumbtack currently doesn't handle these so a corrupt volume (or some other error) can cause the thumbtack process to hang.

The attempt was to run imount --check via subprocess, it was not able to determine where the correct environment was to run that environment's imount command. Now that ralphje/imagemounter#29 has been merged with the imagemounter library, as soon as that gets to PyPI we can use the API call instead and not have to worry about this.

The Pull Request that Greg made to the upstream imagemounter are incorporated in the unreleased 4.x branch as of 10/15/2018. But the latest release (3.1.0) was 14 months ago, so maybe a new one will come soon?

https://imagemounter.readthedocs.io/en/latest/changelog.html

There should be an option to point some/most jobs at mounted directories or even single files. This would give us an option for manually mounting images that didn't mount automatically and running jobs against them.

Files such as /$Extend/$UsnJrnl:$J are inaccessible by the disk image mounter.
The UsnJrnl artifact is the only artifact I have seen impacted by this. /$MFT is supported by disk image mounter, but that seems to be the extent of it.

An alternative would be to make the user manually extract the UsnJrnl and parse it as an additional evidence type, or write in support using TSK to extract arbitrary files with a plugin. This is hard since volume offsets and inode tables change between evidence items.

This is an underlying problem with libewf from what I understand, I dont think we will be able to add this functionality directly, this ticket is to discover a workaround.

Some tools will not run on evidence filenames that contain characters like spaces and apostrophes. We should strip these out beforehand or return escaped strings.

If you're running jobs, they are probably on the same image. Give each image a cool down time, and defer unmounting until then.

Show what is mounted, with mount count (list of job_ids, other meta), unmount buttons. Text box for path to image to mount.

Possibly add a way for it to serve a manually mounted image (give it a path to image and path to mount directory).

• xfs
• exfat (exfat-fuse, exfat-utils)
• mtd
• zfs (baked into Ubuntu > 16.04)
• apple (hfs/+, apfs)
• bsd ufs (baked into Ubuntu
• bsd ufs2 (unknown if imount is smart enough to do -t ufs -o ufstype=ufs2)

Evaluate which of these can be added.

If a mount request is made for an image while a mount request for the same image is already in progress, 2 requests to mount the image will be passed to imagemounter, resulting in duplicate mountpoints being created.

The dependency on the imagemount git repo means that every build environment that transitively depends on thumbtack needs git installed just to install this package. Can we look into publishing this package? I recognize that it's a fork and somebody else owns the name on PyPI, but we have a few options:

• Ask to merge our changes back into original fork.
• Contact PyPI and ask to have the name transferred to us.
• Publish the package under a new name.

Currently we specify the qemu-nbd mount method for .vmdk and .vhdx files.

It may be better to default to using autodetect methods, then attempt to use qemu-nbd if no volumes could be mounted with the default method. This will automatically make a second mount attempt with a new method if the first attempt failed, and eliminates the need to hardcode certain file types as using qemu-nbd.

Recent attempts to mount vmdk images with version "VMware4 disk image" have failed due to an apparent lack of support from libvmdk. A workaround was found to use qemu-nbd to mount the images in a similar manner to the intermediate mount used by ewfmount. To mount a vmdk image, issue the following commands (using Ubuntu 16.04):

sudo modprobe nbd sudo qemu-nbd -c /dev/nbd0 -r /path/to/image.vmdk sudo imount /dev/nbd0

To unmount the nbd device:
sudo qemu-nbd -d /dev/nbd0

One issue with the qemu-nbd technique is that there does not appear to be a good way to determine which image is mounted to which /dev/nbd device, but this can be handled through Thumbtack's internal state tracking.

Currently the paths for specific mountpoints are not clear. The disk mountpoint is displayed, but the more useful information about specific volume mountpoints is accessed via a dropdown button that is not clearly marked. It may be better to just show the mountpoints by default instead of hiding them at first.

Thumbtack is currently unable to mount LVM volumes within an image. We were able to use the following procedures to mount LVM volumes inside an image for analysis. A similar procedure could likely be used to mount the images automatically through Thumbtack.

Mounting LVM with NBD

Connect NBD to hd_image.dd as readonly

qemu-nbd -c /dev/nbd0 -r hd_image.dd

Run fdisk to list the partitions (also check if LVM is present)

sudo fdisk -l /dev/nbd0

Display information about physical volume

sudo pvs

Scan for volume groups

sudo vgscan

Scan for logical volume and see their status

sudo lvscan

Activate newly found volume groups

Note the volume group name
sudo vgchange -ay

List LVM mapping

ls -al /dev/mapper/

Mount as normal; examples below

sudo mount -o loop,ro,noload,noexec,nodev,nosuid,noatime /dev/mapper/vgroup1-root /mnt/linux_mount sudo mount -o loop,ro,noload,noexec,nodev,nosuid,noatime /dev/mapper/vgroup1-home /mnt/linux_mount/home sudo mount -o loop,ro,noload,noexec,nodev,nosuid,noatime /dev/mapper/vgroup1-var /mnt/linux_mount/var

Unmounting Procedure

Unmount LVM partitions

sudo umount /mnt/linux_mount/var /mnt/linux_mount/home /mnt/linux_mount

Deactivate volume group (you should have gotten the volume group name when you ran the activate command previously)

sudo vgchange -an <volume_group_name>

Disconnect hard drive image from NBD

sudo qemu-nbd -d /dev/nbd0

The current regular expression used to detect multipart vhd files ignores some files that should be added to the thumbtack database.

Thumbtack can read the metadata of EWF files, but it can't mount them because the libewf package was updated and no longer has FUSE support. These were the steps to fix this locally:

sudo apt install autoconf automake autopoint libtool pkg-config libfuse-dev zlib1g-dev 
wget https://github.com/libyal/libewf/releases/download/20201230/libewf-experimental-20201230.tar.gz
tar -xavf libewf-experimental-20201230.tar.gz
cd libewf-20201230/
./configure --enable-python3 --with-libfuse
make
Sudo make install

Some volumes are nested within other volumes from imagemounter and are not detected/displayed properly in thumbtack. For example, a volume can be within an LVM volume which is within a LUKS volume. Multiple nested layers make it difficult to detect and display volumes in thumbtack.

When you get info on a disk image from DIM it should include a listing of the VSS within the partitions. When you mount it, you should have the option to mount none (current behavior), all, or select VSS.

Related, the DIM client should include this information (so the worker can see it and add it to the provenance).

This probably just requires passing the correct options to imagemounter (and possibly exposing it as configuration option when submitting mount requests).

Once ralphje/imagemounter#29 is included in a release of imagemounter.

Perhaps ntfs bug? File loaded in FTK image? ewfmount works as expected. mmls can read that there are partitions. Mount.ntfs (sift: mountwin shortcut) fails. Sure enough the NTFS header doesn't match expected (ord of 127 where sectors per cluster should be). dd if=/mnt/ewf/ewf1 skip=$((512*2048)) bs=1 count=200000000 | strings produces nothing when there should be at least some pe headers or other plain text.

Will need a copy of the file that causes this error.

Update description for add mountpoint option to specify that it is for a manually created mountpoint where the image is already mounted.

CamelCase module names are not recommended in Python. They're all pretty small anyway, so we can just put them in resources.py

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

If disk mounting fails for any reason, the worker only sees a 500 error and quits. It would be helpful to include more information in the error page that the Disk Image Mounter API client receives, and propagate that info to the web UI.

This is potentially a huge task. A possible short term milestone is using nginx to display directory listings of the mounted directory.

Separate test/build and deploy into different workflows.

Currently the api allows you to get information of all mounted images. It would be helpful to have an endpoint that allows you to get information for all images, regardless of mount status.

If a bunch of short tasks that auto-unmount are run, then a mount may be processed while an unmount is happening (or vice versa) and things will be in a bad state.

Need to allow users to specify certain subdirectories to skip.

Enable mounting zfs in thumbtack.