saolaolsson / pixiple Goto Github PK

Pixiple is a Windows application that searches your files for images that are similar in pixel and metadata content and presents you with a sorted list of similar image pairs to compare

License: MIT License

C++ 99.44% C 0.56%

pixiple's Introduction

Pixiple

What's Pixiple?

Pixiple is a Windows application that searches your files for images that are similar in pixel and metadata content and presents you with a sorted list of similar image pairs to compare.

Unlike similar programs, Pixiple will attempt to find not only duplicate images but images that have something in common, either in their general appearance or their metadata, and may belong together. Pixiple also lets you compare two images (with synchronised zooming and panning) and check for minute quality differences (with the "swap images" button).

Screenshot

Features

Uses both pixel and metadata content (dates, location, camera) to find related images.
Fast, multi-threaded processing.
Minimalist, resizable, localised, DPI-aware UI.
Portable: single file, no settings saved, no installation required.
Free, open source, no restrictions, no nonsense.
Optimised for cows.

Misfeatures

Support for image file formats supported by Windows Imaging Component only (PNG, JPEG, GIF, TIFF, BMP).
No caching of previous results, so all images must be processed every new scan.
No installer.
Cows not included.

Requirements

Windows 7 or later.

For other versions of Windows, Pixiple requires access to Direct2D and Windows Imaging Component which may be available as updates for download from Microsoft. Pixiple will not work on Windows XP, however.

Download

Download the executable from Releases.

What's similar?

Pixiple will easily detect images that are identical, have identical pixel content, are uniformly resized, flipped, rotated (90, 180, 270 degrees), are cropped along a single edge, or have minor differences in pixel content.

Pixiple is less well able to detect similar images with significant changes to pixel content (cropping or change of brightness, contrast, saturation, etc).

File metadata (name, size, date, format) is ignored when detecting similarity. By default, image paths are also ignored.

pixiple's People

Stargazers

Watchers

Forkers

kairen69 nystud alobp sydr sls463 henro2k ayynonwhymouse licmeth prozacman networkelements midi-in gitubhu ahmadrmusa maxox wmjdgla

pixiple's Issues

Tweak scoring?

I'm getting some very unlikely false positives, even after changing the matching to visual similarity only.

Is there a way to adjust the visual similarity matching? If not, would you mind adding this ability?

Thanks!

Manual installation

Hello!

Please, consider sharing zipped binaries alongside installers.

Crash due to improper error handling

I'm getting crashes with the following stack:

 	pixiple_x64.exe!std::_Associated_state<int>::_Set_exception(std::exception_ptr _Exc, bool _At_thread_exit) Line 334	C++
 	pixiple_x64.exe!`std::_Packaged_state<void __cdecl(void)>::_Call_immediate'::`1'::catch$1() Line 596	C++
 	pixiple_x64.exe!_CallSettingFrame_LookupContinuationIndex() Line 98	Unknown
 	pixiple_x64.exe!__FrameHandler4::CxxCallCatchBlock(_EXCEPTION_RECORD * pExcept) Line 1393	C++
 	ntdll.dll!00007ffd332917a6()	Unknown
 	pixiple_x64.exe!std::_Packaged_state<void __cdecl(void)>::_Call_immediate() Line 593	C++
 	pixiple_x64.exe!`std::_Task_async_state<void>::_Task_async_state<void><std::_Fake_no_copy_callable_adapter<void (__cdecl Pane::*)(ID2D1HwndRenderTarget *)const ,Pane const *,_com_ptr_t<_com_IIID<ID2D1HwndRenderTarget,&_GUID_2cd90698_12e2_11dc_9fed_001143a055f9>> &>>'::`2'::<lambda_1>::operator()() Line 674	C++
 	pixiple_x64.exe!std::invoke<`std::_Task_async_state<void>::_Task_async_state<void><std::_Fake_no_copy_callable_adapter<void (__cdecl Pane::*)(ID2D1HwndRenderTarget *)const ,Pane const *,_com_ptr_t<_com_IIID<ID2D1HwndRenderTarget,&_GUID_2cd90698_12e2_11dc_9fed_001143a055f9>> &>>'::`2'::<lambda_1> &>(std::_Task_async_state<void>::{ctor}::__l2::<lambda_1> & _Obj) Line 1525	C++
 	pixiple_x64.exe!std::_Invoker_ret<void,1>::_Call<`std::_Task_async_state<void>::_Task_async_state<void><std::_Fake_no_copy_callable_adapter<void (__cdecl Pane::*)(ID2D1HwndRenderTarget *)const ,Pane const *,_com_ptr_t<_com_IIID<ID2D1HwndRenderTarget,&_GUID_2cd90698_12e2_11dc_9fed_001143a055f9>> &>>'::`2'::<lambda_1> &>(std::_Task_async_state<void>::{ctor}::__l2::<lambda_1> & _Func) Line 652	C++
 	pixiple_x64.exe!std::_Func_impl_no_alloc<`std::_Task_async_state<void>::_Task_async_state<void><std::_Fake_no_copy_callable_adapter<void (__cdecl Pane::*)(ID2D1HwndRenderTarget *)const ,Pane const *,_com_ptr_t<_com_IIID<ID2D1HwndRenderTarget,&_GUID_2cd90698_12e2_11dc_9fed_001143a055f9>> &>>'::`2'::<lambda_1>,void>::_Do_call() Line 823	C++
 	pixiple_x64.exe!std::_Func_class<void>::operator()() Line 870	C++
 	pixiple_x64.exe!`Concurrency::details::_MakeVoidToUnitFunc'::`2'::<lambda_1>::operator()() Line 2348	C++
 	pixiple_x64.exe!std::invoke<`Concurrency::details::_MakeVoidToUnitFunc'::`2'::<lambda_1> &>(Concurrency::details::_MakeVoidToUnitFunc::__l2::<lambda_1> & _Obj) Line 1525	C++
 	pixiple_x64.exe!std::_Invoker_ret<unsigned char,0>::_Call<`Concurrency::details::_MakeVoidToUnitFunc'::`2'::<lambda_1> &>(Concurrency::details::_MakeVoidToUnitFunc::__l2::<lambda_1> & _Func) Line 661	C++
 	pixiple_x64.exe!std::_Func_impl_no_alloc<`Concurrency::details::_MakeVoidToUnitFunc'::`2'::<lambda_1>,unsigned char>::_Do_call() Line 823	C++
 	pixiple_x64.exe!std::_Func_class<unsigned char>::operator()() Line 870	C++
 	pixiple_x64.exe!Concurrency::task<unsigned char>::_InitialTaskHandle<void,`std::_Task_async_state<void>::_Task_async_state<void><std::_Fake_no_copy_callable_adapter<void (__cdecl Pane::*)(ID2D1HwndRenderTarget *)const ,Pane const *,_com_ptr_t<_com_IIID<ID2D1HwndRenderTarget,&_GUID_2cd90698_12e2_11dc_9fed_001143a055f9>> &>>'::`2'::<lambda_1>,Concurrency::details::_TypeSelectorNoAsync>::_LogWorkItemAndInvokeUserLambda<std::function<unsigned char __cdecl(void)>>(std::function<unsigned char __cdecl(void)> _func) Line 3513	C++
 	pixiple_x64.exe!Concurrency::task<unsigned char>::_InitialTaskHandle<void,`std::_Task_async_state<void>::_Task_async_state<void><std::_Fake_no_copy_callable_adapter<void (__cdecl Pane::*)(ID2D1HwndRenderTarget *)const ,Pane const *,_com_ptr_t<_com_IIID<ID2D1HwndRenderTarget,&_GUID_2cd90698_12e2_11dc_9fed_001143a055f9>> &>>'::`2'::<lambda_1>,Concurrency::details::_TypeSelectorNoAsync>::_Init(Concurrency::details::_TypeSelectorNoAsync __formal) Line 3533	C++
 	pixiple_x64.exe!Concurrency::task<unsigned char>::_InitialTaskHandle<void,`std::_Task_async_state<void>::_Task_async_state<void><std::_Fake_no_copy_callable_adapter<void (__cdecl Pane::*)(ID2D1HwndRenderTarget *)const ,Pane const *,_com_ptr_t<_com_IIID<ID2D1HwndRenderTarget,&_GUID_2cd90698_12e2_11dc_9fed_001143a055f9>> &>>'::`2'::<lambda_1>,Concurrency::details::_TypeSelectorNoAsync>::_Perform() Line 3519	C++
 	pixiple_x64.exe!Concurrency::details::_PPLTaskHandle<unsigned char,Concurrency::task<unsigned char>::_InitialTaskHandle<void,`std::_Task_async_state<void>::_Task_async_state<void><std::_Fake_no_copy_callable_adapter<void (__cdecl Pane::*)(ID2D1HwndRenderTarget *)const ,Pane const *,_com_ptr_t<_com_IIID<ID2D1HwndRenderTarget,&_GUID_2cd90698_12e2_11dc_9fed_001143a055f9>> &>>'::`2'::<lambda_1>,Concurrency::details::_TypeSelectorNoAsync>,Concurrency::details::_TaskProcHandle>::invoke() Line 1469	C++
 	pixiple_x64.exe!Concurrency::details::_TaskProcHandle::_RunChoreBridge(void * _Parameter) Line 158	C++
 	pixiple_x64.exe!Concurrency::details::_DefaultPPLTaskScheduler::_PPLTaskChore::_Callback(void * _Args) Line 51	C++
>	pixiple_x64.exe!Concurrency::details::`anonymous namespace'::_Task_scheduler_callback(_TP_CALLBACK_INSTANCE * _Pci, void * _Args, _TP_WORK * __formal) Line 133	C++
 	ntdll.dll!00007ffd33252260()	Unknown
 	ntdll.dll!00007ffd332431aa()	Unknown
 	kernel32.dll!00007ffd32467614()	Unknown
 	ntdll.dll!00007ffd332426a1()	Unknown
 	[Async Call]	
 	[External Code]	
 	pixiple_x64.exe!Window::paint() Line 612	C++
 	pixiple_x64.exe!Window::window_procedure(HWND__ * hwnd, unsigned int msg, unsigned __int64 wparam, __int64 lparam) Line 465	C++
 	pixiple_x64.exe!Window::static_window_procedure(HWND__ * hwnd, unsigned int msg, unsigned __int64 wparam, __int64 lparam) Line 399	C++
 	[External Code]

As you can see, we're crashing in std lib. It's unclear what exactly is causing the crash because Window::paint() is using std::future to execute Pane::draw() asynchronously. std::future captures and re-throws any exception in the asynchronous routine, so we won't be able to catch original location where the exception occurred.

If we change Visual Studio's exception settings to break when all exceptions are thrown, the stack now becomes:

 	KernelBase.dll!00007ffd30bbcd29()	Unknown
 	[External Code]	
>	pixiple_x64.exe!ErrorReflector::throw_result(const long line, const char * const file, const HRESULT hresult) Line 102	C++
 	pixiple_x64.exe!ErrorReflector::operator=(const HRESULT & hr) Line 69	C++
 	pixiple_x64.exe!Image::get_bitmap(ID2D1HwndRenderTarget * const render_target) Line 838	C++
 	pixiple_x64.exe!Image::draw(ID2D1HwndRenderTarget * const render_target, const D2D_RECT_F & rect_dest, const D2D_RECT_F & rect_src, const D2D1_BITMAP_INTERPOLATION_MODE & interpolation_mode) Line 113	C++
 	pixiple_x64.exe!Pane::draw(ID2D1HwndRenderTarget * const render_target) Line 426	C++
 	[External Code]	
 	[Async Call]	
 	[External Code]	
 	pixiple_x64.exe!Window::paint() Line 612	C++
 	pixiple_x64.exe!Window::window_procedure(HWND__ * hwnd, unsigned int msg, unsigned __int64 wparam, __int64 lparam) Line 465	C++
 	pixiple_x64.exe!Window::static_window_procedure(HWND__ * hwnd, unsigned int msg, unsigned __int64 wparam, __int64 lparam) Line 399	C++
 	[External Code]

Now it's clear what's causing the problem. Image::get_bitmap() uses ErrorReflector to handle all Windows API errors. ErrorReflector throws exceptions when it sees an error - this is what's causing the crash.

std::future uses exceptions for synchronization and Window::paint() is not expecting Pane::draw() to throw any exceptions, so Pane::draw() (including all functions called by it) should not be using ErrorReflector for error handling.

suggestion: "auto batch processing"

Simple idea, little bit advanced to implement:

An extra button to automatically process the batch.
It should have a threshold slider and the following buttons:
Preview, Continue, Cancel (not necessarily in that order)

Preview would preview 5 different (worst/random?) pairs with the set threshold, so the user can find the optimal level before hitting the Continue button and go grab themselves a coffee. Once they get back, the batch process may have completed or be near completion 👍

CreateBitmapFromWicBitmap fails with E_INVALIDARG for extremely long images

pixiple/src/image.cpp

Lines 837 to 838 in 62bcacc

    
           er = render_target->CreateBitmapFromWicBitmap( 
        
           	format_converter, D2D1::BitmapProperties(d2d_pf, dpi.x, dpi.y), &bce.bitmap);

This call is failing and causing the crash in issue #13.

It occurs when processing the following image:
https://i.pximg.net/img-original/img/2012/06/16/00/52/28/27979797_p1.jpg

I'm not too familiar with D2D, so I'm not sure what's causing it to fail. It works fine for thousands of other images, and the only thing that could be contributing to the issue is that the image is extremely long.

Identical images shows different pixel hash due to Image::calculate_hash() not accounting for pixel format differences

Image::calculate_hash() grabs the bitmap data directly without normalizing it to account for alpha channel:

pixiple/src/image.cpp

Lines 724 to 749 in 62bcacc

    
           void Image::calculate_hash() const { 
        
           	std::vector<std::uint8_t> buffer(numeric_cast<std::size_t>(file_size())); 
        
           	auto frame = get_frame(buffer); 
        
           	if (frame == nullptr) 
        
           		return; 
        
           	file_hash = Hash(buffer.data(), buffer.size()); 
        
           	ComPtr<IWICImagingFactory> wic_factory; 
        
           	er = CoCreateInstance( 
        
           		CLSID_WICImagingFactory, 
        
           		nullptr, 
        
           		CLSCTX_INPROC_SERVER, 
        
           		IID_PPV_ARGS(&wic_factory)); 
        
           	ComPtr<IWICBitmap> bitmap; 
        
           	er = wic_factory->CreateBitmapFromSource(frame, WICBitmapNoCache, &bitmap); 
        
           	ComPtr<IWICBitmapLock> bitmap_lock; 
        
           	auto hr = bitmap->Lock(nullptr, WICBitmapLockRead, &bitmap_lock); 
        
           	if (SUCCEEDED(hr)) { 
        
           		UINT size; 
        
           		std::uint8_t* pixel_data; 
        
           		er = bitmap_lock->GetDataPointer(&size, static_cast<BYTE**>(&pixel_data)); 
        
           		pixel_hash = Hash(pixel_data, size); 
        
           	} 
        
           }

Thus when given 2 identical images, one without alpha channel and one with a redundant alpha channel (i.e. all alpha values are 0xFF), the computed pixel hash would be different. The image without the alpha channel should be normalized to a bitmap with alpha channel before computing the pixel hash (it may also be a good idea to indicate the presence of alpha channel in the UI).

As a hack fix, I replaced lines 739-748 with the following chunk from Image::load_pixels() that does conversion to RGBA:

pixiple/src/image.cpp

Lines 432 to 452 in 62bcacc

    
           ComPtr<IWICFormatConverter> format_converter; 
        
           er = wic_factory->CreateFormatConverter(&format_converter); 
        
           er = format_converter->Initialize( 
        
           	frame, 
        
           	GUID_WICPixelFormat32bppPBGRA, 
        
           	WICBitmapDitherTypeNone, 
        
           	nullptr, 
        
           	0, 
        
           	WICBitmapPaletteTypeCustom); 
        
           const auto pixel_stride = 4; 
        
           const auto line_stride = image_size.w * pixel_stride; 
        
           const std::size_t pixel_buffer_size = line_stride * image_size.h; 
        
           assert(pixel_buffer_size > 0); 
        
           std::vector<uint8_t> pixel_buffer(pixel_buffer_size); 
        
           auto hr = format_converter->CopyPixels( 
        
           	nullptr, 
        
           	line_stride, 
        
           	numeric_cast<UINT>(pixel_buffer_size), 
        
           	pixel_buffer.data());

We can then simply do:

pixel_hash = Hash(pixel_buffer.data(), pixel_buffer_size);

This normalizes all images regardless of whether they have an alpha channel. I'm not sure whether this will adversely impact performance. If you want to normalize only those without alpha channel, add the following code (thanks to this SO question):

WICPixelFormatGUID pixelFormat{};
er = frame->GetPixelFormat(&pixelFormat);
ComPtr<IWICComponentInfo> imageInfo;
er = wic_factory->CreateComponentInfo(pixelFormat, &imageInfo);
ComPtr<IWICPixelFormatInfo> imageFormatInfo;
er = imageInfo->QueryInterface(__uuidof(IWICPixelFormatInfo), reinterpret_cast<void**>(&imageFormatInfo));
UINT bitsPerPixel{};
er = imageFormatInfo->GetBitsPerPixel(&bitsPerPixel);

if (bitsPerPixel == 24)
{
	// normalize
}

Suggestion：Add display information...

Suggestion：Add display information: Quantity deleted、Not similar quantity、Pending quantity

Suggestion - click (or double click) on image to open in default viewing application

This would be very useful and more efficient than just opening the folder and having to browse.

Thank you for a useful application. I hope you still have the time to support and develop it.

Suggestion：Add "Not Similar" or "Remove Pair" button...

Suggestion：Add "Not Similar" or "Remove Pair" button, After clicking, Remove pair from queue, And automatically go to the "Next pair"

Suggestion: After clicking "Delete file"...

Suggestion: After clicking "Delete file", Remove pair from queue, And automatically go to the "Next pair"

SignatureDoesNotMatch error when downloading windows binaries.

I'm getting a SignatureDoesNotMatch AWS error xml when I try to download any of the windows binaries.

Suggestion：Multiple folder comparison support...

Suggestion：Multiple folder comparison support, Currently only one folder can be added; If support drag-and-dropfor adding folders, That would be great

Example.
D:\XXX\A
E:\B
F:\XXX\XX\C

suggestion

I think that introduction of a done tag should help out alot.
In detail, there are photos that are marked as possible identical due to huge colour uniformity (?), such are photos of the sea or big green fields. But the details in those are different like people, weather, etc.
creating a log entry that will be set for the user as false positive, would eliminate those photos in a new scan.

Hope it makes sense and is easy to be implemented

By the way the excellent software.

Suggestion: Add keyboard hotkeys

For example:
Arrow keys: next and previous image pair
CTRL+O: Open in default viewing application
CTRL+SHIFT+O: Open containing folder
A: delete left image
D: delete right image
Space: Swap images

	er = render_target->CreateBitmapFromWicBitmap(
	format_converter, D2D1::BitmapProperties(d2d_pf, dpi.x, dpi.y), &bce.bitmap);

	void Image::calculate_hash() const {
	std::vector<std::uint8_t> buffer(numeric_cast<std::size_t>(file_size()));
	auto frame = get_frame(buffer);
	if (frame == nullptr)
	return;

	file_hash = Hash(buffer.data(), buffer.size());

	ComPtr<IWICImagingFactory> wic_factory;
	er = CoCreateInstance(
	CLSID_WICImagingFactory,
	nullptr,
	CLSCTX_INPROC_SERVER,
	IID_PPV_ARGS(&wic_factory));

	ComPtr<IWICBitmap> bitmap;
	er = wic_factory->CreateBitmapFromSource(frame, WICBitmapNoCache, &bitmap);
	ComPtr<IWICBitmapLock> bitmap_lock;
	auto hr = bitmap->Lock(nullptr, WICBitmapLockRead, &bitmap_lock);
	if (SUCCEEDED(hr)) {
	UINT size;
	std::uint8_t* pixel_data;
	er = bitmap_lock->GetDataPointer(&size, static_cast<BYTE**>(&pixel_data));
	pixel_hash = Hash(pixel_data, size);
	}
	}

	ComPtr<IWICFormatConverter> format_converter;
	er = wic_factory->CreateFormatConverter(&format_converter);
	er = format_converter->Initialize(
	frame,
	GUID_WICPixelFormat32bppPBGRA,
	WICBitmapDitherTypeNone,
	nullptr,
	0,
	WICBitmapPaletteTypeCustom);

	const auto pixel_stride = 4;
	const auto line_stride = image_size.w * pixel_stride;
	const std::size_t pixel_buffer_size = line_stride * image_size.h;
	assert(pixel_buffer_size > 0);
	std::vector<uint8_t> pixel_buffer(pixel_buffer_size);

	auto hr = format_converter->CopyPixels(
	nullptr,
	line_stride,
	numeric_cast<UINT>(pixel_buffer_size),
	pixel_buffer.data());

saolaolsson / pixiple Goto Github PK

pixiple's Introduction

Pixiple

What's Pixiple?

Screenshot

Features

Misfeatures

Requirements

Download

What's similar?

pixiple's People

Stargazers

Watchers

Forkers

pixiple's Issues

Recommend Projects

Recommend Topics

Recommend Org