glitchassassin / lackey Goto Github PK

when I use type() function to input "@" character, I find it will input nothing. you can try it. thanks

The MoveMouseDelay setting was not being respected by mouse move commands, a holdover from before the Settings object was implemented. This has been fixed in develop and will be pushed to master with the next version.

I noticed that OCR is still on the TODO.

I started using SikuliX for scripting complex export and import procedures for 3D cad programs, and quickly switched to lacky to drop the java runtime. It's wonderful and have not run into any problems yet. Now I'm cleaning up some scripts and moving parts into Python functions and would like to replace some of the images with OCR to make some of the functions more general.

Is there any plan for how to implement this? would opencv be suitable for this?

In my case it would be enough to render a Python string onto a blank image in the correct font, and search for the the rendered text, but that would not be compatible with SikuliX.
I guess selectRegion() is also required to make OCR run at an acceptable speed.

(I can so a fair bit of C and asm programming, and trying to get more experience in Python. The object oriented stuff is a bit new to me.)

Lackey is having issue interacting with Windows 10 installation dialogues (created with NSIS installer). It does see a button, seems like it simulates a click but nothing happens. Log states that click was performed on desired location.

Focusing application does not seem to work either, as if no interaction is possible.

NOTE: Sikuli has same problem on Windows 10.
NOTE: Same code runs on a Windows 7 machine without problem.

Do you maybe know what could be the problem, and if there is a workaround?

isRegionValid logic is wrong and needs to be reworked.

Sikuli 1.1.0+ supports visual events.

While we're at it, let's listen for the standard Alt+Shift+C to kill a Sikuli script.

Attempt to use doubleClick method always results with the following error:

File "build\bdist.win32\egg\lackey\RegionMatching.py", line 516, in doubleClick
File "build\bdist.win32\egg\lackey\PlatformManagerWindows.py", line 261, in pressKey
TypeError: object of type 'int' has no len()

NOTE: Image does exist on the screen (Found match for pattern 'image.png' at (385,656) with confidence (0.894939541817). Target at (407,674))

OS: Windows 7

class Match(Region):
""" Extended Region object with additional data on click target, match score """
def init(self, score, target, rect):
super(Match, self).init(rect[0][0], rect[0][1], rect[1][0], rect[1][1])
self._score = float(score)
if not target or not isinstance(target, Location):
raise TypeError("Match expected target to be a Location object")
self._target = target

def getScore(self):
	""" Returns confidence score of the match """
	return self._score

def getTarget(self):
	""" Returns the location of the match click target (center by default, but may be offset) """
	return self.getCenter().offset(self._target.x, self._target.y)

def __repr__(self):
	return "Match[{},{} {}x{}] score={.2f}, target={}".format(self.x, self.y, self.w, self.h, self._score, **# self.target.getTuple())**

Hi,
I am working on an open source project to automate FTK Imager application. I have started the work and found that Lackey is not allowing the script to press ALT button in the FTK imager application. Here, the FTK is focusing up as a new window.

from lackey import *
appname=App("FTK Imager")
appname.focus()
appWindow=App.focusedWindow
key1 = Key.ALT
p1 = PlatformManagerWindows()
p1.typeKeys(key1)

**Note: I am including only the logic to press ALT key here.

I have also found that the Lackey is not even focusing the MS word or MS Excel

Could you please guide me on this?

Hello, i'm using python 3.5.x, and windows 7 x64.

When i use this piece of code:

import lackey

pattern = lackey.Pattern(r'C:\Users\rainman\Documents\cam\my.png')
screen = lackey.Screen()
screen.click(pattern) # or screen.find(patter) or screen.capture()

Error happens:

hdc = self._gdi32.CreateDCA(ctypes.c_char_p(device_name), 0, 0, 0)
TypeError: bytes or integer address expected instead of str instance

Settings.MinSimilarity parameter set globally seems to be ignored.

Hint - shouldn't the line 40 in RegionMatching.py be:
self.similarity = Settings.MinSimilarity

instead of:
self.similarity = 0.7

This is a bit tricky to even explain - but it seems impossible to interact with a windows-style pop-up while there is active RDP (tried both with built-in RDP and VNC) to that machine.

Start script on machine A -> script works until need to click on a pop-up thrown by application and fails to recognize it. If there is no RDP connection to machine A, everything works as expected.

Do you know what might cause this kind of issue?

I don't know where to post these questions so I'll sum up here.

1. Is it possible to open any file using windows explorer directly? For instance:
   App.open('explorer.exe' + file_path)
   was working in Sikuli, but it does not work in Lackey. Explorer itself opens correctly.
   The only way around is using subprocess directly.

2. I can't seem to make addImagePath(folder_path) work - still file not found message displayed for any image within folder. List of image paths displayed by getImagePath() is correct though.

I have encountered a trouble in finding a mono-colour image.

1. Set the desktop background to a solid colour.
2. Take a screenshot1 of the blank area of the desktop, e.g. 1500x900 pixels, without any icon. The file is a solid-colour image and its size is about 5k.
3. Take another screenshot2 including left side shortcut icons, e.g. 1900x900 pixels. The file size is about 90k.
4. Open a program on the desktop, e.g. notepad 1000x800 pixels, and position it in the 1500x900 area.
5. Run find(screenshot1.png) and find(screenshot2.png).

The expected result is that both 'find' will fail, because there is a significant notepad dialog in the region blocking the background.

The actual outcome was that only the second 'find' failed. The first 'find' was successful and it reported:
Found match for pattern 'C:\temp\testPNG.sikuli\monoBG.png' at (0,0) with confidence (1.0). Target at (750, 450)

Tested using Sikuli, it was able to fail the find(screenshot1).

Issue was found in 0.4.0a1. (It has been working well for me so I haven't upgraded it yet.)

Thanks.

Hi, I use lackey together with pywinauto form GUI automation on windows. when I import lackey and use pywinauto to fill in forms, it gives error

    itunes2.child_window(control_type="Edit",found_index=0).type_keys(self.email)
  File "C:\Python27\lib\site-packages\pywinauto\base_wrapper.py", line 832, in type_keys
    turn_off_numlock)
  File "C:\Python27\lib\site-packages\pywinauto\keyboard.py", line 634, in SendKeys
    k.run()
  File "C:\Python27\lib\site-packages\pywinauto\keyboard.py", line 364, in run
    ctypes.sizeof(win32structures.INPUT))
ctypes.ArgumentError: argument 2: <type 'exceptions.TypeError'>: expected LP_INPUT instance instead of pointer to INPUT_Array_2

if I don't import lackey, it works fine. is it a bug from ctypes or keyboard? how to fix it?
thanks

Typing text into a box types same text without brackets, e.g.

scr.type("some_text_(1)_end")

results in typing

"some_text_1_end"

Same goes for square brackets, where error is thrown for curly braces {}:

File "C:\Python27\lib\site-packages\lackey\RegionMatching.py", line 731, in type
kb.type(text, typeSpeed)
File "C:\Python27\lib\site-packages\lackey\RegionMatching.py", line 1189, in type
return PlatformManager.typeKeys(text, delay)
File "C:\Python27\lib\site-packages\lackey\PlatformManagerWindows.py", line 340, in typeKeys
raise ValueError("Unrecognized special code {{{}}}".format(special_code))
ValueError: Unrecognized special code {123}

Calling import lackey from the interpreter results in an AttributeError:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "lackey\__init__.py", line 24, in <module>
    from .PlatformManagerWindows import PlatformManagerWindows
File "lackey\PlatformManagerWindows.py", line 16, in <module>
    from .Settings import Debug
File "lackey\Settings.py", line 126, in <module>
    class SettingsMaster(object):
File "lackey\Settings.py", line 160, in SettingsMaster
    BundlePath = os.path.dirname(os.path.abspath(os.path.join(os.getcwd(), __main__.__file__)))
AttributeError: 'module' object has no attribute '__file__'

I've had enough of sikilix to rely on, but I do not know how to use this framework?

Image search is running slow on some server platforms. Is there a way to improve the search algorithm's efficiency?

Traceback output on attempt to call exists() method within a region:

Error
Traceback (most recent call last):
...
if target.exists(images.checkbox_black, 0.5):
File "C:\Python27\lib\site-packages\lackey\RegionMatching.py", line 489, in exists
match = matcher.findBestMatch(needle,pattern.similarity)
File "C:\Python27\lib\site-packages\lackey\TemplateMatchers.py", line 92, in findBestMatch
matches_heatmap = ((numpy.ones if method == cv2.TM_SQDIFF_NORMED else numpy.zeros)((lvl_haystack.shape[0] - lvl_needle.shape[0] + 1,lvl_haystack.shape[1] - lvl_needle.shape[1] + 1),dtype=numpy.float32))
ValueError: negative dimensions are not allowed

I try to launch iTunes with lackey.

import lackey
app = lackey.App("C:\\Program Files\\iTunes\\iTunes.exe").open()

nothing happens.
when I test with app = lackey.App("notepad.exe").open()
the notepad is opened. I change to
app = lackey.App("iTunes.exe").open()
also nothing happens

"*" doesn't replace single character

I'm trying the sample code written on the first page and I'm getting the following error.
I'm using Windows 7 and Python 2.7.

Thanks.

Traceback (most recent call last):
File "C:\Python27\testing_lackey.py", line 6, in
click("Start_Button.png")
File "C:\Python27\lib\site-packages\lackey\RegionMatching.py", line 552, in click
target_location = self.find(target).getTarget()
File "C:\Python27\lib\site-packages\lackey\RegionMatching.py", line 374, in find
match = self.exists(pattern)
File "C:\Python27\lib\site-packages\lackey\RegionMatching.py", line 509, in exists
pattern = Pattern(pattern)
File "C:\Python27\lib\site-packages\lackey\RegionMatching.py", line 39, in init
raise OSError("File not found: {}".format(path))
OSError: File not found: Start_Button.png

Mouse wheel (up/down) stopped working in new 0.5.3 version.

Log:

File "C:\Python27\lib\site-packages\lackey\RegionMatching.py", line 824, in wheel
    Mouse().wheel(direction, steps)
  File "C:\Python27\lib\site-packages\lackey\RegionMatching.py", line 1225, in wheel
    return PlatformManager.mouseWheel(direction, steps)
  File "C:\Python27\lib\site-packages\lackey\PlatformManagerWindows.py", line 342, in mouseWheel
    mouse._os_mouse.queue.put(WheelEvent(wheel_moved, time.time()))
AttributeError: 'module' object has no attribute 'queue'

findAll() throws an error trying to print the debug string. Can't use len() on listiterator.

The findAll() method should only return non-overlapping matches.

Hi,

I am trying to automate a tool using Lackey and using below code to paste the input values.

                          paste("test1") 
                          type(Key.TAB)
                          paste("test2") 
                          type(Key.TAB)

Here the paste function is not working instantly. Ideally the "test1" should be pasted and after a TAB, "test2" should be pasted. But instead, the programming is processing all the TABs and at the end pasting only "test2".

Am I doing something wrong? I have tried to include a sleep() after each paste, but that's not working too.

Can you help me on this?

As of v0.4.0a1

It seems that underscore (_) is being typed as minus (-) and question mark (?) is being typed as forward slash (/) in current (0.5.4) version.

There are issues with following keys - Key.PRINTSCREEN & Key.COMMAND -> typing those in text editor results in typing empty curly braces - {}.

The modifiers+brackets doesn't seem to work with current version (0.5.3)? E.g. typing "+(abc)" results in actually seeing "+(abc)" typed, instead of "ABC" (input string in caps).

For some reason the scanning loop in exists() seems to leave significant chunks of memory allocated. This causes problems for repetitive loops.

Look into potentially caching loaded images as well.

From @jazz1215:

Is there any chance that this issue relates to 32bit or 64bit Python?

I have 2 PCs, one with 32bit Python 2.7.12 and the other with 64bit Python 2.7.12 installed. Both have latest 0.5.2 Lackey installed with pip.

On the machine with 32bit, a failed Find will raise FindFailed exception, which is good. However, on the machine with 64bit Python, a "WIndowsError: exception access violation reading 0x0000000C00000000" will be raised. Please see below for details:

32Bit Python:
Python Version:
2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 32 bit (Intel)]

Error:
[info] Couldn't find 'c:\testimage.png' with enough similarity.
Traceback (most recent call last):
  File "C:\guitest.py", line 11, in <module>
    print find(r'c:\testimage.png')
  File "C:\Python27\lib\site-packages\lackey\RegionMatching.py", line 386, in find
    raise FindFailed("Could not find pattern '{}'".format(path))
lackey.Exceptions.FindFailed: Could not find pattern 'c:\testimage.png'

64Bit Python
Python Version:
2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:24:40) [MSC v.1500 64 bit (AMD64)]

Exception:
Traceback (most recent call last):
  File "E:\guitest.py", line 11, in <module>
    print find(r'E:\testimage.png')
  File "E:\3rdParty\Python-2.7.12\lib\site-packages\lackey\RegionMatching.py", line 383, in find
    match = self.exists(pattern)
  File "E:\3rdParty\Python-2.7.12\lib\site-packages\lackey\RegionMatching.py", line 528, in exists
    matcher = TemplateMatcher(r.getBitmap())
  File "E:\3rdParty\Python-2.7.12\lib\site-packages\lackey\RegionMatching.py", line 349, in getBitmap
    return PlatformManager.getBitmapFromRect(self.x, self.y, self.w, self.h)
  File "E:\3rdParty\Python-2.7.12\lib\site-packages\lackey\PlatformManagerWindows.py", line 348, in getBitmapFromRect
    img = self._getVirtualScreenBitmap()
  File "E:\3rdParty\Python-2.7.12\lib\site-packages\lackey\PlatformManagerWindows.py", line 577, in _getVirtualScreenBitmap
    img = self._captureScreen(monitors[monitor_id]["name"])
  File "E:\3rdParty\Python-2.7.12\lib\site-packages\lackey\PlatformManagerWindows.py", line 438, in _captureScreen
    hdc = self._gdi32.CreateDCA(ctypes.c_char_p(device_name), 0, 0, 0)
WindowsError: exception: access violation reading 0x0000000C00000000

Originally Lackey was designed to be a standalone automation library with optional Sikuli integration. As more Sikuli functionality is implemented, though, it makes more and more sense to just implement it as a drop-in Sikuli replacement.

This will eliminate the need for the sikuli.py shim and make it far easier to port existing Sikuli scripts.

Drag & drop fails with following output:

Error
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\lackey\RegionMatching.py", line 675, in dragDrop
drag(dragFrom)
NameError: global name 'drag' is not defined

Sikuli provides a guide class for improved user interaction. Not core functionality.

Documentation here.

While using someApp.focus(), it seems the PID is not being used for looking for the application if _search is not None.

In the below sample code, after notePadx.open() is executed, below values are generated in the application instance:
self._search str: C:\Windows\system32\notepad.exe
self._pid Int:3328

In _focus_instance method, self._pid will not be used to look for the application, even when the PlatformManager.getWindowByTitle() fails to find the window.

I would suggest that return value check to be added to confirm if PlatformManager.focusWindow(PlatformManager.getWindowByTitle(re.escape(self._search))) is successful. If unsuccessful then use PlatformManager.focusWindow(PlatformManager.getWindowByPID(self._pid)).
My temporary work around is to use 'self._pid' as the first preference.

Demo code: to switch between two notepad instances.

def testFocus(self):
    notePadProgram = r'C:\Windows\system32\notepad.exe'
    notePad1 = App(notePadProgram)
    notePad2 = App(notePadProgram)
    notePad1.open()
    notePad1.focus()
    time.sleep(2)
    type('Note Pad 1')
    notePad2.open()
    notePad1.focus()
    time.sleep(2)        
    type('Note Pad 2')
    raw_input('Press Enter to focus on notePad1')
    notePad1.focus()
    raw_input('Press Enter to focus on notePad2')
    notePad2.focus()

type("s", KeyModifier.SHIFT + KeyModifier.CTRL)
KeyError: 'S' in: \lackey\PlatformManagerWindows.pyc in releaseKey(self, text) line 258

simple workaround:
type("S", KeyModifier.CTRL)
works ok. (I was trying to do "SaveAs" without thinking what the shift modifier does.)

In feature-ocr branch.
Note that type("s", KeyModifier.SHIFT + KeyModifier.CTRL) did work before I updated lackey.

I did uncomment the sikuliGUI import because I could not solve that dependency. If you can't reproduce, let me know and I'll have another look at my libraries.

For some reason using tkinter to clear the clipboard is causing other (non-related) programs to hang until the Lackey script terminates. More troubleshooting is needed, but we may need to outsource the clipboard to a different module.

Somewhat easier thanks to the keyboard library.

Not doing OCR now, but I am still using the OCR branch.
The returned locations are not the middle of the found image.
I have been "correcting" today using targetOffset(), but I'm starting to see a pattern: If I mirror the incorrect location over a diagonal line starting at the top-left corner of the image, I end up in the centre of the image.

Could the X and Y offset within the image be swapped somewhere?
(I'm slowly getting more used to Pyhton. and still thinking about a faster OCR approach.)

App.focus() throws an error because _convert_sikuli_wildcards isn't a classmethod

Hi,

When trying to install lackey via pip, I was getting an error that said something along the lines of no module named keyboard. I managed to fix this error by 'pip install keyboard.' However, I ran into another error, soon after; 'no module named PIL'

I tried 'PIP install PIL' but am running into the following error:

Could not find a version that satisfied the requirement PIL...no matching distribution found for PIL.

Any ideas what I'm doing wrong? Thanks.

The Pattern, Location, Mouse, Keyboard, and probably App classes ought to be split out from the RegionMatching.py file.

The Env class needs to be implemented.

OS: Windows 10
Python version: 2.7.12

Whenever seconds parameter in exists() within RegionMatching is different than 0, Attribute error is thrown:

  File "C:\Python27\lib\site-packages\lackey\RegionMatching.py", line 998, in __repr__
    return "Match[{},{} {}x{}] score={.2f}, target={}".format(self.x, self.y, self.w, self.h, self._score, self._target.getTuple())
AttributeError: 'float' object has no attribute '2f'

Using "pip install Lackey" fails to install.
See image below

Using "pip3 install Lackey" works fine.

Lackey does not work with python 3 because Tkinter module is now named tkinter (lower case t).
Some classes from tkinter also get renamed.

OS : Linux
Command : pip install Lackey

Error:
vagrant@vagrant-ubuntu-trusty-64:$ sudo pip install Lackey
Downloading/unpacking Lackey Downloading Lackey-0.5.4-py2-none-any.whl Requirement already satisfied (use --upgrade to upgrade): numpy in /usr/lib/python2.7/dist-packages (from Lackey) Requirement already satisfied (use --upgrade to upgrade): keyboard in /usr/local/lib/python2.7/dist-packages (from Lackey) Requirement already satisfied (use --upgrade to upgrade): requests in /usr/lib/python2.7/dist-packages (from Lackey) Requirement already satisfied (use --upgrade to upgrade): pillow in /usr/lib/python2.7/dist-packages (from Lackey) Downloading/unpacking opencv-python (from Lackey) Could not find any downloads that satisfy the requirement opencv-python (from Lackey) Cleaning up...
No distributions at all found for opencv-python (from Lackey) Storing debug log for failure in /home/vagrant/.pip/pip.log vagrant@vagrant-ubuntu-trusty-64:~$