C++ dll source code: UIAutomationClient

How to use uiautomation? Run 'automation.py -h'

Understand the arguments of automation.py, and try the following examples:
automation.py -t 0 -n, print current active window's controls, show full name
automation.py -r -d 1 -t 0, print desktop (the root of control tree) and its children (top level windows)

automation.py prints the properties of controls and the patterns they support. You use controls and patterns to retrieve information about controls and interact with them.

A control may support some patterns or conditionally support some patterns according to its control type.

Refer to Control Pattern Mapping for UI Automation Clients for the complete control pattern table.

uiautomation searches controls from the control tree based on the controls' properties you supply.

Suppose the control tree is

root(Name='Desktop', Depth=0)
　　window1(Depth=1)
　　　　control1-001(Depth=2)
　　　　control1-...(Depth=2)
　　　　...
　　　　control1-100(Depth=2)
　　window2(Name='window2', Depth=1)
　　　　control2-1(Depth=2)
　　　　　　control2-1-001(Depth=3)
　　　　　　control2-1-...(Depth=3)
　　　　　　...
　　　　　　control2-1-100(Depth=3)
　　　　control2-2(Depth=2)
　　　　control2-3(Depth=2)
　　　　control2-4(Name='2-4', Depth=2)
　　　　　　editcontrol(Name='myedit1', Depth=3)
　　　　　　editcontrol(Name='myedit2', Depth=3)

If you want to find the EditControl whose name is 'myedit2' and type 'hi',
you can write the following code:

uiautomation.EditControl(searchDepth=3, Name='myedit2').SendKeys('hi')

But this code runs slowly because there are more than 200 controls before myedit2 in the control tree.
uiautomation has to traverse more than 200 controls before finding myedit2 if searching from the root with a search depth of 3.
A better approach is:

window2 = uiautomation.WindowControl(searchDepth=1, Name='window2') # search 2 times
sub = window2.Control(searchDepth=1, Name='2-4')    # search 4 times
edit = sub.EditControl(searchDepth=1, Name='myedit2')   # search 2 times
edit.SendKeys('hi')

This code runs faster than the former approach.
You can also combine the four lines of code into one line.

uiautomation.WindowControl(searchDepth=1, Name='window2').Control(searchDepth=1, Name='2-4').EditControl(searchDepth=1, Name='myedit2').SendKeys('hi')

Now let's take notepad.exe as an example.
Launch notepad.exe and run automation.py -t 3, then switch to Notepad and wait for 5 seconds

automation.py will print the controls of Notepad and save them to @AutomationLog.txt:

ControlType: PaneControl ClassName: #32769 Name: 桌面 Depth: 0 (Desktop window, the root control)
　　ControlType: WindowControl ClassName: Notepad Depth: 1 (Top level window)
　　　　ControlType: EditControl ClassName: Edit Depth: 2
　　　　　　ControlType: ScrollBarControl ClassName: Depth: 3
　　　　　　　　ControlType: ButtonControl ClassName: Depth: 4
　　　　　　　　ControlType: ButtonControl ClassName: Depth: 4
　　　　　　ControlType: ThumbControl ClassName: Depth: 3
　　　　ControlType: TitleBarControl ClassName: Depth: 2
　　　　　　ControlType: MenuBarControl ClassName: Depth: 3
　　　　　　　　ControlType: MenuItemControl ClassName: Depth: 4
　　　　　　ControlType: ButtonControl ClassName: Name: 最小化 Depth: 3 (Minimize Button)
　　　　　　ControlType: ButtonControl ClassName: Name: 最大化 Depth: 3 (Maximize Button)
　　　　　　ControlType: ButtonControl ClassName: Name: 关闭 Depth: 3 (Close Button)
...

Run the following code

# -*- coding: utf-8 -*-
# this script only works with Win32 notepad.exe
# if your notepad.exe is the Windows Store version in Windows 11, you need to uninstall it.
import subprocess
import uiautomation as auto

def test():
    print(auto.GetRootControl())
    subprocess.Popen('notepad.exe', shell=True)
    # you should find the top level window first, then find children from the top level window
    notepadWindow = auto.WindowControl(searchDepth=1, ClassName='Notepad')
    if not notepadWindow.Exists(3, 1):
        print('Can not find Notepad window')
        exit(0)
    print(notepadWindow)
    notepadWindow.SetTopmost(True)
    # find the first EditControl in notepadWindow
    edit = notepadWindow.EditControl()
    # usually you don't need to catch exceptions
    # but if you encounter a COMError exception, put it in a try block
    try:
        # use value pattern to get or set value
        edit.GetValuePattern().SetValue('Hello')  # or edit.GetPattern(auto.PatternId.ValuePattern)
    except auto.comtypes.COMError as ex:
        # maybe you aren't running Python as administrator
        # or the control doesn't have an implementation for the pattern method (there is no workaround for this)
        pass
    edit.Click() # this step is optional, but some edits need it
    edit.SendKeys('{Ctrl}{End}{Enter}World')
    print('current text:', edit.GetValuePattern().Value)
    notepadWindow.CaptureToImage('notepad.png')
    notepadWindow.MenuBarControl(searchDepth=1).CaptureToImage('notepad_menubar.png')

    # generate an animated gif
    bitmap = notepadWindow.ToBitmap(x=0, y=0, width=160, height=160)
    side = int(bitmap.Width * 1.42)
    gifBmp = auto.Bitmap(side, side)
    gifBmp.Clear(0xFFFF_FFFF) # set bitmap background color to white
    gifBmp.Paste(x=(side-bitmap.Width)//2, y=(side-bitmap.Height)//2, bitmap=bitmap)
    gifFrameCount = 20
    bmps = [gifBmp.RotateWithSameSize(gifBmp.Width//2, gifBmp.Height//2, i*360/gifFrameCount) for i in range(0, gifFrameCount)]
    auto.GIF.ToGifFile('notepad_part.gif', bitmaps=bmps, delays=[100]*gifFrameCount)

    # find the first TitleBarControl in notepadWindow,
    # then find the second ButtonControl in TitleBarControl, which is the Maximize button
    maximizeButton = notepadWindow.TitleBarControl().ButtonControl(foundIndex=2)
    maximizeButton.Click(waitTime=2)
    maximizeButton.Click()
    # find the first button in notepadWindow whose Name is '关闭' or 'Close', the close button
    # the relative depth from Close button to Notepad window is 2
    notepadWindow.ButtonControl(searchDepth=2, Compare=lambda c, d: c.Name in ['Close', '关闭']).Click()
    # then notepad will pop up a window asking whether to save, press Alt+N to discard
    notepadWindow.WindowControl(searchDepth=1).CaptureToImage('notepad_save.png')
    auto.SendKeys('{Alt}n')

if __name__ == '__main__':
    test()

The above code automates notepad.exe and generates a GIF file.

auto.GetRootControl() returns the root control (the Desktop window)
auto.WindowControl(searchDepth=1, ClassName='Notepad') creates a WindowControl, the parameters specify how to search the control
The following parameters can be used:
searchFromControl = None,
searchDepth = 0xFFFFFFFF,
searchInterval = SEARCH_INTERVAL,
foundIndex = 1
Name
SubName
RegexName
ClassName
AutomationId
ControlType
Depth
Compare

See Control.__init__ for the comments on the parameters.
See scripts in folder demos for more examples.

Control.Element returns the low level COM object IUIAutomationElement, Almost all methods and properties of Control are implemented via IUIAutomationElement COM API and Win32 API. when calling a control's method or property that indirectly calls Control.Element and Control.Element is None, uiautomation starts searching the control by the properties you supply. uiautomation will raise a LookupError exception if it can't find the control within uiautomation.TIME_OUT_SECOND (default 10 seconds). Control.Element will have a valid value if uiautomation finds the control successfully. You can use Control.Exists(maxSearchSeconds, searchIntervalSeconds) to check whether a control exists, this function doesn't raise any exceptions. Call Control.Refind or Control.Exists to make Control.Element invalid again and uiautomation will start a new search.

For example:

#!python3
# -*- coding:utf-8 -*-
# this script only works with Win32 notepad.exe
# if your notepad.exe is the Windows Store version in Windows 11, you need to uninstall it.
import subprocess
import uiautomation as auto
auto.uiautomation.SetGlobalSearchTimeout(15)  # set new timeout 15


def main():
    subprocess.Popen('notepad.exe', shell=True)
    window = auto.WindowControl(searchDepth=1, ClassName='Notepad')
    # or use Compare for custom search
    # window = auto.WindowControl(searchDepth=1, ClassName='Notepad', Compare=lambda control, depth: control.ProcessId==100)
    edit = window.EditControl()
    # when calling SendKeys, uiautomation starts searching the window and edit controls in 15 seconds
    # because SendKeys indirectly calls Control.Element and Control.Element is None
    # if the window and edit controls don't exist within 15 seconds, a LookupError exception will be raised
    try:
        edit.SendKeys('first not