target_finder_toolkit.targetfinder module¶

Detection of GUI widgets from live screen captures or static images.

Core features:

Real-time screen capture and YOLO-based detection of GUI widgets.
Callback mechanism to provide the current detections, the captured frame (optional), and the lists of added/removed detections.
Optional PyQt overlay to visualize bounding boxes in real time.
One-shot detection on static images with optional annotated export.

Notes

This is the main entry point of the toolkit. Utility helpers live in postprocess.py and demonstration apps are in bubblecursor.py and semanticpointing.py.

class target_finder_toolkit.targetfinder.TargetFinder(model_path=None, change_thresh=100, capture_interval=0.03333333333333333, confidence=0.28, iou=0.3)¶

Bases: object

Initialize the detector.

Parameters:

model_path (str | None, optional) – Path to a YOLO .pt weights file. If None, the detector loads the default model (best.pt) packaged with the toolkit. You can also supply your own trained YOLOv8 model.
change_thresh (int, optional) – Screen change detection threshold (L2 distance on a down-scaled screen). A higher value makes the detector less sensitive to small variations. Default = 100.
capture_interval (float, optional) – Delay in seconds between consecutive screen captures. Lower values = higher refresh rate but also higher CPU/GPU usage. Default = 1/30 ≈ 0.033s (30 FPS).
confidence (float, optional) – Minimum YOLO confidence score required to keep a detection. Must be in the range [0.0 – 1.0]. Default = 0.28.
iou (float, optional) – Intersection-over-Union (IoU) threshold used for YOLO’s Non-Maximum Suppression (NMS). Determines when two overlapping bounding boxes should be considered the same object (higher = stricter merging, lower = more boxes kept). Must be in the range [0.0 – 1.0]. Default = 0.3.

Raises:

FileNotFoundError – If the specified YOLO model file cannot be found.
ValueError – If numeric arguments are outside their valid ranges.
TypeError – If numeric arguments have invalid types.

set_callback(fn, with_frame=False, diff_iou=0.5)¶

The callback is invoked as:: fn(detections, added, removed, frame)

Parameters:

fn (Callable | None) –
User callback function, or None to disable. The function receives:
- detections (list[dict]): All current detections (widgets).
- added (list[dict]): Detections that appeared since the last frame.
- removed (list[dict]): Detections that disappeared since the last frame.
- frame (np.ndarray | None): RGB screen frame (only provided if with_frame=True).
Each detection dictionary has the following keys:
- id (int): Unique identifier for a detection, preserved across frames as long as the widget remains visible.
- x (int): Top-left X coordinate.
- y (int): Top-left Y coordinate.
- width (int): Bounding box width.
- height (int): Bounding box height.
- score (float): Confidence score in [0, 1].
- class_id (int): YOLO class index.
- class_name (str): class label (e.g., “Button”).
with_frame (bool, optional) – If True, include the RGB frame in the callback. Default = False.
diff_iou (float, optional) – IoU threshold used to decide whether a detection is considered the same widget across frames (for ID conservation and added/removed lists). Must be in [0, 1]. Default = 0.5.

Raises:

TypeError – If fn is not callable.
ValueError – If diff_iou is outside [0, 1].

start()¶: Start the capture+inference loop in a separate thread.

stop()¶: Stop the detection loop.

get_detections()¶

Return current detections as tuples. Coordinates are in logical screen space, i.e. they already include DPI-aware scaling to match the primary screen geometry.

Returns:: [(x, y, w, h, score, class_id) ...].

detect_image(image_path, save_annotated=False, save_json=False)¶

Run detection on a single image and optionally save results.

Parameters:

image_path (str) – Path to input image.
save_annotated (bool, optional) – If True, write *_annotated.png next to the image. Defaults to False.
save_json (bool, optional) – If True, write *_detections.json next to the image. Defaults to False.

Returns:

Detections with keys id, x, y, width, height, score, class_id, class_name.

Raises:

FileNotFoundError – If the image cannot be read.

target_finder_toolkit.targetfinder.show_detections(detector)¶

Launch a PyQt application showing detections in a transparent overlay window.

Parameters:: detector (TargetFinder) – The detector instance providing the detections.

target_finder_toolkit.targetfinder.main()¶

CLI entry point for launching the TargetFinder overlay.

Example: python -m target_finder_toolkit.targetfinder --confidence 0.3 --iou 0.4