YOLO-Patch-Based-Inference

This Python library simplifies SAHI-like inference for instance segmentation tasks, enabling the detection of small objects in images. It caters to both object detection and instance segmentation tasks, supporting a wide range of Ultralytics models.

The library also provides a sleek customization of the visualization of the inference results for all models, both in the standard approach (direct network run) and the unique patch-based variant.

Model Support: The library offers support for multiple ultralytics deep learning models, such as YOLOv8, YOLOv8-seg, YOLOv9, YOLOv9-seg, YOLOv10, YOLO11, YOLO11-seg, YOLO12, YOLO12-seg, FastSAM, and RTDETR. Users can select from pre-trained options or utilize custom-trained models to best meet their task requirements.

Explanation of how Patch-Based-Inference works:

Installation

You can install the library via pip:

pip install patched_yolo_infer

- Click here to visit the PyPI page of patched-yolo-infer.

Note: If CUDA support is available, it's recommended to pre-install PyTorch with CUDA support before installing the library. Otherwise, the CPU version will be installed by default.

---

Notebooks

Interactive notebooks are provided to showcase the functionality of the library. These notebooks cover batch-inference procedures for detection, instance segmentation, inference custom visualization, and more. Each notebook is paired with a tutorial on YouTube, making it easy to learn and implement features.

Topic	Notebook	YouTube
[Patch-Based-Inference Example][nb_example1]	[![Open In Colab][colab_badge]][colab_ex1]	[<img width=30% alt="Youtube Video" src=https://raw.githubusercontent.com/ultralytics/assets/main/social/logo-social-youtube-rect.png>][yt_link1]
[Example of custom visualization of usual inference results][nb_example2]	[![Open In Colab][colab_badge]][colab_ex2]	[<img width=30% alt="Youtube Video" src=https://raw.githubusercontent.com/ultralytics/assets/main/social/logo-social-youtube-rect.png>][yt_link2]

For Russian-speaking users, there is a detailed video presentation of the project at the AiConf 2024 conference. The YouTube video is available at this link.

---

Examples:

Detection example:

Instance Segmentation example 1:

Instance Segmentation example 2:

---

Usage

1. Patch-Based-Inference

To carry out patch-based inference of YOLO models using our library, you need to follow a sequential procedure. First, you create an instance of the MakeCropsDetectThem class, providing all desired parameters related to YOLO inference and the patch segmentation principle. Subsequently, you pass the obtained object of this class to CombineDetections, which facilitates the consolidation of all predictions from each overlapping crop, followed by intelligent suppression of duplicates. Upon completion, you receive the result, from which you can extract the desired outcome of frame processing.

The output obtained from the process includes several attributes that can be leveraged for further analysis or visualization:

1. img: This attribute contains the original image on which the inference was performed. It provides context for the detected objects.

2. confidences: This attribute holds the confidence scores associated with each detected object. These scores indicate the model's confidence level in the accuracy of its predictions.

3. boxes: These bounding boxes are represented as a list of lists, where each list contains four values: [x_min, y_min, x_max, y_max]. These values correspond to the coordinates of the top-left and bottom-right corners of each bounding box.

4. polygons: If available, this attribute provides a list containing NumPy arrays of polygon coordinates that represent segmentation masks corresponding to the detected objects. These polygons can be utilized to accurately outline the boundaries of each object.

5. classes_ids: This attribute contains the class IDs assigned to each detected object. These IDs correspond to specific object classes defined during the model training phase.

6. classes_names: These are the human-readable names corresponding to the class IDs. They provide semantic labels for the detected objects, making the results easier to interpret.

import cv2
from patched_yolo_infer import MakeCropsDetectThem, CombineDetections

# Load the image 
img_path = "test_image.jpg"
img = cv2.imread(img_path)

element_crops = MakeCropsDetectThem(
    image=img,
    model_path="yolo11m.pt",
    segment=False,
    shape_x=640,
    shape_y=640,
    overlap_x=25,
    overlap_y=25,
    conf=0.5,
    iou=0.7,
)
result = CombineDetections(element_crops, nms_threshold=0.25)  

# Final Results:
img=result.image
confidences=result.filtered_confidences
boxes=result.filtered_boxes
polygons=result.filtered_polygons
classes_ids=result.filtered_classes_id
classes_names=result.filtered_classes_names

Explanation of possible input arguments:

MakeCropsDetectThem Class implementing cropping and passing crops through a neural network for detection/segmentation:

Argument	Type	Default	Description
image	np.ndarray		The input image in BGR format.
model_path	str	"yolo11m.pt"	Path to the YOLO model.
model	Ultralytics model	None	Pre-initialized model object. If provided, the model will be used directly instead of loading from model_path.
imgsz	int	640	Size of the input image for inference YOLO.
conf	float	0.25	Confidence threshold for detections YOLO.
iou	float	0.7	IoU threshold for non-maximum suppression YOLO of single crop.
classes_list	List[int] or None	None	List of classes to filter detections. If None, all classes are considered.
segment	bool	False	Whether to perform segmentation (if the model supports it).
shape_x	int	700	Size of the crop in the x-coordinate.
shape_y	int	600	Size of the crop in the y-coordinate.
overlap_x	float	25	Percentage of overlap along the x-axis.
overlap_y	float	25	Percentage of overlap along the y-axis.
show_crops	bool	False	Whether to visualize the cropping.
resize_initial_size	bool	True	Whether to resize the results to the original input image size (ps: slow operation).
memory_optimize	bool	True	Memory optimization option for segmentation (less accurate results when enabled).
inference_extra_args	dict	None	Dictionary with extra ultralytics inference parameters (possible keys: half, device, max_det, augment, agnostic_nms and retina_masks)

CombineDetections Class implementing combining masks/boxes from multiple crops + NMS (Non-Maximum Suppression):

Argument	Type	Default	Description
element_crops	MakeCropsDetectThem		Object containing crop information. This can be either a single MakeCropsDetectThem object or a list of objects.
nms_threshold	float	0.3	IoU/IoS threshold for non-maximum suppression. The lower the value, the fewer objects remain after suppression.
match_metric	str	IOS	Matching metric, either 'IOU' or 'IOS'.
class_agnostic_nms	bool	True	Determines the NMS mode in object detection. When set to True, NMS operates across all classes, ignoring class distinctions and suppressing less confident bounding boxes globally. Otherwise, NMS is applied separately for each class.
intelligent_sorter	bool	True	Enable sorting by area and rounded confidence parameter. If False, sorting will be done only by confidence (usual nms).
sorter_bins	int	5	Number of bins to use for intelligent_sorter. A smaller number of bins makes the NMS more reliant on object sizes rather than confidence scores.

---

2. Custom inference visualization:

Visualizes results of patch-based object detection or segmentation on an image.
Possible arguments of the visualize_results function:

Argument	Type	Default	Description