Instance Segmentation with Point Prompts and SAM 3¶

This notebook demonstrates how to use the Segment Anything Model 3 (SAM3) for interactive instance segmentation using point and box prompts.

Installation¶

1	`# %pip install "segment-geospatial[samgeo3]"`

Import Libraries¶

from samgeo import SamGeo3, download_file, show_image

Download Sample Data¶

url = "https://raw.githubusercontent.com/facebookresearch/sam3/refs/heads/main/assets/images/truck.jpg"
image_path = download_file(url)

show_image(image_path, axis="on")

Initialize SAM3¶

To use point and box prompts (SAM1-style interactive segmentation), initialize SAM3 with enable_inst_interactivity=True.

sam = SamGeo3(backend="meta", enable_inst_interactivity=True)

sam.set_image(image_path)

Generate Masks by Point Prompts¶

Select an object by clicking a point on it. Points are input in (x, y) format with labels: - 1 = foreground point (include this region) - 0 = background point (exclude this region)

# Single foreground point - input as Python list
sam.generate_masks_by_points([[520, 375]])

print(sam.masks)

print(sam.scores)

Visualize Point Prompts¶

sam.show_points([[520, 375]], [1])

Show the Results¶

sam.show_anns()

sam.show_masks()

Save Masks¶

sam.save_masks("truck_mask.png", unique=True)

Multiple Points with Labels¶

Use multiple points to refine selection. Use label=0 for background points to exclude regions.

# Two foreground points on the truck
sam.generate_masks_by_points([[500, 375], [1125, 625]], point_labels=[1, 1])

sam.show_points([[500, 375], [1125, 625]], [1, 1])

sam.show_anns()

sam.show_masks()

Using Background Points¶

Add a background point (label=0) to exclude a region from the mask.

# One foreground point on window, one background point on car body
sam.generate_masks_by_points(
    [[500, 375], [1125, 625]], point_labels=[1, 0]  # foreground, background
)

sam.show_points([[500, 375], [1125, 625]], [1, 0])

sam.show_anns()

Box Prompts¶

Use a bounding box in XYXY format (xmin, ymin, xmax, ymax) to select an object.

# Box around the front wheel
sam.generate_masks_by_boxes_inst([[425, 600, 700, 875]])

sam.show_boxes([[425, 600, 700, 875]])

sam.show_anns()

Multiple Box Prompts¶

Process multiple boxes at once for efficient batch segmentation.

boxes = [
    [75, 275, 1725, 850],  # Whole truck
    [425, 600, 700, 875],  # Front wheel
    [1375, 550, 1650, 800],  # Rear wheel
    [1240, 675, 1400, 750],  # License plate area
]

sam.generate_masks_by_boxes_inst(boxes)

sam.show_boxes(boxes)

sam.show_anns()

sam.save_masks("truck_boxes_mask.png", unique=True)

Low-Level API: predict_inst¶

For more control, you can use the lower-level predict_inst() method which returns masks, scores, and logits directly. Input points and boxes can be provided as Python lists.

# Using Python lists for input
masks, scores, logits = sam.predict_inst(
    point_coords=[[520, 375]],
    point_labels=[1],
    multimask_output=True,
)

print(f"Generated {len(masks)} masks")
print(f"Scores: {scores}")

# Show all masks with point overlays
sam.show_inst_masks(masks, scores, point_coords=[[520, 375]], point_labels=[1])

# Box prompt with Python list
masks, scores, logits = sam.predict_inst(
    box=[425, 600, 700, 875],
    multimask_output=False,
)

sam.show_inst_masks(masks, scores, box_coords=[425, 600, 700, 875])

Summary¶

This notebook demonstrated SAM3's interactive instance segmentation capabilities:

High-level API (recommended): - generate_masks_by_points() - Generate masks from point prompts - generate_masks_by_boxes_inst() - Generate masks from box prompts - show_points() / show_boxes() - Visualize prompts - show_anns() / show_masks() - Visualize results - save_masks() - Save masks to file

Low-level API: - predict_inst() - Direct access to masks, scores, and logits - show_inst_masks() - Display masks with overlays

Input formats: - Points and boxes can be provided as Python lists or numpy arrays - Point labels: 1 = foreground, 0 = background