Skip to content

Download notebook

Automated Segmentation of Remote Sensing Imagery with SAM 3

image

In this notebook, we demonstrate the automated segmentation of remote sensing imagery using SAM 3. The process begins with image captioning, which automatically identifies key features within the image. These features can then be utilized as text prompts for SAM 3, enabling precise segmentation.

Installation

First, make sure you have the required dependencies installed:

1
# %pip install "segment-geospatial[samgeo3]"

Import Libraries

1
2
3
import leafmap
from samgeo import SamGeo3, download_file
from samgeo.caption import ImageCaptioner

Download Sample Data

Let's download a sample satellite image covering the University of California, Berkeley, for testing:

1
2
url = "https://huggingface.co/datasets/giswqs/geospatial/resolve/main/uc_berkeley.tif"
image_path = download_file(url)

1
2
3
m = leafmap.Map()
m.add_raster(image_path, layer_name="Satellite image")
m

Image captioning

1
2
3
4
captioner = ImageCaptioner(
    blip_model_name="Salesforce/blip-image-captioning-base",
    spacy_model_name="en_core_web_sm",
)

1
2
3
caption, features = captioner.analyze(image_path)
print(f"Caption: {caption}")
print(f"Features: {features}")

Request access to SAM3

To use SAM3, you need to request access by filling out this form on Hugging Face: https://huggingface.co/facebook/sam3

Once you have access, uncomment the following code block and run it.

1
2
# from huggingface_hub import login
# login()

Initialize SAM3

When initializing SAM3, you can choose the backend from "meta", or "transformers".

1
sam3 = SamGeo3(backend="meta", device=None, checkpoint_path=None, load_from_HF=True)

Set the image

You can set the image by either passing the image path or the image URL.

1
sam3.set_image(image_path)

Generate masks with text prompt

1
2
feature = features[0]
sam3.generate_masks(prompt=feature)

Show the results

1
sam3.show_anns()

annotation

1
sam3.show_masks()

Save Masks

Save the generated masks to a file. If the input is a GeoTIFF, the output will be a GeoTIFF with the same georeferencing. Otherwise, it will be saved as PNG.

1
2
3
# Save masks with unique values for each object
# Since uc_berkeley.tif is a GeoTIFF, the output will also be a GeoTIFF
sam3.save_masks(output=f"{feature}_masks.tif", unique=True)

1
2
# Save as binary mask (all foreground pixels are 255)
sam3.save_masks(output=f"{feature}_masks_binary.tif", unique=False)

Save Masks with Confidence Scores

You can also save the confidence scores for each mask. The scores indicate the model's confidence for each predicted mask.

1
2
3
4
5
6
7
# Save masks and confidence scores
# Each pixel in the scores image will have the confidence value of its mask
sam3.save_masks(
    output=f"{feature}_masks_with_scores.tif",
    save_scores=f"{feature}_scores.tif",
    unique=True,
)

1
sam3.show_masks(cmap="coolwarm")

scores

Visualize Confidence Scores

Let's visualize the confidence scores to see which masks have higher confidence:

1
2
3
4
5
6
7
8
9
m.add_raster(f"{feature}_masks.tif", layer_name=f"{feature} masks", visible=False)
m.add_raster(
    f"{feature}_scores.tif",
    layer_name=f"{feature} scores",
    cmap="coolwarm",
    opacity=0.8,
    nodata=0,
)
m

map