Seedlabs
Source research

SAM2-UNet: segment anything 2 makes strong encoder for natural and medical image segmentation

Xinyu Xiong, Zihuang Wu, Shuangyi Tan, Wenxue Li, Feilong Tang, Ying Chen et al. · 2026 · 99 citationsRead the paper

Abstract Image segmentation plays an important role in vision understanding. Recently, the emerging vision foundation models continuously achieved superior performance on various tasks. Following such success, in this paper, we prove that the Segment Anything Model 2 (SAM2) can be a strong encoder for U-shaped segmentation models. We propose a simple but effective framework, termed SAM2-UNet, for versatile image segmentation. Specifically, SAM2-UNet adopts the Hiera backbone of SAM2 as the encoder, while the decoder uses the classic U-shaped design. Additionally, adapters are inserted into the encoder to enable parameter-efficient fine-tuning. Preliminary experiments on various downstream tasks, such as camouflaged object detection, salient object detection, marine animal segmentation, mirror detection, and polyp segmentation, demonstrate that our SAM2-UNet can outperform existing specialized state-of-the-art methods with minimal additional complexity.

What the full text shows

Working prototype
  • Outperformed FEDER in camouflaged object detection S-measure across four datasets. — 1.1% to 4.8% higher S-measure (CHAMELEON, CAMO, COD10K, and NC4K datasets)
  • Exceeded MENet in salient object detection S-measure across five datasets. — 1.4% to 3.4% higher S-measure (DUTS-TE, DUT-OMRON, HKU-IS, PASCAL-S, and ECSSD datasets)
  • Improved marine animal segmentation mIoU over MASNet. — 0.7% to 5.7% higher mIoU (MAS3K and RMAS datasets)
  • Significantly outperformed HetNet in mirror detection IoU. — 3.8% to 9% higher IoU (MSD and PMD datasets)
  • Surpassed CFA-Net in polyp segmentation mDice on average. — 2.34% higher mDice (Across five datasets (Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB, CVC-300, ETIS))
Setup
Trained on a single NVIDIA RTX 4090 GPU (24GB memory) using 18 datasets across 5 benchmarks; input images resized to 352 x 352.
How
The authors integrated the Hiera backbone of SAM2 as an encoder with a classic U-Net decoder and added adapters for parameter-efficient fine-tuning. The model was trained using a combination of weighted IoU loss and binary cross-entropy loss with deep supervision.
Limitations
Performance was weaker than state-of-the-art on ClinicDB and CVC-300 polyp datasets.

Summarized by Gemma 4 from the open-access full text · 14 Oct 2026. Read it