构建基础模型：SegmentAnything推动图像分割新篇章

需积分: 5 58 浏览量更新于2024-06-26 收藏 14.67MB PDF 举报

SegmentAnything项目是计算机视觉领域的一项创新举措，旨在构建首个针对图像分割的基础模型。该项目由MetaAIResearch和FAIR的专家团队主导，其核心目标是通过三个相互关联的组件来推动图像分割技术的发展：一是可提示的（promptable）图像分割任务，二是名为SegmentAnything Model (SAM) 的强大分割模型，三是专为收集数据而设计的数据引擎和名为SegmentAnything-1B (SA-1B) 的大规模数据集。首先，"promptable segmentation" 是项目的关键概念，它强调了模型能够根据用户的自然语言提示进行实时、灵活的图像分割，这种能力使得模型能够在无需重新训练的情况下适应各种新的场景和对象，从而实现零样本迁移学习。SegmentAnything Model (SAM) 就是这种任务的核心执行者，它不仅具备强大的图像编码功能，还整合了一个轻量级的掩码解码器，使得模型能够高效地根据提示生成准确的像素级分割结果。其次，SA-1B 数据集是SegmentAnything项目的核心组成部分，包含超过10亿个掩码和1100万张图片。这些数据的特点是尊重隐私，所有的图片都经过授权，并且由专业人员进行了标注，确保了数据的质量和多样性。这个大规模的数据集为模型的训练提供了丰富的样本，使得SegmentAnything模型能够在多样化的场景中展现出卓越的性能。最后，数据引擎的设计是项目背后的基础设施，它支持数据的高效收集、管理和更新，确保了SA-1B数据集的持续扩展和维护。通过这种数据驱动的方法，SegmentAnything项目致力于提供一个强大的工具平台，帮助研究人员和开发者快速迭代和改进他们的图像分割算法。 SegmentAnything项目的目标是通过创新的任务设计、先进的模型架构以及庞大的数据资源，建立一个可以广泛应用的通用图像分割基础模型，这将对计算机视觉领域产生深远的影响，推动图像分析在诸如自动驾驶、医疗影像分析、图像编辑等众多场景中的应用发展。

At the start of this stage, SAM was trained using com-

mon public segmentation datasets. After sufﬁcient data an-

notation, SAM was retrained using only newly annotated

masks. As more masks were collected, the image encoder

was scaled from ViT-B to ViT-H and other architectural de-

tails evolved; in total we retrained our model 6 times. Av-

erage annotation time per mask decreased from 34 to 14

seconds as the model improved. We note that 14 seconds

is 6.5× faster than mask annotation for COCO [66] and

only 2× slower than bounding-box labeling with extreme

points [76, 71]. As SAM improved, the average number of

masks per image increased from 20 to 44 masks. Overall,

we collected 4.3M masks from 120k images in this stage.

Semi-automatic stage. In this stage, we aimed to increase

the diversity of masks in order to improve our model’s

ability to segment anything. To focus annotators on less

prominent objects, we ﬁrst automatically detected conﬁdent

masks. Then we presented annotators with images preﬁlled

with these masks and asked them to annotate any additional

unannotated objects. To detect conﬁdent masks, we trained

a bounding box detector [84] on all ﬁrst stage masks using a

generic “object” category. During this stage we collected an

additional 5.9M masks in 180k images (for a total of 10.2M

masks). As in the ﬁrst stage, we periodically retrained our

model on newly collected data (5 times). Average annota-

tion time per mask went back up to 34 seconds (excluding

the automatic masks) as these objects were more challeng-

ing to label. The average number of masks per image went

from 44 to 72 masks (including the automatic masks).

Fully automatic stage. In the ﬁnal stage, annotation was

fully automatic. This was feasible due to two major en-

hancements to our model. First, at the start of this stage, we

had collected enough masks to greatly improve the model,

including the diverse masks from the previous stage. Sec-

ond, by this stage we had developed the ambiguity-aware

model, which allowed us to predict valid masks even in am-

biguous cases. Speciﬁcally, we prompted the model with a

32×32 regular grid of points and for each point predicted

a set of masks that may correspond to valid objects. With

the ambiguity-aware model, if a point lies on a part or sub-

part, our model will return the subpart, part, and whole ob-

ject. The IoU prediction module of our model is used to se-

lect conﬁdent masks; moreover, we identiﬁed and selected

only stable masks (we consider a mask stable if threshold-

ing the probability map at 0.5 − δ and 0.5 + δ results in

similar masks). Finally, after selecting the conﬁdent and

stable masks, we applied non-maximal suppression (NMS)

to ﬁlter duplicates. To further improve the quality of smaller

masks, we also processed multiple overlapping zoomed-in

image crops. For further details of this stage, see §B. We

applied fully automatic mask generation to all 11M images

in our dataset, producing a total of 1.1B high-quality masks.

We describe and analyze the resulting dataset, SA-1B, next.

Figure 5: Image-size normalized mask center distributions.

5. Segment Anything Dataset

Our dataset, SA-1B, consists of 11M diverse, high-

resolution, licensed, and privacy protecting images and

1.1B high-quality segmentation masks collected with our

data engine. We compare SA-1B with existing datasets

and analyze mask quality and properties. We are releasing

SA-1B to aid future development of foundation models for

computer vision. We note that SA-1B will be released un-

der a favorable license agreement for certain research uses

and with protections for researchers.

Images. We licensed a new set of 11M images from a

provider that works directly with photographers. These im-

ages are high resolution (3300×4950 pixels on average),

and the resulting data size can present accessibility and stor-

age challenges. Therefore, we are releasing downsampled

images with their shortest side set to 1500 pixels. Even af-

ter downsampling, our images are signiﬁcantly higher reso-

lution than many existing vision datasets (e.g., COCO [66]

images are

∼

480×640 pixels). Note that most models today

operate on much lower resolution inputs. Faces and vehicle

license plates have been blurred in the released images.

Masks. Our data engine produced 1.1B masks, 99.1% of

which were generated fully automatically. Therefore, the

quality of the automatic masks is centrally important. We

compare them directly to professional annotations and look

at how various mask properties compare to prominent seg-

mentation datasets. Our main conclusion, as borne out in

the analysis below and the experiments in §7, is that our

automatic masks are high quality and effective for training

models. Motivated by these ﬁndings, SA-1B only includes

automatically generated masks.

Mask quality. To estimate mask quality, we randomly sam-

pled 500 images (

∼

50k masks) and asked our professional

annotators to improve the quality of all masks in these im-

ages. Annotators did so using our model and pixel-precise

“brush” and “eraser” editing tools. This procedure resulted

in pairs of automatically predicted and professionally cor-

rected masks. We computed IoU between each pair and

found that 94% of pairs have greater than 90% IoU (and

97% of pairs have greater than 75% IoU). For comparison,

prior work estimates inter-annotator consistency at 85-91%

IoU [44, 60]. Our experiments in §7 conﬁrm by human rat-

ings that mask quality is high relative to a variety of datasets

and that training our model on automatic masks is nearly as

good as using all masks produced by the data engine.

剩余29页未读，继续阅读

anjushi_

粉丝: 177
资源: 7

构建基础模型：SegmentAnything推动图像分割新篇章

segment-anything

segmentation papers

segment-anything代码

paper

paper.rar_paper_paper语音_voice to text

Design Con paper 2019 PAPER 01

Design Con paper 2019 PAPER 02

DPT.rar_Dirty Paper code_dirty paper_dpt.rar_paper_watermarking

New folder.rar_LMI_LMI + paper_LMI paper_lmi dependent_paper

Design Con paper 2019 PAPER 01_03

最新资源