.hYddlmZddlZddlmcmZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZmZdd lmZmZdd lmZmZd ZGd dej2ZGddejj2Zy)) annotationsN)nn) trunc_normal_)MLP)LOGGER)SAM2TwoWayTransformer) MaskDecoderSAM2MaskDecoder)ImageEncoderViT PromptEncoder)get_1d_sine_peselect_closest_cond_framesgcVeZdZUdZdZded< d dfd ZdZxZS) SAMModela Segment Anything Model (SAM) for object segmentation tasks. This class combines image encoders, prompt encoders, and mask decoders to predict object masks from images and input prompts. Attributes: mask_threshold (float): Threshold value for mask prediction. image_encoder (ImageEncoderViT): Backbone for encoding images into embeddings. prompt_encoder (PromptEncoder): Encoder for various types of input prompts. mask_decoder (MaskDecoder): Predicts object masks from image and prompt embeddings. pixel_mean (torch.Tensor): Mean values for normalizing pixels in the input image. pixel_std (torch.Tensor): Standard deviation values for normalizing pixels in the input image. Methods: set_imgsz: Set image size to make model compatible with different image sizes. Examples: >>> image_encoder = ImageEncoderViT(...) >>> prompt_encoder = PromptEncoder(...) >>> mask_decoder = MaskDecoder(...) >>> sam_model = SAMModel(image_encoder, prompt_encoder, mask_decoder) >>> # Further usage depends on SAMPredictor class Notes: All forward() operations are implemented in the SAMPredictor class. floatmask_thresholdc(t|||_||_||_|j dt j|jdddd|j dt j|jddddy)a Initialize the SAMModel class to predict object masks from an image and input prompts. Args: image_encoder (ImageEncoderViT): The backbone used to encode the image into image embeddings. prompt_encoder (PromptEncoder): Encodes various types of input prompts. mask_decoder (MaskDecoder): Predicts masks from the image embeddings and encoded prompts. pixel_mean (list[float]): Mean values for normalizing pixels in the input image. pixel_std (list[float]): Standard deviation values for normalizing pixels in the input image. Examples: >>> image_encoder = ImageEncoderViT(...) >>> prompt_encoder = PromptEncoder(...) >>> mask_decoder = MaskDecoder(...) >>> sam_model = SAMModel(image_encoder, prompt_encoder, mask_decoder) >>> # Further usage depends on SAMPredictor class Notes: All forward() operations moved to SAMPredictor. pixel_meanrF pixel_stdN) super__init__ image_encoderprompt_encoder mask_decoderregister_buffertorchTensorview)selfrrrrr __class__s `/var/www/html/ai-service/venv/lib/python3.12/site-packages/ultralytics/models/sam/modules/sam.pyrzSAMModel.__init__8s8 *,( \5<< +C+H+HQPQ+RTYZ [%,,y*A*F*Fr1a*PRWXct|jdr|jj|||j_|Dcgc]}|dz c}|j_|d|j_ycc}w)CSet image size to make model compatible with different image sizes. set_imgszrN)hasattrrr(rinput_image_sizeimage_embedding_sizeimg_sizer"imgszxs r$r(zSAMModel.set_imgsz[si 4%%{ 3    ( ( //4,EJ3KAG3K0&+Ah#4Ls A:))g33333^@gR]@gRY@)g(\2M@g(\L@gL@) rr rr rr r list[float]rr1returnNone) __name__ __module__ __qualname____doc__r__annotations__rr( __classcell__r#s@r$rrsh8 NE#>> model = SAM2Model(image_encoder, memory_attention, memory_encoder) >>> image_batch = torch.rand(1, 3, 512, 512) >>> features = model.forward_image(image_batch) >>> track_results = model.track_step(0, True, features, None, None, None, {}) rrrc$Vt$|||_| |_| rdnd|_||_||_|r(tjjdddd|_ ||_ |r|sJ||_ ||_ ||_||_|j |_||_|j"|_t)|j$drRt)|j$j*dr2|j$j*j,j.d|_||_tjj3tj4|dd|j&|_t9|j6d tjj3tj4dd|j"|_tjj3tj4dd|j"|_t9|j:d t9|j<d | |_||_ ||_!| |_"||_#||_$| |_%||_&||_'||_(||_)||_*||_+||_,||_-|"|_.||_/||_0||_1||_2|jbr|j^sJ|j sJ|j^re|j rYtjj3tj4d|j"|_3t9|jfd | |_4d |_5|!rYtjj3tj4d|j&|_5t9|jjd |jm| |_7|#rRtqjrd tjt|jjvd d d|j_;y y )a Initialize the SAM2Model for video object segmentation with memory-based tracking. Args: image_encoder (nn.Module): Visual encoder for extracting image features. memory_attention (nn.Module): Module for attending to memory features. memory_encoder (nn.Module): Encoder for generating memory representations. num_maskmem (int): Number of accessible memory frames. image_size (int): Size of input images. backbone_stride (int): Stride of the image backbone output. sigmoid_scale_for_mem_enc (float): Scale factor for mask sigmoid probability. sigmoid_bias_for_mem_enc (float): Bias factor for mask sigmoid probability. binarize_mask_from_pts_for_mem_enc (bool): Whether to binarize sigmoid mask logits on interacted frames with clicks during evaluation. use_mask_input_as_output_without_sam (bool): Whether to directly output the input mask without using SAM prompt encoder and mask decoder on frames with mask input. max_cond_frames_in_attn (int): Maximum number of conditioning frames to participate in memory attention. directly_add_no_mem_embed (bool): Whether to directly add no-memory embedding to image feature on the first frame. use_high_res_features_in_sam (bool): Whether to use high-resolution feature maps in the SAM mask decoder. multimask_output_in_sam (bool): Whether to output multiple masks for the first click on initial conditioning frames. multimask_min_pt_num (int): Minimum number of clicks to use multimask output in SAM. multimask_max_pt_num (int): Maximum number of clicks to use multimask output in SAM. multimask_output_for_tracking (bool): Whether to use multimask output for tracking. use_multimask_token_for_obj_ptr (bool): Whether to use multimask tokens for object pointers. iou_prediction_use_sigmoid (bool): Whether to use sigmoid to restrict IoU prediction to [0-1]. memory_temporal_stride_for_eval (int): Memory bank's temporal stride during evaluation. non_overlap_masks_for_mem_enc (bool): Whether to apply non-overlapping constraints on object masks in memory encoder during evaluation. use_obj_ptrs_in_encoder (bool): Whether to cross-attend to object pointers from other frames in the encoder. max_obj_ptrs_in_encoder (int): Maximum number of object pointers from other frames in encoder cross-attention. add_tpos_enc_to_obj_ptrs (bool): Whether to add temporal positional encoding to object pointers in the encoder. proj_tpos_enc_in_obj_ptrs (bool): Whether to add an extra linear projection layer for temporal positional encoding in object pointers. use_signed_tpos_enc_to_obj_ptrs (bool): Whether to use signed distance in the temporal positional encoding in the object pointers. only_obj_ptrs_in_the_past_for_eval (bool): Whether to only attend to object pointers in the past during evaluation. pred_obj_scores (bool): Whether to predict if there is an object in the frame. pred_obj_scores_mlp (bool): Whether to use an MLP to predict object scores. fixed_no_obj_ptr (bool): Whether to have a fixed no-object pointer when there is no object present. soft_no_obj_ptr (bool): Whether to mix in no-object pointer softly for easier recovery and error mitigation. use_mlp_for_obj_ptr_proj (bool): Whether to use MLP for object pointer projection. no_obj_embed_spatial (bool): Whether add no obj embedding to spatial frames. sam_mask_decoder_extra_args (dict | None): Extra arguments for constructing the SAM mask decoder. compile_image_encoder (bool): Whether to compile the image encoder for faster inference. Examples: >>> image_encoder = ImageEncoderViT(...) >>> memory_attention = SAM2TwoWayTransformer(...) >>> memory_encoder = nn.Sequential(...) >>> model = SAM2Model(image_encoder, memory_attention, memory_encoder) >>> image_batch = torch.rand(1, 3, 512, 512) >>> features = model.forward_image(image_batch) >>> track_results = model.track_step(0, True, features, None, None, None, {}) r) kernel_sizestrideout_projweightrg{Gz?)stdNzFImage encoder compilation is enabled. First forward pass will be slow.z max-autotuneTF)mode fullgraphdynamic)$'>$ "$)88??1aQq?#QD (@% $+ ++)B&/N,2T/!1*22- 4&& 3@S@S@\@\^f8g..77>>DDQGDL& % 2 25;;{AqRVR^R^3_ `d++6!HH..u{{1a/QR#hh00Q4??1STd''T2d))t4)B&*C&(@%2T/-J*/N,5Y1'>$$8!$8!-J*/N,*D'%.+F(.#6 0.  '' ''// //   D$@$@#hh00Q1PQDO $//t 4(@%$(! (-(:(:5;;q$,,;W(XD % $33 > '>$ ! KK` a).""**# *D   & !r%cHt|jjS)z=Return the device on which the model's parameters are stored.)next parametersdevicer"s r$r~zSAM2Model.devicelsDOO%&---r%ctd)zWProcess image and prompt inputs to generate object masks and scores in video sequences.zPlease use the corresponding methods in SAM2VideoPredictor for inference.See notebooks/video_predictor_example.ipynb for an example.)NotImplementedError)r"argskwargss r$ryzSAM2Model.forwardqs! J  r%c |j|_|j|jz|_t |j|j|jf|j|jfd|_td dtd|jdd|jdd|j|j|j|j|jd |jxsi|_|j rwt"j$j'|j|j|_|j*rUt-|j|j|jd|_n#t"j$j/|_|j0r:t"j$j'|j|j2|_y t"j$j/|_y ) zMBuild SAM-style prompt encoder and mask decoder for image segmentation tasks.r)) embed_dimr,r+ mask_in_chansr>i)depth embedding_dimmlp_dim num_heads) num_multimask_outputs transformertransformer_dimiou_head_depthiou_head_hidden_dimuse_high_res_featuresrjrnroriN)rTsam_prompt_embed_dimrkrlsam_image_embedding_sizer sam_prompt_encoderr r rHrjrnrorirmsam_mask_decoderrJrrLinear obj_ptr_projrsrIdentityrOrVobj_ptr_tpos_projrs r$ruzSAM2Model._build_sam_headsxs$(OO!(,4;O;O(O%#0//----"#oot?# !0! "#-"77  !55 #"&"C"C'+'F'F 00 $ 8 8,0,P,P! //52!! $  ' ' % QD ,,$'$//[\$]! % 1 1 3D   ) )&+XX__T__dll%SD "%*XX%6%6%8D "r%c |jd}|j}|jd|jk(sJ|jd|jk(sJ|jd|jk(sJ|0|d}|d} |jd|k(r| jd|k(sNJt j |dd||j}t j|dt j|  } |t|jd k(r|jdd|dfk(sJ|jd d|jjk7r=tj|j|jjd d d} n|} nd} |j|| fd| \} } |j!||jj#| | |d |\} }}}|j$r(|dkD}t j&|ddddf| t(} tj| |j*|j*fd d }|dddf}|rvt j,|d}t j.||}| ||fj1d}|||fj1d}|jddkDr |||f}n| |}}|j3|}|j$r^|j4r|j7}nj9|j}|j:r||z}|d|z |j<zz}| ||||||fS)a{ Forward pass through SAM prompt encoders and mask heads. This method processes image features and optional point/mask inputs to generate object masks and scores. Args: backbone_features (torch.Tensor): Image features with shape (B, C, H, W). point_inputs (dict[str, torch.Tensor] | None): Dictionary containing point prompts. 'point_coords': Tensor of shape (B, P, 2) with float32 dtype, containing absolute pixel-unit coordinates in (x, y) format for P input points. 'point_labels': Tensor of shape (B, P) with int32 dtype, where 1 means positive clicks, 0 means negative clicks, and -1 means padding. mask_inputs (torch.Tensor | None): Mask of shape (B, 1, H*16, W*16), float or bool, with the same spatial size as the image. high_res_features (list[torch.Tensor] | None): List of two feature maps with shapes (B, C, 4*H, 4*W) and (B, C, 2*H, 2*W) respectively, used as high-resolution feature maps for SAM decoder. multimask_output (bool): If True, output 3 candidate masks and their IoU estimates; if False, output only 1 mask and its IoU estimate. Returns: low_res_multimasks (torch.Tensor): Tensor of shape (B, M, H*4, W*4) with SAM output mask logits. high_res_multimasks (torch.Tensor): Tensor of shape (B, M, H*16, W*16) with upsampled mask logits. ious (torch.Tensor): Tensor of shape (B, M) with estimated IoU for each output mask. low_res_masks (torch.Tensor): Tensor of shape (B, 1, H*4, W*4) with the best low-resolution mask. high_res_masks (torch.Tensor): Tensor of shape (B, 1, H*16, W*16) with the best high-resolution mask. obj_ptr (torch.Tensor): Tensor of shape (B, C) with object pointer vector for the output mask. object_score_logits (torch.Tensor): Tensor of shape (B) with object score logits. Examples: >>> backbone_features = torch.rand(1, 256, 32, 32) >>> point_inputs = {"point_coords": torch.rand(1, 2, 2), "point_labels": torch.tensor([[1, 0]])} >>> mask_inputs = torch.rand(1, 1, 512, 512) >>> results = model._forward_sam_heads(backbone_features, point_inputs, mask_inputs) >>> ( ... low_res_multimasks, ... high_res_multimasks, ... ious, ... low_res_masks, ... high_res_masks, ... obj_ptr, ... object_score_logits, ... ) = results rrrr>N point_coords point_labelsr~dtype)rr~r?FbilinearTsize align_cornersrE antialias)pointsboxesmasks)image_embeddingsimage_pesparse_prompt_embeddingsdense_prompt_embeddingsmultimask_output repeat_imagehigh_res_features)rrErrdimr~)rWr~rrrrrZronesint32lenrmask_input_sizeF interpolaterr get_dense_pernwhere NO_OBJ_SCORErkargmaxarange unsqueezerrqsigmoidtorprr)r"backbone_features point_inputs mask_inputsrrBr~sam_point_coordssam_point_labelssam_mask_promptsparse_embeddingsdense_embeddingslow_res_multimasksioussam_output_tokensobject_score_logitsis_obj_appearinghigh_res_multimaskssam_output_token best_iou_inds batch_inds low_res_maskshigh_res_masksobj_ptrlambda_is_obj_appearings r$_forward_sam_headszSAM2Model._forward_sam_headssh  # #A &")) %%a(D,E,EEEE %%a(D,I,IIII %%a(D,I,IIII  #+N; +N; #))!,16F6L6LQ6OST6T TT %{{1a6IZI`I`a  % 1au{{6 RR   "{(()Q.;3D3DRa3HQPQF3R RR  %)@)@)P)PP"#--%%'00@@"'#" ##.#O.2.E.E$&67!/F/ ++ LPK`K`.,,99;%6$4-/LaL HD"35H   2Q6 "'-=atm-LN`bn!o  mm //4??3  -QT2 !LL26Ma7J.z=/HISSTUVM0]1JKUUVWXN %%a(1,#4Z5N#O ,>@S>M##$45   ##*=*E*E*G'*:*=*=gmm*L'$$1G;%s* 8'--/*Y6A  %%b)Q.0C0CB0G10LM  ##K$5$5a$8!<BBD++/@/HL]Lekk+"3"3A"6P[PbPbcG)-(?(?"3 001BC"3)@) %Aq!Q7A!99[%8%8%;%A%A%Cc%IqQ+I6"2"8"8":'*AAHL   $$1G;%}'t7j@|#| |djB}(tE|(|&z |'}(|jG|(}(|(jIdjKd| |j>}(n&|%jMt |#| |j>}(|j>| kro|%jOd| | |j>z|j>}%|%j ddddj#dd}%|(jQ| |j>zd}(|j|%|j|(|%jRd}nd}n|jTr9|d|jVz})|)j dddj | | | | })|)S|jVjKd| |j>g}|jXjKd| |j>g}t7jZ|d}*t7jZ|d}+|j]|||*|+|})|)j dddj | | | | })|)Scc}wcc}}wcc}}w)zePrepare memory-conditioned features by fusing current frame's visual features with previous memories.rrrrcond_frame_outputsnon_cond_frame_outputsNmaskmem_featurescuda)r~ non_blockingmaskmem_pos_encrrrrr>)currcurr_posmemory memory_posnum_obj_ptr_tokens)/rrTr~rXrr!rrrvvaluestrainingrcrangegetappendrtyperr[rJminrKrQitemsrPabsziprstackrNrOrVtensorrrrrexpand new_zerosreshaperepeat_interleaverWr^r\r]catrR),r" frame_idxis_init_cond_framecurrent_vision_featscurrent_vision_pos_embedsr output_dict num_framestrack_in_reverserCHWr~r tpos_sign_mul to_cat_memoryto_cat_memory_pos_embed cond_outputsselected_cond_outputsunselected_cond_outputsoutt_pos_and_prevsrt_post_relprev_frame_idxprevfeats maskmem_encrKtptr_cond_outputs pos_and_ptrst_diffpos_list ptrs_listobj_ptrs t_diff_maxtpos_dimobj_pospix_feat_with_memrmemory_pos_embeds, r$$_prepare_memory_conditioned_featuresz.SAM2Model._prepare_memory_conditioned_featuress ! $ ) )! , OO"~1%b)00   q '+33Aq!<AA!Q1M M.A !572M{#789A= ==&'; : !#:4I3O3O3QRC3xROR ]](L(LAq$"2"23 5((50A::JY%6PY\aPaN)(11}&:a%?N%3uqyAo%EN*3Q'71'<%=%AN%3uqyAo%EN!":;??PTU;255ndKC&&s|4- 50 / < t</0336PVP[P[_ePe3f$$U]]1%5%=%=aA%FG"#45b9<ANAN3($( (=$#3"8"8": 3 $CC']m;!$Y]!3I   $A'>?FF.> F*IPVDVA1u!7AO%&>?CCAG^GbGbcdfjGklC$++VS^,DE F *-|*<'Hi${{9!CCAq!QO  ASd( s X.X!1X%c|djd}|j}|d\}} |djdddj|||| } |jr|j s|j |}|jxr|} | r+|j s|dkDj| j} ntj|} |jdk7r| |jz} |jdk7r| |jz} |j| | d} | d }| d }|jE|dkDj!}|d|d z |jd j"|j$zz }||fS) zXEncode frame features and masks into a new memory representation for video segmentation.rrrr?rT)skip_mask_sigmoidvision_featuresr).NN)rrTrr!rbr"_apply_non_overlapping_constraintsrarrrrr_r`rUrtrr rW)r"rrpred_masks_high_resris_mask_from_ptsrrrrpix_featbinarize mask_for_mem maskmem_outrrrs r$_encode_new_memoryzSAM2Model._encode_new_memory*s ! $ ) )! , OO"~1'+33Aq!<AA!Q1M  - -dmm#'"I"IJ]"^ ::O?O DMM/!377GL!==)<=L  ) )S 0'$*H*HHL  ( (C /'$*G*GGL))(LTX)Y &'89%&67  $ $ 0 3a 7>>@  %5o%F!FK$JcJcKf&,,K.!. .  00r%c ^t|dkDrft|dd|ddD cgc]H\} } | jdddj| j d| j dg| J}} } nd}|W|j rK|djddd}|jd|j g|d}|j|||}nT|j|||dd|dd|dd|| | }| ||J| }|j||}|j|||||}|||fScc} } w)hPerform a single tracking step, updating object masks and memory features based on current frame inputs.rNrrr)rrrrrrrr)rrrrr) rrrr!rrdrTrr5_use_multimaskr)r"rrrrrrrrrrprev_sam_mask_logitsr0srr= sam_outputsrs r$ _track_stepzSAM2Model._track_stepVs # $q ( 4Sb 9:cr?K!Aq( !Q"''q 166!9AqA! ! !%   "t'P'P,B/771a@H$x}}RJ:b>JH22;J[\K@@##5%9"#%>*CBC*H%bc?'%!1A H$/#/K4GGG2 #223E|T 11"*)'"3!1 2K-x77O!sA D)c|r5|jdkDr&|j|||||du\}} ||d<| |d<yd|d<d|d<y)z^Run memory encoder on predicted mask to encode it into a new memory feature for future frames.rN)rrr;rr<rr)rXrA) r"rrrrun_mem_encoderrr current_outrrs r$_encode_memory_in_outputz"SAM2Model._encode_memory_in_outputsr t//!3040G0G%9%$2$7".d": 1H1 - o/?K* +-s r$ set_binarizezSAM2Model.set_binarizes 2:/r%c|d|_||j_|Dcgc]}|dz c}|j_|j|jz|_ycc}w)r'rr)N)rkrr+r,rlrr.s r$r(zSAM2Model.set_imgszsU(380IN7OAR7O4(,4;O;O(O%8Ps A) ir)r7rFFrFFFrrFFFrFFr)TFFFFFFFFFNF)riboolrnr^ror^rpr^rqr^rsr^rtr^rzr^)NNNF)NN)rz torch.Tensor)F)FTN)r4r5r6r7rr8rpropertyr~ryrurrrrr5rArHrLrOrD staticmethodr:r[r(r9r:s@r$r<r<dszDL NE"%!$+0-2 ""'%* %&+05#(()&+ % "!%"'(-+0 %$)!& %).%*$(&+I}&*.'}:;}<"=}>?}@A}B#'C}D#E}H $I}~.. -9d U n- ^ I0b!H*1X88t2H !':x ";Pr%r<) __future__rrtorch.nn.functionalr functionalr torch.nn.initrultralytics.nn.modulesrultralytics.utilsrblocksr decodersr r encodersr r utilsrrrModulerr<rr%r$rls] # '&$)24= H/ryyH/VePePr%