.hdddlmZddlZddlmZddlmZmZGddejZGddejZ y) ) annotationsN)nn)MLP LayerNorm2dceZdZdZdej ddf dfd Z ddZ d dZxZ S) MaskDecodera Decoder module for generating masks and their associated quality scores using a transformer architecture. This class predicts masks given image and prompt embeddings, utilizing a transformer to process the inputs and generate mask predictions along with their quality scores. Attributes: transformer_dim (int): Channel dimension for the transformer module. transformer (nn.Module): Transformer module used for mask prediction. num_multimask_outputs (int): Number of masks to predict for disambiguating masks. iou_token (nn.Embedding): Embedding for the IoU token. num_mask_tokens (int): Number of mask tokens. mask_tokens (nn.Embedding): Embedding for the mask tokens. output_upscaling (nn.Sequential): Neural network sequence for upscaling the output. output_hypernetworks_mlps (nn.ModuleList): Hypernetwork MLPs for generating masks. iou_prediction_head (nn.Module): MLP for predicting mask quality. Methods: forward: Predict masks given image and prompt embeddings. predict_masks: Internal method for mask prediction. Examples: >>> decoder = MaskDecoder(transformer_dim=256, transformer=transformer_module) >>> masks, iou_pred = decoder( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, multimask_output=True ... ) >>> print(f"Predicted masks shape: {masks.shape}, IoU predictions shape: {iou_pred.shape}") c t|||_||_||_t j d||_|dz|_t j |j||_ t jt j||dzddt|dz|t j|dz|dzdd||_ t jt|jDcgc]}t!|||dzdc}|_t!|||j||_ycc}w)a Initialize the MaskDecoder module for generating masks and their associated quality scores. Args: transformer_dim (int): Channel dimension for the transformer module. transformer (nn.Module): Transformer module used for mask prediction. num_multimask_outputs (int): Number of masks to predict for disambiguating masks. activation (Type[nn.Module]): Type of activation to use when upscaling masks. iou_head_depth (int): Depth of the MLP used to predict mask quality. iou_head_hidden_dim (int): Hidden dimension of the MLP used to predict mask quality. Examples: >>> transformer = nn.TransformerEncoder(nn.TransformerEncoderLayer(d_model=256, nhead=8), num_layers=6) >>> decoder = MaskDecoder(transformer_dim=256, transformer=transformer) >>> print(decoder)  kernel_sizestrider N)super__init__transformer_dim transformernum_multimask_outputsr Embedding iou_tokennum_mask_tokens mask_tokens SequentialConvTranspose2droutput_upscaling ModuleListrangeroutput_hypernetworks_mlpsiou_prediction_head) selfrrr activationiou_head_depthiou_head_hidden_dim_ __class__s e/var/www/html/ai-service/venv/lib/python3.12/site-packages/ultralytics/models/sam/modules/decoders.pyrzMaskDecoder.__init__)s$2 .&%:"a94q8<<(<(>> decoder = MaskDecoder(transformer_dim=256, transformer=transformer_module) >>> image_emb = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_emb = torch.rand(1, 2, 256) >>> dense_emb = torch.rand(1, 256, 64, 64) >>> masks, iou_pred = decoder(image_emb, image_pe, sparse_emb, dense_emb, multimask_output=True) >>> print(f"Masks shape: {masks.shape}, IoU predictions shape: {iou_pred.shape}") )image_embeddingsimage_pesparse_prompt_embeddingsdense_prompt_embeddingsr Nr) predict_masksslice) r#r+r,r-r.multimask_outputmasksiou_pred mask_slices r)forwardzMaskDecoder.forwardYsk>,,-%=$; - x(8U1d^U1a[ aQ)*AzM*hc tj|jj|jjgd}|j dj |jddd}tj||fd}tj||jdd}||z}tj||jdd}|j\} } } } |j|||\} }| dddddf}| dddd|jzddf}|jddj| | | | }|j|}t|jDcgc]!}|j||dd|ddf#}}tj |d}|j\} } } } ||j| | | | zzj| d| | }|j#|}||fScc}w)z`Predict masks and quality scores using image and prompt embeddings via transformer architecture.rdimr Nr)torchcatrweightr unsqueezeexpandshaperepeat_interleaverr transposeviewrr r!stackr")r#r+r,r-r. output_tokenstokenssrcpos_srcbchwhs iou_token_outmask_tokens_outupscaled_embeddingi hyper_in_listhyper_inr2r3s r)r/zMaskDecoder.predict_maskss 4>>#8#8$:J:J:Q:Q"RXYZ %//299:R:X:XYZ:[]_acd M+CD!L%%&6 QQO++))(FLLOKYY 1a""38C1a7 QQ)=)=%= >ABmmAq!&&q!Q2!2237QVW[WkWkQl- LM -D * *1 -oaAg.F G- - ;;}!4'-- 1a.33Aq!a%@@FFq"aQRS++M:h- s3&H)rintr nn.ModulerrTr$type[nn.Module]r%rTr&rTreturnNone) r+ torch.Tensorr,rYr-rYr.rYr1boolrW!tuple[torch.Tensor, torch.Tensor]) r+rYr,rYr-rYr.rYrWr[) __name__ __module__ __qualname____doc__rGELUrr5r/ __classcell__r(s@r)rr sB&'&(gg#&.s.s.s # .s $ .s  .s!.s .s`+&++#/ + ". +  + ++Z%&%%#/ % ". % + %r6rc eZdZdZdej ddddddddddf d fd Z d ddZ d dd Zd Z d Z xZ S)SAM2MaskDecodera Transformer-based decoder for predicting instance segmentation masks from image and prompt embeddings. This class extends the functionality of the MaskDecoder, incorporating additional features such as high-resolution feature processing, dynamic multimask output, and object score prediction. Attributes: transformer_dim (int): Channel dimension of the transformer. transformer (nn.Module): Transformer used to predict masks. num_multimask_outputs (int): Number of masks to predict when disambiguating masks. iou_token (nn.Embedding): Embedding for IOU token. num_mask_tokens (int): Total number of mask tokens. mask_tokens (nn.Embedding): Embedding for mask tokens. pred_obj_scores (bool): Whether to predict object scores. obj_score_token (nn.Embedding): Embedding for object score token. use_multimask_token_for_obj_ptr (bool): Whether to use multimask token for object pointer. output_upscaling (nn.Sequential): Upscaling layers for output. use_high_res_features (bool): Whether to use high-resolution features. conv_s0 (nn.Conv2d): Convolutional layer for high-resolution features (s0). conv_s1 (nn.Conv2d): Convolutional layer for high-resolution features (s1). output_hypernetworks_mlps (nn.ModuleList): List of MLPs for output hypernetworks. iou_prediction_head (MLP): MLP for IOU prediction. pred_obj_score_head (nn.Linear | MLP): Linear layer or MLP for object score prediction. dynamic_multimask_via_stability (bool): Whether to use dynamic multimask via stability. dynamic_multimask_stability_delta (float): Delta value for dynamic multimask stability. dynamic_multimask_stability_thresh (float): Threshold for dynamic multimask stability. Methods: forward: Predict masks given image and prompt embeddings. predict_masks: Predict instance segmentation masks from image and prompt embeddings. _get_stability_scores: Compute mask stability scores based on IoU between thresholds. _dynamic_multimask_via_stability: Dynamically select the most stable mask output. Examples: >>> image_embeddings = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_prompt_embeddings = torch.rand(1, 2, 256) >>> dense_prompt_embeddings = torch.rand(1, 256, 64, 64) >>> decoder = SAM2MaskDecoder(256, transformer) >>> masks, iou_pred, sam_tokens_out, obj_score_logits = decoder.forward( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, True, False ... ) r r Fg?g\(\?c 4t|||_||_||_t j d||_|dz|_t j |j||_ | |_ |jrt j d||_ ||_ t jt j||dzddt|dz|t j|dz|dzdd||_||_|rBt j$||dzdd|_t j$||dzdd|_t j*t-|jDcgc]}t/|||dzdc}|_t/|||j|||_|jr0t j4|d|_| rt/||dd|_| |_| |_| |_ycc}w) a Initialize the SAM2MaskDecoder module for predicting instance segmentation masks. This decoder extends the functionality of MaskDecoder, incorporating additional features such as high-resolution feature processing, dynamic multimask output, and object score prediction. Args: transformer_dim (int): Channel dimension of the transformer. transformer (nn.Module): Transformer used to predict masks. num_multimask_outputs (int): Number of masks to predict when disambiguating masks. activation (Type[nn.Module]): Type of activation to use when upscaling masks. iou_head_depth (int): Depth of the MLP used to predict mask quality. iou_head_hidden_dim (int): Hidden dimension of the MLP used to predict mask quality. use_high_res_features (bool): Whether to use high-resolution features. iou_prediction_use_sigmoid (bool): Whether to use sigmoid for IOU prediction. dynamic_multimask_via_stability (bool): Whether to use dynamic multimask via stability. dynamic_multimask_stability_delta (float): Delta value for dynamic multimask stability. dynamic_multimask_stability_thresh (float): Threshold for dynamic multimask stability. pred_obj_scores (bool): Whether to predict object scores. pred_obj_scores_mlp (bool): Whether to use MLP for object score prediction. use_multimask_token_for_obj_ptr (bool): Whether to use multimask token for object pointer. Examples: >>> transformer = nn.TransformerEncoder(nn.TransformerEncoderLayer(d_model=256, nhead=8), num_layers=6) >>> decoder = SAM2MaskDecoder(transformer_dim=256, transformer=transformer) >>> print(decoder) r r rrrr )sigmoidN)rrrrrrrrrrpred_obj_scoresobj_score_tokenuse_multimask_token_for_obj_ptrrrrruse_high_res_featuresConv2dconv_s0conv_s1rr rr!r"Linearpred_obj_score_headdynamic_multimask_via_stability!dynamic_multimask_stability_delta"dynamic_multimask_stability_thresh)r#rrrr$r%r&rjiou_prediction_use_sigmoidrprqrrrgpred_obj_scores_mlprir'r(s r)rzSAM2MaskDecoder.__init__sX .&%:"a94q8<<(<(>> image_embeddings = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_prompt_embeddings = torch.rand(1, 2, 256) >>> dense_prompt_embeddings = torch.rand(1, 256, 64, 64) >>> decoder = SAM2MaskDecoder(256, transformer) >>> masks, iou_pred, sam_tokens_out, obj_score_logits = decoder.forward( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, True, False ... ) )r+r,r-r. repeat_imagehigh_res_featuresNr r)r/rptraining _dynamic_multimask_via_stabilityri) r#r+r,r-r.r1rvrwr2r3rOobject_score_logitssam_tokens_outs r)r5zSAM2MaskDecoder.forward8sLAE@R@R-%=$;%/ ASA =x*= !QRA+&E12H  1 1$--"CCE8TOE8!QqS!Q,'E1Q3'H  D D,QU3N-Q!V4Nh0CCCr6c d}|jrYtj|jj|j j|j jgd}d}nAtj|j j|j jgd}|jdj|jddd}tj||fd} |r&tj|| jdd} n#|jd| jdk(sJ|} | |z} |jddk(sJdtj|| jdd} | j\} } }}|j| | | \}} |dd|ddf}|dd|dz|dz|jzddf}| jddj| | ||} |jr||j!| }n?|j \}}}}}|\}}|||| |z}||||z}t#|jDcgc]!}|j$||dd|ddf#}}tj&|d}|j\} } }}||j| | ||zzj| d||}|j)|}|jr#|dk(sJ|j+|dddddf} n"d|j-|jddz} |||| fScc}w) zYPredict instance segmentation masks from image and prompt embeddings using a transformer.rr8r r:z@image_pe should have size 1 in batch dim (from `get_dense_pe()`)Nrg$@)rgr;r<rhr=rrr>r?r@rArrrBrCrjrr r!rDr"ronew_ones)!r#r+r,r-r.rvrwsrErFrGrHrIrJrKrLrMrNrOrPdc1ln1act1dc2act2feat_s0feat_s1rQrRrSr2r3rzs! r)r/zSAM2MaskDecoder.predict_masks}sd    !II((//NN))$$++  MA!IIt~~'<'N>N>U>U&V\]^M%//299:R:X:XYZ:[]_acd M+CD!L ))*:FLLOQRSC#))!, Q? ??"C++~~a A%i'ii%))(FLLOKYY 1a""38C1a7 QAQ1E1E)E FIJmmAq!&&q!Q2))->-F!%!6!6s!; (,(=(= %CdC0 GW!%c#c(W*<&=!> !%c*<&=&G!H RWW[WkWkQl- LM -D * *1 -oaAg.F G- - ;;}!4'-- 1a.33Aq!a%@@FFq"aQRS++M:   6M6"&":":2aAg;"G #'):):8>>!;La)P"P h1DDD!- s-&Mc|jd}|j}tj||kDdj }tj|| kDdj }tj |dkD||z dS)zNCompute mask stability scores based on IoU between upper and lower thresholds.r:r8rg?)flattenrqr;sumfloatwhere)r# mask_logitsstability_deltaarea_iarea_us r)_get_stability_scoresz%SAM2MaskDecoder._get_stability_scoressw!))"- @@;8bAGGI;/)99rBHHJ{{6A:v<>> decoder = SAM2MaskDecoder(...) >>> all_mask_logits = torch.rand(2, 4, 256, 256) # 2 images, 4 masks each >>> all_iou_scores = torch.rand(2, 4) >>> mask_logits, iou_scores = decoder._dynamic_multimask_via_stability(all_mask_logits, all_iou_scores) >>> print(mask_logits.shape, iou_scores.shape) torch.Size([2, 1, 256, 256]) torch.Size([2, 1]) Nr r:r8r)device).NN) r;argmaxaranger@rr>rrrr expand_as)r#all_mask_logitsall_iou_scoresmultimask_logitsmultimask_iou_scoresbest_scores_inds batch_indsbest_multimask_logitsbest_multimask_iou_scoressinglemask_logitssinglemask_iou_scoresstability_scores is_stablemask_logits_outiou_scores_outs r)ryz0SAM2MaskDecoder._dynamic_multimask_via_stabilitysL6+1ab!Q;7-ae4 <<(<"E\\"6"<"rs8# 3`"))`FS/biiS/r6