.hzdZddlmZddlZddlZddlmZddlmcmZ ddl m Z m Z ddl mZddlmZddlmZmZmZd ZGd d ej,ZGd d eZGddej,ZGddej,ZGddej,ZGddej,ZGddej,ZGddej,ZGddej,ZGddej,Z y)zTransformer modules.) annotationsN) constant_xavier_uniform_) TORCH_1_11)Conv) _get_clonesinverse_sigmoid#multi_scale_deformable_attn_pytorch) TransformerEncoderLayerTransformerLayerTransformerBlockMLPBlock LayerNorm2dAIFIDeformableTransformerDecoder!DeformableTransformerDecoderLayer MSDeformAttnMLPceZdZdZdddej df d fd Zed d dZ d ddZ d dd Z d dd Z xZ S)r a A single layer of the transformer encoder. This class implements a standard transformer encoder layer with multi-head attention and feedforward network, supporting both pre-normalization and post-normalization configurations. Attributes: ma (nn.MultiheadAttention): Multi-head attention module. fc1 (nn.Linear): First linear layer in the feedforward network. fc2 (nn.Linear): Second linear layer in the feedforward network. norm1 (nn.LayerNorm): Layer normalization after attention. norm2 (nn.LayerNorm): Layer normalization after feedforward network. dropout (nn.Dropout): Dropout layer for the feedforward network. dropout1 (nn.Dropout): Dropout layer after attention. dropout2 (nn.Dropout): Dropout layer after feedforward network. act (nn.Module): Activation function. normalize_before (bool): Whether to apply normalization before attention and feedforward. Fct|ddlm}|s t dt j |||d|_t j|||_ t j|||_ t j||_ t j||_ t j||_t j||_t j||_||_||_y)a Initialize the TransformerEncoderLayer with specified parameters. Args: c1 (int): Input dimension. cm (int): Hidden dimension in the feedforward network. num_heads (int): Number of attention heads. dropout (float): Dropout probability. act (nn.Module): Activation function. normalize_before (bool): Whether to apply normalization before attention and feedforward. ) TORCH_1_9z]TransformerEncoderLayer() requires torch>=1.9 to use nn.MultiheadAttention(batch_first=True).T)dropout batch_firstN)super__init__utils.torch_utilsrModuleNotFoundErrornnMultiheadAttentionmaLinearfc1fc2 LayerNormnorm1norm2Dropoutrdropout1dropout2actnormalize_before) selfc1cm num_headsrr/r0r __class__s `/var/www/html/ai-service/venv/lib/python3.12/site-packages/ultralytics/nn/modules/transformer.pyr z TransformerEncoderLayer.__init__4s( 2%o ''IwTXY99R$99R$\\"% \\"% zz'*  7+  7+  0c||S||zS)z2Add position embeddings to the tensor if provided.tensorposs r6with_pos_embedz&TransformerEncoderLayer.with_pos_embed]v6&3,6r7c l|j||x}}|j|||||d}||j|z}|j|}|j |j |j |j|}||j|z}|j|S)a Perform forward pass with post-normalization. Args: src (torch.Tensor): Input tensor. src_mask (torch.Tensor, optional): Mask for the src sequence. src_key_padding_mask (torch.Tensor, optional): Mask for the src keys per batch. pos (torch.Tensor, optional): Positional encoding. Returns: (torch.Tensor): Output tensor after attention and feedforward. value attn_maskkey_padding_maskr) r=r%r-r*r(rr/r'r.r+)r1srcsrc_masksrc_key_padding_maskr<qksrc2s r6 forward_postz$TransformerEncoderLayer.forward_postbs&##C--Awwq!3(MawbcdeDMM$''jjoxx TXXdhhsm%<=>DMM$''zz#r7c l|j|}|j||x}}|j|||||d}||j|z}|j |}|j |j |j|j|}||j|zS)a Perform forward pass with pre-normalization. Args: src (torch.Tensor): Input tensor. src_mask (torch.Tensor, optional): Mask for the src sequence. src_key_padding_mask (torch.Tensor, optional): Mask for the src keys per batch. pos (torch.Tensor, optional): Positional encoding. Returns: (torch.Tensor): Output tensor after attention and feedforward. r@r) r*r=r%r-r+r(rr/r'r.)r1rDrErFr<rIrGrHs r6 forward_prez#TransformerEncoderLayer.forward_pre}s&zz###D#..Awwq!48NbwcdefDMM$''zz#xx TXXdhhtn%=>?T]]4(((r7cj|jr|j||||S|j||||S)a Forward propagate the input through the encoder module. Args: src (torch.Tensor): Input tensor. src_mask (torch.Tensor, optional): Mask for the src sequence. src_key_padding_mask (torch.Tensor, optional): Mask for the src keys per batch. pos (torch.Tensor, optional): Positional encoding. Returns: (torch.Tensor): Output tensor after transformer encoder layer. )r0rLrJ)r1rDrErFr<s r6forwardzTransformerEncoderLayer.forwards=&  ##C3GM M  h0DcJJr7 r2intr3rPr4rPrfloatr/ nn.Moduler0boolNr; torch.Tensorr<torch.Tensor | NonereturnrVNNN) rDrVrErWrFrWr<rWrXrV) __name__ __module__ __qualname____doc__r#GELUr staticmethodr=rJrLrN __classcell__r5s@r6r r s;, !&'1 '1 '1 '1  '1  '1'1R77)-48#'  &2  !   <)-48#' ) )&)2 ) ! )  )<)-48#' K K&K2 K ! K  Kr7r ceZdZdZdddej df d fd Zd fd Ze d d dZ xZ S) rz AIFI transformer layer for 2D data with positional embeddings. This class extends TransformerEncoderLayer to work with 2D feature maps by adding 2D sine-cosine positional embeddings and handling the spatial dimensions appropriately. rrrFc.t|||||||y)a Initialize the AIFI instance with specified parameters. Args: c1 (int): Input dimension. cm (int): Hidden dimension in the feedforward network. num_heads (int): Number of attention heads. dropout (float): Dropout probability. act (nn.Module): Activation function. normalize_before (bool): Whether to apply normalization before attention and feedforward. N)rr )r1r2r3r4rr/r0r5s r6r z AIFI.__init__s( RGS:JKr7cd|jdd\}}}|j|||}t| |j dj ddd|j |j|j}|j dddjd|||gjS)z Forward pass for the AIFI transformer layer. Args: x (torch.Tensor): Input tensor with shape [B, C, H, W]. Returns: (torch.Tensor): Output tensor with shape [B, C, H, W]. rNr)devicedtype)r<) shape"build_2d_sincos_position_embeddingrrNflattenpermutetorfrgview contiguous)r1xchw pos_embedr5s r6rNz AIFI.forwards''!"+1a;;Aq!D GOAIIaL00Aq9y||STS[S[cdcjcj|?kO lyyAq!&&Aq!}5@@BBr7c|dzdk(sJdtj|tj}tj|tj}trtj||dntj||\}}|dz}tj|tj|z }d||zz }|j d|d z}|j d|d z} tj tj|tj|tj| tj| gd d S) a Build 2D sine-cosine position embedding. Args: w (int): Width of the feature map. h (int): Height of the feature map. embed_dim (int): Embedding dimension. temperature (float): Temperature for the sine/cosine functions. Returns: (torch.Tensor): Position embedding with shape [1, embed_dim, h*w]. rzHEmbed dimension must be divisible by 4 for 2D sin-cos position embeddingrgij)indexingg?.NNr) torcharangefloat32rmeshgridrkcatsincos) rsrr embed_dim temperaturegrid_wgrid_hpos_dimomegaout_wout_hs r6rjz'AIFI.build_2d_sincos_position_embeddings 1}!m#mm!au}}5au}}5JTFZ_ZhZhioqwZxq. WEMM:WD{E)* +eDk9 +eDk9yy%))E*EIIe,NPUPYPYZ_P`acdefjkkr7rOrprVrXrV)g@) rsrPrrrPrrPrrQrXrV) rZr[r\r]r#r^r rNr_rjr`ras@r6rrs !&L L L L  L  LL,C CJl ll#&l;@l llr7rc,eZdZdZdfd ZddZxZS)r zeTransformer layer https://arxiv.org/abs/2010.11929 (LayerNorm layers removed for better performance).c|t|tj||d|_tj||d|_tj||d|_tj|||_tj||d|_ tj||d|_ y)z Initialize a self-attention mechanism using linear transformations and multi-head attention. Args: c (int): Input and output channel dimension. num_heads (int): Number of attention heads. F)bias)rr4N) rr r#r&rGrHvr$r%r'r()r1rqr4r5s r6r zTransformerLayer.__init__s 1ae,1ae,1ae,''!yI99Q.99Q.r7c|j|j||j||j|d|z}|j |j ||zS)z Apply a transformer block to the input x and return the output. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Output tensor after transformer layer. r)r%rGrHrr(r'r1rps r6rNzTransformerLayer.forwardsT GGDFF1Itvvay$&&) 4Q 7! ;xx $q((r7)rqrPr4rPrrZr[r\r]r rNr`ras@r6r r so/ )r7r c,eZdZdZdfd ZddZxZS)ra Vision Transformer block based on https://arxiv.org/abs/2010.11929. This class implements a complete transformer block with optional convolution layer for channel adjustment, learnable position embedding, and multiple transformer layers. Attributes: conv (Conv, optional): Convolution layer if input and output channels differ. linear (nn.Linear): Learnable position embedding. tr (nn.Sequential): Sequential container of transformer layers. c2 (int): Output channel dimension. ct|d|_|k7rt||_t j |_t jfdt|D|_ |_ y)aU Initialize a Transformer module with position embedding and specified number of heads and layers. Args: c1 (int): Input channel dimension. c2 (int): Output channel dimension. num_heads (int): Number of attention heads. num_layers (int): Number of transformer layers. Nc36K|]}tywrT)r ).0_c2r4s r6 z,TransformerBlock.__init__..:s!]a"22y"A!]s) rr convrr#r&linear Sequentialrangetrr)r1r2rr4 num_layersr5s `` r6r zTransformerBlock.__init__+s`  8R DIiiB' --!]5Q[K\!]^r7cB|j|j|}|j\}}}}|jdjddd}|j ||j |zjdddj ||j||S)z Forward propagate the input through the transformer block. Args: x (torch.Tensor): Input tensor with shape [b, c1, w, h]. Returns: (torch.Tensor): Output tensor with shape [b, c2, w, h]. rerr)rrirkrlrrreshaper)r1rpbrrsrrps r6rNzTransformerBlock.forward=s 99  ! AWW 1a IIaL Aq )wwq4;;q>)*221a;CCAtwwPQSTUUr7)r2rPrrPr4rPrrPrrras@r6rrs $Vr7rcDeZdZdZej fdfd ZddZxZS)rz+A single block of a multi-layer perceptron.ct|tj|||_tj|||_||_y)a  Initialize the MLPBlock with specified embedding dimension, MLP dimension, and activation function. Args: embedding_dim (int): Input and output dimension. mlp_dim (int): Hidden dimension. act (nn.Module): Activation function. N)rr r#r&lin1lin2r/)r1 embedding_dimmlp_dimr/r5s r6r zMLPBlock.__init__Qs= IImW5 IIg}5 5r7c`|j|j|j|S)z Forward pass for the MLPBlock. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Output tensor after MLP block. )rr/rrs r6rNzMLPBlock.forward_s$yy$))A,/00r7)rrPrrPr) rZr[r\r]r#r^r rNr`ras@r6rrNs5=?WW  1r7rcXeZdZdZej df dfd ZddZxZS)ra A simple multi-layer perceptron (also called FFN). This class implements a configurable MLP with multiple linear layers, activation functions, and optional sigmoid output activation. Attributes: num_layers (int): Number of layers in the MLP. layers (nn.ModuleList): List of linear layers. sigmoid (bool): Whether to apply sigmoid to the output. act (nn.Module): Activation function. Fct|||_|g|dz z}tjdt |g|z||gzD|_||_||_y)a Initialize the MLP with specified input, hidden, output dimensions and number of layers. Args: input_dim (int): Input dimension. hidden_dim (int): Hidden dimension. output_dim (int): Output dimension. num_layers (int): Number of layers. act (nn.Module): Activation function. sigmoid (bool): Whether to apply sigmoid to the output. rc3NK|]\}}tj||ywrT)r#r&)rnrHs r6rzMLP.__init__..s#g1BIIaO#gs#%N) rr rr# ModuleListziplayerssigmoidr/) r1 input_dim hidden_dim output_dimrr/rrrr5s r6r z MLP.__init__zse $ LJN +mm#gYKRSOUVZdYeUe@f#gg  5r7ct|jD]J\}}||jdz kr+t|dt j ||n||}Lt|ddr|j S|S)z Forward pass for the entire MLP. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Output tensor after MLP. rr/rF) enumeraterrgetattrr#ReLUr)r1rpilayers r6rNz MLP.forwardsx"$++. cHAu=>STAT=T/eRWWY/a9Z_`aZbA c%dIu=qyy{D1Dr7) rrPrrPrrPrrPrrSr) rZr[r\r]r#rr rNr`ras@r6rrlsK VXU\U\ns*-;>LOgk* Er7rc.eZdZdZddfd ZddZxZS)rau 2D Layer Normalization module inspired by Detectron2 and ConvNeXt implementations. This class implements layer normalization for 2D feature maps, normalizing across the channel dimension while preserving spatial dimensions. Attributes: weight (nn.Parameter): Learnable scale parameter. bias (nn.Parameter): Learnable bias parameter. eps (float): Small constant for numerical stability. References: https://github.com/facebookresearch/detectron2/blob/main/detectron2/layers/batch_norm.py https://github.com/facebookresearch/ConvNeXt/blob/main/models/convnext.py ct|tjt j ||_tjt j||_||_ y)z Initialize LayerNorm2d with the given parameters. Args: num_channels (int): Number of channels in the input. eps (float): Small constant for numerical stability. N) rr r# Parameterr{onesweightzerosreps)r1 num_channelsrr5s r6r zLayerNorm2d.__init__sI ll5::l#;< LL\!:; r7c|jdd}||z jdjdd}||z tj||jzz }|j ddddf|z|j ddddfzS)z Perform forward pass for 2D layer normalization. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Normalized output tensor. rTkeepdimreN)meanpowr{sqrtrrr)r1rpuss r6rNzLayerNorm2d.forwards FF1dF # UKKN  4  0 UejjTXX. .{{1dD=)A- !T4-0HHHr7)gư>)rrPrrQrrras@r6rrs  Ir7rcNeZdZdZddfd ZdZ d ddZxZS) ra Multiscale Deformable Attention Module based on Deformable-DETR and PaddleDetection implementations. This module implements multiscale deformable attention that can attend to features at multiple scales with learnable sampling locations and attention weights. Attributes: im2col_step (int): Step size for im2col operations. d_model (int): Model dimension. n_levels (int): Number of feature levels. n_heads (int): Number of attention heads. n_points (int): Number of sampling points per attention head per feature level. sampling_offsets (nn.Linear): Linear layer for generating sampling offsets. attention_weights (nn.Linear): Linear layer for generating attention weights. value_proj (nn.Linear): Linear layer for projecting values. output_proj (nn.Linear): Linear layer for projecting output. References: https://github.com/fundamentalvision/Deformable-DETR/blob/main/models/ops/modules/ms_deform_attn.py ct|||zdk7rtd|d|||z}||z|k(sJdd|_||_||_||_||_tj|||z|zdz|_ tj|||z|z|_ tj|||_ tj|||_ |jy)aG Initialize MSDeformAttn with the given parameters. Args: d_model (int): Model dimension. n_levels (int): Number of feature levels. n_heads (int): Number of attention heads. n_points (int): Number of sampling points per attention head per feature level. rz.d_model must be divisible by n_heads, but got z and z(`d_model` must be divisible by `n_heads`@reN)rr ValueError im2col_stepd_modeln_levelsn_headsn_pointsr#r&sampling_offsetsattention_weights value_proj output_proj_reset_parameters)r1rrrr _d_per_headr5s r6r zMSDeformAttn.__init__s  W  !MgYV[\c[def f( W$/[1[[/      " '7X3E3PST3T U!#7Gh4F4Q!R))GW599Wg6  r7cHt|jjjdt j |j tjdtjz|j z z}t j|j|jgd}||jjdddz j|j ddd j!d|j"|j$d}t'|j$D]}|d d d d |d d fxx|dzzcc<t j(5t+j,|jd|j_d d d t|j0jjdt|j0j.jdt3|j4jjt|j4j.jdt3|j6jjt|j6j.jdy #1swYxYw) zReset module parameters.rrwg@rhTrrrreN)rrrdatar{r|rr}mathpistackrrabsmaxrnrepeatrrrno_gradr#rrrrrr)r1thetas grid_initrs r6rzMSDeformAttn._reset_parameterss$''..33S9dll%--@C$''MTXT`T`D`aKKvzz| qA A T$,,1a ( VAt}}dmmQ 7  t}}% +A aAqj !QU * ! + ]]_ J)+innR6H)ID ! ! & J$((//44c:$((--22C8..334$//&&++S1((//445$""'',,c2 J Js 4JJ!c |jdd\}}|jd}td|D|k(sJ|j|}||j|dt d}|j |||j |j|j z}|j|j |||j |j|jd} |j|j |||j |j|jz} tj| dj |||j |j|j} |jd} | dk(rdtj||j |j"j%d} | | ddddddddfz } |ddddddddddf| z}nQ| d k(r=| |jz |ddddddddddfzd z} |ddddddddddf| z}nt'd | d t)|||| }|j+|S) a Perform forward pass for multiscale deformable attention. Args: query (torch.Tensor): Query tensor with shape [bs, query_length, C]. refer_bbox (torch.Tensor): Reference bounding boxes with shape [bs, query_length, n_levels, 2], range in [0, 1], top-left (0,0), bottom-right (1, 1), including padding area. value (torch.Tensor): Value tensor with shape [bs, value_length, C]. value_shapes (list): List with shape [n_levels, 2], [(H_0, W_0), (H_1, W_1), ..., (H_{L-1}, W_{L-1})]. value_mask (torch.Tensor, optional): Mask tensor with shape [bs, value_length], True for non-padding elements, False for padding elements. Returns: (torch.Tensor): Output tensor with shape [bs, Length_{query}, C]. References: https://github.com/PaddlePaddle/PaddleDetection/blob/develop/ppdet/modeling/transformers/deformable_transformer.py Nrerc32K|]}|d|dzyw)rrNr9)rrs r6rz'MSDeformAttn.forward..2s511Q4!A$;5srzrrh)rgrfrvg?z5Last dim of reference_points must be 2 or 4, but got .)risumr masked_fillrQrnrrrrrrFsoftmaxr{ as_tensorrgrffliprr r)r1query refer_bboxrA value_shapes value_maskbslen_qlen_vrr num_pointsoffset_normalizeraddsampling_locationsoutputs r6rNzMSDeformAttn.forwardsH4KKO E A5 55>>>&  !%%j&;U1XFE 2udllDLLDLL4PQ007<>r5$,,X\XeXehlhuhuXuvII&7<AA"eT\\[_[h[hjnjwjwx%%b) ? % EKKX]XdXd e j jkm n "%6tT4DRS7S%TTC!+Aq$4,B!Cc!I  1_"T]]2Z1dAtUVUW@W5XX[^^C!+Aq$4!,C!Ds!J TU_T``abc c4ULJ\^op''r7)rrvrrv)rrPrrPrrPrrPrT) rrVrrVrArVrlistrrWrXrV)rZr[r\r]r rrNr`ras@r6rrsV*!>36+/ 1(1(!1( 1(  1( ( 1( 1(r7rceZdZdZddddej ddf d fd Zed dZd d Z d dd Z xZ S)ra; Deformable Transformer Decoder Layer inspired by PaddleDetection and Deformable-DETR implementations. This class implements a single decoder layer with self-attention, cross-attention using multiscale deformable attention, and a feedforward network. Attributes: self_attn (nn.MultiheadAttention): Self-attention module. dropout1 (nn.Dropout): Dropout after self-attention. norm1 (nn.LayerNorm): Layer normalization after self-attention. cross_attn (MSDeformAttn): Cross-attention module. dropout2 (nn.Dropout): Dropout after cross-attention. norm2 (nn.LayerNorm): Layer normalization after cross-attention. linear1 (nn.Linear): First linear layer in the feedforward network. act (nn.Module): Activation function. dropout3 (nn.Dropout): Dropout in the feedforward network. linear2 (nn.Linear): Second linear layer in the feedforward network. dropout4 (nn.Dropout): Dropout after the feedforward network. norm3 (nn.LayerNorm): Layer normalization after the feedforward network. References: https://github.com/PaddlePaddle/PaddleDetection/blob/develop/ppdet/modeling/transformers/deformable_transformer.py https://github.com/fundamentalvision/Deformable-DETR/blob/main/models/deformable_transformer.py rrirrvcht|tj||||_tj ||_tj||_t|||||_ tj ||_ tj||_ tj|||_||_tj ||_tj|||_tj ||_tj||_y)a Initialize the DeformableTransformerDecoderLayer with the given parameters. Args: d_model (int): Model dimension. n_heads (int): Number of attention heads. d_ffn (int): Dimension of the feedforward network. dropout (float): Dropout probability. act (nn.Module): Activation function. n_levels (int): Number of feature levels. n_points (int): Number of sampling points. )rN)rr r#r$ self_attnr,r-r)r*r cross_attnr.r+r&linear1r/dropout3linear2dropout4norm3) r1rrd_ffnrr/rrr5s r6r z*DeformableTransformerDecoderLayer.__init__ds, ..wQ 7+ \\'* 'w'8L 7+ \\'* yy%0  7+ yy0  7+ \\'* r7c||S||zS)z;Add positional embeddings to the input tensor, if provided.r9r:s r6r=z0DeformableTransformerDecoderLayer.with_pos_embedr>r7c |j|j|j|j|}||j |z}|j |S)z Perform forward pass through the Feed-Forward Network part of the layer. Args: tgt (torch.Tensor): Input tensor. Returns: (torch.Tensor): Output tensor after FFN. )rrr/rrr)r1tgttgt2s r6 forward_ffnz-DeformableTransformerDecoderLayer.forward_ffnsN||DMM$((4<<3D*EFGDMM$''zz#r7c|j||x}} |j|jdd| jdd|jdd|djdd} ||j| z}|j |}|j |j|||j d|||} ||j| z}|j|}|j|S)aH Perform the forward pass through the entire decoder layer. Args: embed (torch.Tensor): Input embeddings. refer_bbox (torch.Tensor): Reference bounding boxes. feats (torch.Tensor): Feature maps. shapes (list): Feature shapes. padding_mask (torch.Tensor, optional): Padding mask. attn_mask (torch.Tensor, optional): Attention mask. query_pos (torch.Tensor, optional): Query position embeddings. Returns: (torch.Tensor): Output tensor after decoder layer. rr)rBre) r=r transposer-r*r unsqueezer.r+r ) r1embedrfeatsshapes padding_maskrB query_posrGrHrs r6rNz)DeformableTransformerDecoderLayer.forwards4##E955AnnQ[[A. Aq0A5??STVWCXdmnn  )Aq/  c** 5!oo   y 1:3G3G3JESY[g  c** 5!&&r7)rrPrrPrrPrrQr/rRrrPrrPrU)rrVrXrVrY)rrVrrVrrVrrrrWrBrWrrWrXrV) rZr[r\r]r#rr r_r=r rNr`ras@r6rrJs6 (+(+(+ (+  (+  (+(+(+T77 (-1)-)-)')'!)' )'  )' * )'')'')' )'r7rcVeZdZdZddfd Z d ddZxZS)rav Deformable Transformer Decoder based on PaddleDetection implementation. This class implements a complete deformable transformer decoder with multiple decoder layers and prediction heads for bounding box regression and classification. Attributes: layers (nn.ModuleList): List of decoder layers. num_layers (int): Number of decoder layers. hidden_dim (int): Hidden dimension. eval_idx (int): Index of the layer to use during evaluation. References: https://github.com/PaddlePaddle/PaddleDetection/blob/develop/ppdet/modeling/transformers/deformable_transformer.py ct|t|||_||_||_|dk\r||_y||z|_y)aU Initialize the DeformableTransformerDecoder with the given parameters. Args: hidden_dim (int): Hidden dimension. decoder_layer (nn.Module): Decoder layer module. num_layers (int): Number of decoder layers. eval_idx (int): Index of the layer to use during evaluation. rN)rr r rrreval_idx)r1r decoder_layerrrr5s r6r z%DeformableTransformerDecoder.__init__sD !-< $$$,M zH7L r7c |} g} g} d} |j}t|jD]\}}|| |||| |||} ||| }tj|t |z}|j rb| j ||| |dk(r| j |nm| j tj|t | zn<||jk(r-| j ||| | j |n#|} |j r|jn|}tj| tj| fS)a Perform the forward pass through the entire decoder. Args: embed (torch.Tensor): Decoder embeddings. refer_bbox (torch.Tensor): Reference bounding boxes. feats (torch.Tensor): Image features. shapes (list): Feature shapes. bbox_head (nn.Module): Bounding box prediction head. score_head (nn.Module): Score prediction head. pos_mlp (nn.Module): Position MLP. attn_mask (torch.Tensor, optional): Attention mask. padding_mask (torch.Tensor, optional): Padding mask. Returns: dec_bboxes (torch.Tensor): Decoded bounding boxes. dec_cls (torch.Tensor): Decoded classification scores. Nr) rrrr{r trainingappendrdetachr)r1rrrr bbox_head score_headpos_mlprBrr dec_bboxesdec_clslast_refined_bboxrrbbox refined_bboxs r6rNz$DeformableTransformerDecoder.forwardsN<  '') !$++. RHAu6:uflIW^_iWjkF9Q<'D == 0K)KLL}}}z!}V456%%l3%%emmD?K\;]4]&^_dmm#}z!}V45!!,/ , 26--,,.\J% R({{:& G(<<r,s" 44TT MKbiiMK`Il "IlX)ryy)B.Vryy.Vb1ryy1