.hn:ddlmZddlZddlZddlmZmZddlmZGddejZ GddejZ Gd d ejZ y) ) annotationsN)Tensornn)MLPBlockcpeZdZdZej df dfd Z ddZxZS)TwoWayTransformera A Two-Way Transformer module for simultaneous attention to image and query points. This class implements a specialized transformer decoder that attends to an input image using queries with supplied positional embeddings. It's useful for tasks like object detection, image segmentation, and point cloud processing. Attributes: depth (int): Number of layers in the transformer. embedding_dim (int): Channel dimension for input embeddings. num_heads (int): Number of heads for multihead attention. mlp_dim (int): Internal channel dimension for the MLP block. layers (nn.ModuleList): List of TwoWayAttentionBlock layers composing the transformer. final_attn_token_to_image (Attention): Final attention layer from queries to image. norm_final_attn (nn.LayerNorm): Layer normalization applied to final queries. Methods: forward: Process image and point embeddings through the transformer. Examples: >>> transformer = TwoWayTransformer(depth=6, embedding_dim=256, num_heads=8, mlp_dim=2048) >>> image_embedding = torch.randn(1, 256, 32, 32) >>> image_pe = torch.randn(1, 256, 32, 32) >>> point_embedding = torch.randn(1, 100, 256) >>> output_queries, output_image = transformer(image_embedding, image_pe, point_embedding) >>> print(output_queries.shape, output_image.shape) c `t|||_||_||_||_t j|_t|D]/}|jjt||||||dk(1t||||_ t j||_y)ak Initialize a Two-Way Transformer for simultaneous attention to image and query points. Args: depth (int): Number of layers in the transformer. embedding_dim (int): Channel dimension for input embeddings. num_heads (int): Number of heads for multihead attention. Must divide embedding_dim. mlp_dim (int): Internal channel dimension for the MLP block. activation (Type[nn.Module], optional): Activation function to use in the MLP block. attention_downsample_rate (int, optional): Downsampling rate for attention mechanism. r) embedding_dim num_headsmlp_dim activationattention_downsample_rateskip_first_layer_pedownsample_rateN)super__init__depthr r r r ModuleListlayersrangeappendTwoWayAttentionBlock Attentionfinal_attn_token_to_image LayerNormnorm_final_attn) selfrr r r rri __class__s h/var/www/html/ai-service/venv/lib/python3.12/site-packages/ultralytics/models/sam/modules/transformer.pyrzTwoWayTransformer.__init__*s(  *" mmo u A KK  $"/'#).G)*a   *3=)]v)w&!||M:cB|jdjddd}|jdjddd}|}|}|jD]}|||||\}}||z}||z}|j|||} || z}|j |}||fS)a Process image and point embeddings through the Two-Way Transformer. Args: image_embedding (torch.Tensor): Image to attend to, with shape (B, embedding_dim, H, W). image_pe (torch.Tensor): Positional encoding to add to the image, with same shape as image_embedding. point_embedding (torch.Tensor): Embedding to add to query points, with shape (B, N_points, embedding_dim). Returns: queries (torch.Tensor): Processed point embeddings with shape (B, N_points, embedding_dim). keys (torch.Tensor): Processed image embeddings with shape (B, H*W, embedding_dim). r r)querieskeysquery_pekey_peqkv)flattenpermuterrr) rimage_embeddingimage_pepoint_embeddingr&r'layerr+r,attn_outs r"forwardzTwoWayTransformer.forwardTs&*11!4<>> embedding_dim, num_heads = 256, 8 >>> block = TwoWayAttentionBlock(embedding_dim, num_heads) >>> queries = torch.randn(1, 100, embedding_dim) >>> keys = torch.randn(1, 1000, embedding_dim) >>> query_pe = torch.randn(1, 100, embedding_dim) >>> key_pe = torch.randn(1, 1000, embedding_dim) >>> processed_queries, processed_keys = block(queries, keys, query_pe, key_pe) ir Fct|t|||_t j ||_t||||_t j ||_t||||_ t j ||_ t j ||_ t||||_ ||_y)a Initialize a TwoWayAttentionBlock for simultaneous attention to image and query points. This block implements a specialized transformer layer with four main components: self-attention on sparse inputs, cross-attention of sparse inputs to dense inputs, MLP block on sparse inputs, and cross-attention of dense inputs to sparse inputs. Args: embedding_dim (int): Channel dimension of the embeddings. num_heads (int): Number of attention heads in the attention layers. mlp_dim (int, optional): Hidden dimension of the MLP block. activation (Type[nn.Module], optional): Activation function for the MLP block. attention_downsample_rate (int, optional): Downsampling rate for the attention mechanism. skip_first_layer_pe (bool, optional): Whether to skip positional encoding in the first layer. rN)rrr self_attnrrnorm1cross_attn_token_to_imagenorm2rmlpnorm3norm4cross_attn_image_to_tokenr)rr r r rrrr!s r"rzTwoWayAttentionBlock.__init__s0 "=)<\\-0 )2=)]v)w&\\-0 M7J?\\-0 \\-0 )2=)]v)w r#c|jr|j|||}n||z}|j|||}||z}|j|}||z}||z}|j|||}||z}|j |}|j |}||z}|j |}||z}||z}|j|||}||z}|j|}||fS)a Apply two-way attention to process query and key embeddings in a transformer block. Args: queries (torch.Tensor): Query embeddings with shape (B, N_queries, embedding_dim). keys (torch.Tensor): Key embeddings with shape (B, N_keys, embedding_dim). query_pe (torch.Tensor): Positional encodings for queries with same shape as queries. key_pe (torch.Tensor): Positional encodings for keys with same shape as keys. Returns: queries (torch.Tensor): Processed query embeddings with shape (B, N_queries, embedding_dim). keys (torch.Tensor): Processed key embeddings with shape (B, N_keys, embedding_dim). r*) rrFrGrHrIrJrKrMrL) rr&r'r(r)r+r4r,mlp_outs r"r5zTwoWayAttentionBlock.forwards"  # #nnw'WnEG("A~~Q'~:H(G**W% h  6M11Ad1CH$**W%((7#G#**W% h  6M11Ag1Fhzz$}r#)r r6r r6r r6rr7rr6rboolr8r9) r&r:r'r:r(r:r)r:r8r;r<rCs@r"rrsF&(gg)*$)%7%7%7 %7 $ %7 $' %7"%7 %7N,#,+7,CO,Ye, *,r#rcheZdZdZ d dfd ZeddZed dZd dZxZ S) ra An attention layer with downscaling capability for embedding size after projection. This class implements a multi-head attention mechanism with the option to downsample the internal dimension of queries, keys, and values. Attributes: embedding_dim (int): Dimensionality of input embeddings. kv_in_dim (int): Dimensionality of key and value inputs. internal_dim (int): Internal dimension after downsampling. num_heads (int): Number of attention heads. q_proj (nn.Linear): Linear projection for queries. k_proj (nn.Linear): Linear projection for keys. v_proj (nn.Linear): Linear projection for values. out_proj (nn.Linear): Linear projection for output. Methods: _separate_heads: Separate input tensor into attention heads. _recombine_heads: Recombine separated attention heads. forward: Compute attention output for given query, key, and value tensors. Examples: >>> attn = Attention(embedding_dim=256, num_heads=8, downsample_rate=2) >>> q = torch.randn(1, 100, 256) >>> k = v = torch.randn(1, 50, 256) >>> output = attn(q, k, v) >>> print(output.shape) torch.Size([1, 100, 256]) ct|||_||n||_||z|_||_|j|zdk(sJdt j||j|_t j|j|j|_ t j|j|j|_ t j|j||_ y)a? Initialize the Attention module with specified dimensions and settings. Args: embedding_dim (int): Dimensionality of input embeddings. num_heads (int): Number of attention heads. downsample_rate (int, optional): Factor by which internal dimensions are downsampled. kv_in_dim (int | None, optional): Dimensionality of key and value inputs. If None, uses embedding_dim. Raises: AssertionError: If num_heads does not evenly divide the internal dim (embedding_dim / downsample_rate). Nrz$num_heads must divide embedding_dim.) rrr kv_in_dim internal_dimr rLinearq_projk_projv_projout_proj)rr r rrSr!s r"rzAttention.__init__s& *&/&;)_<"  9,1Y3YY1ii t/@/@A ii0A0AB ii0A0AB  $"3"3]C r#ct|j\}}}|j|||||z}|jddS)zGSeparate the input tensor into the specified number of attention heads.r%r )shapereshape transpose)xr bncs r"_separate_headszAttention._separate_heads5s<''1a IIaIqI~ 6{{1a  r#ct|j\}}}}|jdd}|j||||zS)z9Recombine separated attention heads into a single tensor.r%r )r[r]r\)r^r_n_headsn_tokens c_per_heads r"_recombine_headszAttention._recombine_heads<s>,-77(7Hj KK1 yyHg &:;;r#c |j|}|j|}|j|}|j||j}|j||j}|j||j}|j \}}}}||j ddddz}|tj|z }tj|d}||z}|j|}|j|S)a Apply multi-head attention to query, key, and value tensors with optional downsampling. Args: q (torch.Tensor): Query tensor with shape (B, N_q, embedding_dim). k (torch.Tensor): Key tensor with shape (B, N_k, embedding_dim). v (torch.Tensor): Value tensor with shape (B, N_k, embedding_dim). Returns: (torch.Tensor): Output tensor after attention with shape (B, N_q, embedding_dim). rr%r )dim) rVrWrXrbr r[r/mathsqrttorchsoftmaxrgrY)rr+r,r-_rfattnouts r"r5zAttention.forwardCs KKN KKN KKN  DNN 3  DNN 3  DNN 3 gg1a199Q1a((dii ++}}Tr*Qh##C(}}S!!r#)r%N) r r6r r6rr6rSr6r8r9)r^r:r r6r8r:)r^rr8r)r+r:r,r:r-r:r8r:) r=r>r?r@r staticmethodrbrgr5rBrCs@r"rrsyD ! DDD D  D  D>!! << "r#r) __future__rrlrnrrultralytics.nn.modulesrModulerrrr#r"rxsN# +q qhs299slk" k"r#