&VfhddlmZddlmZddlmZedGddeZdS))ops) keras_export) Attentionzkeras.layers.AdditiveAttentionc>eZdZdZ dfd ZdZdZfdZxZS) AdditiveAttentionaV Additive attention layer, a.k.a. Bahdanau-style attention. Inputs are a list with 2 or 3 elements: 1. A `query` tensor of shape `(batch_size, Tq, dim)`. 2. A `value` tensor of shape `(batch_size, Tv, dim)`. 3. A optional `key` tensor of shape `(batch_size, Tv, dim)`. If none supplied, `value` will be used as `key`. The calculation follows the steps: 1. Calculate attention scores using `query` and `key` with shape `(batch_size, Tq, Tv)` as a non-linear sum `scores = reduce_sum(tanh(query + key), axis=-1)`. 2. Use scores to calculate a softmax distribution with shape `(batch_size, Tq, Tv)`. 3. Use the softmax distribution to create a linear combination of `value` with shape `(batch_size, Tq, dim)`. Args: use_scale: If `True`, will create a scalar variable to scale the attention scores. dropout: Float between 0 and 1. Fraction of the units to drop for the attention scores. Defaults to `0.0`. Call Args: inputs: List of the following tensors: - `query`: Query tensor of shape `(batch_size, Tq, dim)`. - `value`: Value tensor of shape `(batch_size, Tv, dim)`. - `key`: Optional key tensor of shape `(batch_size, Tv, dim)`. If not given, will use `value` for both `key` and `value`, which is the most common case. mask: List of the following tensors: - `query_mask`: A boolean mask tensor of shape `(batch_size, Tq)`. If given, the output will be zero at the positions where `mask==False`. - `value_mask`: A boolean mask tensor of shape `(batch_size, Tv)`. If given, will apply the mask such that values at positions where `mask==False` do not contribute to the result. return_attention_scores: bool, it `True`, returns the attention scores (after masking and softmax) as an additional output argument. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). use_causal_mask: Boolean. Set to `True` for decoder self-attention. Adds a mask such that position `i` cannot attend to positions `j > i`. This prevents the flow of information from the future towards the past. Defaults to `False`. Output: Attention outputs of shape `(batch_size, Tq, dim)`. (Optional) Attention scores after masking and softmax with shape `(batch_size, Tq, Tv)`. Tc @tjd||d|dS)N) use_scaledropout)super__init__)selfr r kwargs __class__s j/var/www/html/software/conda/lib/python3.11/site-packages/keras/src/layers/attention/additive_attention.pyrzAdditiveAttention.__init__<s/ H9gHHHHHHHc|||dd}d|_|jr%|d|gd|jd|_d|_dS)Nrscaleglorot_uniformT)nameshape initializerdtype trainable)_validate_inputsrr add_weightrbuilt)r input_shapedims rbuildzAdditiveAttention.buildDso k***!nR  > e,j )DJ rctj|d}tj|d}|jr|jnd}tj|tj||zzdS)aCalculates attention scores as a nonlinear sum of query and key. Args: query: Query tensor of shape `(batch_size, Tq, dim)`. key: Key tensor of shape `(batch_size, Tv, dim)`. Returns: Tensor of shape `(batch_size, Tq, Tv)`. )axisg?r)r expand_dimsr rsumtanh)rquerykey q_reshaped k_reshapedrs r_calculate_scoresz#AdditiveAttention._calculate_scoresRsg_U444 _Sr222 "n5 #wusx Z(?@@@rJJJJrcNt}|d=|S)N score_mode)r get_config)r base_configrs rr1zAdditiveAttention.get_configds&gg((**  %r)Tr) __name__ __module__ __qualname____doc__rr"r.r1 __classcell__)rs@rrrs22lIIIIII   KKK$rrN) keras.srcrkeras.src.api_exportr$keras.src.layers.attention.attentionrrr rrr;s------::::::.//````` ``0/```r