&Vfq.tddlmZddlmZddlmZddlmZedGddeZdS) )backend)ops) keras_export)Layerzkeras.layers.AttentioncveZdZdZ dfd ZdZdZdd Zd Z dd Z dd Z d Z ddZ fdZ xZS) Attentiona} Dot-product attention layer, a.k.a. Luong-style attention. Inputs are a list with 2 or 3 elements: 1. A `query` tensor of shape `(batch_size, Tq, dim)`. 2. A `value` tensor of shape `(batch_size, Tv, dim)`. 3. A optional `key` tensor of shape `(batch_size, Tv, dim)`. If none supplied, `value` will be used as a `key`. The calculation follows the steps: 1. Calculate attention scores using `query` and `key` with shape `(batch_size, Tq, Tv)`. 2. Use scores to calculate a softmax distribution with shape `(batch_size, Tq, Tv)`. 3. Use the softmax distribution to create a linear combination of `value` with shape `(batch_size, Tq, dim)`. Args: use_scale: If `True`, will create a scalar variable to scale the attention scores. dropout: Float between 0 and 1. Fraction of the units to drop for the attention scores. Defaults to `0.0`. seed: A Python integer to use as random seed incase of `dropout`. score_mode: Function to use to compute attention scores, one of `{"dot", "concat"}`. `"dot"` refers to the dot product between the query and key vectors. `"concat"` refers to the hyperbolic tangent of the concatenation of the `query` and `key` vectors. Call Args: inputs: List of the following tensors: - `query`: Query tensor of shape `(batch_size, Tq, dim)`. - `value`: Value tensor of shape `(batch_size, Tv, dim)`. - `key`: Optional key tensor of shape `(batch_size, Tv, dim)`. If not given, will use `value` for both `key` and `value`, which is the most common case. mask: List of the following tensors: - `query_mask`: A boolean mask tensor of shape `(batch_size, Tq)`. If given, the output will be zero at the positions where `mask==False`. - `value_mask`: A boolean mask tensor of shape `(batch_size, Tv)`. If given, will apply the mask such that values at positions where `mask==False` do not contribute to the result. return_attention_scores: bool, it `True`, returns the attention scores (after masking and softmax) as an additional output argument. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). use_causal_mask: Boolean. Set to `True` for decoder self-attention. Adds a mask such that position `i` cannot attend to positions `j > i`. This prevents the flow of information from the future towards the past. Defaults to `False`. Output: Attention outputs of shape `(batch_size, Tq, dim)`. (Optional) Attention scores after masking and softmax with shape `(batch_size, Tq, Tv)`. FdotNc tjdi|||_||_||_|jdkr%t j||_|jdvrtd|dS)Nrseed)r concatz_Invalid value for argument score_mode. Expected one of {'dot', 'concat'}. Received: score_mode=) super__init__ use_scale score_modedropoutrrandom SeedGeneratorseed_generator ValueError)selfrrrr kwargs __class__s a/var/www/html/software/conda/lib/python3.11/site-packages/keras/src/layers/attention/attention.pyrzAttention.__init__As ""6""""$ ">D">"I"ID  ?"3 3 35(255  4 3c||d|_d|_|jr$|ddd|jd|_|jdkr$|ddd|jd|_d|_dS)NscaleronesT)nameshape initializerdtype trainablerconcat_score_weight)_validate_inputsrr&r add_weightr$rbuiltr input_shapes rbuildzAttention.buildWs k*** #' > "j )DJ ?h & &'+*"j (7((D $ rc|jdkr=tj|tj|gd}|j ||jz}n|jdkrtj|d}tj|d}|j<|jtjtj|j||zzd z}n3|jtjtj||zd z}|S) aCalculates attention scores as a query-key dot product. Args: query: Query tensor of shape `(batch_size, Tq, dim)`. key: Key tensor of shape `(batch_size, Tv, dim)`. Returns: Tensor of shape `(batch_size, Tq, Tv)`. r )r)axesNraxis) rrmatmul transposer expand_dimsr&sumtanh)rquerykeyscores q_reshaped k_reshapeds r_calculate_scoreszAttention._calculate_scoresms ?e # #Zs}Syyy'I'I'IJJFz%$*$ _ ( (R888J2666Jz%1CGHTZ: +BCDD25551CGHZ*455B555 rcZ|Dtj|}|jdkrdnd}||tj||jzz}tj|d}|r7|jdkr,t j||j|j }tj |||fS) aApplies attention scores to the given value tensor. To use this method in your attention layer, follow the steps: * Use `query` tensor of shape `(batch_size, Tq)` and `key` tensor of shape `(batch_size, Tv)` to calculate the attention `scores`. * Pass `scores` and `value` tensors to this method. The method applies `scores_mask`, calculates `attention_distribution = softmax(scores)`, then returns `matmul(attention_distribution, value). * Apply `query_mask` and return the result. Args: scores: Scores float tensor of shape `(batch_size, Tq, Tv)`. value: Value tensor of shape `(batch_size, Tv, dim)`. scores_mask: A boolean mask tensor of shape `(batch_size, 1, Tv)` or `(batch_size, Tq, Tv)`. If given, scores at positions where `scores_mask==False` do not contribute to the result. It must contain at least one `True` value in each line along the last dimension. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). Returns: Tensor of shape `(batch_size, Tq, dim)`. Attention scores after masking and softmax with shape `(batch_size, Tq, Tv)`. Nfloat16g@geAr$r5r2rr ) r logical_notr$castsoftmaxrrrrr6)rr=value scores_masktraining padding_mask max_valueweightss r _apply_scoreszAttention._apply_scoress<  "?;77L$*<9#<#<%I i#(JzAAAI 92666I 92666I+IyAAK!b999v{;;; Mrc||||d}|d}t|dkr|dn|}|r|dnd} |r|dnd} |||} || | |} || || |\} }| 4t j| d} | t j| | j z} |r| |fS| S) Ninputsmaskrr/r.)r;r<)r=rGrHrIr5r2rC) r'lenr@r[rMrr8rEr$)rr^r_rIreturn_attention_scoresrTqvkq_maskrSr=rHresultattention_scoress rcallzAttention.calls V$777 1I 1IVqF1IIa *ad *ad''aQ'7700 FO  $(#5#5 h$6$ $     _V"555F chvV\::: :F " ,++ + rc||||||ddStj|dS)Nr]r)r'rconvert_to_tensor)rr^r_s r compute_maskzAttention.compute_masksB V$777 <47?4$T!W---rcDg|ddd|ddRS)z>Returns shape of value tensor dim, but for query tensor lengthrNr5r/rr*s rcompute_output_shapezAttention.compute_output_shapes+9Q$9k!nR&8999rc|jj}t|tst |d|dt |dkst |dkr"t |dt |d|ht|tst |d|dt |dkst |dkrt |d|d |ddSdS) z'Validates arguments of the call method.zj layer must be called on a list of inputs, namely [query, value] or [query, value, key]. Received: inputs=.r.zl layer accepts inputs list of length 2 or 3, namely [query, value] or [query, value, key]. Received length: NzL layer mask must be a list, namely [query_mask, value_mask]. Received: mask=z< layer accepts mask list of length 2 or 3. Received: inputs=z, mask=)r__name__ isinstancelistrr`)rr^r_ class_names rr'zAttention._validate_inputssa^, &$'' ..$*...  v;;??c&kkAoo33$'KK333   dD))  !OOGKOOO4yy1}}D A  !??(.??7;???   !. rczt}|j|j|jd}i||S)N)rrr)r get_configrrr)r base_configconfigrs rrvzAttention.get_configsEgg((** /|   )+(((r)Fr r N)NF)NFFF)N)rq __module__ __qualname____doc__rr,r@rMr[rhrkrmr'rv __classcell__)rs@rrrs 66t  ,,>,3,3,3,3\4 % :.... :::6)))))))))rrN) keras.srcrrkeras.src.api_exportrkeras.src.layers.layerrrrrrrs------((((((&''R)R)R)R)R)R)R)('R)R)R)r