&VfXddlZddlZddlmZddlmZddlmZddlmZddlm Z ddl m Z ddl m Z dd lmZGd d e Zd Zd ZdZdS)N)backend) initializers)ops)learning_rate_schedule)serialization_lib) KerasSaveable)tracking) auto_nameceZdZ d1dZdZejdZdZe dZ d Z d2d Z d3d Z dZdZdZdZdZdZd4dZdZdZdZdZdZdZe dZejdZdZdZdZd Z d!Z!d"Z"d#Z#d5d$Z$d%Z%d&Z&d'Z'd(Z(d)Z)d*Z*d+Z+d,Z,e-d4d-Z.fd.Z/d4d/Z0d0Z1xZ2S)6 BaseOptimizerNFGz?c  d|_| ddtjd| rt d| | t |jj} | |_||_ ||_ ||_ ||_ ||_ | |_| |_| r| dkst d| |rM|dks|dkrt d ||r-t!|t"r|dkrt d |||_||_t)d |||fD} | dkrt d |d |d|d|_g|_g|_t1jdd|jfi|_i|_t9j|j|5t9jddddd}dddn #1swxYwY||||_ t!|tBj"r ||_#dStI|r ||_#dSt!|tJst d|t9j|j|5t9j|dt9j&dd}dddn #1swxYwY||||_#dS)NFdecayz= 2. Received: gradient_accumulation_steps=rzC`ema_momentum` must be in the range [0, 1]. Received: ema_momentum=z]`ema_overwrite_frequency` must be an integer >= 1 or None. Received: ema_overwrite_frequency=c3K|]}|duV dSN).0as `/var/www/html/software/conda/lib/python3.11/site-packages/keras/src/optimizers/base_optimizer.py z)BaseOptimizer.__init__..Ms7  ATM      zYOnly one of `clipnorm`, `clipvalue` and `global_clipnorm` can be set. Received: clipnorm=z , clipvalue=z, global_clipnorm= variablesc6t|tjSr) isinstancerVariable)xs rz(BaseOptimizer.__init__..^sjG,<==rcaller iterationintonly_first_replica)namedtype trainable aggregationzArgument `learning_rate` should be float, or an instance of LearningRateSchedule, or a callable (that takes in the current iteration value and returns the corresponding learning rate value). Received instead: learning_rate= learning_rate)'_lockpopwarningswarn ValueErrorr __class____name__r% weight_decayclipnormglobal_clipnorm clipvalueuse_emaloss_scale_factorgradient_accumulation_stepsrr# ema_momentumema_overwrite_frequencysumbuilt _variables_trainable_variablesr Tracker_tracker_trainable_variables_indicesr name_scoper_track_variable iterationsrLearningRateSchedule_learning_ratecallablefloatfloatx)selfr)r1r2r4r3r5r8r9r6r7r%kwargs clip_args_sumrCs r__init__zBaseOptimizer.__init__sQ ::gt $ $ 0 MN     FDFDDEE E <T^455D (  ." !2+F( & .!33 5255  a$  $,i#I      1  L.6LL&LL:ILL   $&! (==O    -/)   $ 7 7 7   ) 0 J                Z((($  1F   0#0D    m $ $ 0"/D   mU33  G8E GG#DId;;;   ' 0!(!.**# 4 !!!                   / / /"/D   s$$G  GG.+J%%J),J)c<|jd|dS)Nr)r? add_to_storerIvariables rrBzBaseOptimizer._track_variables  "";99999rc|jrg|_|jrg|_t |D]\}}||j||<|jr/|j||d|jr/|j||d|dd|_ d|_ dS)Naverage)r%gradient_accumulatorT) r5_model_variables_moving_averager7_accumulated_gradients enumerater@_var_keyappendadd_variable_from_referencer=r;)rIrirPs rbuildzBaseOptimizer.builds < 635D 0  + -*,D '$Y//  KAxIJD -dmmH.E.E F| 4;;44 &5 / +2244 35 %.aaaL! rc t|Sr)idrOs rrWzBaseOptimizer._var_keys(||rc |jddSr)r<rIs rrzBaseOptimizer.variablessqqq!!rcB|j||Sr)r@rWrOs r_get_variable_indexz!BaseOptimizer._get_variable_indexs0x1H1HIIrzerosmeanc |tj|}tj|j|5tj|||d||}dddn #1swxYwY|||S)Nr F) initializershaper&r'r(r%)_check_super_calledrgetrrAr%rrB)rIrfrer&r(r%rPs r add_variablezBaseOptimizer.add_variables   """"&{33   $ 7 7 7  ''' H                X&&&sA++A/2A/c8|pd}t|dr"|jdddz|z}nBt|jdddddz|z}||j||j|S)z\Add an all-zeros variable with the shape and dtype of a reference variable. varpath/_:)rfrer&r%)hasattrrlreplacestrr%rirfr&)rIreference_variabler%res rrYz)BaseOptimizer.add_variable_from_references }u %v . . %*223<>FFsCPP    $*#$* !   rcn|D]1}|||jvrtd|d2dS)NzUnknown variable: z. This optimizer can only be called for the variables it was originally built with. When working with a new set of variables, you should recreate a new optimizer instance.)rWr@r.)rIrvs r_check_variables_are_knownz(BaseOptimizer._check_variables_are_knownsc  A}}Qt'HHH 9999I  rc0||dS)aAssign a value to a variable. This should be used in optimizers instead of `variable.assign(value)` to support backend specific optimizations. Note that the variable can be a model variable or an optimizer variable; it can be a backend native variable or a Keras variable. Args: variable: The variable to update. value: The value to add to the variable. N)assignrIrPvalues rrxzBaseOptimizer.assigns rc0||dS)aAdd a value to a variable. This should be used in optimizers instead of `variable.assign_add(value)` to support backend specific optimizations. Note that the variable can be a model variable or an optimizer variable; it can be a backend native variable or a Keras variable. Args: variable: The variable to update. value: The value to add to the variable. N) assign_addrys rr|zBaseOptimizer.assign_add E"""""rc0||dS)aSubtract a value from a variable. This should be used in optimizers instead of `variable.assign_sub(value)` to support backend specific optimizations. Note that the variable can be a model variable or an optimizer variable; it can be a backend native variable or a Keras variable. Args: variable: The variable to update. value: The value to add to the variable. N) assign_subrys rrzBaseOptimizer.assign_subr}rctr)NotImplementedError)rIgradientrPr)s r update_stepzBaseOptimizer.update_steps!!rcTt|\}}||||jSr)zipapplyrC)rIgrads_and_varsgradstrainable_variabless rapply_gradientszBaseOptimizer.apply_gradientss/%(.%9"" 5-...rcZt|dkrdS|x|jstdt|t|jkr5tdt|dt|jd|j}nzt |}|jsOt j|j|5| |dddn #1swxYwYd|_| |t j|j|5| ||\}}| ||\}}tt |dkr ddddS|j fd |D}||}||||||D]1}|j(|||2 ddddS#1swxYwYdS) a3Update traininable variables according to provided gradient values. `grads` should be a list of gradient tensors with 1:1 mapping to the list of variables the optimizer was built with. `trainable_variables` can be provided on the first call to build the optimizer. rNzWhen passing `grads` without `variables`, the optimizer must already be built on a list of variables. Call `optimizer.build(trainable_variables)` first. zWhen passing `grads` as a list of gradient tensors, the gradients must match `optimizer.variables` one-to-on. Received a list of z* gradients, but the optimizer is tracking z trainable variables.r Tc"g|] }||n|z  Srr)rgscales r z'BaseOptimizer.apply..Xs%FFF1aiQYFFFr)lenr;r.r@r=listrrAr%r[rv,_overwrite_variables_directly_with_gradients_filter_empty_gradientsr6_clip_gradients_apply_weight_decay_backend_apply_gradients constraintrx)rIrrrPrs @rrzBaseOptimizer.applys@ u::?? F  &:  J 5zzS!BCCCC +*-e**++.11J-K-K+++#'";  "&':";"; : "' $???44JJ2333444444444444444!  + +,? @ @ @   $ 7 7 7 C CAA. 'E&*.)E)E*** &E&4;;1$$ C C C C C C C C"*E FFFFFFF((//E  $ $%8 9 9 9  ) )%1D E E E/ C C&2OOH$7$7$A$ABBB C9 C C C C C C C C C C C C C C C C C Cs,C$$C(+C(&AH B H  H$'H$cjrKjdzjzdk}fdDfdtj|fdfdnjjrQjj r0jdzj zdk}tj|fddj dd S) arApply method that can be overridden by different backends. JAX overrides it in order to deal with statelessness in gradient accumulation and EMA handling. The below implementation is intended to be generally backend-agnostic, but may not work with all backends. This method does 4 things: - Call the optimizer's update_step() to update trainable variables and optimizer variables. - Update EMA variables, if EMA is configured. - Update gradient accumulators, if gradient accumulation is configured. - Update the iteration counter. rrcPg|]"}j|#Sr)rUra)rrurIs rrz:BaseOptimizer._backend_apply_gradients..|s?+D,D,DQ,G,GHrcjfdt|D}||jdS)Nc&g|] \}}||zz Srr)rracc_gstepss rrzSBaseOptimizer._backend_apply_gradients.._update_step_fn..s3,4AuQY%'r)r7r_backend_update_stepr)$_backend_reset_gradient_accumulators)rrr acc_gradsrIs @r_update_step_fnz?BaseOptimizer._backend_apply_gradients.._update_step_fns}88;E98M8M)).0B99;;;;;rcSrr)rrrsrrz8BaseOptimizer._backend_apply_gradients..s/BCCrc0Sr)(_backend_increment_gradient_accumulators)rrrIsrrz8BaseOptimizer._backend_apply_gradients..sEE9rc8jSr)-_overwrite_model_variables_with_average_valuer=r_srrz8BaseOptimizer._backend_apply_gradients..sDNN1rcdSrrrrrrz8BaseOptimizer._backend_apply_gradients..sDrN) r7rCrcondrr)r5&_update_model_variables_moving_averager=r9r|)rIrris_update_stepshould_overwrite_model_varsrrs``` @@rrz&BaseOptimizer._backend_apply_gradientses  +" !#01456N ,I < < < < < < HCCCCCC      % %*D,>    <   7 7)   + Oa'0/145/6+/!L  ""1%%%%%rc`t||D]\}}||||dS)zCollective update_step that can be overridden by the backend. It is overridden by torch for performance reasons, and by TF to support tf.distribute. N)rr)rIrrr)gradrks rrz"BaseOptimizer._backend_update_stepsG U$788 7 7ID#   T3 6 6 6 6 7 7rc|jD]5}|tj|j|j6dSN)r&)rUrxrrbrfr&)rIg_accs rrz2BaseOptimizer._backend_reset_gradient_accumulatorssJ0 D DE LL5;ekBBB C C C C D Drcdt||D}t||D]\}}||dS)Ncg|] \}}||z Srr)rrrs rrzJBaseOptimizer._backend_increment_gradient_accumulators..s HHHhaq5yHHHr)rrx)rIrr new_g_accsn_g_accrs rrz6BaseOptimizer._backend_increment_gradient_accumulatorss[HH#eY2G2GHHH !*i88 " "NGU LL ! ! ! ! " "rc B||jstd|jjdt |t |jkrBtd|jjdt |dt |jdt |t |jkr5tdt |dt |jdtt|j|tt|j|z}tj | 5}| |dddn #1swxYwYg}|jD]D}| |}|||/||Eg}|jD]D}| |}|||/||E||fS) NzTo call `stateless_apply`, zx must be built (i.e. its variables must have been created). You can build it via `optimizer.build(trainable_variables)`.zNArgument `optimizer_variables` must be a list of tensors corresponding 1:1 to z(().variables. Received list with length z, but expected z variables.zArgument `optimizer_variables` must be a list of tensors corresponding 1:1 to the trainable variables list that the optimizer was built with. Received len(trainable_variables) == z& whereas the optimizer was built with ) state_mapping)rgr;r.r/r0rrr=rrrStatelessScoperget_current_valuerX)rIoptimizer_variablesrrmappingscoperunew_vs rstateless_applyzBaseOptimizer.stateless_applys   """z Odn.EOOO  " # #s4>':': : :=(,(?==-01D-E-E== //===  " # #s4+D'E'E E E?033F/G/G?? t011 ???  )+> ? ?  T^%899 : :;  #' : : : e JJu                  !* . .A++A..E #**51111#**1----  . .A++A..E #**51111#**1----"$777sE66E:=E:c(|j ||jzS|S)zScale the loss before computing gradients. Scales the loss before gradients are computed in a `train_step`. This is primarily useful during mixed precision training to prevent numeric underflow. )r6)rIlosss r scale_losszBaseOptimizer.scale_losss   ! -$00 0 rc*|Sr)_get_current_learning_rater_s rr)zBaseOptimizer.learning_rates..000rct|jtjr|j}nd}t|tjr||_n_t |r||_nHt|jtjrtd|j||6t|jtjs| |dSdSdS)NaThis optimizer was created with a `LearningRateSchedule` object as its `learning_rate` constructor argument, hence its learning rate is not settable. If you need the learning rate to be settable, you should instantiate the optimizer with a float `learning_rate` argument.) rrErrrrDrF TypeErrorrx_untrack_variable)rIr) prev_lr_vars rr)zBaseOptimizer.learning_rates d)7+; < < -KKK  1F   6#0D   m $ $ 6"/D  #%;%P  K   & &} 5 5 5  ":  !1, , "  " "; / / / / / # " " "rc R|jstdt|j|D]z\}}|j|jkrPtd||dt |jdt |jd||{dS)z!Set the weights of the optimizer.zYou are calling `set_weights()` on an optimizer that has not yet been built. Please call `optimizer.build(trainable_variables)` to create the optimizer weights before calling `set_weights()`.zOptimizer variable z has shape z+ not compatible with provided weight shape .N)r;r.rr<rfrWrrrx)rIweightsrPweights r set_weightszBaseOptimizer.set_weights sz D  !$DOW = = $ $ Hf~-- 9$--*A*A998>**99$' $5$5999 OOF # # # # $ $rct|jD])\}}||t|<*dS)z'Get the state of this optimizer object.N)rVrnumpyrr)rIstorerZrPs rsave_own_variablesz BaseOptimizer.save_own_variables2sD$T^44 - -KAx$NN,,E#a&&MM - -rc t|t|jkrwd|jdt|jdt|d}t|jdkr|dz }t j|dd St |jD]-\}}||t|.d S) z'Set the state of this optimizer object.z)Skipping variable loading for optimizer 'z', because it has z+ variables whereas the saved optimizer has z variables. rzCThis is likely because the optimizer has not been called/built yet.r) stacklevelN) rkeysrr%r,r-rVrxrr)rIrmsgrZrPs rload_own_variablesz BaseOptimizer.load_own_variables7s uzz||  DN 3 3 3 3KDIKK"%dn"5"5KK+.uzz||+<+<KKK  4>""a''( M#! , , , , F$T^44 + +KAx OOE#a&&M * * * * + +rct|jtjr||jSt |jr|S|jSr)rrErrDrCrFr_s rrz(BaseOptimizer._get_current_learning_rateIsf   !7!L   )&&t77 7 d) * * )&&(( (""rc t|dds||fSt|}t|}tt|dz ddD]}||||c jr|jr|j}|jdz|zdk}|j|  tj | fd fd}tj | fd fd tj | fd  fd } |  |n | || |||fS) a.Overwrite the variables directly by their gradients. This method is designed for a special case where we want to overwrite the variable directly with its computed gradient. For example, in float8 training, new `scale` and `amax_history` are computed as gradients, and we want to overwrite them directly instead of following the typical procedure such as gradient descent with a learning rate, gradient clipping and weight decaying. After the update, the processed pairs will be filtered out. roverwrite_with_gradientrcDtjjjSr)rrbrfr&rsrrzLBaseOptimizer._overwrite_variables_directly_with_gradients..vs !' A A Arc.tjSrrmaximumrrsrrzLBaseOptimizer._overwrite_variables_directly_with_gradients..w Au 5 5rc.tjSrrrsrrzLBaseOptimizer._overwrite_variables_directly_with_gradients..{rrcSrrrsrrzLBaseOptimizer._overwrite_variables_directly_with_gradients..|srcSrr)new_gsrrzLBaseOptimizer._overwrite_variables_directly_with_gradients..srcjSr)rz)rusrrzLBaseOptimizer._overwrite_variables_directly_with_gradients..s qwr) rprrangerrr7rCrUrarrrxr+)rIrvarsfiltered_grads filtered_varsrZrr new_g_accrrrrrus @@@@rrz:BaseOptimizer._overwrite_variables_directly_with_gradientsRstAw 9:: $; eT s>**Q.B77 % %A!!$mA&6DAq( %3  **Q.B77 1 1Aa (""1%%%!%%a((!((000 HFGG G   M#!23344###    },,rcjrjdkrfd|DSjr jdkrt|jSjr jdkrjfd|DS|S)NrcBg|]}||n|Sr) _clip_by_norm)rrrIs rrz1BaseOptimizer._clip_gradients..s=BC""1%%%ArcFg|]}|tj| n|Sr)rclip)rrrus rrz1BaseOptimizer._clip_gradients..s2NNN1!-CHQA&&&QNNNr)r2r3clip_by_global_normr4)rIrrus` @rrzBaseOptimizer._clip_gradientss = T]Q..GL  ! d&:Q&>&>&ud.BCC C ^  2 2ANNNNNNN NLrctdrjrtd|r!tfd|D_nt_|rMt |dkr:t jdt|_ nd_ t_ dS)aExclude variables from weight decay. This method must be called before the optimizer's `build` method is called. You can set specific variables to exclude out, or set a list of strings as the anchor words, if any of which appear in a variable's name, then the variable is excluded. Args: var_list: A list of `Variable`s to exclude from weight decay. var_names: A list of strings. If any string in `var_names` appear in the model variable's name, then this model variable is excluded from weight decay. For example, `var_names=['bias']` excludes all bias variables from weight decay. _builtzR`exclude_from_weight_decay()` can only be configued before the optimizer is built.c3BK|]}|VdSr)rW)rrPrIs rrz:BaseOptimizer.exclude_from_weight_decay..s@22,4 h''222222rr|N) rprr.set_exclude_from_weight_decayrrecompilejoin"_exclude_from_weight_decay_patterndict _exclude_from_weight_decay_cache)rIvar_list var_namess` rexclude_from_weight_decayz'BaseOptimizer.exclude_from_weight_decays 4 " " t{ *   4.122228@222//D + +/2eeD +  ;Y!++68jY((77D 3 37;D 315---rc||}t|dst|_||jvr |j|St |dt }t |dd}||vr d|j|<dS|'t j||j d|j|<dSd|j|<dS)NrrrFT) rWrprrgetattrrrsearchr%)rIrP variable_idr!exclude_from_weight_decay_patterns r_use_weight_decayzBaseOptimizer._use_weight_decaysmmH-- t?@@ ;48FFD 1 $? ? ?8E E%, .% % !-4 6- - ) 3 3 3AFD 1+ >5 , 8 ;X]KKFK5kBu=A-k:trc|jdS|D]s}||r\tj|j|j}tj|j|j}||||z|zz tdSr)r1rrcastr)r&rx)rIrrPlrwds rrz!BaseOptimizer._apply_weight_decays   $ F! ? ?H%%h// ?Xd0(.AAXd/@@8b=2+= =>>>  ? ?rc`t|dstd|jjddS)Nr*zIn optimizer 'zh', you forgot to call `super().__init__()` as the first statement in the `__init__()` method. Go add it!)rp RuntimeErrorr/r0r_s rrgz!BaseOptimizer._check_super_calledsKtW%% !8   rc|jrxt||jD]d\}}tj|jd}tj||j|jz}| ||zd|z |zzcdSdS)z8Update the stored moving average using the latest value.rrN) r5rrTr not_equalrCr r&r8rx)rIrrkrRnot_first_stepmomentums rrz4BaseOptimizer._update_model_variables_moving_averages < J ##T%I!! J J W"%t!B!BH^SY77$:KKx'1Q\S4HHIIII J J J Jrct|t|jkr5tdt|dt|jdt||jD]\}}||dS)z2Overwrite model variables with its moving average.zThe length of model variables (zT) to override does not match the length of model variables stored in the optimizer (z:). Please check if the optimizer was called on your model.N)rrTr.rrx)rIrrk average_vars rrz;BaseOptimizer._overwrite_model_variables_with_average_values " # #s  0( (   C#6I2J2JCC<==CCC !$ !E! !  $ $ C JJ{ # # # # $ $rcB|jr||dSdS)aSet the final value of model's trainable variables. Sometimes there are some extra steps before ending the variable updates, such as overriding the model variables with its average value. Args: var_list: list of model variables. N)r5r)rIrs rfinalize_variable_valuesz&BaseOptimizer.finalize_variable_values#s7 < I  > >x H H H H H  I IrcdS)N Optimizerrr_s r _obj_typezBaseOptimizer._obj_type2s{rc :t|jtjrtj|j}nt|jt jr't|j}n[tj |jrt|j}n-t|jrtj |j}|j||j|j|j|j|j|j|j|j|jd }|S)aReturns the config of the optimizer. An optimizer config is a Python dictionary (serializable) containing the configuration of an optimizer. The same optimizer can be reinstantiated later (without any saved state) from this configuration. Subclass optimizer should override this method to include other hyperparameters. Returns: Python dictionary. ) r%r)r1r2r3r4r5r8r9r6r7)rrErrD serializerrrGrr is_tensorrFrserialize_keras_objectr%r1r2r3r4r5r8r9r6r7)rIr)configs r get_configzBaseOptimizer.get_config5s   !7!L  3<#MM+W-= > > !$"5";";"="=>>MM ]4. / / !$"566MM d) * * -D#M I* - #3| -'+'C!%!7+/+K    rcd|vr:t|dtrtj|d||d<|di|S)aCreates an optimizer from its config. This method is the reverse of `get_config`, capable of instantiating the same optimizer from the config dictionary. Args: config: A Python dictionary, typically the output of get_config. custom_objects: A Python dictionary mapping names to additional user-defined Python objects needed to recreate this optimizer. Returns: An optimizer instance. r))custom_objectsr)rrrdeserialize_keras_object)clsrr s r from_configzBaseOptimizer.from_configbsc f $ $&1488 %>/' s}}V}}rc|dkr|t|dr|j|}t ||S)Nr*r?)rgrpr?tracksuper __setattr__)rIr%rzr/s rr'zBaseOptimizer.__setattr__zsa 7??  $ $ & & & 4 $ $ /M''..Eww""4///rctjtj||d}|dk}tj||tj|}tj|tj||}tj||j}tj|tj ||jz }|S)NT)keepdimsr) rr:squarewhere ones_likesqrtmultiplyr2convert_to_tensorr) rIvaluesaxesl2sumpred l2sum_safel2norm intermediate values_clips rrzBaseOptimizer._clip_by_norms 6**D4@@@qyYtUCM%,@,@AA 4*!5!5u==|FDM:: +L99CK DM= =  rc|jj}|j|j||dur|jdSdS)NT)r?lockedunlockuntracklock)rIrPprevious_lock_states rrzBaseOptimizer._untrack_variablesd"m2  h''' $ & & M    ' &r) NNNNFr NNNN)rbNrcN)Nrbr)NN)3r0 __module__ __qualname__rLrBr no_automatic_dependency_trackingr[rWpropertyrrarirYrvrxr|rrrrrrrrrrr)setterrrrrrrrrrrrgrrrrr classmethodr#r'rr __classcell__)r/s@rr r sk $$( }0}0}0}0~:::./.0""X"JJJ   .:A    ,    # # # # # #""" ECECECECNF&F&F&P777DDD""" 282828h   11X1000:$$$$--- +++$###5-5-5-n---.   &7&7&7&7P:??? J J J$$$& I I I+++Z[. 0 0 0 0 0    !!!!!!!rr aR name: String. The name to use for momentum accumulator weights created by the optimizer. weight_decay: Float. If set, weight decay is applied. clipnorm: Float. If set, the gradient of each weight is individually clipped so that its norm is no higher than this value. clipvalue: Float. If set, the gradient of each weight is clipped to be no higher than this value. global_clipnorm: Float. If set, the gradient of all weights is clipped so that their global norm is no higher than this value. use_ema: Boolean, defaults to `False`. If `True`, exponential moving average (EMA) is applied. EMA consists of computing an exponential moving average of the weights of the model (as the weight values change after each training batch), and periodically overwriting the weights with their moving average. ema_momentum: Float, defaults to 0.99. Only used if `use_ema=True`. This is the momentum to use when computing the EMA of the model's weights: `new_average = ema_momentum * old_average + (1 - ema_momentum) * current_variable_value`. ema_overwrite_frequency: Int or None, defaults to None. Only used if `use_ema=True`. Every `ema_overwrite_frequency` steps of iterations, we overwrite the model variable by its moving average. If None, the optimizer does not overwrite model variables in the middle of training, and you need to explicitly overwrite the variables at the end of training by calling `optimizer.finalize_variable_values()` (which updates the model variables in-place). When using the built-in `fit()` training loop, this happens automatically after the last epoch, and you don't need to do anything. loss_scale_factor: Float or `None`. If a float, the scale factor will be multiplied the loss before computing gradients, and the inverse of the scale factor will be multiplied by the gradients before updating variables. Useful for preventing underflow during mixed precision training. Alternately, `keras.optimizers.LossScaleOptimizer` will automatically set a loss scale factor. gradient_accumulation_steps: Int or `None`. If an int, model & optimizer variables will not be updated at every step; instead they will be updated every `gradient_accumulation_steps` steps, using the average value of the gradients since the last update. This is known as "gradient accumulation". This can be useful when your batch size is very small, in order to reduce gradient noise at each update step. cd|D}tjtj|}tj|S)z-Computes the global norm of multiple tensors.c^g|]*}|tjtj|+Sr)rr:r*)rrus rrzglobal_norm..s2#$am 1 mmmr)rr:stackr-) value_list squared_norms squared_norms r global_normrKsK(2M739]3344L 8L ! !!rct|}|tjd|z d|z z}|||z zfd|DS)Ng?c"g|] }||zn| Srr)rrurs rrz'clip_by_global_norm..s% B B B!AIIA B B Br)rKrminimum)rH clip_normuse_normscale_for_finiters @rrrsZ:&&H 3;sX~sY#O#OO 8 3 4E B B B Bz B B BBr)rr, keras.srcrrrkeras.src.optimizers.schedulesrkeras.src.savingrkeras.src.saving.keras_saveablerkeras.src.utilsr keras.src.utils.namingr r base_optimizer_keyword_argsrKrrrrrYs """"""AAAAAA......999999$$$$$$,,,,,,I!I!I!I!I!MI!I!I!X.b"""CCCCCr