&VfgddlZddlZddlmZddlmZddlmZddlm Z ddl m Z ddl m Z ddl mZdd lmZed Gd d e Zed GddeZdS)N)backend)tree) keras_export)Layer)serialization_lib) jax_utils)tracking)jaxzkeras.layers.JaxLayerceZdZdZ d fd ZdZejdZdZ dZ dZ dd Z fd Z efd ZxZS)JaxLayera Keras Layer that wraps a JAX model. This layer enables the use of JAX components within Keras when using JAX as the backend for Keras. ## Model function This layer accepts JAX models in the form of a function, `call_fn`, which must take the following arguments with these exact names: - `params`: trainable parameters of the model. - `state` (*optional*): non-trainable state of the model. Can be omitted if the model has no non-trainable state. - `rng` (*optional*): a `jax.random.PRNGKey` instance. Can be omitted if the model does not need RNGs, neither during training nor during inference. - `inputs`: inputs to the model, a JAX array or a `PyTree` of arrays. - `training` (*optional*): an argument specifying if we're in training mode or inference mode, `True` is passed in training mode. Can be omitted if the model behaves the same in training mode and inference mode. The `inputs` argument is mandatory. Inputs to the model must be provided via a single argument. If the JAX model takes multiple inputs as separate arguments, they must be combined into a single structure, for instance in a `tuple` or a `dict`. ## Model weights initialization The initialization of the `params` and `state` of the model can be handled by this layer, in which case the `init_fn` argument must be provided. This allows the model to be initialized dynamically with the right shape. Alternatively, and if the shape is known, the `params` argument and optionally the `state` argument can be used to create an already initialized model. The `init_fn` function, if provided, must take the following arguments with these exact names: - `rng`: a `jax.random.PRNGKey` instance. - `inputs`: a JAX array or a `PyTree` of arrays with placeholder values to provide the shape of the inputs. - `training` (*optional*): an argument specifying if we're in training mode or inference mode. `True` is always passed to `init_fn`. Can be omitted regardless of whether `call_fn` has a `training` argument. ## Models with non-trainable state For JAX models that have non-trainable state: - `call_fn` must have a `state` argument - `call_fn` must return a `tuple` containing the outputs of the model and the new non-trainable state of the model - `init_fn` must return a `tuple` containing the initial trainable params of the model and the initial non-trainable state of the model. This code shows a possible combination of `call_fn` and `init_fn` signatures for a model with non-trainable state. In this example, the model has a `training` argument and an `rng` argument in `call_fn`. ```python def stateful_call(params, state, rng, inputs, training): outputs = ... new_state = ... return outputs, new_state def stateful_init(rng, inputs): initial_params = ... initial_state = ... return initial_params, initial_state ``` ## Models without non-trainable state For JAX models with no non-trainable state: - `call_fn` must not have a `state` argument - `call_fn` must return only the outputs of the model - `init_fn` must return only the initial trainable params of the model. This code shows a possible combination of `call_fn` and `init_fn` signatures for a model without non-trainable state. In this example, the model does not have a `training` argument and does not have an `rng` argument in `call_fn`. ```python def stateless_call(params, inputs): outputs = ... return outputs def stateless_init(rng, inputs): initial_params = ... return initial_params ``` ## Conforming to the required signature If a model has a different signature than the one required by `JaxLayer`, one can easily write a wrapper method to adapt the arguments. This example shows a model that has multiple inputs as separate arguments, expects multiple RNGs in a `dict`, and has a `deterministic` argument with the opposite meaning of `training`. To conform, the inputs are combined in a single structure using a `tuple`, the RNG is split and used the populate the expected `dict`, and the Boolean flag is negated: ```python def my_model_fn(params, rngs, input1, input2, deterministic): ... if not deterministic: dropout_rng = rngs["dropout"] keep = jax.random.bernoulli(dropout_rng, dropout_rate, x.shape) x = jax.numpy.where(keep, x / dropout_rate, 0) ... ... return outputs def my_model_wrapper_fn(params, rng, inputs, training): input1, input2 = inputs rng1, rng2 = jax.random.split(rng) rngs = {"dropout": rng1, "preprocessing": rng2} deterministic = not training return my_model_fn(params, rngs, input1, input2, deterministic) keras_layer = JaxLayer(my_model_wrapper_fn, params=initial_params) ``` ## Usage with Haiku modules `JaxLayer` enables the use of [Haiku](https://dm-haiku.readthedocs.io) components in the form of [`haiku.Module`](https://dm-haiku.readthedocs.io/en/latest/api.html#module). This is achieved by transforming the module per the Haiku pattern and then passing `module.apply` in the `call_fn` parameter and `module.init` in the `init_fn` parameter if needed. If the model has non-trainable state, it should be transformed with [`haiku.transform_with_state`]( https://dm-haiku.readthedocs.io/en/latest/api.html#haiku.transform_with_state). If the model has no non-trainable state, it should be transformed with [`haiku.transform`]( https://dm-haiku.readthedocs.io/en/latest/api.html#haiku.transform). Additionally, and optionally, if the module does not use RNGs in "apply", it can be transformed with [`haiku.without_apply_rng`]( https://dm-haiku.readthedocs.io/en/latest/api.html#without-apply-rng). The following example shows how to create a `JaxLayer` from a Haiku module that uses random number generators via `hk.next_rng_key()` and takes a training positional argument: ```python class MyHaikuModule(hk.Module): def __call__(self, x, training): x = hk.Conv2D(32, (3, 3))(x) x = jax.nn.relu(x) x = hk.AvgPool((1, 2, 2, 1), (1, 2, 2, 1), "VALID")(x) x = hk.Flatten()(x) x = hk.Linear(200)(x) if training: x = hk.dropout(rng=hk.next_rng_key(), rate=0.3, x=x) x = jax.nn.relu(x) x = hk.Linear(10)(x) x = jax.nn.softmax(x) return x def my_haiku_module_fn(inputs, training): module = MyHaikuModule() return module(inputs, training) transformed_module = hk.transform(my_haiku_module_fn) keras_layer = JaxLayer( call_fn=transformed_module.apply, init_fn=transformed_module.init, ) ``` Args: call_fn: The function to call the model. See description above for the list of arguments it takes and the outputs it returns. init_fn: the function to call to initialize the model. See description above for the list of arguments it takes and the ouputs it returns. If `None`, then `params` and/or `state` must be provided. params: A `PyTree` containing all the model trainable parameters. This allows passing trained parameters or controlling the initialization. If both `params` and `state` are `None`, `init_fn` is called at build time to initialize the trainable parameters of the model. state: A `PyTree` containing all the model non-trainable state. This allows passing learned state or controlling the initialization. If both `params` and `state` are `None`, and `call_fn` takes a `state` argument, then `init_fn` is called at build time to initialize the non-trainable state of the model. seed: Seed for random number generator. Optional. Nc ztjdkr#tdtj|||tdtjd i|||_||_tj||_| |d|_ | |d|_ |j |j d|_||dhdd h|_d |jv|_|r"||d hd d h|_dSdS)Nr zBJaxLayer is only supported with the JAX backend. Current backend: z5`init_fn`, `params` and `state` cannot all be `None`.T trainableFcall_fn>rngstateinputsparamstrainingrrinit_fn>rrr)r ValueErrorsuper__init__rrrandom SeedGeneratorseed_generator_create_variablestracked_params tracked_staterrbuilt_validate_signaturecall_fn_arguments has_stateinit_fn_arguments)selfrrrrseedkwargs __class__s V/var/www/html/software/conda/lib/python3.11/site-packages/keras/src/utils/jax_layer.pyrzJaxLayer.__init__sx ?   % %0#O--00  ?v~%-G  ""6"""  %n::4@@"44Vt4LL!33EU3KK ; "dj&<DJ!%!9!9   < < < J " "  !D$::  %)%=%=$A$A$AH:&&D " " "  c Ptj|j}|D]}||vrtd|d|dg}|D]V}|j|vr1td|d|jdd|d||jW|S)NzMissing required argument in `z`: ``zUnsupported argument in `z`, supported arguments are `z`, `)inspect signature parametersrvaluesnamejoinappend) r&fnfn_nameallowedrequired fn_parametersparameter_nameparameter_names parameters r*r"zJaxLayer._validate_signatures)"--8 &  N]22 *W**&***3 &--// 3 3I~W,, HHHY^HH06 G0D0DHHH  " "9> 2 2 2 2r+cfd}tj||}r|_n|_tj|\}}|S)aCreate a structure of variables from a structure of JAX arrays. `values` is traversed via JAX's `tree_map`. When a leaf is a JAX array or a tensor-like object, a corresponding variable is created with it as the initial value. The resulting structure of variables is assigned to `self.params` or `self.state` depending on `trainable`. Then, a flattened version of the variables is returned for tracking. `self.params` or `self.state` are intentionally not tracked because structures like `TrackedList` interfere with `jax.tree_utils`. Note that leaf objects that are not JAX arrays and not tensor-like are left intact as they are assumed to be configuration used by the model. Args: values: the structure of values to traverse. trainable: whether to create trainable variables. Returns: flat list of variables initialized with `values` for tracking. cxtj|st|tjr4|jd}|||St|tjttfr/dd}|||S|S)Nzeros) initializerrr) r is_tensor isinstancenpndarray add_weightshapeassigngenericintfloat)valuevariabler&rs r*create_variablez3JaxLayer._create_variables..create_variable%s '' :eRZ+H+H ??KW +&&&EBJU#;<< ??Gy+&&& r+)r tree_utiltree_maprr tree_flatten)r&r1rrM variablesflat_variables_s` ` r*rzJaxLayer._create_variablesss,      "M**?FCC  ##DKK"DJM66yAAr+c4|jS)a Returns a JAX `PRNGKey` or structure of `PRNGKey`s to pass to `init_fn`. By default, this returns a single `PRNGKey` retrieved by calling `self.seed_generator.next()`. Override this to return a different structure. Returns: a JAX `PRNGKey` or structure of `PRNGKey`s that will be passed as the `rng` argument of `init_fn`. rnextr&s r* _get_init_rngzJaxLayer._get_init_rng@s"'')))r+c<|r|jSdS)a Returns a JAX `PRNGKey` or structure of `PRNGKey`s to pass to `call_fn`. By default, this returns a single `PRNGKey` retrieved by calling `self.seed_generator.next()` when `training` is `True`, and `None` when `training` is `False`. Override this to return a different structure or to pass RNGs in inference mode too. Returns: a JAX `PRNGKey` or structure of `PRNGKey`s that will be passed as the `rng` argument of `call_fn`. NrUr&rs r* _get_call_rngzJaxLayer._get_call_rngNs&  &++-- -4r+c0|j|jdStjrt dd}t j||}g}|jD]g}|dkr(|| 0|dkr||L|dkr|dh|j |}|j r|\}}n|d}}| |d|_ | |d|_d|_dS) Nz+'JaxLayer' cannot be built in tracing scopecXd|D}tj|S)Ncg|]}||nd S)Nr).0ds r* z8JaxLayer.build..create_input..ks >>>1!-QQQ>>>r+)r numpyones)rFs r* create_inputz$JaxLayer.build..create_inputjs)>>>>>E9>>%(( (r+rrrTrF)rrris_in_jax_tracing_scoperrmap_shape_structurer%r4rXrr$rrr r!) r& input_shapere init_inputs init_args argument_name init_result init_params init_states r*buildzJaxLayer.build`sZ ; "dj&< F  , . . LJKK K ) ) ).|[II  !3 ' 'M%%  !3!3!5!56666(**  ----*,,  &&&"dlI. > 8&1 #K&14K"44 45  "33J%3PP r+Fczd}g}|jD]}|dkr9|tj||jA|dkr9|tj||j|dkr)||||dkr|||dkr||d}|jr5|j |\}}tj|||j|S|j |S)Nc|dn|jSN)rK)rLs r*unwrap_variablez&JaxLayer.call..unwrap_variables#+44 ?r+rrrrrcnt|dstd||dS)NrGzStructure mismatch: the structure of the state returned by `call` does not match the structure of the state at initialization time.)hasattrrrG)rKrLs r*assign_state_to_variablez/JaxLayer.call..assign_state_to_variablesE8X..  + OOE " " " " "r+) r#r4r rNrOrrr[r$r) r&rrrs call_argsrkrv predictions new_states r*callz JaxLayer.calls @ @ @ !3 + +M((  M**?DKHH'))  M**?DJGG%''  !3!3H!=!=>>>>(**  ((((*,,  *** # # # > ,%1T\9%= "K M " "()TZ    4<+ +r+cJtj|jtj|jd}t }t t|t|zS)N)rr) rserialize_keras_objectrrr get_configdictlistitems)r&config base_configr)s r*r}zJaxLayer.get_configs}(? MM(? MM  gg((** D**,,--V\\^^0D0DDEEEr+ctj|d}tj|d}||d<||d<t|S)Nrr)rdeserialize_keras_objectr from_config)clsrrrr)s r*rzJaxLayer.from_configsY#.apply_with_training-sF;$$33FEBB{%! % r+ctj||||jS)N)rrrr)rrrrrr&s r*apply_without_trainingz2FlaxLayer.__init__..apply_without_training7sC;$$33FEBB{% % r+cpj||j|S)N)rr_variables_to_params_and_staterinitr)rrrr&s r*init_with_trainingz.FlaxLayer.__init__..init_with_training@sC66   ;% ! r+cnj||jS)N)rr)rrr&s r*init_without_trainingz1FlaxLayer.__init__..init_without_trainingJs@66   ;! r+r)rrrrr) flax.corerrrrrDenyListr.r/__call__r0rrr)r&rrrQr( flax_scoperrrrrrrrrr)s` @r*rzFlaxLayer.__init__s 211111 ?   % %0#O--00    "++XJ77                         !:6?;;F G G 34FWGG57LWG;;IFF           r+c&|r |ri||S|S|r|SiSrrr)r&rrs r*rz(FlaxLayer._params_and_state_to_variableses9   *&*E**  L r+c|dSd|vri|fSt|dkr|ifSd|di}d|D}||fS)NNNrr_c&i|]\}}|dk ||S)rr)r`kvs r* z.{s#EEE$!QqH}}A}}}r+)lenr)r&rQrrs r*rz(FlaxLayer._variables_to_params_and_stateoss  : 9 $ $y= y>>Q  b= Ih/0EE)//"3"3EEEu}r+ch|j|jdS)N)rdropoutrUrWs r*rXzFlaxLayer._get_init_rng~s5)..00*//11   r+c@|rd|jiSiS)NrrUrZs r*r[zFlaxLayer._get_call_rngs)  t27799: :Ir+c|j}t|jdr!|jj|jkr |jj}t j|jt j|d}t}| d| dtt| t| zS)N__self__)rrrr) rrurrrrr|rr}popr~rr)r& config_methodrrr)s r*r}zFlaxLayer.get_configs DK , , 1 $ 33!K0M'>t{KK'>}MM  gg((**  """ """D**,,--V\\^^0D0DDEEEr+ctj|d}tj|d}t|dtrt ||}||d<||d<|di|S)Nrrr)rrrBstrgetattr)rrrrs r*rzFlaxLayer.from_configsx";F8rsK------((((((......%%%%%%$$$$$$,,,,,,%&&j+j+j+j+j+uj+j+'&j+Z &''ggggggg('gggr+