&Vf%ddZddlZddlZddlmZddZdZdZdZ dZ d Z d Z d Z d Zd ZdS)z!!!DO NOT USE!!! Distribution related class for JAX backend. This is just a prototype and we might want to unify it with other backends in the future. N) jax_utilsct|r|nd}tj|}d|DS)aReturn all the available devices based on the device type. Note that this should return the global devices in a distributed setting. Args: device_type: string of `"cpu"`, `"gpu"` or `"tpu"`. Defaults to `"gpu"` or `"tpu"` if available when device_type is not provided. Otherwise will return the `"cpu"` devices. Return: List of devices that are available for distribute computation. Nbackendc0g|]}|jd|jS):)platformid).0devices c/var/www/html/software/conda/lib/python3.11/site-packages/keras/src/backend/jax/distribution_lib.py z list_devices..s* G G Gv , , , , G G G)lowerjaxdevices) device_type jax_devicess r list_devicesrsE*5>+##%%%$K+k222K G G; G G GGrc`t|tjjst |}ttjtjjfr0j|tj rS|j rtj |S| j }tj fd|Dt|}tjj ||}|S)aCreate a distributed variable for JAX. Since JAX doesn't have a variable class, this will just return a `jax.Array` with the corresponding layout/sharding specified. Note that this function should be used in eager context, not in jitted function. Args: value: the initial value of the variable. layout: `TensorLayout` for the created variable, or a `jax.sharding.Sharding` instance. Returns: jax.Array which is the distributed variable. )ndimc g|] }| Sr)r ivalues r rz'distribute_variable..As 0 0 0!U1X 0 0 0r) isinstancershardingSharding_to_jax_layoutArraynumpyndarrayis_equivalent_tolenshapeis_fully_addressable device_putaddressable_devices_indices_mapvalueslistkeys$make_array_from_single_device_arrays)rlayoutmapping local_values global_values` r distribute_variabler1!s" fcl3 4 4(''  39,- . ) )&s5;7G7G ) H H  " ~eV,,,88EE~ 0 0 0 0w~~// 0 0 0$w||~~2F2F  ? K  rct|tjjst |}t jr tj|S|j rtj |S| j }tj fd| Dt|}tjj ||}|S)aqDistribute the tensor based on the layout. Note that this function can be used both in eager context, or within a jitted function. Args: tensor: `jax.Array` that need to be distributed. layout: `TensorLayout` for the distribution information, or a `jax.sharding.Sharding` instance. Returns: Distributed value. c g|] }| Srr)r rtensors r rz%distribute_tensor..es 1 1 11VAY 1 1 1r)rrrrrris_in_jax_tracing_scopelaxwith_sharding_constraintr&r'r(r%r)r*r+r,)r4r-r.r/r0s` r distribute_tensorr8Is fcl3 4 4(''(**@w//??? " ~ff---88FF~ 1 1 1 1 0 0 1 1 14 3G3G  ? L&,  rc t|tjjst |}|jrtj|S|j}t|j }j d}|dkrotj }||z}||zdkrtd|d||tj z}tj |d}n|dkrt|j d} tj } | | kr|}fdt#| D}nB|| | zz}tj | d}ntd|j |fj dd z} tj| |d t'||jD } | S) aDistribute the input data with the corresponding layout. Note that the inputs here is a local worker batch. Within the local worker, the data need to be further partitioned to map to the each of the devices. Args: inputs: `jax.Array` that is already sharded to a local process size. layout: `TensorLayout` for the distribution information, or a `jax.sharding.Sharding` instance. Returns: Distributed inputs thats been properly put to local devices. rzThe local batch size z1 is notdivisible by the number of local replicas )axiscg|]}Srr)r _inputss r rz)distribute_data_input..s"M"M"Ma6"M"M"MrzEOnly 1D or 2D mesh is supported at the moment. Received mesh shape = Nc>g|]\}}tj||Sr)rr')r batchr s r rz)distribute_data_input..s8   v N5& ) )   r)arrays)rrrrrr&r'meshr$r%local_device_count ValueError process_countr!splitr*r)ranger,zipaddressable_devices) r?r-jax_mesh mesh_rankper_process_batch_size num_splitper_replica_batch_sizeglobal_batch_sizeper_replica_batchesmesh_batch_dim_sizerD global_shapeglobal_batch_arrays ` r distribute_data_inputrUmsJ fcl3 4 4('' ".~ff---{HHN##I#\!_A~~*,, !79!D !$: :a ? ?(>  3S5F5H5HH!ioofiaoHH a#8>#8#8#:#:;;A> 355 !3 3 3 6 "M"M"M"M59K3L3L"M"M"M  !7#'99! #&)//*#2##   6%-^ 6 6   &'&,qrr*::LA  !$#V%?""       rc |r\d|vrX|d}|8|t|kr%td|dt|d||d}n|}tj|||dS)N,zThe provided job_addresses z has z jobs, but num_processes is r)coordinator_address num_processes process_id)rGr$rEr distributed initialize) job_addressesrYrZrXs r r\r\s, --&++C00   $#m:L:L)L)L#m##}%%## ##  ,A.+O/#rc(tjS)zDReturn the number of processes for the current distribution setting.)rrFrrr rYrY    rc(tjS)z;Return the current process ID for the distribution setting.)r process_indexrrr rZrZr_rc t|tjr|S|d\}}t |}tj|}|t |krtd|||S)NrrzUnknown device: )rrDevicerGintrr$rE) device_idrindexrs r _to_jax_devicergs)SZ(("--K JJEk+...G G 7I77888 5>rc|jj}d|jD}tj||}t j||j S)zConvert the DeviceMesh to JAX backend specific Mesh. Args: device_mesh: DeviceMesh instance to convert. Returns: A `jax.sharding.Mesh` instance. c,g|]}t|Sr)rg)r ds r rz _to_jax_mesh..s HHHQ~a  HHHr) rr%flattennparrayreshaperrMesh axis_names) device_meshr%rs r _to_jax_meshrrsi   %EHH+*=*E*E*G*GHHHGhw''..G <  Wk&< = ==rc|jtdtjj|j}t |j}tj||S)zConvert the TensorLayout to JAX backend specific Sharding. Args: tensor_layout: TensorLayout instance to convert. Returns: A `jax.sharding.NamedSharding` instance. NzDCannot create sharding when device mesh is not set for TensorLayout.)rqrErr PartitionSpecaxesrr NamedSharding) tensor_layoutpartition_specrKs r rrs` (    \/1CDNM566H < % %h ? ??r)N)__doc__rr!rlkeras.src.utilsrrr1r8rUr\rYrZrgrrrrrr r{s %%%%%%HHHH$%%%P!!!HNNNb0   > > >@@@@@r