o 0GfV@sddlmZddlmZmZmZddlmZddlm Z ddl Z dddZ ddd Z dd d Zd d ZddZ  dddZdddZifddZGdddZGdddeZdS))RegularizedResults)_calc_nodewise_row_calc_nodewise_weight_calc_approx_inv_cov)LikelihoodModelResults)OLSNcC"|durtd|jdi|jS)aestimates the regularized fitted parameters. Parameters ---------- mod : statsmodels model class instance The model for the current partition. pnum : scalar Index of current partition partitions : scalar Total number of partitions fit_kwds : dict-like or None Keyword arguments to be given to fit_regularized Returns ------- An array of the parameters for the regularized fit NzD_est_regularized_naive currently requires that fit_kwds not be None.) ValueErrorfit_regularizedparamsmodpnum partitionsfit_kwdsr r Glib/python3.10/site-packages/statsmodels/base/distributed_estimation.py_est_regularized_naiveKrcCr)aestimates the unregularized fitted parameters. Parameters ---------- mod : statsmodels model class instance The model for the current partition. pnum : scalar Index of current partition partitions : scalar Total number of partitions fit_kwds : dict-like or None Keyword arguments to be given to fit Returns ------- An array of the parameters for the fit NzF_est_unregularized_naive currently requires that fit_kwds not be None.r )r fitr r r r r_est_unregularized_naiveerrcCsNt|d}t|}t|}|D]}||7}q||}d|t||k<|S)a joins the results from each run of _est__naive and returns the mean estimate of the coefficients Parameters ---------- params_l : list A list of arrays of coefficients. threshold : scalar The threshold at which the coefficients will be cut. r)lennpzerosabs)Zparams_l thresholdpr params_mnr r r r _join_naives   rcCs.|jt|fi| }||d|7}|S)acalculates the log-likelihood gradient for the debiasing Parameters ---------- mod : statsmodels model class instance The model for the current partition. params : array_like The estimated coefficients for the current partition. alpha : scalar or array_like The penalty weight. If a scalar, the same penalty weight applies to all variables in the model. If a vector, it must have the same length as `params`, and contains a penalty weight for each coefficient. L1_wt : scalar The fraction of the penalty given to the L1 penalty term. Must be between 0 and 1 (inclusive). If 0, the fit is a ridge fit, if 1 it is a lasso fit. score_kwds : dict-like or None Keyword arguments for the score function. Returns ------- An array-like object of the same dimension as params Notes ----- In general: gradient l_k(params) where k corresponds to the index of the partition For OLS: X^T(y - X^T params) )Zscorerasarray)rr alphaL1_wt score_kwdsgradr r r _calc_grads&r%cCs4t|jt|fi|}|dddf|jS)acalculates the weighted design matrix necessary to generate the approximate inverse covariance matrix Parameters ---------- mod : statsmodels model class instance The model for the current partition. params : array_like The estimated coefficients for the current partition. hess_kwds : dict-like or None Keyword arguments for the hessian function. Returns ------- An array-like object, updated design matrix, same dimension as mod.exog N)rZsqrtZhessian_factorr exog)rr hess_kwdsZrhessr r r_calc_wdesign_matsr(cCs|durin|}|durin|}|durtd|d}d|vr%|d}nd}|jj\}} ttd| |} |jdi|j} t|| ||||} t || |} g}g}t || t |d| | D]}t | ||}| |t| |||}| |qc| | ||fS)aestimates the regularized fitted parameters, is the default estimation_method for class DistributedModel. Parameters ---------- mod : statsmodels model class instance The model for the current partition. mnum : scalar Index of current partition. partitions : scalar Total number of partitions. fit_kwds : dict-like or None Keyword arguments to be given to fit_regularized score_kwds : dict-like or None Keyword arguments for the score function. hess_kwds : dict-like or None Keyword arguments for the Hessian function. Returns ------- A tuple of parameters for regularized fit An array-like object of the fitted parameters, params An array-like object for the gradient A list of array like objects for nodewise_row A list of array like objects for nodewise_weight NzG_est_regularized_debiased currently requires that fit_kwds not be None.r!r"rg?r )r r&shapeintrZceilr r r%r(rangeminrappendr)rZmnumrrr#r'r!r"ZnobsrZp_partr r$Zwexognodewise_row_lnodewise_weight_lidxZ nodewise_rowZnodewise_weightr r r_est_regularized_debiaseds.        r1c Cst|dd}t|}t|}t|}g}g}|D]}||d7}||d7}||d||dqt|}t|}||}|d|9}t||} || |} d| t| |k<| S)ajoins the results from each run of _est_regularized_debiased and returns the debiased estimate of the coefficients Parameters ---------- results_l : list A list of tuples each one containing the params, grad, nodewise_row and nodewise_weight values for each partition. threshold : scalar The threshold at which the coefficients will be cut. rrg)rrrextendZarrayrdotr) results_lrrrrZgrad_mnr.r/rZapprox_inv_covZdebiased_paramsr r r_join_debiaseds&         r8c CsJ|j}|||j||fi|}|j|||jfd|i|j}|S)ahandles the model fitting for each machine. NOTE: this is primarily handled outside of DistributedModel because joblib cannot handle class methods. Parameters ---------- self : DistributedModel class instance An instance of DistributedModel. pnum : scalar index of current partition. endog : array_like endogenous data for current partition. exog : array_like exogenous data for current partition. fit_kwds : dict-like Keywords needed for the model fitting. init_kwds_e : dict-like Additional init_kwds to add for each partition. Returns ------- estimation_method result. For the default, _est_regularized_debiased, a tuple. r) init_kwdscopyupdate model_classestimation_methodrestimation_kwds) selfrendogr&r init_kwds_eZtemp_init_kwdsmodelresultsr r r_helper_fit_partitionHs  rDc@sHeZdZdZ    d ddZ  d ddZ ddd Z dd d ZdS)DistributedModela Distributed model class Parameters ---------- partitions : scalar The number of partitions that the data will be split into. model_class : statsmodels model class The model class which will be used for estimation. If None this defaults to OLS. init_kwds : dict-like or None Keywords needed for initializing the model, in addition to endog and exog. init_kwds_generator : generator or None Additional keyword generator that produces model init_kwds that may vary based on data partition. The current usecase is for WLS and GLS estimation_method : function or None The method that performs the estimation for each partition. If None this defaults to _est_regularized_debiased. estimation_kwds : dict-like or None Keywords to be passed to estimation_method. join_method : function or None The method used to recombine the results from each partition. If None this defaults to _join_debiased. join_kwds : dict-like or None Keywords to be passed to join_method. results_class : results class or None The class of results that should be returned. If None this defaults to RegularizedResults. results_kwds : dict-like or None Keywords to be passed to results class. Attributes ---------- partitions : scalar See Parameters. model_class : statsmodels model class See Parameters. init_kwds : dict-like See Parameters. init_kwds_generator : generator or None See Parameters. estimation_method : function See Parameters. estimation_kwds : dict-like See Parameters. join_method : function See Parameters. join_kwds : dict-like See Parameters. results_class : results class See Parameters. results_kwds : dict-like See Parameters. Notes ----- Examples -------- Nc Cs||_|dur t|_n||_|duri|_n||_|dur!t|_n||_|dur,i|_n||_|dur7t|_n||_|durBi|_ n||_ |durMt |_ n||_ | durYi|_ dS| |_ dSN) rrr<r9r1r=r>r8 join_method join_kwdsr results_class results_kwds) r?rr<r9r=r>rGrHrIrJr r r__init__s2  zDistributedModel.__init__ sequentialc Cs|duri}|dkr||||}n|dkr|||||}ntd||j|fi|j}|jdgdgfi|j}|j||fi|jS)aePerforms the distributed estimation using the corresponding DistributedModel Parameters ---------- data_generator : generator A generator that produces a sequence of tuples where the first element in the tuple corresponds to an endog array and the element corresponds to an exog array. fit_kwds : dict-like or None Keywords needed for the model fitting. parallel_method : str type of distributed estimation to be used, currently "sequential", "joblib" and "dask" are supported. parallel_backend : None or joblib parallel_backend object used to allow support for more complicated backends, ex: dask.distributed init_kwds_generator : generator or None Additional keyword generator that produces model init_kwds that may vary based on data partition. The current usecase is for WLS and GLS Returns ------- join_method result. For the default, _join_debiased, it returns a p length array. NrLZjoblibz.parallel_method: %s is currently not supportedr) fit_sequential fit_joblibr rGrHr<r9rIrJ) r?data_generatorrZparallel_methodparallel_backendinit_kwds_generatorr6r Zres_modr r rrs"zDistributedModel.fitc Csg}|dur t|D]\}\}}t|||||}||q |Stt||} | D]\}\\}}} t|||||| }||q)|S)a*Sequentially performs the distributed estimation using the corresponding DistributedModel Parameters ---------- data_generator : generator A generator that produces a sequence of tuples where the first element in the tuple corresponds to an endog array and the element corresponds to an exog array. fit_kwds : dict-like Keywords needed for the model fitting. init_kwds_generator : generator or None Additional keyword generator that produces model init_kwds that may vary based on data partition. The current usecase is for WLS and GLS Returns ------- join_method result. For the default, _join_debiased, it returns a p length array. N) enumeraterDr-zip) r?rOrrQr6rr@r&rCtup_genrAr r rrMs"     zDistributedModel.fit_sequentialc sFddlm}|tj\}}|dur(|dur(|fddt|D}|S|durT|durT||fddt|D}Wd|S1sMwY|S|durr|durrtt||} |fdd| D}|S|dur|durtt||} ||fdd| D}Wd|S1swY|S) aPerforms the distributed estimation in parallel using joblib Parameters ---------- data_generator : generator A generator that produces a sequence of tuples where the first element in the tuple corresponds to an endog array and the element corresponds to an exog array. fit_kwds : dict-like Keywords needed for the model fitting. parallel_backend : None or joblib parallel_backend object used to allow support for more complicated backends, ex: dask.distributed init_kwds_generator : generator or None Additional keyword generator that produces model init_kwds that may vary based on data partition. The current usecase is for WLS and GLS Returns ------- join_method result. For the default, _join_debiased, it returns a p length array. r) parallel_funcNc3(|]\}\}}|||VqdSrFr .0rr@r&frr?r r c z.DistributedModel.fit_joblib..c3rVrFr rWrYr rr[ir\c3.|]\}\\}}}||||VqdSrFr rXrr@r&r9rYr rr[oc3r]rFr r^rYr rr[vr_)Zstatsmodels.tools.parallelrUrDrrRrS) r?rOrrPrQrUZparZn_jobsr6rTr rYrrNDs@    zDistributedModel.fit_joblib)NNNNNNNN)NrLNNrF)__name__ __module__ __qualname____doc__rKrrMrNr r r rrEms? / : 0rEcs(eZdZdZfddZddZZS)DistributedResultsaT Class to contain model results Parameters ---------- model : class instance Class instance for model used for distributed data, this particular instance uses fake data and is really only to allow use of methods like predict. params : ndarray Parameter estimates from the fit model. cst||dSrF)superrK)r?rBr  __class__r rrKszDistributedResults.__init__cOs|jj|j|g|Ri|S)aCalls self.model.predict for the provided exog. See Results.predict. Parameters ---------- exog : array_like NOT optional The values for which we want to predict, unlike standard predict this is NOT optional since the data in self.model is fake. *args : Some models can take additional arguments. See the predict method of the model for the details. **kwargs : Some models can take additional keywords arguments. See the predict method of the model for the details. Returns ------- prediction : ndarray, pandas.Series or pandas.DataFrame See self.model.predict )rBpredictr )r?r&argskwargsr r rrhszDistributedResults.predict)r`rarbrcrKrh __classcell__r r rfrrd}s rdrF)r)NNN)Zstatsmodels.base.elastic_netrZ(statsmodels.stats.regularized_covariancerrrZstatsmodels.base.modelrZ#statsmodels.regression.linear_modelrZnumpyrrrrr%r(r1r8rDrErdr r r rs*    C  + A. %