c1UdZddlZddlZddlZddlZddlZddlZdZdZ ej e Z ej dZej dZgdZgdZejdZed krejejZned krejejZndZd ZeeeZejd eZ d Zd dZd!dZ d"dZ!dZ"d#dZ#dZ$dZ%dZ&dZ'dZ(dZ)dZ*dZ+dZ,dZ-dS)$uLoad models from the native binary format released by Facebook. The main entry point is the :func:`~gensim.models._fasttext_bin.load` function. It returns a :class:`~gensim.models._fasttext_bin.Model` namedtuple containing everything loaded from the binary. Examples -------- Load a model from a binary file: .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> from gensim.models.fasttext_bin import load >>> with open(datapath('crime-and-punishment.bin'), 'rb') as fin: ... model = load(fin) >>> model.nwords 291 >>> model.vectors_ngrams.shape (391, 5) >>> sorted(model.raw_vocab, key=lambda w: len(w), reverse=True)[:5] ['останавливаться', 'изворачиваться,', 'раздражительном', 'exceptionally', 'проскользнуть'] See Also -------- `FB Implementation `_. N iO/) )dimi)wsrepochr min_countrnegr word_ngramsrlossrmodelrbucketrminnrmaxnrlr_update_ratertd) rr r rrrrrrrrz@fc#KttzD]\}}|ds|VdVdVdVdVdVdVdS)N_ raw_vocab vocab_sizenwordsvectors_ngrams hidden_outputntokens)_OLD_HEADER_FORMAT_NEW_HEADER_FORMAT startswith)namer"s ;lib/python3.11/site-packages/gensim/models/_fasttext_bin.py_yield_field_namesr.ks%(::as## JJJ   NNN   OOOOOModelcztj|}tj|||S)N)structcalcsizeunpackread)finfmt num_bytess r-_struct_unpackr9s/$$I =chhy11 2 22r/utf-8cdt|d\}}}|dkrtdtd||jt|dd}|rt|d\}t j}t|D]} tj } | d} | tkr5| | | d} | tk5| } | |} nC#t$r6| |d} td | | YnwxYwt|d \} }| || <|r"t|D]}t|d ||||fS) a`Load a vocabulary from a FB binary. Before the vocab is ready for use, call the prepare_vocab function and pass in the relevant parameters from the model. Parameters ---------- fin : file An open file pointer to the binary. new_format: boolean True if the binary is of the newer format. encoding : str The encoding to use when decoding binary data into words. Returns ------- tuple The loaded vocabulary. Keys are words, values are counts. The vocabulary size. The number of words. The number of tokens. z@3irz,Supervised fastText models are not supportedz+loading %s words for fastText model from %sz@qbackslashreplace)errorszQfailed to decode invalid unicode bytes %r; replacing invalid characters, using %rz@qb@2i)r9NotImplementedErrorloggerinfor, collections OrderedDictrangeioBytesIOr5_END_OF_WORD_MARKERwritegetvaluedecodeUnicodeDecodeErrorerror)r6 new_formatencodingr$r%nlabelsr( pruneidx_sizer#r word_bytes char_bytewordcountr"js r- _load_vocabrWs.#1e"<"<J{R!"PQQQ KK=z38TTTS$''*G3'T22 '))I :    Z\\ HHQKK .. $   Y ' ' ' I.. $ ((**  $$X..DD!   $$X6H$IID LLcD       "#u--q $'}%% ' 'A 3 & & & & j&' 11sD,,=E,+E,Tcttdtz|rt|dt|d\}}||z}t |t jr1tdt|t|}ntj |t|}|j |fksJd|d|j | ||f}|S)aALoad a matrix from fastText native format. Interprets the matrix dimensions and type from the file stream. Parameters ---------- fin : file A file handle opened for reading. new_format : bool, optional True if the quant_input variable precedes the matrix declaration. Should be True for newer versions of fastText. Returns ------- :class:`numpy.array` The vectors as an array. Each vector will be a row in the array. The number of columns of the array will correspond to the vector size. Nzbad _FLOAT_SIZE: %r@?@2qzLoading model from a compressed .gz file. This can be slow. This is a work-around for a bug in NumPy: https://github.com/numpy/numpy/issues/13470. Consider decompressing your model file for a faster load. z expected (z ,), got ) _FLOAT_DTYPE ValueError _FLOAT_SIZEr9 isinstancegzipGzipFilerAwarning _fromfilenpfromfileshapereshape)r6rN num_vectorsrrUmatrixs r- _load_matrixris*>.<==="sD!!!%c511K # E#t}%%7 I   3 e44S,66