o tdfI@sHddlZddlZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z m Z mZddlZddlmZddlmZddlmZmZmZmZmZmZmZmZmZer^ddlmZmZGd d d e ej!e"Z#Gd d d e#e"Z$Gd dde#e"Z%dZ&ej'dddkre&d7Z&ej'dddkre&d7Z&e&d7Z&dddZ(dS)N)Module)IterableIteratorListOptionalTupleAnycast TYPE_CHECKING)xrange) LineNumbers) Token match_tokenis_non_coding_tokenpatched_generate_tokens last_stmtannotate_fstring_nodesgenerate_tokens is_moduleis_stmt)AstNode TokenInfoc@s6eZdZddZejddZd ddZd dd Zd S) ASTTextBasecCs$||_t|}||_t||_dSN) _filenamesix ensure_text_textr _line_numbers)self source_textfilenamer#\/var/www/html/software/conda/envs/catlas/lib/python3.10/site-packages/asttokens/asttokens.py__init__%s zASTTextBase.__init__cCst) Returns two ``(lineno, col_offset)`` tuples for the start and end of the given node. If the positions can't be determined, or the nodes don't correspond to any particular text, returns ``(1, 0)`` for both. ``padded`` corresponds to the ``padded`` argument to ``ast.get_source_segment()``. This means that if ``padded`` is True, the start position will be adjusted to include leading whitespace if ``node`` is a multiline statement. )NotImplementedErrorr nodepaddedr#r#r$get_text_positions4s zASTTextBase.get_text_positionsTcCs(|||\}}|jj||jj|fS)a Returns the (startpos, endpos) positions in source text corresponding to the given node. Returns (0, 0) for nodes (like `Load`) that don't correspond to any particular text. See ``get_text_positions()`` for details on the ``padded`` argument. )r+rline_to_offsetr r)r*startendr#r#r$get_text_rangeBs  zASTTextBase.get_text_rangecCs|||\}}|j||S)z Returns the text corresponding to the given node. Returns '' for nodes (like `Load`) that don't correspond to any particular text. See ``get_text_positions()`` for details on the ``padded`` argument. )r0rr-r#r#r$get_textPszASTTextBase.get_textN)T) __name__ __module__ __qualname__r%abcabstractmethodr+r0r1r#r#r#r$r$s   rcseZdZdZd%fdd ZddZd d Zed d Zed dZ eddZ eddZ ddZ ddZ ddZd&ddZd&ddZd'ddZ d&dd Zd&d!d"Zd#d$ZZS)( ASTTokensa ASTTokens maintains the text of Python code in several forms: as a string, as line numbers, and as tokens, and is used to mark and access token and position information. ``source_text`` must be a unicode or UTF8-encoded string. If you pass in UTF8 bytes, remember that all offsets you'll get are to the unicode text, which is available as the ``.text`` property. If ``parse`` is set, the ``source_text`` will be parsed with ``ast.parse()``, and the resulting tree marked with token info and made available as the ``.tree`` property. If ``tree`` is given, it will be marked and made available as the ``.tree`` property. In addition to the trees produced by the ``ast`` module, ASTTokens will also mark trees produced using ``astroid`` library . If only ``source_text`` is given, you may use ``.mark_tokens(tree)`` to mark the nodes of an AST tree created separately. FN csvtt||||rt||n||_|durt|j}t| ||_ dd|j D|_ |jr9| |jdSdS)NcSsg|]}|jqSr#)startpos).0tokr#r#r$ sz&ASTTokens.__init__..) superr7r%astparse_treerrlist_translate_tokens_tokens_token_offsets mark_tokens)r r!r?treer"tokens __class__r#r$r%ps zASTTokens.__init__cCsddlm}|||dS)ap Given the root of the AST or Astroid tree produced from source_text, visits all nodes marking them with token and position information by adding ``.first_token`` and ``.last_token``attributes. This is done automatically in the constructor when ``parse`` or ``tree`` arguments are set, but may be used manually with a separate AST or Astroid tree. r ) MarkTokensN)rErJZ visit_tree)r Z root_noderJr#r#r$rEs zASTTokens.mark_tokensc csftt|D])\}}|\}}}}}t|||||||j|d|d|j|d|dVqdS)zS Translates the given standard library tokens into our own representation. rr N) enumeraterrrr,) r Zoriginal_tokensindexr;tok_typetok_strr.r/liner#r#r$rBszASTTokens._translate_tokenscC|jS)z,The source code passed into the constructor.)rr r#r#r$textzASTTokens.textcCrP)zIThe list of tokens corresponding to the source code from the constructor.)rCrQr#r#r$rGrSzASTTokens.tokenscCrP)zTThe root of the AST tree passed into the constructor or parsed from the source code.)r@rQr#r#r$rFrSzASTTokens.treecCrP)zThe filename that was parsed)rrQr#r#r$r"rSzASTTokens.filenamecCs|jt|j|dS)z Returns the token containing the given character offset (0-based position in source text), or the preceeding token if the position is between tokens. r )rCbisectrD)r offsetr#r#r$get_token_from_offsetszASTTokens.get_token_from_offsetcCs||j||S)z Returns the token containing the given (lineno, col_offset) position, or the preceeding token if the position is between tokens. )rVrr,r lineno col_offsetr#r#r$ get_tokens zASTTokens.get_tokencCs|||j||S)zd Same as get_token(), but interprets col_offset as a UTF8 offset, which is what `ast` uses. )rZr from_utf8_colrWr#r#r$get_token_from_utf8szASTTokens.get_token_from_utf8cCs@|jd}|st|j|jr|d7}t|j|js|j|S)z Returns the next token after the given one. If include_extra is True, includes non-coding tokens from the tokenize module, such as NL and COMMENT. r rLrrCtyper r; include_extrair#r#r$ next_token  zASTTokens.next_tokencCs@|jd}|st|j|jr|d8}t|j|js|j|S)z Returns the previous token before the given one. If include_extra is True, includes non-coding tokens from the tokenize module, such as NL and COMMENT. r r]r_r#r#r$ prev_tokenrczASTTokens.prev_tokencCsT|}|r|jn|j}t|||s(t|js(||dd}t|||s(t|jr|S)z Looks for the first token, starting at start_token, that matches tok_type and, if given, the token string. Searches backwards if reverse is True. Returns ENDMARKER token if not found (you can check it with `token.ISEOF(t.type)`). Tr`)rdrbrtokenISEOFr^)r Z start_tokenrMrNreversetadvancer#r#r$ find_tokens  zASTTokens.find_tokenccs>t|j|jdD]}|st|j|js|j|Vq dS)z Yields all tokens in order from first_token through and including last_token. If include_extra is True, includes non-coding tokens such as tokenize.NL and .COMMENT. r N)r rLrrCr^)r first_token last_tokenr`rar#r#r$ token_ranges   zASTTokens.token_rangecCs|j|j|j|dS)z Yields all tokens making up the given node. If include_extra is True, includes non-coding tokens such as tokenize.NL and .COMMENT. re)rnrlrm)r r)r`r#r#r$ get_tokensszASTTokens.get_tokenscCsNt|dsdS|jj}|jj}|r#tdd||Dr#|ddf}||fS)r&rlr rrqcss|] }t|tjVqdSr)rrfNEWLINE)r:rir#r#r$ sz/ASTTokens.get_text_positions..r)hasattrrlr.rmr/anyror-r#r#r$r+s  zASTTokens.get_text_positions)FNr8N)F)NF)r2r3r4__doc__r%rErBpropertyrRrGrFr"rVrZr\rbrdrkrnror+ __classcell__r#r#rHr$r7\s.        r7csJeZdZdZdfdd ZeddZedd Zd d Zd d Z Z S)ASTTextaF Supports the same ``get_text*`` methods as ``ASTTokens``, but uses the AST to determine the text positions instead of tokens. This is faster than ``ASTTokens`` as it requires less setup work. It also (sometimes) supports nodes inside f-strings, which ``ASTTokens`` doesn't. Some node types and/or Python versions are not supported. In these cases the ``get_text*`` methods will fall back to using ``ASTTokens`` which incurs the usual setup cost the first time. If you want to avoid this, check ``supports_tokenless(node)`` before calling ``get_text*`` methods. Nr8cs6tt|||||_|jdurt|jd|_dSr)r=ryr%r@r _asttokens)r r!rFr"rHr#r$r%,s    zASTText.__init__cCs,|jdurt|j|j|_t|j|jSr)r@r>r?rrrrQr#r#r$rF9s  z ASTText.treecCs&|jdurt|j|j|jd|_|jS)N)rFr")rzr7rrFrrQr#r#r$ asttokensAs zASTText.asttokensc Cstjdddkr tdt|rd|jt|jfSt|dddur&dS|s*Jt|dg}|s>t|d d}t|d g}|rE|d }n|}|j }t |}|rg||j ksd||j krgt|d drgt |rgd }n|j ||j}||f} tt|j } tt|j} |j | | } | | f} | | fS) zF Version of ``get_text_positions()`` that doesn't use tokens. NzPThis method should only be called internally after checking supports_tokenless()rqrXrpdecorator_list decoratorsnodesrZdoc_node)sys version_infoAssertionErrorrrZoffset_to_linelenrgetattrrXr end_linenorr[rYr intend_col_offset) r r)r*rZdecorators_nodeZ start_nodeZ start_linenoZend_nodeZstart_col_offsetr.rrr/r#r#r$_get_text_positions_tokenlessLs>         z%ASTText._get_text_positions_tokenlesscCs2t|ddrdSt|r|||S|j||S)r&Z_broken_positionsNrp)rsupports_tokenlessrr{r+r(r#r#r$r+s  zASTText.get_text_positions)Nr8) r2r3r4rvr%rwrFr{rr+rxr#r#rHr$rys    Fryr#r|r}) arguments Argumentswithitem)argStarred)SliceExtSliceIndexkeywordcCsdt|jtvo1t|tj o |duo t|jdko t|jjdv o1tjdddko1dtj vS)a Returns True if the Python version and the node (if given) are supported by the ``get_text*`` methods of ``ASTText`` without falling back to ``ASTTokens``. See ``ASTText`` for why this matters. The following cases are not supported: - Python 3.7 and earlier - PyPy - ``ast.arguments`` / ``astroid.Arguments`` - ``ast.withitem`` - ``astroid.Comprehension`` - ``astroid.AssignName`` inside ``astroid.Arguments`` or ``astroid.ExceptHandler`` - The following nodes in Python 3.8 only: - ``ast.arg`` - ``ast.Starred`` - ``ast.Slice`` - ``ast.ExtSlice`` - ``ast.Index`` - ``ast.keyword`` NZ AssignName)r ExceptHandlerr|r}pypy) r^r2_unsupported_tokenless_types isinstancer>ASTparentrrversionlower)r)r#r#r$rs  rr))r5r>rTrrfrtypingrrrrrrr r rZ six.movesr line_numbersr utilrrrrrrrrrrrwith_metaclassABCMetaobjectrr7ryrrrr#r#r#r$s2 (  ,8D