3ddZddlZddlmZmZmZmZddlTddlmZm Z ddl m Z dZ dZ d Zdd Zd Zd Zd ZdZdZdZ ddZdS)z This module contains general purpose URL functions not found in the standard library. Some of the functions that used to be imported from this module have been moved to the w3lib.url module. Always import those from there instead. N) ParseResult urldefragurlparse urlunparse)*) _safe_chars _unquotepath) to_unicodect|jsdSd|D}tfd|DS)z:Return True if the url belongs to any of the given domainsFc6g|]}|S)lower).0ds 0lib/python3.11/site-packages/scrapy/utils/url.py z*url_is_from_any_domain..s ***Qqwwyy***c3TK|]"}|kpd|V#dS).Nendswith)rrhosts r z)url_is_from_any_domain..s>HH1 6t}}WWW55HHHHHHr) parse_urlnetlocrany)urldomainsrs @rurl_is_from_any_domainrs` S>> & & ( (D u**'***G HHHHHHH H HHrc lt||jgtt|dgzS)z2Return True if the url belongs to the given spiderallowed_domains)rnamelistgetattr)rspiders rurl_is_from_spiderr&s7 ! fk]T'&2CR"H"HII I  rct|jtfd|DS)z?Return True if the url ends with one of the extensions providedc3BK|]}|VdSNr)rextlowercase_paths rrz(url_has_any_extension..&s1BB~&&s++BBBBBBr)rpathrr)r extensionsr+s @rurl_has_any_extensionr.#s@s^^(..00N BBBBzBBB B BBrcjt|tr|Stt||S)z\Return urlparsed url from the given argument (which could be an already parsed url) ) isinstancerrr )rencodings rrr)s3#{## JsH-- . ..rct|\}}|ds|St|d|ddS)a Return the crawlable url according to: https://developers.google.com/webmasters/ajax-crawling/docs/getting-started >>> escape_ajax("www.example.com/ajax.html#!key=value") 'www.example.com/ajax.html?_escaped_fragment_=key%3Dvalue' >>> escape_ajax("www.example.com/ajax.html?k1=v1&k2=v2#!key=value") 'www.example.com/ajax.html?k1=v1&k2=v2&_escaped_fragment_=key%3Dvalue' >>> escape_ajax("www.example.com/ajax.html?#!key=value") 'www.example.com/ajax.html?_escaped_fragment_=key%3Dvalue' >>> escape_ajax("www.example.com/ajax.html#!") 'www.example.com/ajax.html?_escaped_fragment_=' URLs that are not "AJAX crawlable" (according to Google) returned as-is: >>> escape_ajax("www.example.com/ajax.html#key=value") 'www.example.com/ajax.html#key=value' >>> escape_ajax("www.example.com/ajax.html#") 'www.example.com/ajax.html#' >>> escape_ajax("www.example.com/ajax.html") 'www.example.com/ajax.html' !_escaped_fragment_N)r startswithadd_or_replace_parameter)rdefragfrags r escape_ajaxr:2sG.S>>LFD ??3   #F,@$qrr( K KKrctjd|tj}|st|}|jrdnd}||z}|S)z=Add http as the default scheme if it is missing from the url.z^\w+://flagszhttp:zhttp://)rematchIrr)rr?partsschemes radd_http_if_no_schemerCOsK HZBD 1 1 1E  !L7isl Jrc^ttjd|tjS)Na ^ # start with... ( \. # ...a single dot, ( \. | [^/\.]+ # optionally followed by )? # either a second dot or some characters | ~ # $HOME )? # optional match of ".", ".." or ".blabla" / # at least one "/" for a file path, . # and something after the "/" r<)boolr>r?VERBOSEstrings r_is_posix_pathrIZs7    *     rcxttjd|tjtjzS)Nzg ^ ( [a-z]:\\ | \\\\ ) r<)rEr>r? IGNORECASErFrGs r_is_windows_pathrLps>    -"*,    rc>t|pt|Sr))rIrLrGs r_is_filesystem_pathrNs & ! ! =%5f%=%==rc\t|rt|St|S)zWAdd an URL scheme if missing: file:// for filepath-like input or http:// otherwise.)rN any_to_urirC)rs r guess_schemerQs.3#  % %%rTFcrt|}|j}|s|r)|js|jr|dd}|r5|jr.|j|jfdvr|d|jd}t|j||rdn|j |rdn|j |rdn|j |rdn|j fS)aStrip URL string from some of its components: - ``strip_credentials`` removes "user:password@" - ``strip_default_port`` removes ":80" (resp. ":443", ":21") from http:// (resp. https://, ftp://) URLs - ``origin_only`` replaces path component with "/", also dropping query and fragment components ; it also strips credentials - ``strip_fragment`` drops any #fragment component @))httpP)httpsi)ftp:/) rrusernamepasswordsplitportrBreplacerr,paramsqueryfragment)rstrip_credentialsstrip_default_port origin_onlystrip_fragment parsed_urlrs r strip_urlrjs$#J  F'['')2'c""2&?jo?  z /4   ? ^^$9 $9$92>>F     3CCJO 4BB:#4 3BB:#3 9BBj&9     rr))TTFT)__doc__r> urllib.parserrrr w3lib.urlrr scrapy.utils.pythonr rr&r.rr:rCrIrLrNrQrjr rrrosF EEEEEEEEEEEE////////******IIICCC ////LLL:,    >>>&&& ((((((r