[CVE-2024-11168] urlparse incorrectly retrieves IPv4 and regular name hosts from inside of brackets

[JohnJamesUtley]

Background

RFC 3986 defines a host as follows

host = IP-literal / IPv4address / reg-name

Where

IP-literal = "[" ( IPv6address / IPvFuture  ) "]"
reg-name = *( unreserved / pct-encoded / sub-delims )
IPv4address = dec-octet "." dec-octet "." dec-octet "." dec-octet

WhatWG says that "A valid host string must be a valid domain string, a valid IPv4-address string, or: U+005B ([), followed by a valid IPv6-address string, followed by U+005D (])."

The Bug

This is code from Lib/urllib/parse.py:196-208 used for retrieving the hostname from the netloc

    @property
    def _hostinfo(self):
        netloc = self.netloc
        _, _, hostinfo = netloc.rpartition('@')
        _, have_open_br, bracketed = hostinfo.partition('[')
        if have_open_br:
            hostname, _, port = bracketed.partition(']')
            _, _, port = port.partition(':')
        else:
            hostname, _, port = hostinfo.partition(':')
        if not port:
            port = None
        return hostname, port

It will incorrectly retrieve IPv4 addresses and regular name hosts from inside brackets. This is in violation of both specifications.

Minimally reproducible example:

from urllib.parse import urlsplit

parsedURL = urlsplit('scheme://user@[regname]/Path')
print(parsedURL.hostname) # Prints 'regname'

Your environment

• CPython versions tested on:
  • 3.12a7 (23cf1e2)
  • 3.10.10
• Operating system and architecture:
  • Arch Linux x86_64

Linked PRs

• gh-103848: Adds checks to ensure that bracketed hosts found by urlsplit are of IPv6 or IPvFuture format #103849
• [3.11] gh-103848: Adds checks to ensure that bracketed hosts found by urlsplit are of IPv6 or IPvFuture format (GH-103849) #104349
• [3.10] gh-103848: Adds checks to ensure that bracketed hosts found by urlsplit are of IPv6 or IPvFuture format (#103849) #126975
• [3.9] gh-103848: Adds checks to ensure that bracketed hosts found by urlsplit are of IPv6 or IPvFuture format (#103849) #126976

[mjpieters]

As implemented, the URL scheme://foobar.[::1]/ is considered correct, but results in the hostname value of ::1. The hostname portion of the netloc is invalid however, as it doesn't start with [. Shouldn't the parser at least validate that the [ bracket is the first character or directly following a @?

[MarcoGlauser]

This change caused an issue on our system. We have brackets in a password. Since 3.11.4, urlparse tries to parse an ip address if it finds brackets. The brackets don't even have to be in the correct order.

Running this code:

from urllib.parse import urlparse
urlparse("https://user:some]password[@host.com")

In 3.11.3 it works as expected:

ParseResult(scheme='https', netloc='user:some]password[@host.com', path='', params='', query='', fragment='')

In 3.11.4 it throws an exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.11/urllib/parse.py", line 395, in urlparse
    splitresult = urlsplit(url, scheme, allow_fragments)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/urllib/parse.py", line 500, in urlsplit
    _check_bracketed_host(bracketed_host)
  File "/usr/local/lib/python3.11/urllib/parse.py", line 446, in _check_bracketed_host
    ip = ipaddress.ip_address(hostname) # Throws Value Error if not IPv6 or IPv4
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ipaddress.py", line 54, in ip_address
    raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
ValueError: '@host.com' does not appear to be an IPv4 or IPv6 address

[mjpieters]

This change caused an issue on our system. We have brackets in a password. Since 3.11.4, urlparse tries to parse an ip address if it finds brackets. The brackets don't even have to be in the correct order.

Running this code:

from urllib.parse import urlparse
urlparse("https://user:some]password[@host.com")

In 3.11.3 it works as expected:

ParseResult(scheme='https', netloc='user:some]password[@host.com', path='', params='', query='', fragment='')

That this worked before was a bug. urlparse() was violating the various and miriad URL standards before; [ and ] are reserved characters with meaning in the netloc portion of a URL (the part between the scheme:// and /path) and must be URL encoded. See the WhatWG living URL standard percent-encoding section; for the userinfo part (username + password), you need to encode:

• U+002F (/), U+003A (:), U+003B (;), U+003D (=), U+0040 (@), U+005B ([) to U+005E (^), inclusive, and U+007C (|).
• U+003F (?), U+0060 (`), U+007B ({), and U+007D (}) (path percent-encode set).
• U+0020 SPACE, U+0022 ("), U+0023 (#), U+003C (<), and U+003E (>) (query percent-encode set).
• U+0000 NULL to U+001F INFORMATION SEPARATOR ONE, inclusive, and all codepoints past U+007E (~) (C0 control percent-encode set)

So this works:

>>> from urllib.parse import urlparse
>>> urlparse("https://user:some%5Dpassword%5B@host.com")
ParseResult(scheme='https', netloc='user:some%5Dpassword%5B@host.com', path='', params='', query='', fragment='')
>>> _.password
'some%5Dpassword%5B'

The password is still URL encoded, use urllib.parse.unquote() for that.

Further references:

• Wikipedia: URL encoding
• RFC 3986, section 2.2 Reserved Characters