48. Fun with ureq, webpki, and IP addresses

If we try to open an HTTPS connection with ureq, and the host part of the URL is an IP address, then we get an InvalidDNSNameError. Let’s figure out what’s going on and how to work around it.

To start off, let’s see the actual error. We use ureq::get to construct the GET request to https://1.1.1.1, we execute it with call, and we convert the response to a string to make sure there’s no laziness happening.

fn experiment1() -> OrError<()> {
    let resp: String = ureq::get("https://1.1.1.1").call()?.into_string()?;
    println!("Response:\n{}", resp);
    Ok(())
}

Running it, experiment1 fails with the following error:

$ ./target/debug/post-48-demo experiment1
Error: https://1.1.1.1/: Dns Failed: InvalidDNSNameError

Caused by:
    InvalidDNSNameError

Where’s that error coming from? Following the code and skipping the uninteresting bits:

• ureq::get constructs a Request struct with the URL as a string,

• the URL is then parsed at request.rs:107,

• the parsed URL is passed to Unit::connect,

• we dispatch on the scheme (http or https) in unit.rs:343, and finally,

• the fateful call is made to webpki::DNSNameRef::try_from_ascii_str.

That is the call that fails with InvalidDNSNameError. The issue for this is webpki#54, and that roughly says that IP addresses aren’t valid CNs according to the spec.

So, what do we do? A comment in issue ureq#393 suggests a workaround, so let’s try that. The gist of it is that we’re going to replace the IP address in the URL with a dummy hostname, configure a DNS resolver to respond with the IP address for the dummy host, and configure a certificate verifier which ignores the hostname.

fn experiment3() -> OrError<()> {
    use certificate_verifier_no_hostname::CertificateVerifierNoHostname;
    use fixed_resolver::FixedResolver;
    use rustls::ClientConfig;
    use std::sync::Arc;
    use url::Url;

    // `original_url` is our original URL, `ip_addr` is the extracted
    // IP address, and `url` is the updated URL with the dummy
    // hostname.
    let original_url: Url = "https://1.1.1.1".parse()?;
    let ip_addr = original_url.host().unwrap().to_string();
    #[allow(clippy::redundant_clone)]
    let mut url = original_url.clone();
    url.set_host(Some("dummy-hostname-for-workaround"))?;

    // Create a TLS config with the default certificate roots and a
    // certificate verifier that ignores hostnames.
    //
    // TODO Configure just your own CA here to get any security from
    // the certificate verification.  Otherwise, any valid certificate
    // will be accepted.
    let tls_config = ClientConfig::builder().with_safe_defaults();
    let tls_config =
        tls_config.with_custom_certificate_verifier(Arc::new(CertificateVerifierNoHostname {
            trustroots: &webpki_roots::TLS_SERVER_ROOTS,
        }));
    let tls_config = tls_config.with_no_client_auth();

    // Execute the `ureq` call with the special TLS config and a
    // resolver that always returns `ip_addr`.
    //
    // The extra tricks here are that we have to specify a port so
    // that we can parse a `SocketAddr` (although it's not used for
    // anything), and we have to set the `Host` header in case the
    // remote host is dispatching on that.
    let resp: String = ureq::builder()
        .tls_config(Arc::new(tls_config))
        .resolver(FixedResolver {
            ip_addr: format!("{}:443", ip_addr).parse()?,
        })
        .build()
        .request_url("GET", &url)
        .set("Host", &ip_addr)
        .call()?
        .into_string()?;
    println!("Response: {} chars", resp.len());
    Ok(())
}

mod fixed_resolver {
    //! A DNS resolver that always returns the same value regardless of
    //! the host it was queried for.

    use std::net::SocketAddr;

    pub struct FixedResolver {
        pub ip_addr: SocketAddr,
    }

    impl ureq::Resolver for FixedResolver {
        fn resolve(&self, _netloc: &str) -> std::io::Result<Vec<SocketAddr>> {
            Ok(vec![self.ip_addr])
        }
    }
}

mod certificate_verifier_no_hostname {
    use rustls::{
        client::{ServerCertVerified, ServerCertVerifier},
        Certificate, Error as TlsError, ServerName,
    };
    use std::time::SystemTime;
    use webpki::{EndEntityCert, SignatureAlgorithm, TlsServerTrustAnchors};

    pub struct CertificateVerifierNoHostname<'a> {
        pub trustroots: &'a TlsServerTrustAnchors<'a>,
    }

    static SUPPORTED_SIG_ALGS: &[&SignatureAlgorithm] = &[
        &webpki::ECDSA_P256_SHA256,
        &webpki::ECDSA_P256_SHA384,
        &webpki::ECDSA_P384_SHA256,
        &webpki::ECDSA_P384_SHA384,
        &webpki::ED25519,
        &webpki::RSA_PSS_2048_8192_SHA256_LEGACY_KEY,
        &webpki::RSA_PSS_2048_8192_SHA384_LEGACY_KEY,
        &webpki::RSA_PSS_2048_8192_SHA512_LEGACY_KEY,
        &webpki::RSA_PKCS1_2048_8192_SHA256,
        &webpki::RSA_PKCS1_2048_8192_SHA384,
        &webpki::RSA_PKCS1_2048_8192_SHA512,
        &webpki::RSA_PKCS1_3072_8192_SHA384,
    ];

    impl ServerCertVerifier for CertificateVerifierNoHostname<'_> {
        /// Will verify the certificate is valid in the following ways:
        /// - Signed by a valid root
        /// - Not Expired
        ///
        /// Based on a https://github.com/ctz/rustls/issues/578#issuecomment-816712636
        fn verify_server_cert(
            &self,
            end_entity: &Certificate,
            intermediates: &[Certificate],
            _server_name: &ServerName,
            _scts: &mut dyn Iterator<Item = &[u8]>,
            ocsp_response: &[u8],
            _now: SystemTime,
        ) -> Result<ServerCertVerified, TlsError> {
            let end_entity_cert = webpki::EndEntityCert::try_from(end_entity.0.as_ref())
                .map_err(|err| TlsError::General(err.to_string()))?;

            let chain: Vec<&[u8]> = intermediates.iter().map(|cert| cert.0.as_ref()).collect();

            // Validate the certificate is valid, signed by a trusted root, and not
            // expired.
            let now = SystemTime::now();
            let webpki_now =
                webpki::Time::try_from(now).map_err(|_| TlsError::FailedToGetCurrentTime)?;

            let _cert: EndEntityCert = end_entity_cert
                .verify_is_valid_tls_server_cert(
                    SUPPORTED_SIG_ALGS,
                    self.trustroots,
                    &chain,
                    webpki_now,
                )
                .map_err(|err| TlsError::General(err.to_string()))
                .map(|_| end_entity_cert)?;

            if !ocsp_response.is_empty() {
                //trace!("Unvalidated OCSP response: {:?}", ocsp_response.to_vec());
            }
            Ok(ServerCertVerified::assertion())
        }
    }
}

The comments in the source explain what does what. More importantly, the code has several limitations:

• The custom certificate verifier is scary because it will accept any valid certificate, regardless of who it was issued for. In practice, this might be fine if we’re using our own certificates. In that case, we can configure our CA as the only root CA (add_server_trust_anchors), and then an adversary would have to present one of our own certificates in order to spoof our server. However, that shouldn’t be possible unless we’ve already been compromised.

• The certificate verifier has a hardcoded list of ciphers, which might get outdated.

• The code doesn’t handle redirects at all.

• The ureq::Agent struct can’t be reused on different hosts because of the dummy resolver.

To conclude, connecting to a host specified by its IP address over HTTPS doesn’t work in ureq at the moment, but with a bit of misdirection and a hundred lines of code, we can work around it.