Wget and https

GNU Wget is a popular file download program, being installed by default on many Linux distributions. Recent Mac OS versions don’t ship Wget, though – Apple ships cURL instead.

Fink provides a wget package that installs Wget. It includes SSL (https) support provided by Mac OS built-in OpenSSL.  There’s a problem with that, though: on Mac OS versions earlier than 10.6, Apple’s OpenSSL doesn’t use the trusted root certificates available on the system (the ones listed by Keychain.app), so it is not able to validate SSL certificates on its own. Note that OpenSSL itself (independently of being shipped with Mac OS) isn’t distributed with root certificates by default.

Because of this, on Mac OS versions earlier than 10.6 the command

wget https://fedorahosted.org

won’t work:

ERROR: cannot verify fedorahosted.org's certificate,
issued by `/C=US/O=Equifax/OU=Equifax Secure
Certificate Authority':
Unable to locally verify the issuer's authority.
To connect to fedorahosted.org insecurely, use
`--no-check-certificate'.
Unable to establish SSL connection.

There are a couple of options to circumvent this. As the error message says, it’s possible to use –no-check-certificate, which is insecure. Another option is –ca-certificate=file where file is a bundle of trusted certification authority certificates. Fink provides a package called ca-bundle that installs a convenient file containing a bundle of CA certificates commonly used by open source Web browsers. After running

fink install ca-bundle

you should be able to use /sw/etc/ssl/certs/ca-bundle.crt with Wget:

wget --ca-certificate=/sw/etc/ssl/certs/ca-bundle.crt \
https://fedorahosted.org

Fortunately, you may specify that option in one of Wget’s startup files (e.g. $HOME/.wgetrc or /sw/etc/wgetrc) by adding the following line to your startup file of choice:

ca_certificate = /sw/etc/ssl/certs/ca-bundle.crt

And voilà!, you may use wget as usual:

wget https://fedorahosted.org

This is particularly useful if you’re using Wget as your DownloadMethod and Fink needs to download a source file from an https URL.

7 responses to “Wget and https

  1. Weird, on what MacOS X version is this? On my 10.6, wget does not complain for that host, even when I don’t configure a CA and “/System/Library/OpenSSL/certs/” is empty (I do have the ca-bundle though).

    One explanation is that I remembered that Apple modified OpenSSL to check the “System Roots” keychain. The best evidence I can find of this is here:
    http://opensource.apple.com/source/OpenSSL098/OpenSSL098-27/src/crypto/x509/x509_vfy_apple.c

  2. Thanks, Bruno. I see they’ve patched 10.6’s OpenSSL to use Mac OS security API if OpenSSL alone isn’t able to verify a X.509 certificate:

    http://opensource.apple.com/source/OpenSSL098/OpenSSL098-27/src/crypto/x509/x509_vfy_apple.h

    I use Mac OS 10.5.8 (OpenSSL 0.97l) which hasn’t got that patch:

    http://opensource.apple.com/source/OpenSSL097/OpenSSL097-16/openssl/crypto/x509/x509_vfy.c

  3. I’ve added this information to be viewable under “fink info wget”. I didn’t want to add a dependency on ca-bundle and to patch %p/etc/wgetrc; at least not yet.

  4. However, I’ve found that even on 10.6 with a wgetrc modified as above, I can’t fetch packages from github:

    wget -U 'fink/0.29.99.cvs' --verbose --passive-ftp -O pdfkittool-1.0-1.tar.gz http://github.com/fjoachim/pdfkittool/tarball/c872a2a49f9ec85056f7d855ee78ca5bac18e48a
    --2011-02-13 19:04:13--  http://github.com/fjoachim/pdfkittool/tarball/c872a2a49f9ec85056f7d855ee78ca5bac18e48a
    Resolving github.com (github.com)... 207.97.227.239
    Connecting to github.com (github.com)|207.97.227.239|:80... connected.
    HTTP request sent, awaiting response... 301 Moved Permanently
    Location: https://github.com/fjoachim/pdfkittool/tarball/c872a2a49f9ec85056f7d855ee78ca5bac18e48a [following]
    --2011-02-13 19:04:15--  https://github.com/fjoachim/pdfkittool/tarball/c872a2a49f9ec85056f7d855ee78ca5bac18e48a
    Connecting to github.com (github.com)|207.97.227.239|:443... connected.
    ERROR: certificate common name “*.github.com” doesn’t match requested host name “github.com”.
    To connect to github.com insecurely, use ‘--no-check-certificate’.
    ### execution of wget failed, exit code 5
    Downloading the file "pdfkittool-1.0-1.tar.gz" failed.
  5. wget –no-check-certificate –secure-protocol=auto https://IP

    It is using self signed certificate
    I have tried the above but still getting error
    Unable to establish SSL connection.

    any advice

  6. Öncelikle yazınız için teşekkür.ederiz. Böyle yazıların bilgilendirici nitelikte olduğunu düşünüyoruz. Tekrar teşekkürler.

  7. Pingback: Configuring Wget to Make a Readable Offline Copy of a WordPress.com Blog | Ray Woodcock's Latest

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s