Nginx: 99: Cannot assign requested address for upstream

A few days ago, I ran into the following interesting Nginx error message.

[crit] 12889#0: *32401195 connect() to 127.0.0.1:80 failed (99: Cannot assign requested address) while connecting to upstream

My configuration was very simple. This was an Nginx proxy that did all the SSL encryption and sent traffic to a Varnish instance, running on port :80 on localhost. The big takeway here is that it was a pretty high traffic Nginx proxy.
Even with keepalive enabled in the nginx upstream, the error popped up. But what did it mean?
TCP ports and limits
It's good to know a thing or two about networking besides just servers once in a while. The problem occurred because the server couldn't get a free TCP port quickly enough to make the connection to  127.0.0.1 .

$ ss -s
Total: 3130 (kernel 3431)
TCP:   51582 (estab 2866, closed 48611, orphaned 92, synrecv 0, timewait 48611/0), ports 35279

The ss tool gives you stats on the sockets/ports on the server. In this case, I had 51.582 TCP sessions in use (either active, closed, awaiting to be closed, ...).
A normal server has around 28.000 possible TCP ports it can use to make a TCP connection to a remote (or local) system. Everything that talks via an IP address will pick a free port from this range to serve as source port for the outgoing connection. This port range is defined by the  ip_local_port_range  sysctl parameter.

$ cat /proc/sys/net/ipv4/ip_local_port_range
32768	61000

$ sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 32768	61000

The format is "minimum maximum" port. So 61000 – 32768 = 28 232 available source ports.
An nginx SSL proxy that connects to a Varnish instance running on localhost will look like this in your  netstat .

$ netstat -anp | grep 127.0.0.1
...
tcp        0      0 127.0.0.1:37713         127.0.0.1:80            TIME_WAIT   -

The key takeaways here the source connection  127.0.0.1:37713  that connects to its endpoint  127.0.0.1:80 . For every source connection a new TCP source port is selected from the range in the  ip_local_port_range  parameter.
The combination of a source IP, source port, destination IP and destination IP needs to be unique. This is what's called a quadruplet in networking terms. You likely can't (easily) change the source IP. The source port is dynamically picked. That only leaves the destination IP and the destination port that are free to play with.
Solving the source port limitation
There are a couple of easy fixes. First, the  ip_local_port_range  can be increased on a Linux machine (for more reading material, see increase ip_local_port_range TCP port range in Linux).

$ echo 15000 64000 > /proc/sys/net/ipv4/ip_local_port_range

This effectively increases the total port range from its default 28 232 ports to 49 000 ports.
If that's not enough, you can add more destination IPs to connect to. Remember that each connection consists of the 4 parts (called quadruplets) with source IP and source port, destination IP and destination port. If you can't change the source port or IP, just change the destination IPs.
Consider this kind of upstream definition in Nginx;

upstream varnish {
  server 127.0.0.1:80;
}

Such a definition can be used in your nginx configurations with the  proxy_pass  directive.

server {
  listen 443;
  ...
  location / {
    proxy_pass http://varnish;
    ...
  }
}

Now if you know that each server usually has 2 IPs or more, it's very easy to add more quadruplets to your networking stack by adding an addition IP to your Nginx upstream. You've already added  127.0.0.1 , but your server will have another IP (its public or DHCP IP) that you can safely add too, if your webserver binds to all ports.

upstream varnish {
  server 127.0.0.1:80;
  server 31.193.180.217:80;
  server 10.50.5.1:80;
  ...
}

Every IP you add in your upstream is effectively adding 49.000 local ports to your networking stack. You can even add non-routable local IPs to your server, as interface aliases, just to use as new destination IPs for your proxy configurations.
Hi! My name is Mattias Geniar. I'm a Support Manager at Nucleus Hosting in Belgium, a general web geek, public speaker and podcaster. If you're interested in keeping up with me, have a look at my podcast and weekly newsletter below. For more updates, follow me on Twitter as @mattiasgeniar.
zhuanzi:  https://ma.ttias.be/nginx-cannot-assign-requested-address-for-upstream/