Nginx with Varnish : all listen directive pointing to 808* ports but nginx still listen to 80

I’m running instances of Symfony or Drupal websites on two Debian servers, with Nginx listening to 443, Varnish listening to 80 and passing to nginx on listening custom ports 80** for each vhost.

Recently I added a new website to one of the servers. Then I began to run in this well documented error nginx: [emerg] bind() to [::]:80 failed (98: Address already in use).

Despite there is no nginx server block at all listening to :80 port, neither any server block without listen directive, Nginx began to listen on port 80 all together with the custom ports.

sudo netstat -tlpn| grep nginx
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      4191/nginx: master  
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      4191/nginx: master  
tcp        0      0 0.0.0.0:8081            0.0.0.0:*               LISTEN      4191/nginx: master  
tcp        0      0 x.x.x.x:8082            0.0.0.0:*               LISTEN      4191/nginx: master  
tcp        0      0 y.y.y.y:8083            0.0.0.0:*               LISTEN      4191/nginx: master  
tcp        0      0 z.z.z.z:8084            0.0.0.0:*               LISTEN      4191/nginx: master  
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      4191/nginx: master  
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      4191/nginx: master  
tcp6       0      0 :::8080                 :::*                    LISTEN      4191/nginx: master  
tcp6       0      0 :::80                   :::*                    LISTEN      4191/nginx: master  
tcp6       0      0 :::8081                 :::*                    LISTEN      4191/nginx: master  
tcp6       0      0 :::443                  :::*                    LISTEN      4191/nginx: master  
tcp6       0      0 :::8000                 :::*                    LISTEN      4191/nginx: master

I also already read all the docs and posts about handling dual-stack IPv4 and IPv6 correct new syntax, and tried all possible syntaxes such as below, no way.

Working directive before crash : listen x.x.x.x:8082;
Tried adding listen [::]:8082 ipv6only=on;. No change.

I listed, and killed process many times with sudo fuser -k 80/tcp before restarting systemctl varnish, nginx, even daemon-reload…

Last, I checked my history but can’t find what could have caused this sudden behavior. The lone point I’m not sure about is I changed a couple of sysctl.conf params, but hopefully reverted them, just in case, I’m not used to this part od administration : cat /etc/sysctl.conf | grep net.ipv4.conf

#net.ipv4.conf.default.rp_filter=1
#net.ipv4.conf.all.rp_filter=1
#net.ipv4.conf.all.accept_redirects = 0
# net.ipv4.conf.all.secure_redirects = 1
#net.ipv4.conf.all.send_redirects = 0
#net.ipv4.conf.all.accept_source_route = 0
#net.ipv4.conf.all.log_martians = 1

Here’s my configuration.

cat /etc/nginx/nginx.conf (relevant 2 lines, no html / server block in it)

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;

cat /etc/nginx/conf.d/default.conf

server {
        listen 8000 default_server;
        listen [::]:8000 ipv6only=on default_server;
        server_name _;

        listen 443 ssl default_server;
        listen [::]:443 ssl ipv6only=on default_server;
}

One of the sites-available vhosts (they all follow exactly same pattern) :

server { # this block only redirects www to non www
        listen x.x.x.x:443 ssl;
        server_name www.example.com;

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_certificate /var/www/clients/client0/web3/ssl/example.com-le.crt;
        ssl_certificate_key /var/www/clients/client0/web3/ssl/example.com-le.key;

        return 301 https://example.com$request_uri;
        }

server {
        listen x.x.x.x:443 ssl;
        server_name example.com

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_certificate /var/www/clients/client0/web3/ssl/example.com-le.crt;
        ssl_certificate_key /var/www/clients/client0/web3/ssl/example.com-le.key;

        location / {
            # Pass the request on to Varnish.
            proxy_pass  http://127.0.0.1;
 
            # Pass some headers to the downstream server, so it can identify the host.
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 
            # Tell any web apps like Drupal that the session is HTTPS.
            proxy_set_header X-Forwarded-Proto https;
            proxy_redirect     off;
        }
        
}
server {
        listen x.x.x.x:8082;
#       listen [::]:8082 ipv6only=on;

        server_name example.com www.example.com;

        root   /var/www/example.com/web/public;

        location / {
            # try to serve file directly, fallback to index.php
            try_files $uri /index.php$is_args$args;
        }

       location ~ ^/index.php(/|$) {
            fastcgi_pass 127.0.0.1:8998;
            fastcgi_split_path_info ^(.+.php)(/.*)$;
            include fastcgi_params;
            fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
            fastcgi_param DOCUMENT_ROOT $realpath_root;
            internal;
        }
        location ~ .php$ {
           # return 404;
        }

        error_log /var/log/ispconfig/httpd/example.com/error.log;
        access_log /var/log/ispconfig/httpd/example.com/access.log combined;

        location ~ /. {
                        deny all;
        }

        location ^~ /.well-known/acme-challenge/ {
             access_log off;
             log_not_found off;
             root /usr/local/ispconfig/interface/acme/;
             autoindex off;
             try_files $uri $uri/ =404;
        }

        location = /favicon.ico {
            log_not_found off;
            access_log off;
            expires max;
            add_header Cache-Control "public, must-revalidate, proxy-revalidate";
        }

        location = /robots.txt {
            allow all;
            log_not_found off;
            access_log off;
        }
}

cat /etc/default/varnish relevant part

DAEMON_OPTS="-a :80 
             -T localhost:6082 
             -f /etc/varnish/default.vcl 
             -S /etc/varnish/secret 
             -s malloc,3G"

I’m wondering what could have caused a config I’m working with since years to bug ?

I carefully studied these Q&A and a bunch of doc or posts, with no success : Nginx tries to run on port 80 but the configs have been removed ; Nginx will not start (Address already in use) ; nginx – bind() to 0.0.0.0:80 failed (98: Address already in use)

Go to Source
Author: Kojo