高并发架构实践指南——资源部署（负载均衡器Nginx）

xuej@xuej-VirtualBox:~$ sudo apt list nginx -a
[sudo] password for xuej: 
Listing... Done
nginx/bionic-updates,bionic-updates,bionic-security,bionic-security 1.14.0-0ubuntu1.10 all
nginx/bionic,bionic 1.14.0-0ubuntu1 all

xuej@xuej-VirtualBox:~$ sudo apt install nginx
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream nginx-common nginx-core
Suggested packages:
  fcgiwrap nginx-doc
The following NEW packages will be installed:
  libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream nginx nginx-common nginx-core
0 upgraded, 8 newly installed, 0 to remove and 209 not upgraded.
Need to get 599 kB of archives.
After this operation, 2,120 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/main amd64 nginx-common all 1.14.0-0ubuntu1.10 [37.4 kB]
Get:2 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/main amd64 libnginx-mod-http-geoip amd64 1.14.0-0ubuntu1.10 [11.3 kB]
Get:3 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/main amd64 libnginx-mod-http-image-filter amd64 1.14.0-0ubuntu1.10 [14.6 kB]
Get:4 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/main amd64 libnginx-mod-http-xslt-filter amd64 1.14.0-0ubuntu1.10 [13.0 kB]
Get:5 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/main amd64 libnginx-mod-mail amd64 1.14.0-0ubuntu1.10 [42.1 kB]
Get:6 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/main amd64 libnginx-mod-stream amd64 1.14.0-0ubuntu1.10 [63.8 kB]
Get:7 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/main amd64 nginx-core amd64 1.14.0-0ubuntu1.10 [413 kB]
Get:8 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/main amd64 nginx all 1.14.0-0ubuntu1.10 [3,596 B]
Fetched 599 kB in 2s (297 kB/s)
Preconfiguring packages ...
Selecting previously unselected package nginx-common.
(Reading database ... 126590 files and directories currently installed.)
Preparing to unpack .../0-nginx-common_1.14.0-0ubuntu1.10_all.deb ...
Unpacking nginx-common (1.14.0-0ubuntu1.10) ...
Selecting previously unselected package libnginx-mod-http-geoip.
Preparing to unpack .../1-libnginx-mod-http-geoip_1.14.0-0ubuntu1.10_amd64.deb ...
Unpacking libnginx-mod-http-geoip (1.14.0-0ubuntu1.10) ...
Selecting previously unselected package libnginx-mod-http-image-filter.
Preparing to unpack .../2-libnginx-mod-http-image-filter_1.14.0-0ubuntu1.10_amd64.deb ...
Unpacking libnginx-mod-http-image-filter (1.14.0-0ubuntu1.10) ...
Selecting previously unselected package libnginx-mod-http-xslt-filter.
Preparing to unpack .../3-libnginx-mod-http-xslt-filter_1.14.0-0ubuntu1.10_amd64.deb ...
Unpacking libnginx-mod-http-xslt-filter (1.14.0-0ubuntu1.10) ...
Selecting previously unselected package libnginx-mod-mail.
Preparing to unpack .../4-libnginx-mod-mail_1.14.0-0ubuntu1.10_amd64.deb ...
Unpacking libnginx-mod-mail (1.14.0-0ubuntu1.10) ...
Selecting previously unselected package libnginx-mod-stream.
Preparing to unpack .../5-libnginx-mod-stream_1.14.0-0ubuntu1.10_amd64.deb ...
Unpacking libnginx-mod-stream (1.14.0-0ubuntu1.10) ...
Selecting previously unselected package nginx-core.
Preparing to unpack .../6-nginx-core_1.14.0-0ubuntu1.10_amd64.deb ...
Unpacking nginx-core (1.14.0-0ubuntu1.10) ...
Selecting previously unselected package nginx.
Preparing to unpack .../7-nginx_1.14.0-0ubuntu1.10_all.deb ...
Unpacking nginx (1.14.0-0ubuntu1.10) ...
Setting up nginx-common (1.14.0-0ubuntu1.10) ...
Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /lib/systemd/system/nginx.service.
Setting up libnginx-mod-http-image-filter (1.14.0-0ubuntu1.10) ...
Setting up libnginx-mod-mail (1.14.0-0ubuntu1.10) ...
Setting up libnginx-mod-http-xslt-filter (1.14.0-0ubuntu1.10) ...
Setting up libnginx-mod-http-geoip (1.14.0-0ubuntu1.10) ...
Setting up libnginx-mod-stream (1.14.0-0ubuntu1.10) ...
Setting up nginx-core (1.14.0-0ubuntu1.10) ...
Setting up nginx (1.14.0-0ubuntu1.10) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ufw (0.36-0ubuntu0.18.04.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for systemd (237-3ubuntu10.52) ...
xuej@xuej-VirtualBox:~$ sudo service nginx status 
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2022-07-21 22:42:40 CST; 1s ago
     Docs: man:nginx(8)
  Process: 6685 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
  Process: 6687 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 6686 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 6688 (nginx)
    Tasks: 2 (limit: 1118)
   CGroup: /system.slice/nginx.service
           ├─6688 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           └─6689 nginx: worker process

7月 21 22:42:40 xuej-VirtualBox systemd[1]: Stopped A high performance web server and a reverse proxy server.
7月 21 22:42:40 xuej-VirtualBox systemd[1]: Starting A high performance web server and a reverse proxy server...
7月 21 22:42:40 xuej-VirtualBox systemd[1]: nginx.service: Failed to parse PID from file /run/nginx.pid: Invalid argument
7月 21 22:42:40 xuej-VirtualBox systemd[1]: Started A high performance web server and a reverse proxy server.

xuej@xuej-VirtualBox:~$ echo "<h1>Welcome to nginx 1</h1>" | sudo tee /var/www/html/index.html
<h1>Welcome to nginx 1</h1>
xuej@xuej-VirtualBox:~$ curl localhost
<h1>Welcome to nginx 1</h1>

xuej@xuej-VirtualBox:~$ echo "<h1>Welcome to nginx 2</h1>" | sudo tee /var/www/html/index.html
<h1>Welcome to nginx 2</h1>
xuej@xuej-VirtualBox:~$ curl localhost
<h1>Welcome to nginx 2</h1>

xuej@xuej-VirtualBox:~$ sudo apt list keepalived -a
[sudo] password for xuej: 
Listing... Done
keepalived/bionic-updates,bionic-security 1:1.3.9-1ubuntu0.18.04.3 amd64
keepalived/bionic 1:1.3.9-1build1 amd64

xuej@xuej-VirtualBox:~$ sudo apt install keepalived
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  ipvsadm libnl-route-3-200
Suggested packages:
  heartbeat ldirectord
The following NEW packages will be installed:
  ipvsadm keepalived libnl-route-3-200
0 upgraded, 3 newly installed, 0 to remove and 209 not upgraded.
Need to get 425 kB of archives.
After this operation, 1,497 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/main amd64 libnl-route-3-200 amd64 3.2.29-0ubuntu3 [146 kB]
Get:2 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/main amd64 keepalived amd64 1:1.3.9-1ubuntu0.18.04.3 [244 kB]
Get:3 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/main amd64 ipvsadm amd64 1:1.28-3ubuntu0.18.04.1 [35.5 kB]
Fetched 425 kB in 1s (446 kB/s)
Selecting previously unselected package libnl-route-3-200:amd64.
(Reading database ... 126519 files and directories currently installed.)
Preparing to unpack .../libnl-route-3-200_3.2.29-0ubuntu3_amd64.deb ...
Unpacking libnl-route-3-200:amd64 (3.2.29-0ubuntu3) ...
Selecting previously unselected package keepalived.
Preparing to unpack .../keepalived_1%3a1.3.9-1ubuntu0.18.04.3_amd64.deb ...
Unpacking keepalived (1:1.3.9-1ubuntu0.18.04.3) ...
Selecting previously unselected package ipvsadm.
Preparing to unpack .../ipvsadm_1%3a1.28-3ubuntu0.18.04.1_amd64.deb ...
Unpacking ipvsadm (1:1.28-3ubuntu0.18.04.1) ...
Setting up libnl-route-3-200:amd64 (3.2.29-0ubuntu3) ...
Setting up ipvsadm (1:1.28-3ubuntu0.18.04.1) ...
Setting up keepalived (1:1.3.9-1ubuntu0.18.04.3) ...
Created symlink /etc/systemd/system/multi-user.target.wants/keepalived.service → /lib/systemd/system/keepalived.service.
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for dbus (1.12.2-1ubuntu1.2) ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for libc-bin (2.27-3ubuntu1.4) ...
Processing triggers for systemd (237-3ubuntu10.52) ...
xuej@xuej-VirtualBox:~$ sudo service keepalived start
xuej@xuej-VirtualBox:~$ sudo service keepalived status
● keepalived.service - Keepalive Daemon (LVS and VRRP)
   Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
Condition: start condition failed at Wed 2022-07-20 16:08:11 CST; 1s ago
           └─ ConditionFileNotEmpty=/etc/keepalived/keepalived.conf was not met

xuej@xuej-VirtualBox:~$ sudo vi /etc/keepalived/keepalived.conf 
[sudo] password for xuej:
global_defs {
  # Keepalived process identifier
  router_id nginx
}

# Script to check whether Nginx is running or not
vrrp_script check_nginx {
  script "/bin/check_nginx.sh"
  interval 2
  weight 50
}

# Virtual interface - The priority specifies the order in which the assigned interface to take over in a failover
vrrp_instance VI_01 {
  state MASTER
  interface enp0s8
  virtual_router_id 101
  priority 100

  # The virtual ip address shared between the two NGINX Web Server which will float
  virtual_ipaddress {
    192.168.246.100/24
  }
  track_script {
    check_nginx
  }
  authentication {
    auth_type PASS
    auth_pass 1111
  }
}
xuej@xuej-VirtualBox:~$ sudo vi /bin/check_nginx.sh
#!/bin/sh
if [ -z "`pidof nginx`" ]; then
  exit 1
fi
xuej@xuej-VirtualBox:~$ sudo chmod 755 /bin/check_nginx.sh
xuej@xuej-VirtualBox:~$ sudo systemctl restart keepalived
xuej@xuej-VirtualBox:~$ sudo systemctl status keepalived
● keepalived.service - Keepalive Daemon (LVS and VRRP)
   Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2022-07-21 22:31:44 CST; 3s ago
  Process: 4796 ExecStart=/usr/sbin/keepalived $DAEMON_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 4799 (keepalived)
    Tasks: 3 (limit: 1118)
   CGroup: /system.slice/keepalived.service
           ├─4799 /usr/sbin/keepalived
           ├─4800 /usr/sbin/keepalived
           └─4801 /usr/sbin/keepalived

7月 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: WARNING - default user 'keepalived_script' for script execution does not exist - please create.
7月 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
7月 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: Using LinkWatch kernel netlink reflector...
7月 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: VRRP_Script(check_nginx) succeeded
7月 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: VRRP_Instance(VI_01) Transition to MASTER STATE
7月 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: VRRP_Instance(VI_01) Received advert with higher priority 140, ours 100
7月 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: VRRP_Instance(VI_01) Entering BACKUP STATE
7月 21 22:31:45 xuej-VirtualBox Keepalived_vrrp[4801]: VRRP_Instance(VI_01) Changing effective priority from 100 to 150
7月 21 22:31:45 xuej-VirtualBox Keepalived_vrrp[4801]: VRRP_Instance(VI_01) forcing a new MASTER election
7月 21 22:31:46 xuej-VirtualBox Keepalived_vrrp[4801]: VRRP_Instance(VI_01) Transition to MASTER STATE
xuej@xuej-VirtualBox:~$ ip add show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:c7:81:a3 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute enp0s3
       valid_lft 69570sec preferred_lft 69570sec
    inet6 fe80::b771:3489:ec2a:1901/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:e1:c4:b7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.246.46/24 brd 192.168.246.255 scope global dynamic noprefixroute enp0s8
       valid_lft 498sec preferred_lft 498sec
    inet 192.168.246.100/24 scope global secondary enp0s8
       valid_lft forever preferred_lft forever
    inet6 fe80::57b0:d3e4:facf:7a3f/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
xuej@xuej-VirtualBox:~$ ip addr show enp0s8
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:e1:c4:b7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.246.46/24 brd 192.168.246.255 scope global dynamic noprefixroute enp0s8
       valid_lft 445sec preferred_lft 445sec
    inet 192.168.246.100/24 scope global secondary enp0s8
       valid_lft forever preferred_lft forever
    inet6 fe80::57b0:d3e4:facf:7a3f/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
xuej@xuej-VirtualBox:~$ tail -40 /var/log/syslog
Jul 21 22:31:43 xuej-VirtualBox systemd[1]: Stopping Keepalive Daemon (LVS and VRRP)...
Jul 21 22:31:43 xuej-VirtualBox Keepalived[30473]: Stopping
Jul 21 22:31:43 xuej-VirtualBox Keepalived_vrrp[30475]: VRRP_Instance(VI_01) sent 0 priority
Jul 21 22:31:43 xuej-VirtualBox avahi-daemon[507]: Withdrawing address record for 192.168.246.100 on enp0s8.
Jul 21 22:31:43 xuej-VirtualBox Keepalived_healthcheckers[30474]: Stopped
Jul 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[30475]: Stopped
Jul 21 22:31:44 xuej-VirtualBox Keepalived[30473]: Stopped Keepalived v1.3.9 (10/21,2017)
Jul 21 22:31:44 xuej-VirtualBox systemd[1]: Stopped Keepalive Daemon (LVS and VRRP).
Jul 21 22:31:44 xuej-VirtualBox systemd[1]: Starting Keepalive Daemon (LVS and VRRP)...
Jul 21 22:31:44 xuej-VirtualBox Keepalived[4796]: Starting Keepalived v1.3.9 (10/21,2017)
Jul 21 22:31:44 xuej-VirtualBox Keepalived[4796]: Opening file '/etc/keepalived/keepalived.conf'.
Jul 21 22:31:44 xuej-VirtualBox systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Jul 21 22:31:44 xuej-VirtualBox Keepalived[4799]: Starting Healthcheck child process, pid=4800
Jul 21 22:31:44 xuej-VirtualBox Keepalived[4799]: Starting VRRP child process, pid=4801
Jul 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: Registering Kernel netlink reflector
Jul 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: Registering Kernel netlink command channel
Jul 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: Registering gratuitous ARP shared channel
Jul 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: Opening file '/etc/keepalived/keepalived.conf'.
Jul 21 22:31:44 xuej-VirtualBox Keepalived_healthcheckers[4800]: Opening file '/etc/keepalived/keepalived.conf'.
Jul 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: WARNING - default user 'keepalived_script' for script execution does not exist - please create.
Jul 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Jul 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: Using LinkWatch kernel netlink reflector...
Jul 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: VRRP_Script(check_nginx) succeeded
Jul 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: VRRP_Instance(VI_01) Transition to MASTER STATE
Jul 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: VRRP_Instance(VI_01) Received advert with higher priority 140, ours 100
Jul 21 22:31:44 xuej-VirtualBox Keepalived_vrrp[4801]: VRRP_Instance(VI_01) Entering BACKUP STATE
Jul 21 22:31:45 xuej-VirtualBox Keepalived_vrrp[4801]: VRRP_Instance(VI_01) Changing effective priority from 100 to 150
Jul 21 22:31:45 xuej-VirtualBox Keepalived_vrrp[4801]: VRRP_Instance(VI_01) forcing a new MASTER election
Jul 21 22:31:46 xuej-VirtualBox Keepalived_vrrp[4801]: VRRP_Instance(VI_01) Transition to MASTER STATE
Jul 21 22:31:47 xuej-VirtualBox Keepalived_vrrp[4801]: VRRP_Instance(VI_01) Entering MASTER STATE
Jul 21 22:31:47 xuej-VirtualBox avahi-daemon[507]: Registering new address record for 192.168.246.100 on enp0s8.IPv4.
Jul 21 22:32:45 xuej-VirtualBox NetworkManager[476]: <info>  [1658413965.9765] connectivity: (enp0s3) timed out
Jul 21 22:32:45 xuej-VirtualBox NetworkManager[476]: <info>  [1658413965.9766] manager: NetworkManager state is now CONNECTED_SITE
Jul 21 22:32:45 xuej-VirtualBox whoopsie[892]: [22:32:45] offline
Jul 21 22:32:45 xuej-VirtualBox dbus-daemon[452]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.12' (uid=0 pid=476 comm="/usr/sbin/NetworkManager --no-daemon " label="unconfined")
Jul 21 22:32:45 xuej-VirtualBox systemd[1]: Starting Network Manager Script Dispatcher Service...
Jul 21 22:32:45 xuej-VirtualBox dbus-daemon[452]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Jul 21 22:32:45 xuej-VirtualBox systemd[1]: Started Network Manager Script Dispatcher Service.
Jul 21 22:32:45 xuej-VirtualBox nm-dispatcher: req:1 'connectivity-change': new request (1 scripts)
Jul 21 22:32:45 xuej-VirtualBox nm-dispatcher: req:1 'connectivity-change': start running ordered scripts...

xuej@xuej-VirtualBox:~$ sudo vi /etc/keepalived/keepalived.conf 
[sudo] password for xuej:
global_defs {
  # Keepalived process identifier
  router_id nginx
}

# Script to check whether Nginx is running or not
vrrp_script check_nginx {
  script "/bin/check_nginx.sh"
  interval 2
  weight 50
}

# Virtual interface - The priority specifies the order in which the assigned interface to take over in a failover
vrrp_instance VI_01 {
  state BACKUP
  interface enp0s8
  virtual_router_id 101
  priority 90

  # The virtual ip address shared between the two NGINX Web Server which will float
  virtual_ipaddress {
    192.168.246.100/24
  }
  track_script {
    check_nginx
  }
  authentication {
    auth_type PASS
    auth_pass 1111
  }
}
xuej@xuej-VirtualBox:~$ sudo vi /bin/check_nginx.sh
#!/bin/sh
if [ -z "`pidof nginx`" ]; then
  exit 1
fi
xuej@xuej-VirtualBox:~$ sudo chmod 755 /bin/check_nginx.sh
xuej@xuej-VirtualBox:~$ sudo systemctl restart keepalived

xuej@xuej-VirtualBox:~$ curl 192.168.246.100
<h1>Welcome to nginx 1</h1>
xuej@xuej-VirtualBox:~$ sudo service nginx stop
[sudo] password for xuej: 
xuej@xuej-VirtualBox:~$ curl 192.168.246.100
curl: (7) Failed to connect to 192.168.246.100 port 80: Connection refused
xuej@xuej-VirtualBox:~$ curl 192.168.246.100
<h1>Welcome to nginx 2</h1>
xuej@xuej-VirtualBox:~$ sudo service nginx start
xuej@xuej-VirtualBox:~$ curl 192.168.246.100
<h1>Welcome to nginx 1</h1>