This article systematically breaks down the complete implementation of Nginx TCP Layer 4 port proxy and mTLS mutual encryption authentication. It covers core technical principles (TLS/mTLS mechanisms), certificate generation (root CA/server/client workflows), Nginx configuration (Stream module, SSL parameter optimization), and function verification (valid/invalid connection testing) with practical commands. It helps DevOps engineers and developers quickly build secure communication channels, addressing risks like data leakage and unauthorized access in traditional proxy architectures, suitable for encrypted proxy scenarios of TCP services such as Redis and databases.
1. Exploring the Technical Background
In the process of digital transformation, network security and efficient proxy technology have become core components of modern network architectures. As enterprises expand their business scales and business scenarios become more complex, network communications face multiple security threats such as man-in-the-middle attacks, data theft, and information tampering, placing higher demands on the security and reliability of communication architectures.
Traditional network proxy solutions have limitations in addressing complex security scenarios. As a high-performance HTTP and reverse proxy server, Nginx occupies a crucial position in the proxy field due to its stability, efficiency, and rich functional modules. Nginx not only supports Layer 7 proxy for the HTTP protocol but also enables Layer 4 proxy for TCP and UDP protocols through the Stream module, providing a flexible and high-performance solution for network communications.
mTLS (mutual TLS) mutual encryption authentication technology is a key means to ensure secure network communications. Traditional one-way TLS authentication only enables the client to authenticate the server’s identity, posing the risk of attackers impersonating legitimate servers to steal data. mTLS mutual authentication requires both communicating parties to verify each other’s identities. Through a certificate verification mechanism, it ensures the legitimacy of both parties’identities, effectively preventing man-in-the-middle attacks and data leakage, and safeguarding the confidentiality, integrity, and availability of data transmission.
In data-sensitive industries—such as scenarios like customer transaction information processing in the financial sector, private medical record transmission in the healthcare industry, and user information protection on e-commerce platforms—the combined application of Nginx TCP Layer 4 proxy and mTLS mutual authentication holds significant importance. This technical combination not only meets enterprises’demands for network communication performance but also provides comprehensive protection for data security, serving as a core technical support for safeguarding network architecture security during enterprises’digital transformation.
2. Nginx Proxy for TCP Layer 4 Ports
2.1 Preparation
Before configuring the Nginx TCP Layer 4 port proxy, ensure that Nginx has been installed on the server. For Ubuntu systems, the installation can be performed using the following command:
sudo apt update && sudo apt install nginx -y
For CentOS systems, the yum command is used for installation:
sudo yum install epel-release -y && sudo yum install nginx -y
Meanwhile, the OpenSSL dependency library must be installed to provide SSL/TLS encryption support, which is a prerequisite for the subsequent mTLS mutual authentication configuration. The installation command for Ubuntu systems is as follows:
sudo apt install openssl -y
Installation command for CentOS systems:
sudo yum install openssl -y
After installation, run the nginx -V command to check the Nginx compilation parameters and confirm whether the --with-stream module is included. This module is the core component for implementing TCP/UDP Layer 4 proxy; if it is missing, Nginx needs to be recompiled with this parameter added.
2.2 Detailed Configuration Steps
- Create a dedicated Stream configuration directoryTo maintain the clarity and manageability of the Nginx configuration structure, create a dedicated directory for storing Stream module configuration files:
sudo mkdir -p /etc/nginx/stream.d
- Modify the main Nginx configuration fileEdit the main Nginx configuration file (
/etc/nginx/nginx.conf) and add an entry to include the Stream configuration directory at the end of the file. This ensures that Nginx loads the TCP proxy configuration when starting up:
# Add the following content at the end of /etc/nginx/nginx.conf
stream {include /etc/nginx/stream.d/*.conf;}
- Configure the TCP Layer 4 proxyCreate a TCP proxy configuration file (e.g.,
redis-proxy.conf) in the/etc/nginx/stream.ddirectory. Taking the proxy for a Redis service (running on192.168.1.100:6379) as an example, the configuration content is as follows:
upstream redis_backend {
server 192.168.1.100:6379; # Backend Redis service address and port
keepalive 32; # Maintain persistent connections to improve performance
}
server {
listen 6380; # Port on which Nginx listens for TCP requests
proxy_pass redis_backend; # Forward requests to the backend upstream cluster
proxy_timeout 300s; # Set the proxy connection timeout period
proxy_buffer_size 16k; # Set the proxy buffer size
}
2.3 Configuration Example Demonstration
After completing the configuration, verify the correctness of the Nginx configuration using the following command:
sudo nginx -t
If the output shows nginx: configuration file /etc/nginx/nginx.conf test is successful, the configuration is valid. Then, reload the Nginx configuration to make the changes take effect:
sudo systemctl reload nginx
To confirm that the TCP proxy port is listening normally, run the ss command to check the port status:
ss -tulnp | grep 6380
If the output includes LISTEN 0 128 *:6380 *:* users:(("nginx",pid=xxxx,fd=xx)), it indicates that the Nginx TCP Layer 4 proxy has been started successfully.
3. Enabling mTLS Mutual Encryption Authentication
3.1 Analysis of mTLS Principles
mTLS (mutual TLS) extends the security mechanism of traditional one-way TLS by requiring both the client and the server to present and verify each other’s digital certificates during the TLS handshake process. This two-way identity verification ensures that:
- The client can confirm that the connected server is a legitimate target (preventing man-in-the-middle attacks by fake servers);
- The server can verify that the accessing client has the required access permissions (avoiding unauthorized access by malicious clients).
The core principle of mTLS relies on a trusted certificate chain: both parties’certificates are issued by a trusted Certificate Authority (CA). During authentication, each party verifies the validity of the other’s certificate (including checking the certificate’s expiration date, signature integrity, and whether it has been revoked) to confirm the other’s identity.
3.2 Generating Certificates and Keys
This section uses the CFSSL tool (a command-line toolset for TLS/SSL certificate management) to generate the required certificates for mTLS, including the root CA certificate, server certificate, and client certificate.
3.2.1 Installing the CFSSL Tool
Download and install the CFSSL toolset in the /usr/local/bin directory (applicable to Linux x86_64 systems):
wget -q -O /usr/local/bin/cfssl https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64
wget -q -O /usr/local/bin/cfssljson https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64
wget -q -O /usr/local/bin/cfssl-certinfo https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl-certinfo_1.6.4_linux_amd64
# Add executable permissions to the tools
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo
Verify the installation by running cfssl version; a version number output indicates successful installation.
3.2.2 Generating the Root CA Certificate
- Create a CA configuration directory
mkdir -p /etc/nginx/certs && cd /etc/nginx/certs
- Create the CA policy file (
ca-config.json)This file defines the validity period and usage scope of the certificates issued by the CA:
{
"signing": {
"default": {"expiry": "87600h" # Root CA validity period (10 years)
},
"profiles": {
"server": {"expiry": "43800h", # Server certificate validity period (5 years)
"usages": [
"signing",
"key encipherment",
"server auth" # Certificate usage: server authentication
]
},
"client": {"expiry": "43800h", # Client certificate validity period (5 years)
"usages": [
"signing",
"key encipherment",
"client auth" # Certificate usage: client authentication
]
}
}
}
}
- Create the CA certificate signing request (CSR) configuration file (
ca-csr.json)This file contains the basic information of the root CA (such as organization name and region):
{
"CN": "MyEnterpriseRootCA", # Common Name of the root CA
"key": {
"algo": "rsa", # Encryption algorithm: RSA
"size": 2048 # Key length: 2048 bits
},
"names": [
{
"C": "CN", # Country/Region
"L": "Beijing", # City
"ST": "Beijing", # Province/State
"O": "MyEnterprise", # Organization name
"OU": "IT Department" # Organizational Unit
}
],
"ca": {"expiry": "87600h"}
}
- Generate the root CA certificate and private key
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
After execution, the following files will be generated in the current directory:
ca.pem: Root CA public certificate (used to verify server/client certificates)ca-key.pem: Root CA private key (used to sign server/client certificates; keep it secure)ca.csr: CA certificate signing request file (for reference only)
3.2.3 Generating the Server Certificate
- Create the server CSR configuration file (
server-csr.json)Note that thehostsfield must include the actual domain name or IP address of the Nginx server (to ensure certificate domain verification passes):
{"CN": "nginx-proxy.example.com", # Common Name of the server (consistent with the access domain name)
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "MyEnterprise",
"OU": "IT Department"
}
],
"hosts": [
"127.0.0.1",
"192.168.1.200", # Nginx server IP address
"nginx-proxy.example.com" # Nginx server domain name
]
}
- Generate the server certificate and private keyUse the root CA to sign the server certificate:
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server-csr.json | cfssljson -bare server
Generated files:
server.pem: Server public certificateserver-key.pem: Server private key
3.2.4 Generating the Client Certificate
- Create the client CSR configuration file (
client-csr.json)
{"CN": "mtls-client-001", # Client identifier (can be customized, e.g., client ID)
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "MyEnterprise",
"OU": "Operations Department"
}
]
}
- Generate the client certificate and private key
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client-csr.json | cfssljson -bare client
Generated files:
client.pem: Client public certificateclient-key.pem: Client private key
3.2.5 Verifying Certificate Validity
Use the openssl command to verify the integrity of the generated certificates and the validity of the certificate chain:
# Verify the server certificate (using the root CA)
openssl verify -CAfile ca.pem server.pem
# Verify the client certificate (using the root CA)
openssl verify -CAfile ca.pem client.pem
If the output shows server.pem: OK and client.pem: OK, the certificates are valid and the certificate chain is intact.
3.3 Configuring mTLS
Modify the Nginx TCP proxy configuration file (/etc/nginx/stream.d/redis-proxy.conf) to enable mTLS mutual authentication. The updated configuration is as follows:
upstream redis_backend {
server 192.168.1.100:6379;
keepalive 32;
}
server {
listen 6380 ssl; # Enable SSL for the listening port
# Server certificate configuration
ssl_certificate /etc/nginx/certs/server.pem; # Server public certificate path
ssl_certificate_key /etc/nginx/certs/server-key.pem; # Server private key path
# mTLS client authentication configuration
ssl_client_certificate /etc/nginx/certs/ca.pem; # Root CA certificate (used to verify client certificates)
ssl_verify_client on; # Enable client certificate verification (mandatory)
# ssl_verify_depth 2; # Set the certificate verification depth (default is 1; adjust if using intermediate CAs)
# SSL/TLS security optimization parameters
ssl_protocols TLSv1.2 TLSv1.3; # Disable insecure protocols (e.g., TLSv1.0, TLSv1.1)
ssl_prefer_server_ciphers on; # Prioritize server-side cipher suites
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH"; # Secure cipher suite list
# Proxy forwarding configuration
proxy_pass redis_backend;
proxy_timeout 300s;
proxy_buffer_size 16k;
}
After modifying the configuration, verify and reload Nginx:
sudo nginx -t && sudo systemctl reload nginx
4. Configuration Verification and Testing
4.1 Checking Configuration Correctness
In addition to using nginx -t to verify the syntax of the configuration file, you can also check the Nginx error log to confirm whether the mTLS configuration is loaded normally:
sudo tail -f /var/log/nginx/error.log
If no error messages (such as “SSL_CTX_load_verify_locations failed” or “invalid certificate”) appear, the mTLS configuration has been loaded successfully.
4.2 Testing Mutual Encryption Authentication
Use the openssl s_client tool on the client to simulate a TLS connection and test the mTLS authentication process. Ensure the client has the client.pem, client-key.pem, and ca.pem files.
4.2.1 Testing a Valid Client Connection
Run the following command to establish a TLS connection to the Nginx proxy port (6380):
openssl s_client -connect 192.168.1.200:6380 \
-cert /path/to/client.pem \
-key /path/to/client-key.pem \
-CAfile /path/to/ca.pem
If the connection is successful, the output will include the following key information:
Verify return code: 0 (ok)(indicating successful certificate verification)- Detailed information about the server certificate (such as issuer, validity period)
Next, test the Redis service access (enter the following commands in the openssl s_client interactive interface):
# Enter Redis authentication password (if the backend Redis has authentication enabled)
AUTH YourRedisPassword
# Expected response: +OK
# Test the Redis PING command
PING
# Expected response: +PONG
# Test data writing and reading
SET test_key "mtls_test_value"
# Expected response: +OK
GET test_key
# Expected response: $13\nmtls_test_value
4.2.2 Testing an Invalid Client Connection
To verify the security of mTLS, test scenarios with invalid client certificates (e.g., using an unsigned certificate or omitting the client certificate):
# Scenario 1: Omit the client certificate
openssl s_client -connect 192.168.1.200:6380 -CAfile /path/to/ca.pem
# Scenario 2: Use an untrusted client certificate
openssl s_client -connect 192.168.1.200:6380 \
-cert /path/to/untrusted-client.pem \
-key /path/to/untrusted-client-key.pem \
-CAfile /path/to/ca.pem
In both scenarios, the connection should be rejected, and the output will include Verify return code: 19 (self-signed certificate in certificate chain) or SSL alert number 116 (indicating client certificate verification failure), confirming that mTLS effectively blocks unauthorized access.
5. Conclusion
This document systematically elaborates on the implementation process of Nginx TCP Layer 4 port proxy and mTLS mutual encryption authentication, covering technical background, configuration steps, and verification methods. By combining Nginx’s high-performance proxy capabilities with mTLS’s strict mutual authentication mechanism, enterprises can establish a secure and reliable network communication channel, effectively addressing security risks such as unauthorized access and data leakage in traditional proxy architectures.
