Ultimate Guide: Optimize Your Linux Server for 10 Gbit/s Network Performance

Achieving 10 Gbit/s network performance on Linux servers requires careful optimization of the network stack, kernel parameters, and system configuration. This comprehensive guide will walk you through every aspect of maximizing network throughput for high-performance applications.

1. Hardware Requirements & Verification

Essential Hardware Components

10 Gbit/s NIC: Intel X520/X540, Mellanox ConnectX, or similar
PCIe 3.0+ Slot: Ensure sufficient bandwidth (x8 or x16 lanes)
CPU Power: Modern multi-core processor (Intel Xeon, AMD EPYC)
RAM: Minimum 16GB DDR4, preferably 32GB+
SSD Storage: NVMe preferred for I/O intensive operations

Verify Hardware Capabilities

# Check network interface speed
sudo ethtool eth0 | grep "Speed:"

# Verify PCIe link width and speed
sudo lspci -vvv | grep -A 10 "Ethernet controller"

# Check CPU and memory info
lscpu
free -h

# Verify NIC driver
sudo ethtool -i eth0

2. Kernel Network Stack Optimization

Advanced Sysctl Configuration

# Create optimized network config
sudo nano /etc/sysctl.d/99-network-performance.conf

Add these parameters for maximum performance:

# === Network Buffer Optimization ===
# Increase socket buffer sizes for high throughput
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.core.rmem_default = 67108864
net.core.wmem_default = 67108864
net.core.netdev_max_backlog = 5000

# TCP buffer sizes
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.ipv4.tcp_mem = 196608 262144 393216

# === TCP Performance Tuning ===
# Enable window scaling for high-speed networks
net.ipv4.tcp_window_scaling = 1

# Increase TCP buffer sizes
net.ipv4.tcp_rfc1337 = 1

# TCP congestion control (bbr for modern networks)
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

# TCP timestamps and selective ACKs
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1

# === Connection Tracking Optimization ===
# Increase connection tracking table size
net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_tcp_timeout_established = 7200

# === Network Interface Optimization ===
# Increase network device backlog
net.core.netdev_budget = 600
net.core.netdev_max_backlog = 5000

# === IRQ Balancing ===
# Enable IRQ affinity for network cards
# (Will be configured separately per interface)

Apply the configuration:

sudo sysctl -p /etc/sysctl.d/99-network-performance.conf

3. Network Interface Card (NIC) Optimization

Intel X520/X540 Optimization

# Install ethtool and intel drivers
sudo apt install ethtool

# Optimize Intel NIC settings
sudo ethtool -G eth0 rx 4096 tx 4096
sudo ethtool -L eth0 combined 16
sudo ethtool -K eth0 tso on gso on gro on lro on
sudo ethtool -C eth0 adaptive-rx on adaptive-tx on rx-usecs 100 tx-usecs 100

Mellanox ConnectX Optimization

# Install Mellanox drivers
sudo apt install mlnx-ofed-linux

# Enable hardware offload features
sudo ethtool -K eth0 hw-tc-offload on
sudo ethtool -K eth0 tx-tcp-segmentation on
sudo ethtool -K eth0 rx-gro-list on

# Set number of queues
sudo ethtool -L eth0 combined 32

Generic NIC Optimization

# Enable all available offload features
sudo ethtool -K eth0 tso on gso on gro on lro on tx-vlan-hw-insert on rx-vlan-hw-strip on

# Set ring buffer sizes
sudo ethtool -G eth0 rx 4096 tx 4096 rx-jumbo 4096

# Optimize interrupt coalescing
sudo ethtool -C eth0 rx-usecs 50 tx-usecs 50 adaptive-rx on adaptive-tx on

4. CPU and IRQ Affinity

IRQ Balancing Setup

# Install irqbalance
sudo apt install irqbalance

# Configure irqbalance for network performance
sudo nano /etc/default/irqbalance

Add these settings:

# Enable irqbalance with optimized settings
IRQBALANCE_BANNED_CPUS=""
IRQBALANCE_ARGS="--foreground"

Manual IRQ Affinity

# Check current IRQ distribution
cat /proc/interrupts | grep eth0

# Set CPU affinity for network interrupts
# Distribute across available CPU cores
sudo sh -c 'echo "1-7" > /proc/irq/32/smp_affinity'  # Adjust IRQ number
sudo sh -c 'echo "1-7" > /proc/irq/33/smp_affinity'  # Adjust IRQ number

# Verify affinity
cat /proc/irq/32/smp_affinity
cat /proc/irq/33/smp_affinity

Process Affinity for Network Applications

# Use taskset to bind applications to specific CPUs
sudo taskset -c 2,3,4,5 nginx -g 'daemon off;'
sudo taskset -c 6,7 redis-server

# Or use systemd CPUAffinity
sudo systemctl edit nginx

Add to the override file:

[Service]
CPUAffinity=2 3 4 5

5. Application-Level Optimization

Nginx High Performance Configuration

# Edit nginx.conf
sudo nano /etc/nginx/nginx.conf

Optimize worker processes and connections:

worker_processes auto;
worker_rlimit_nofile 1048576;

events {
    worker_connections 65535;
    use epoll;
    multi_accept on;
}

http {
    # TCP optimizations
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    
    # Keep-alive optimizations
    keepalive_timeout 65;
    keepalive_requests 10000;
    
    # Buffer sizes
    client_body_buffer_size 128k;
    client_max_body_size 10m;
    client_header_buffer_size 3m;
    large_client_header_buffers 4 256k;
    
    # Output compression
    gzip on;
    gzip_vary on;
    gzip_min_length 10240;
    gzip_comp_level 6;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
}

Redis Performance Tuning

# Edit redis.conf
sudo nano /etc/redis/redis.conf

Optimize for high throughput:

# Memory optimization
maxmemory 8gb
maxmemory-policy allkeys-lru

# Network optimization
tcp-keepalive 300
tcp-backlog 511

# Persistence optimization (if needed)
save 900 1
save 300 10
save 60 10000

# Performance tuning
rdbcompression yes
rdbchecksum yes

6. Network Queue Discipline (QDisc)

FQ (Fair Queueing) Setup

# Set FQ as default queue discipline
sudo tc qdisc add dev eth0 root fq

# Verify current qdisc
tc qdisc show dev eth0

Advanced QDisc Configuration

# Create hierarchical QDisc for better control
sudo tc qdisc add dev eth0 root handle 1: htb default 30
sudo tc class add dev eth0 parent 1: classid 1:1 htb rate 10gbit
sudo tc class add dev eth0 parent 1:1 classid 1:10 htb rate 8gbit ceil 10gbit prio 1
sudo tc class add dev eth0 parent 1:1 classid 1:20 htb rate 1gbit ceil 2gbit prio 2
sudo tc class add dev eth0 parent 1:1 classid 1:30 htb rate 1gbit ceil 10gbit prio 3

7. Monitoring and Performance Testing

Network Performance Tools

# Install performance monitoring tools
sudo apt install iperf3 nethogs iftop bmon sysstat

iperf3 Testing

# Server side
iperf3 -s -i 1 -p 5201

# Client side (test 10Gbit throughput)
iperf3 -c server_ip -t 60 -i 1 -P 8 -w 2M

Real-time Monitoring

# Monitor network interface statistics
sudo sar -n DEV 1

# Monitor network processes
sudo nethogs eth0

# Monitor interface bandwidth
sudo iftop -i eth0

# Monitor buffer usage
cat /proc/net/softnet_stat

Advanced Monitoring

# Monitor TCP statistics
cat /proc/net/netstat | grep -i tcp
cat /proc/net/snmp | grep -i tcp

# Monitor network queues
cat /proc/net/softnet_stat

# Monitor interrupt statistics
cat /proc/interrupts | grep eth0

# Monitor CPU usage per core
mpstat -P ALL 1

8. Security Considerations

Performance vs Security Balance

# Disable unnecessary security features for performance
# (Only in trusted network environments)

# Disable reverse path filtering (use with caution)
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0

# Disable TCP syncookies (if not needed for DDoS protection)
net.ipv4.tcp_syncookies = 0

Firewall Optimization

# Use nftables for better performance
sudo apt install nftables

# Create high-performance ruleset
sudo nft add table inet filter
sudo nft add chain inet filter input { type filter hook input priority 0\; }
sudo nft add chain inet filter forward { type filter hook forward priority 0\; }
sudo nft add chain inet filter output { type filter hook output priority 0\; }

# Accept established connections (high performance)
sudo nft add rule inet filter input ct state related,established accept
sudo nft add rule inet filter forward ct state related,established accept

9. Persistent Configuration

Network Interface Persistence

# Create network interface configuration
sudo nano /etc/network/interfaces.d/eth0

Add interface optimizations:

auto eth0
iface eth0 inet static
    address 192.168.1.100
    netmask 255.255.255.0
    gateway 192.168.1.1
    
    # Performance optimizations
    post-up /usr/sbin/ethtool -G eth0 rx 4096 tx 4096
    post-up /usr/sbin/ethtool -K eth0 tso on gso on gro on lro on
    post-up /usr/sbin/ethtool -C eth0 adaptive-rx on adaptive-tx on

Systemd Service for Network Optimization

# Create optimization service
sudo nano /etc/systemd/system/network-optimization.service

Add service configuration:

[Unit]
Description=Network Performance Optimization
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/optimize-network.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Optimization Script

# Create optimization script
sudo nano /usr/local/bin/optimize-network.sh

Add script content:

#!/bin/bash

# Apply network optimizations
echo "Applying network performance optimizations..."

# Set sysctl parameters
sysctl -p /etc/sysctl.d/99-network-performance.conf

# Optimize network interface
ethtool -G eth0 rx 4096 tx 4096
ethtool -K eth0 tso on gso on gro on lro on
ethtool -C eth0 adaptive-rx on adaptive-tx on

# Set IRQ affinity
# (Add your specific IRQ affinity commands here)

echo "Network optimizations applied successfully!"

exit 0

Make script executable and enable service:

sudo chmod +x /usr/local/bin/optimize-network.sh
sudo systemctl enable network-optimization.service
sudo systemctl start network-optimization.service

10. Troubleshooting Common Issues

Performance Bottlenecks

# Check for dropped packets
cat /proc/net/dev | grep eth0

# Monitor network errors
ethtool -S eth0 | grep -i error

# Check CPU usage
top -p $(pgrep nginx)
top -p $(pgrep redis)

Common Issues and Solutions

Low throughput: Check NIC settings and driver version
High CPU usage: Verify IRQ distribution and CPU affinity
Packet drops: Increase buffer sizes and check queue discipline
Connection limits: Adjust ulimit and connection tracking settings

11. Validation and Benchmarking

Comprehensive Testing

# Test single connection throughput
iperf3 -c server_ip -t 30 -w 2M

# Test multiple parallel connections
iperf3 -c server_ip -t 30 -P 16 -w 2M

# Test with different packet sizes
iperf3 -c server_ip -t 30 -l 64K
iperf3 -c server_ip -t 30 -l 128K
iperf3 -c server_ip -t 30 -l 1M

Performance Metrics

✅ Target: 9.5+ Gbit/s sustained throughput
✅ CPU usage: <50% under full load
✅ Packet loss: <0.1%
✅ Latency: <1ms for local connections
✅ Connection tracking: >100,000 concurrent connections

12. Conclusion

Achieving 10 Gbit/s network performance requires careful optimization across multiple layers: hardware, kernel, network interface, and applications. This comprehensive guide provides the foundation for maximizing network throughput on Linux servers.

Remember that performance optimization is an iterative process. Monitor your system continuously, adjust parameters based on your specific workload, and always test changes in a staging environment before applying to production.

With proper optimization and monitoring, your Linux server can achieve and maintain 10 Gbit/s network performance for demanding applications and workloads.

Tags:

Live Chat	Get instant assistance from our experts, available 24/7 for real-time support with an average response time of under 30 seconds.	Chat Now
Ticket System	Submit detailed inquiries and track their progress through our efficient ticket system. We aim to respond to all tickets within 1-2 hours.	Open a Ticket
Knowledge Base	Access a comprehensive library of articles and guides for self-help.	Browse Articles