09-部署运维指南
·
Mall-Swarm 微服务商城系统开发指南
第九章：部署运维指南
9.1 部署运维概述
本章将详细介绍Mall-Swarm微服务商城系统的部署和运维方案，包括Docker容器化部署、Kubernetes集群部署、监控告警、日志管理、性能优化等内容。通过完善的部署运维体系，确保系统的高可用性、可扩展性和可维护性。
9.2 部署架构设计
9.2.1 部署架构图
负载均衡器 (Nginx/LB)
|
┌─────────────────┐
│ API Gateway │
│ (mall-gateway) │
└─────────────────┘
|
┌─────────────────┼─────────────────┐
| | |
┌─────────┐ ┌─────────┐ ┌─────────┐
│认证服务 │ │后台服务 │ │门户服务 │
│(auth) │ │(admin) │ │(portal) │
└─────────┘ └─────────┘ └─────────┘
| | |
└─────────────────┼─────────────────┘
|
┌─────────────────┐
│ 搜索服务 │
│ (search) │
└─────────────────┘
|
┌─────────────────┼─────────────────┐
| | |
┌─────────┐ ┌─────────┐ ┌─────────┐
│基础设施 │ │数据存储 │ │监控系统 │
│(Infra) │ │(Data) │ │(Monitor)│
└─────────┘ └─────────┘ └─────────┘
9.2.2 环境分层
- 开发环境(Dev): 开发人员本地开发测试
- 测试环境(Test): 功能测试、集成测试
- 预生产环境(Staging): 生产前验证
- 生产环境(Prod): 正式生产环境
9.3 Docker容器化部署
9.3.1 基础镜像制作
JDK基础镜像Dockerfile:
FROM openjdk:17-jre-slim
# 设置时区
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN echo 'Asia/Shanghai' >/etc/timezone
# 创建应用目录
RUN mkdir -p /app
# 设置工作目录
WORKDIR /app
# 添加应用用户
RUN addgroup --system app && adduser --system --group app
USER app
# 暴露端口
EXPOSE 8080
# 健康检查
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
CMD curl -f http://localhost:8080/actuator/health || exit 1
# 启动命令
ENTRYPOINT ["java", "-jar", "-Dspring.profiles.active=prod", "app.jar"]
9.3.2 服务镜像制作
认证服务Dockerfile:
FROM mall/base-jdk:17
MAINTAINER macrozheng
COPY mall-auth-1.0-SNAPSHOT.jar /app/app.jar
EXPOSE 8401
ENV JAVA_OPTS="-Xms512m -Xmx1024m -XX:+UseG1GC"
ENTRYPOINT ["sh", "-c", "java $JAVA_OPTS -jar app.jar"]
网关服务Dockerfile:
FROM mall/base-jdk:17
MAINTAINER macrozheng
COPY mall-gateway-1.0-SNAPSHOT.jar /app/app.jar
EXPOSE 8201
ENV JAVA_OPTS="-Xms1024m -Xmx2048m -XX:+UseG1GC"
ENTRYPOINT ["sh", "-c", "java $JAVA_OPTS -jar app.jar"]
9.3.3 Docker Compose部署
docker-compose.yml:
version: '3.8'
services:
# 基础设施服务
mysql:
image: mysql:5.7
container_name: mall-mysql
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: mall
MYSQL_USER: mall
MYSQL_PASSWORD: mall123
ports:
- "3306:3306"
volumes:
- mysql_data:/var/lib/mysql
- ./document/sql/mall.sql:/docker-entrypoint-initdb.d/mall.sql
command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
networks:
- mall-network
redis:
image: redis:7.0-alpine
container_name: mall-redis
ports:
- "6379:6379"
volumes:
- redis_data:/data
command: redis-server --appendonly yes
networks:
- mall-network
elasticsearch:
image: elasticsearch:7.17.3
container_name: mall-elasticsearch
environment:
- discovery.type=single-node
- ES_JAVA_OPTS=-Xms512m -Xmx512m
- "http.cors.enabled=true"
- "http.cors.allow-origin=*"
ports:
- "9200:9200"
- "9300:9300"
volumes:
- es_data:/usr/share/elasticsearch/data
networks:
- mall-network
kibana:
image: kibana:7.17.3
container_name: mall-kibana
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
- I18N_LOCALE=zh-CN
ports:
- "5601:5601"
depends_on:
- elasticsearch
networks:
- mall-network
rabbitmq:
image: rabbitmq:3.10.5-management-alpine
container_name: mall-rabbitmq
environment:
RABBITMQ_DEFAULT_USER: mall
RABBITMQ_DEFAULT_PASS: mall123
ports:
- "5672:5672"
- "15672:15672"
volumes:
- rabbitmq_data:/var/lib/rabbitmq
networks:
- mall-network
mongodb:
image: mongo:5.0
container_name: mall-mongodb
environment:
MONGO_INITDB_ROOT_USERNAME: mall
MONGO_INITDB_ROOT_PASSWORD: mall123
ports:
- "27017:27017"
volumes:
- mongodb_data:/data/db
networks:
- mall-network
nacos:
image: nacos/nacos-server:v2.3.0
container_name: mall-nacos
environment:
MODE: standalone
SPRING_DATASOURCE_PLATFORM: mysql
MYSQL_SERVICE_HOST: mysql
MYSQL_SERVICE_DB_NAME: nacos
MYSQL_SERVICE_PORT: 3306
MYSQL_SERVICE_USER: root
MYSQL_SERVICE_PASSWORD: root
ports:
- "8848:8848"
depends_on:
- mysql
volumes:
- nacos_data:/home/nacos/data
networks:
- mall-network
# 微服务应用
mall-auth:
build:
context: ./mall-auth
dockerfile: Dockerfile
image: mall/mall-auth:1.0
container_name: mall-auth
environment:
SPRING_PROFILES_ACTIVE: prod
ports:
- "8401:8401"
depends_on:
- mysql
- redis
- nacos
volumes:
- app_logs:/app/logs
networks:
- mall-network
mall-gateway:
build:
context: ./mall-gateway
dockerfile: Dockerfile
image: mall/mall-gateway:1.0
container_name: mall-gateway
environment:
SPRING_PROFILES_ACTIVE: prod
ports:
- "8201:8201"
depends_on:
- mall-auth
- redis
- nacos
volumes:
- app_logs:/app/logs
networks:
- mall-network
mall-admin:
build:
context: ./mall-admin
dockerfile: Dockerfile
image: mall/mall-admin:1.0
container_name: mall-admin
environment:
SPRING_PROFILES_ACTIVE: prod
ports:
- "8080:8080"
depends_on:
- mysql
- redis
- nacos
volumes:
- app_logs:/app/logs
networks:
- mall-network
mall-portal:
build:
context: ./mall-portal
dockerfile: Dockerfile
image: mall/mall-portal:1.0
container_name: mall-portal
environment:
SPRING_PROFILES_ACTIVE: prod
ports:
- "8085:8085"
depends_on:
- mysql
- redis
- mongodb
- rabbitmq
- nacos
volumes:
- app_logs:/app/logs
networks:
- mall-network
mall-search:
build:
context: ./mall-search
dockerfile: Dockerfile
image: mall/mall-search:1.0
container_name: mall-search
environment:
SPRING_PROFILES_ACTIVE: prod
ports:
- "8081:8081"
depends_on:
- mysql
- elasticsearch
- nacos
volumes:
- app_logs:/app/logs
networks:
- mall-network
# 负载均衡
nginx:
image: nginx:1.22-alpine
container_name: mall-nginx
ports:
- "80:80"
- "443:443"
volumes:
- ./document/docker/nginx.conf:/etc/nginx/nginx.conf
- nginx_logs:/var/log/nginx
depends_on:
- mall-gateway
networks:
- mall-network
volumes:
mysql_data:
redis_data:
es_data:
rabbitmq_data:
mongodb_data:
nacos_data:
app_logs:
nginx_logs:
networks:
mall-network:
driver: bridge
9.3.4 Nginx配置
nginx.conf:
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
use epoll;
multi_accept on;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# Gzip压缩
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_types
text/plain
text/css
text/xml
text/javascript
application/json
application/javascript
application/xml+rss
application/atom+xml
image/svg+xml;
# 上游服务器配置
upstream mall-gateway {
server mall-gateway:8201 max_fails=3 fail_timeout=30s;
keepalive 32;
}
# 限流配置
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
server {
listen 80;
server_name localhost;
# API请求代理
location /api/ {
limit_req zone=api burst=20 nodelay;
proxy_pass http://mall-gateway/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 30s;
proxy_send_timeout 30s;
proxy_read_timeout 30s;
proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 8 4k;
}
# 静态资源
location /static/ {
alias /usr/share/nginx/html/static/;
expires 30d;
add_header Cache-Control "public, immutable";
}
# 健康检查
location /health {
access_log off;
return 200 "healthy\n";
add_header Content-Type text/plain;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
}
9.4 Kubernetes集群部署
9.4.1 命名空间配置
namespace.yaml:
apiVersion: v1
kind: Namespace
metadata:
name: mall-system
labels:
name: mall-system
env: production
9.4.2 配置映射
configmap.yaml:
apiVersion: v1
kind: ConfigMap
metadata:
name: mall-config
namespace: mall-system
data:
application.yml: |
spring:
profiles:
active: prod
cloud:
nacos:
discovery:
server-addr: nacos-service:8848
config:
server-addr: nacos-service:8848
logging:
level:
root: INFO
com.macro.mall: DEBUG
management:
endpoints:
web:
exposure:
include: "*"
endpoint:
health:
show-details: always
9.4.3 密钥配置
secret.yaml:
apiVersion: v1
kind: Secret
metadata:
name: mall-secret
namespace: mall-system
type: Opaque
data:
mysql-username: bWFsbA== # mall
mysql-password: bWFsbDEyMw== # mall123
redis-password: ""
jwt-secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXo=
9.4.4 持久化存储
storage.yaml:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: mall-storage
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
fsType: ext4
reclaimPolicy: Retain
allowVolumeExpansion: true
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pvc
namespace: mall-system
spec:
accessModes:
- ReadWriteOnce
storageClassName: mall-storage
resources:
requests:
storage: 20Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-pvc
namespace: mall-system
spec:
accessModes:
- ReadWriteOnce
storageClassName: mall-storage
resources:
requests:
storage: 5Gi
9.4.5 基础设施服务部署
mysql-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql
namespace: mall-system
spec:
replicas: 1
selector:
matchLabels:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- name: mysql
image: mysql:5.7
env:
- name: MYSQL_ROOT_PASSWORD
value: "root"
- name: MYSQL_DATABASE
value: "mall"
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: mall-secret
key: mysql-username
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: mall-secret
key: mysql-password
ports:
- containerPort: 3306
volumeMounts:
- name: mysql-data
mountPath: /var/lib/mysql
- name: init-sql
mountPath: /docker-entrypoint-initdb.d
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
livenessProbe:
exec:
command:
- mysqladmin
- ping
- -h
- localhost
initialDelaySeconds: 30
timeoutSeconds: 5
readinessProbe:
exec:
command:
- mysql
- -h
- localhost
- -u
- root
- -proot
- -e
- "SELECT 1"
initialDelaySeconds: 10
timeoutSeconds: 1
volumes:
- name: mysql-data
persistentVolumeClaim:
claimName: mysql-pvc
- name: init-sql
configMap:
name: mysql-init-sql
---
apiVersion: v1
kind: Service
metadata:
name: mysql-service
namespace: mall-system
spec:
selector:
app: mysql
ports:
- port: 3306
targetPort: 3306
type: ClusterIP
9.4.6 微服务应用部署
mall-gateway-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: mall-gateway
namespace: mall-system
labels:
app: mall-gateway
version: v1.0
spec:
replicas: 3
selector:
matchLabels:
app: mall-gateway
template:
metadata:
labels:
app: mall-gateway
version: v1.0
spec:
containers:
- name: mall-gateway
image: mall/mall-gateway:1.0
ports:
- containerPort: 8201
env:
- name: SPRING_PROFILES_ACTIVE
value: "prod"
- name: JAVA_OPTS
value: "-Xms1024m -Xmx2048m -XX:+UseG1GC"
volumeMounts:
- name: config-volume
mountPath: /app/config
- name: logs-volume
mountPath: /app/logs
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
livenessProbe:
httpGet:
path: /actuator/health
port: 8201
initialDelaySeconds: 120
periodSeconds: 30
timeoutSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet:
path: /actuator/health
port: 8201
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumes:
- name: config-volume
configMap:
name: mall-config
- name: logs-volume
emptyDir: {}
imagePullSecrets:
- name: mall-registry-secret
---
apiVersion: v1
kind: Service
metadata:
name: mall-gateway-service
namespace: mall-system
labels:
app: mall-gateway
spec:
selector:
app: mall-gateway
ports:
- name: http
port: 8201
targetPort: 8201
protocol: TCP
type: ClusterIP
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: mall-gateway-hpa
namespace: mall-system
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: mall-gateway
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
9.4.7 Ingress配置
ingress.yaml:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: mall-ingress
namespace: mall-system
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/rate-limit: "100"
nginx.ingress.kubernetes.io/rate-limit-window: "1m"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
tls:
- hosts:
- api.mall.com
secretName: mall-tls-secret
rules:
- host: api.mall.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: mall-gateway-service
port:
number: 8201
9.5 监控告警系统
9.5.1 Prometheus配置
prometheus.yaml:
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
namespace: mall-system
data:
prometheus.yml: |
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- "/etc/prometheus/rules/*.yml"
alerting:
alertmanagers:
- static_configs:
- targets:
- alertmanager:9093
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'mall-gateway'
metrics_path: '/actuator/prometheus'
static_configs:
- targets: ['mall-gateway-service:8201']
- job_name: 'mall-admin'
metrics_path: '/actuator/prometheus'
static_configs:
- targets: ['mall-admin-service:8080']
- job_name: 'mall-portal'
metrics_path: '/actuator/prometheus'
static_configs:
- targets: ['mall-portal-service:8085']
- job_name: 'mall-search'
metrics_path: '/actuator/prometheus'
static_configs:
- targets: ['mall-search-service:8081']
- job_name: 'mall-auth'
metrics_path: '/actuator/prometheus'
static_configs:
- targets: ['mall-auth-service:8401']
- job_name: 'kubernetes-nodes'
kubernetes_sd_configs:
- role: node
relabel_configs:
- source_labels: [__address__]
regex: '(.*):10250'
target_label: __address__
replacement: '${1}:9100'
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus
namespace: mall-system
spec:
replicas: 1
selector:
matchLabels:
app: prometheus
template:
metadata:
labels:
app: prometheus
spec:
containers:
- name: prometheus
image: prom/prometheus:v2.40.0
args:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus/'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--storage.tsdb.retention.time=200h'
- '--web.enable-lifecycle'
ports:
- containerPort: 9090
volumeMounts:
- name: prometheus-config
mountPath: /etc/prometheus/
- name: prometheus-storage
mountPath: /prometheus/
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
volumes:
- name: prometheus-config
configMap:
name: prometheus-config
- name: prometheus-storage
emptyDir: {}
9.5.2 Grafana配置
grafana.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
namespace: mall-system
spec:
replicas: 1
selector:
matchLabels:
app: grafana
template:
metadata:
labels:
app: grafana
spec:
containers:
- name: grafana
image: grafana/grafana:9.3.0
ports:
- containerPort: 3000
env:
- name: GF_SECURITY_ADMIN_PASSWORD
value: "admin123"
- name: GF_INSTALL_PLUGINS
value: "grafana-kubernetes-app"
volumeMounts:
- name: grafana-storage
mountPath: /var/lib/grafana
- name: grafana-config
mountPath: /etc/grafana/provisioning
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"
volumes:
- name: grafana-storage
emptyDir: {}
- name: grafana-config
configMap:
name: grafana-config
---
apiVersion: v1
kind: Service
metadata:
name: grafana-service
namespace: mall-system
spec:
selector:
app: grafana
ports:
- port: 3000
targetPort: 3000
type: ClusterIP
9.5.3 告警规则配置
alert-rules.yaml:
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-rules
namespace: mall-system
data:
mall-rules.yml: |
groups:
- name: mall.rules
rules:
# 服务可用性告警
- alert: ServiceDown
expr: up == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Service {{ $labels.job }} is down"
description: "Service {{ $labels.job }} has been down for more than 1 minute."
# CPU使用率告警
- alert: HighCPUUsage
expr: (100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > 80
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU usage on {{ $labels.instance }}"
description: "CPU usage is above 80% for 5 minutes on {{ $labels.instance }}"
# 内存使用率告警
- alert: HighMemoryUsage
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
for: 5m
labels:
severity: warning
annotations:
summary: "High memory usage on {{ $labels.instance }}"
description: "Memory usage is above 85% for 5 minutes on {{ $labels.instance }}"
# 磁盘空间告警
- alert: DiskSpaceUsage
expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 80
for: 5m
labels:
severity: warning
annotations:
summary: "High disk usage on {{ $labels.instance }}"
description: "Disk usage is above 80% for 5 minutes on {{ $labels.instance }}"
# 应用响应时间告警
- alert: HighResponseTime
expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job)) > 2
for: 2m
labels:
severity: warning
annotations:
summary: "High response time for {{ $labels.job }}"
description: "95th percentile response time is above 2 seconds for {{ $labels.job }}"
# 错误率告警
- alert: HighErrorRate
expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (job) / sum(rate(http_requests_total[5m])) by (job) * 100 > 5
for: 2m
labels:
severity: critical
annotations:
summary: "High error rate for {{ $labels.job }}"
description: "Error rate is above 5% for {{ $labels.job }}"
9.6 日志管理系统
9.6.1 ELK Stack部署
elasticsearch-cluster.yaml:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: elasticsearch
namespace: mall-system
spec:
serviceName: elasticsearch
replicas: 3
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- name: elasticsearch
image: elasticsearch:7.17.3
env:
- name: cluster.name
value: "mall-cluster"
- name: node.name
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: discovery.seed_hosts
value: "elasticsearch-0.elasticsearch,elasticsearch-1.elasticsearch,elasticsearch-2.elasticsearch"
- name: cluster.initial_master_nodes
value: "elasticsearch-0,elasticsearch-1,elasticsearch-2"
- name: ES_JAVA_OPTS
value: "-Xms1g -Xmx1g"
ports:
- containerPort: 9200
name: http
- containerPort: 9300
name: transport
volumeMounts:
- name: elasticsearch-data
mountPath: /usr/share/elasticsearch/data
resources:
requests:
memory: "2Gi"
cpu: "500m"
limits:
memory: "4Gi"
cpu: "1000m"
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: "mall-storage"
resources:
requests:
storage: 50Gi
logstash.yaml:
apiVersion: v1
kind: ConfigMap
metadata:
name: logstash-config
namespace: mall-system
data:
logstash.conf: |
input {
beats {
port => 5044
}
}
filter {
if [fields][service] {
mutate {
add_field => { "service_name" => "%{[fields][service]}" }
}
}
# 解析Java日志
if [fields][service] =~ /mall-/ {
grok {
match => {
"message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{NUMBER:pid} --- \[%{DATA:thread}\] %{DATA:logger} : %{GREEDYDATA:msg}"
}
}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss.SSS" ]
}
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "mall-logs-%{+YYYY.MM.dd}"
template_name => "mall-logs"
template => "/usr/share/logstash/templates/mall-logs.json"
template_overwrite => true
}
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: logstash
namespace: mall-system
spec:
replicas: 2
selector:
matchLabels:
app: logstash
template:
metadata:
labels:
app: logstash
spec:
containers:
- name: logstash
image: logstash:7.17.3
ports:
- containerPort: 5044
volumeMounts:
- name: logstash-config
mountPath: /usr/share/logstash/pipeline
env:
- name: LS_JAVA_OPTS
value: "-Xmx1g -Xms1g"
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
volumes:
- name: logstash-config
configMap:
name: logstash-config
9.6.2 Filebeat配置
filebeat-daemonset.yaml:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat
namespace: mall-system
spec:
selector:
matchLabels:
app: filebeat
template:
metadata:
labels:
app: filebeat
spec:
containers:
- name: filebeat
image: elastic/filebeat:7.17.3
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env:
- name: ELASTICSEARCH_HOST
value: elasticsearch
- name: ELASTICSEARCH_PORT
value: "9200"
- name: LOGSTASH_HOST
value: logstash
- name: LOGSTASH_PORT
value: "5044"
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: varlog
mountPath: /var/log
readOnly: true
resources:
requests:
memory: "200Mi"
cpu: "100m"
limits:
memory: "500Mi"
cpu: "200m"
volumes:
- name: config
configMap:
name: filebeat-config
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: varlog
hostPath:
path: /var/log
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate
9.7 CI/CD流水线
9.7.1 Jenkins Pipeline
Jenkinsfile:
pipeline {
agent any
environment {
DOCKER_REGISTRY = 'registry.mall.com'
DOCKER_REPO = 'mall'
K8S_NAMESPACE = 'mall-system'
SONAR_HOST = 'http://sonarqube:9000'
}
stages {
stage('Checkout') {
steps {
checkout scm
script {
env.GIT_COMMIT_SHORT = sh(
script: "git rev-parse --short HEAD",
returnStdout: true
).trim()
env.BUILD_TAG = "${env.BUILD_NUMBER}-${env.GIT_COMMIT_SHORT}"
}
}
}
stage('Build') {
steps {
sh 'mvn clean compile -DskipTests'
}
}
stage('Test') {
parallel {
stage('Unit Tests') {
steps {
sh 'mvn test'
}
post {
always {
publishTestResults testResultsPattern: 'target/surefire-reports/*.xml'
publishCoverage adapters: [
jacocoAdapter('target/site/jacoco/jacoco.xml')
]
}
}
}
stage('Code Quality') {
steps {
withSonarQubeEnv('SonarQube') {
sh 'mvn sonar:sonar'
}
}
}
}
}
stage('Package') {
steps {
sh 'mvn package -DskipTests'
archiveArtifacts artifacts: '**/target/*.jar', fingerprint: true
}
}
stage('Build Images') {
parallel {
stage('Build mall-gateway') {
steps {
script {
def image = docker.build("${DOCKER_REGISTRY}/${DOCKER_REPO}/mall-gateway:${BUILD_TAG}", "./mall-gateway")
docker.withRegistry("https://${DOCKER_REGISTRY}", 'docker-registry-credentials') {
image.push()
image.push('latest')
}
}
}
}
stage('Build mall-admin') {
steps {
script {
def image = docker.build("${DOCKER_REGISTRY}/${DOCKER_REPO}/mall-admin:${BUILD_TAG}", "./mall-admin")
docker.withRegistry("https://${DOCKER_REGISTRY}", 'docker-registry-credentials') {
image.push()
image.push('latest')
}
}
}
}
stage('Build mall-portal') {
steps {
script {
def image = docker.build("${DOCKER_REGISTRY}/${DOCKER_REPO}/mall-portal:${BUILD_TAG}", "./mall-portal")
docker.withRegistry("https://${DOCKER_REGISTRY}", 'docker-registry-credentials') {
image.push()
image.push('latest')
}
}
}
}
stage('Build mall-search') {
steps {
script {
def image = docker.build("${DOCKER_REGISTRY}/${DOCKER_REPO}/mall-search:${BUILD_TAG}", "./mall-search")
docker.withRegistry("https://${DOCKER_REGISTRY}", 'docker-registry-credentials') {
image.push()
image.push('latest')
}
}
}
}
stage('Build mall-auth') {
steps {
script {
def image = docker.build("${DOCKER_REGISTRY}/${DOCKER_REPO}/mall-auth:${BUILD_TAG}", "./mall-auth")
docker.withRegistry("https://${DOCKER_REGISTRY}", 'docker-registry-credentials') {
image.push()
image.push('latest')
}
}
}
}
}
}
stage('Security Scan') {
steps {
sh """
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \\
-v \$PWD:/tmp/.trivy-cache \\
aquasec/trivy image ${DOCKER_REGISTRY}/${DOCKER_REPO}/mall-gateway:${BUILD_TAG}
"""
}
}
stage('Deploy to Dev') {
when {
branch 'develop'
}
steps {
script {
kubernetesDeploy(
configs: 'k8s/dev/*.yaml',
kubeconfigId: 'k8s-dev-config'
)
}
}
}
stage('Deploy to Test') {
when {
branch 'test'
}
steps {
script {
kubernetesDeploy(
configs: 'k8s/test/*.yaml',
kubeconfigId: 'k8s-test-config'
)
}
// 运行集成测试
sh 'mvn verify -Pintegration-test'
}
}
stage('Deploy to Production') {
when {
branch 'main'
}
steps {
script {
// 需要人工确认
input message: 'Deploy to production?', ok: 'Deploy'
// 蓝绿部署
sh """
kubectl set image deployment/mall-gateway \\
mall-gateway=${DOCKER_REGISTRY}/${DOCKER_REPO}/mall-gateway:${BUILD_TAG} \\
-n ${K8S_NAMESPACE}
kubectl set image deployment/mall-admin \\
mall-admin=${DOCKER_REGISTRY}/${DOCKER_REPO}/mall-admin:${BUILD_TAG} \\
-n ${K8S_NAMESPACE}
kubectl set image deployment/mall-portal \\
mall-portal=${DOCKER_REGISTRY}/${DOCKER_REPO}/mall-portal:${BUILD_TAG} \\
-n ${K8S_NAMESPACE}
kubectl set image deployment/mall-search \\
mall-search=${DOCKER_REGISTRY}/${DOCKER_REPO}/mall-search:${BUILD_TAG} \\
-n ${K8S_NAMESPACE}
kubectl set image deployment/mall-auth \\
mall-auth=${DOCKER_REGISTRY}/${DOCKER_REPO}/mall-auth:${BUILD_TAG} \\
-n ${K8S_NAMESPACE}
"""
// 等待部署完成
sh """
kubectl rollout status deployment/mall-gateway -n ${K8S_NAMESPACE}
kubectl rollout status deployment/mall-admin -n ${K8S_NAMESPACE}
kubectl rollout status deployment/mall-portal -n ${K8S_NAMESPACE}
kubectl rollout status deployment/mall-search -n ${K8S_NAMESPACE}
kubectl rollout status deployment/mall-auth -n ${K8S_NAMESPACE}
"""
}
}
}
}
post {
always {
cleanWs()
}
success {
slackSend channel: '#deployments',
color: 'good',
message: "✅ Mall-Swarm deployment succeeded: ${env.JOB_NAME} - ${env.BUILD_NUMBER}"
}
failure {
slackSend channel: '#deployments',
color: 'danger',
message: "❌ Mall-Swarm deployment failed: ${env.JOB_NAME} - ${env.BUILD_NUMBER}"
}
}
}
9.8 备份恢复策略
9.8.1 数据库备份
mysql-backup-cronjob.yaml:
apiVersion: batch/v1
kind: CronJob
metadata:
name: mysql-backup
namespace: mall-system
spec:
schedule: "0 2 * * *" # 每天凌晨2点执行
jobTemplate:
spec:
template:
spec:
containers:
- name: mysql-backup
image: mysql:5.7
command:
- /bin/bash
- -c
- |
BACKUP_FILE="/backup/mall-$(date +%Y%m%d-%H%M%S).sql"
mysqldump -h mysql-service -u root -p$MYSQL_ROOT_PASSWORD --single-transaction --routines --triggers mall > $BACKUP_FILE
gzip $BACKUP_FILE
# 删除7天前的备份
find /backup -name "mall-*.sql.gz" -mtime +7 -delete
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: root-password
volumeMounts:
- name: backup-storage
mountPath: /backup
volumes:
- name: backup-storage
persistentVolumeClaim:
claimName: backup-pvc
restartPolicy: OnFailure
9.8.2 Redis备份
redis-backup-script.sh:
#!/bin/bash
# Redis备份脚本
REDIS_HOST="redis-service"
REDIS_PORT="6379"
BACKUP_DIR="/backup/redis"
DATE=$(date +%Y%m%d-%H%M%S)
# 创建备份目录
mkdir -p $BACKUP_DIR
# 执行Redis备份
redis-cli -h $REDIS_HOST -p $REDIS_PORT --rdb $BACKUP_DIR/dump-$DATE.rdb
# 压缩备份文件
gzip $BACKUP_DIR/dump-$DATE.rdb
# 删除7天前的备份
find $BACKUP_DIR -name "dump-*.rdb.gz" -mtime +7 -delete
echo "Redis backup completed: dump-$DATE.rdb.gz"
9.9 灾难恢复方案
9.9.1 多区域部署
multi-region-deployment.yaml:
# 主区域部署
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: mall-primary
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/macrozheng/mall-swarm
targetRevision: HEAD
path: k8s/primary
destination:
server: https://primary-cluster-api-server
namespace: mall-system
syncPolicy:
automated:
prune: true
selfHeal: true
---
# 备份区域部署
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: mall-secondary
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/macrozheng/mall-swarm
targetRevision: HEAD
path: k8s/secondary
destination:
server: https://secondary-cluster-api-server
namespace: mall-system
syncPolicy:
automated:
prune: true
selfHeal: true
9.9.2 数据同步策略
data-sync-strategy.md:
# 数据同步策略
## MySQL主从复制
- 主库：primary-mysql
- 从库：secondary-mysql
- 同步方式：异步复制
- 监控指标：复制延迟、复制状态
## Redis集群
- 主集群：primary-redis-cluster
- 备份集群：secondary-redis-cluster
- 同步方式：redis-shake工具
- 监控指标：同步延迟、数据一致性
## Elasticsearch集群
- 主集群：primary-es-cluster
- 备份集群：secondary-es-cluster
- 同步方式：跨集群复制
- 监控指标：索引同步状态、文档数量
## 文件存储
- 主存储：primary-s3-bucket
- 备份存储：secondary-s3-bucket
- 同步方式：跨区域复制
- 监控指标：同步状态、存储使用量
9.10 性能优化建议
9.10.1 JVM优化
jvm-optimization.md:
# JVM优化配置
## 内存配置
```bash
# 网关服务（高并发）
JAVA_OPTS="-Xms2g -Xmx4g -XX:NewRatio=1 -XX:+UseG1GC -XX:MaxGCPauseMillis=200"
# 业务服务（中等负载）
JAVA_OPTS="-Xms1g -Xmx2g -XX:NewRatio=2 -XX:+UseG1GC -XX:MaxGCPauseMillis=100"
# 搜索服务（内存密集）
JAVA_OPTS="-Xms2g -Xmx4g -XX:NewRatio=1 -XX:+UseG1GC -XX:G1HeapRegionSize=16m"
GC优化
# G1GC调优参数
-XX:+UseG1GC
-XX:MaxGCPauseMillis=200
-XX:G1HeapRegionSize=16m
-XX:G1NewSizePercent=30
-XX:G1MaxNewSizePercent=40
-XX:G1MixedGCLiveThresholdPercent=85
-XX:G1MixedGCCountTarget=8
-XX:G1OldCSetRegionThresholdPercent=20
监控参数
# GC日志
-Xloggc:/app/logs/gc.log
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintGCDateStamps
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=100M
# JVM监控
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
#### 9.10.2 数据库优化
**database-optimization.sql:**
```sql
-- MySQL优化配置
[mysqld]
# 基本配置
default-storage-engine = InnoDB
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
# 内存配置
innodb_buffer_pool_size = 2G
innodb_log_file_size = 256M
innodb_log_buffer_size = 16M
key_buffer_size = 256M
tmp_table_size = 64M
max_heap_table_size = 64M
# 连接配置
max_connections = 500
max_connect_errors = 100000
wait_timeout = 28800
interactive_timeout = 28800
# 查询缓存
query_cache_type = 1
query_cache_size = 256M
query_cache_limit = 2M
# 慢查询日志
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 2
-- 索引优化建议
-- 商品表索引
CREATE INDEX idx_product_category_brand ON pms_product(product_category_id, brand_id);
CREATE INDEX idx_product_status ON pms_product(publish_status, delete_status);
CREATE INDEX idx_product_name ON pms_product(name);
-- 订单表索引
CREATE INDEX idx_order_member_status ON oms_order(member_id, status);
CREATE INDEX idx_order_create_time ON oms_order(create_time);
-- 用户表索引
CREATE INDEX idx_member_username ON ums_member(username);
CREATE INDEX idx_member_phone ON ums_member(phone);
9.11 安全加固方案
9.11.1 网络安全
network-policy.yaml:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: mall-network-policy
namespace: mall-system
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress:
# 只允许同namespace内的pod访问
- from:
- namespaceSelector:
matchLabels:
name: mall-system
# 允许从ingress-nginx访问
- from:
- namespaceSelector:
matchLabels:
name: ingress-nginx
egress:
# 允许访问DNS
- to: []
ports:
- protocol: UDP
port: 53
# 允许访问外部API
- to: []
ports:
- protocol: TCP
port: 443
9.11.2 Pod安全策略
pod-security-policy.yaml:
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: mall-psp
spec:
privileged: false
allowPrivilegeEscalation: false
requiredDropCapabilities:
- ALL
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
- 'persistentVolumeClaim'
runAsUser:
rule: 'MustRunAsNonRoot'
seLinux:
rule: 'RunAsAny'
fsGroup:
rule: 'RunAsAny'
9.12 运维脚本工具
9.12.1 部署脚本
deploy.sh:
#!/bin/bash
set -e
# 配置变量
NAMESPACE="mall-system"
ENVIRONMENT=${1:-dev}
VERSION=${2:-latest}
echo "Deploying Mall-Swarm to $ENVIRONMENT environment..."
# 创建namespace
kubectl create namespace $NAMESPACE --dry-run=client -o yaml | kubectl apply -f -
# 部署配置
kubectl apply -f k8s/common/ -n $NAMESPACE
kubectl apply -f k8s/$ENVIRONMENT/ -n $NAMESPACE
# 部署基础设施
echo "Deploying infrastructure..."
kubectl apply -f k8s/infrastructure/ -n $NAMESPACE
# 等待基础设施就绪
kubectl wait --for=condition=ready pod -l app=mysql -n $NAMESPACE --timeout=300s
kubectl wait --for=condition=ready pod -l app=redis -n $NAMESPACE --timeout=300s
kubectl wait --for=condition=ready pod -l app=nacos -n $NAMESPACE --timeout=300s
# 部署应用
echo "Deploying applications..."
kubectl apply -f k8s/apps/ -n $NAMESPACE
# 设置镜像版本
if [ "$VERSION" != "latest" ]; then
kubectl set image deployment/mall-gateway mall-gateway=mall/mall-gateway:$VERSION -n $NAMESPACE
kubectl set image deployment/mall-admin mall-admin=mall/mall-admin:$VERSION -n $NAMESPACE
kubectl set image deployment/mall-portal mall-portal=mall/mall-portal:$VERSION -n $NAMESPACE
kubectl set image deployment/mall-search mall-search=mall/mall-search:$VERSION -n $NAMESPACE
kubectl set image deployment/mall-auth mall-auth=mall/mall-auth:$VERSION -n $NAMESPACE
fi
# 等待部署完成
echo "Waiting for deployments to be ready..."
kubectl rollout status deployment/mall-gateway -n $NAMESPACE
kubectl rollout status deployment/mall-admin -n $NAMESPACE
kubectl rollout status deployment/mall-portal -n $NAMESPACE
kubectl rollout status deployment/mall-search -n $NAMESPACE
kubectl rollout status deployment/mall-auth -n $NAMESPACE
echo "Deployment completed successfully!"
# 显示服务状态
kubectl get pods -n $NAMESPACE
kubectl get svc -n $NAMESPACE
9.12.2 健康检查脚本
health-check.sh:
#!/bin/bash
NAMESPACE="mall-system"
SERVICES=("mall-gateway" "mall-admin" "mall-portal" "mall-search" "mall-auth")
echo "Checking health of Mall-Swarm services..."
for service in "${SERVICES[@]}"; do
echo "Checking $service..."
# 检查Pod状态
pod_status=$(kubectl get pods -l app=$service -n $NAMESPACE -o jsonpath='{.items[0].status.phase}')
if [ "$pod_status" != "Running" ]; then
echo "❌ $service pod is not running (status: $pod_status)"
continue
fi
# 检查健康端点
pod_name=$(kubectl get pods -l app=$service -n $NAMESPACE -o jsonpath='{.items[0].metadata.name}')
health_check=$(kubectl exec $pod_name -n $NAMESPACE -- curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/actuator/health)
if [ "$health_check" = "200" ]; then
echo "✅ $service is healthy"
else
echo "❌ $service health check failed (HTTP $health_check)"
fi
done
echo "Health check completed."
9.13 故障排查指南
9.13.1 常见问题排查
troubleshooting-guide.md:
# 故障排查指南
## 1. 服务无法启动
### 问题现象
- Pod状态为CrashLoopBackOff
- 服务无法注册到Nacos
### 排查步骤
```bash
# 查看Pod日志
kubectl logs -f <pod-name> -n mall-system
# 查看Pod描述
kubectl describe pod <pod-name> -n mall-system
# 查看事件
kubectl get events -n mall-system --sort-by='.lastTimestamp'
常见原因
- 配置文件错误
- 依赖服务未就绪
- 资源限制不足
- 镜像拉取失败
2. 服务间调用失败
问题现象
- 接口返回504超时
- Feign调用失败
排查步骤
# 检查服务注册状态
curl http://nacos:8848/nacos/v1/ns/catalog/services
# 检查网络连通性
kubectl exec -it <pod-name> -- nslookup <service-name>
# 查看负载均衡状态
kubectl get endpoints -n mall-system
3. 数据库连接问题
问题现象
- 连接池耗尽
- 数据库连接超时
排查步骤
# 检查MySQL状态
kubectl exec -it mysql-pod -- mysql -u root -p -e "SHOW PROCESSLIST;"
# 检查连接数
kubectl exec -it mysql-pod -- mysql -u root -p -e "SHOW STATUS LIKE 'Threads_connected';"
# 查看慢查询
kubectl exec -it mysql-pod -- mysql -u root -p -e "SELECT * FROM mysql.slow_log ORDER BY start_time DESC LIMIT 10;"
4. 内存泄漏问题
问题现象
- Pod内存使用持续增长
- 频繁GC但内存不释放
排查步骤
# 生成堆转储
kubectl exec -it <pod-name> -- jcmd 1 GC.run_finalization
kubectl exec -it <pod-name> -- jcmd 1 VM.classloader_stats
# 查看GC日志
kubectl logs <pod-name> | grep GC
# 使用jstat监控
kubectl exec -it <pod-name> -- jstat -gc 1 5s
9.14 小结
通过本章的学习，我们完成了Mall-Swarm微服务商城系统的完整部署运维方案，主要内容包括：
- 容器化部署: Docker镜像制作、Docker Compose部署
- Kubernetes集群: K8s资源配置、服务部署、自动扩缩容
- 监控告警: Prometheus监控、Grafana可视化、告警规则配置
- 日志管理: ELK Stack日志收集、分析和存储
- CI/CD流水线: Jenkins自动化构建、测试、部署
- 备份恢复: 数据备份策略、灾难恢复方案
- 性能优化: JVM调优、数据库优化、系统调优
- 安全加固: 网络安全、访问控制、安全策略
- 运维工具: 自动化脚本、健康检查、故障排查
这套完整的部署运维方案确保了Mall-Swarm系统的高可用性、可扩展性和可维护性，为系统的稳定运行提供了坚实的保障。
开源鸿蒙跨平台开发社区汇聚开发者与厂商，共建“一次开发，多端部署”的开源生态，致力于降低跨端开发门槛，推动万物智联创新。
更多推荐
0
0- 0
已为社区贡献1条内容
所有评论(0)