K8s集群Java项目上云部署
K8s集群Java项目上云部署（storageclass的环境）
- 部署前准备工作
- 项目基础镜像准备
本次发布一个java项目，以war包方式发布，需要使用tomcat做为项目基础镜像,然后定制。
定制tomcat镜像：
[root@localhost ~]# mkdir /root/tomcat
[root@localhost ~]# cd /root/tomcat/
[root@localhost tomcat]# vim Dockerfile
添加：
FROM centos:7
MAINTAINER "admin<admin@test.com>"
ENV VERSION=9.0.102
ENV JAVA_HOME=/usr/local/jdk
ADD ./wget-1.14-18.el7_6.1.x86_64.rpm /wget-1.14-18.el7_6.1.x86_64.rpm
RUN rpm -i /wget-1.14-18.el7_6.1.x86_64.rpm
RUN wget https://dlcdn.apache.org/tomcat/tomcat-9/v${VERSION}/bin/apache-tomcat-${VERSION}.tar.gz --no-check-certificate
RUN tar xf apache-tomcat-${VERSION}.tar.gz
RUN mv apache-tomcat-${VERSION} /usr/local/tomcat
RUN rm -rf apache-tomcat-${VERSION}.tar.gz /usr/local/tomcat/webapps/*
RUN mkdir /usr/local/tomcat/webapps/ROOT
ADD ./jdk /usr/local/jdk
RUN echo "export TOMCAT_HOME=/usr/local/tomcat" >> /etc/profile
RUN echo "export JAVA_HOME=/usr/local/jdk" >> /etc/profile
RUN echo "export PATH=$TOMCAT_HOME/bin:$JAVA_HOME/bin:$PATH" >> /etc/profile
RUN echo "export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar" >> /etc/profile
RUN source /etc/profile
EXPOSE 8080
CMD ["/usr/local/tomcat/bin/catalina.sh","run"]
保存退出
准备wget下载工具：（可挂载关盘复制）：
[root@localhost tomcat]# mkdir /cdrom
[root@localhost tomcat]# mount /dev/cdrom /cdrom/
[root@localhost tomcat]# cp /cdrom/Packages/wget-1.14-18.el7_6.1.x86_64.rpm .
复制jdk8u91到harbor主机，解包，改名为jdk：
[root@localhost ~]# cd ~
[root@localhost ~]# tar xf jdk-8u191-linux-x64.tar.gz
[root@localhost ~]# mv jdk1.8.0_191/ tomcat/jdk
[root@localhost ~]# cd /root/tomcat
[root@localhost tomcat]# ls
开启路由转发：
[root@localhost tomcat]# vim /etc/sysctl.conf
文件末尾添加：net.ipv4.ip_forward = 1
保存退出
[root@localhost tomcat]# sysctl -p
构建镜像：
[root@localhost tomcat]# docker build -t harbor.test.com/java-project/tomcat:9102 . //需等待较长时间
[root@localhost ~]# docker images
上传镜像：
[root@localhost ~]# docker ps
[root@localhost ~]# cd /usr/local/harbor/
[root@localhost harbor]# docker-compose restart
[root@localhost ~]# docker login harbor.test.com
windows浏览器访问harbor服务器 ：http://192.168.10.14
并创建java-project项目
上传tomcat镜像：
[root@localhost ~]# docker push harbor.test.com/java-project/tomcat:9102
Harbor上验证tomcat镜像是否上传：
项目部署工作流程：
- 基于项目源码制作容器镜像
- 编写项目部署资源清单文件
- 使用资源清单文件创建项目 Service
- 使用ingress暴露服务实现项目对外可访问
- 存储准备
本次使用NFS服务做为K8S集群后端存储，实现kubernetes集群持久存储动态供给，详细可见kubernetes集群公共服务章节。
[root@master01 ~]# kubectl get storageclass
[root@master01 ~]# kubectl get pod
如果报错解决方法：
[root@xxx ~]# yum install nfs-utils -y //所有主机安装
[root@localhost ~]# mkdir -p /data/nfs
[root@localhost ~]# vim /etc/exports
添加：/data/nfs *(rw,sync,no_root_squash)
保存退出
[root@localhost ~]# systemctl start nfs-server
[root@localhost ~]# systemctl enable nfs-server
[root@localhost ~]# showmount -e
两台worker主机上挂载nfs目录：（master没必要挂载，master上不运行pod）
[root@master01 ~]# kubectl get pod
下载并创建（storageclass动态存储供给机制）
[root@master01 ~]# wget https://raw.githubusercontent.com/kubernetes-sigs/nfs-subdir-external-provisioner/master/deploy/class.yaml
或者复制storageclass-nfs.yml包到master01
[root@master01 ~]# mv class.yaml storageclass-nfs.yml
[root@master01 ~]# vim storageclass-nfs.yml
添加：
apiVersion: storage.k8s.io/v1
kind: StorageClass # 类型
metadata:
name: nfs-client # 名称,要使用就需要调用此名称
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # 动态供给插件
parameters:
archiveOnDelete: "false" # 删除数据时是否存档，false表示不存档，true表示存档
保存退出
应用：[root@master01 ~]# kubectl apply -f storageclass-nfs.yml
查看：[root@master01 ~]# kubectl get storageclass
下载配置清单文件，创建rbac（创建账号，并授权）：
[root@master01 ~]# wget https://raw.githubusercontent.com/kubernetes-sigs/nfs-subdir-external-provisioner/master/deploy/rbac.yaml
[root@master01 ~]# mv rbac.yaml storageclass-nfs-rbac.yaml
[root@master01 ~]# cat storageclass-nfs-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-client-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: Role
name: leader-locking-nfs-client-provisioner
apiGroup: rbac.authorization.k8s.io
应用：[root@master01 ~]# kubectl apply -f storageclass-nfs-rbac.yaml
创建动态供给的deployment
需要一个deployment来专门实现pv与pvc的自动创建
[root@master01 ~]# vim deploy-nfs-client-provisioner.yml
添加：
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-client-provisioner
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccount: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: registry.cn-beijing.aliyuncs.com/pylixm/nfs-subdir-external-provisioner:v4.0.0
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: k8s-sigs.io/nfs-subdir-external-provisioner
- name: NFS_SERVER
value: 192.168.10.14
- name: NFS_PATH
value: /data/nfs
volumes:
- name: nfs-client-root
nfs:
server: 192.168.10.14
path: /data/nfs
保存退出
应用：[root@master01 ~]# kubectl apply -f deploy-nfs-client-provisioner.yml
查看：[root@master01 ~]# kubectl get pods |grep nfs-client-provisioner
创建缓慢可以去相应的节点拉取镜像：
[root@worker02 ~]# docker pull registry.cn-beijing.aliyuncs.com/pylixm/nfs-subdir-external-provisioner:v4.0.0
[root@worker02 ~]# systemctl restart docker
- 项目容器镜像仓库及项目源码准备
- 项目源码及编译环境准备
- 项目源码准备（文件在“k8s相关资料”java子目录中）
复制java源码包javaproject.tar.gz到当前位置，并解包
[root@localhost ~]# cd ~
[root@localhost ~]# tar xf javaproject.tar.gz
[root@localhost ~]# cd /root/javaproject/
[root@localhost javaproject]# ls
[root@localhost javaproject]# ls project-source/
db pro-source
[root@localhost javaproject]# ls project-source/db
test.sql
[root@localhost javaproject]# ls project-source/pro-source/
java-project
[root@localhost javaproject]# ls project-source/pro-source/java-project/
Dockerfile LICENSE pom.xml README.md src
[root@localhost javaproject]# ls project-source/pro-source/java-project/src
Main源代码目录
[root@localhost ~]# cd /root/javaproject/project-source/pro-source/java-project/
[root@localhost java-project]# rm -rf Dockerfile
[root@localhost java-project]# vim Dockerfile
添加：
FROM harbor.test.com/java-project/tomcat:9102
LABEL maintainer "admin <admin@test.com>"
RUN rm -rf /usr/local/tomcat/webapps/*
ADD target/*.war /usr/local/tomcat/webapps/ROOT.war
保存退出
[root@localhost java-project]# cat /root/javaproject/project-source/pro-source/java-project/README.md
[root@localhost java-project]# cat /root/javaproject/project-source/pro-source/java-project/src/main/resources/application.yml
- 项目编译环境准备jdk&maven
用于对java项目代码进行编译打包代码
下载maven：
[root@localhost java-project]# cd ~
[root@localhost ~]# wget https://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.8.8/binaries/apache-maven-3.8.8-bin.tar.gz
或者复制apache-maven-3.8.8-bin.tar.gz包到10.14主机：
安装maven:
必须要有官方的jdk，不然maven会报错
[root@localhost ~]# ls
[root@localhost ~]# tar xf jdk-8u191-linux-x64.tar.gz
[root@localhost ~]# mv jdk1.8.0_191 /usr/local/jdk
[root@localhost ~]# tar xf apache-maven-3.8.8-bin.tar.gz
[root@localhost ~]# mv apache-maven-3.8.8 /usr/local/maven
[root@localhost ~]# vim /etc/profile.d/maven.sh
添加：
export JAVA_HOME=/usr/local/jdk
export MAVEN_HOME=/usr/local/maven
export PATH=${MAVEN_HOME}/bin:${JAVA_HOME}/bin:$PATH
保存退出
[root@localhost ~]# source /etc/profile
[root@localhost ~]# mvn -v
- 项目部署
- 本项目部署思路
- 数据库 MySQL
- Web应用 Tomcat
- 有状态应用与无状态应用部署规划
Mysql
-
-
- statefulset控制器
- headless service
- pv,pvc 存储资源动态供给
-
tomcat（java应用）
-
-
- deployment控制器
- ClusterIP Service
- Ingress 服务暴露实现集群外访问
-
- 项目资源清单文件准备
- 项目文件总览
[root@localhost ~]# ls /root/javaproject/project-source/pro-source/java-project
- 资源清单文件准备
[root@harbor ~]# mkdir /usr/local/nginx/html/java_project
[root@harbor ~]# cd /usr/local/nginx/html/java_project
-
- namespace资源清单文件用于实现项目隔离
[root@harbor java_project]# vim 01_ns.yaml
添加：
apiVersion: v1
kind: Namespace
metadata:
name: javaproject
保存退出
-
- 部署java项目资源清单文件用于部署java项目
[root@harbor java_project]# vim 02_deployment.yaml
添加：
apiVersion: apps/v1
kind: Deployment
metadata:
name: java-project
namespace: javaproject
spec:
replicas: 2
selector:
matchLabels:
project: www
app: java-demo
template:
metadata:
labels:
project: www
app: java-demo
spec:
containers:
- name: tomcat
image: harbor.test.com/java-project/java-project:v1 #镜像
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
name: web
protocol: TCP
resources:
requests:
cpu: 0.5
memory: 1Gi
limits:
cpu: 1
memory: 2Gi
livenessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 20
readinessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 20
保存退出
-
- 项目服务（service）资源清单文件用于创建service
[root@harbor java_project]# vim 03_service.yaml
添加：
apiVersion: v1
kind: Service
metadata:
name: java-project
namespace: javaproject
spec:
selector:
project: www
app: java-demo
ports:
- name: web
port: 80
targetPort: 8080
保存退出
-
- Ingress对象资源清单文件需要提前部署 ingress控制器
[root@harbor java_project]# vim 04_ingress.yaml
添加：
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: java-project
namespace: javaproject
annotations:
ingressclass.kubernetes.io/is-default-class: "true"
spec:
ingressClassName: nginx
rules:
- host: javaweb.test.com
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: java-project
port:
number: 80
保存退出
-
- mysql部署资源清单文件
[root@harbor java_project]# vim 05_mysql.yaml
（如果是rocky9时可使用mysql:8.0的容器镜像）
添加：
apiVersion: v1
kind: Service
metadata:
name: mysql
namespace: javaproject
spec:
ports:
- port: 3306
name: mysql
clusterIP: None
selector:
app: mysql-public
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: db
namespace: javaproject
spec:
selector:
matchLabels:
app: mysql-public
serviceName: "mysql"
template:
metadata:
labels:
app: mysql-public
spec:
containers:
- name: mysql
image: mysql:5.7
env:
- name: MYSQL_ROOT_PASSWORD
value: "123456"
- name: MYSQL_DATABASE
value: test
ports:
- containerPort: 3306
volumeMounts:
- mountPath: "/var/lib/mysql"
name: nfsdata
volumeClaimTemplates:
- metadata:
name: nfsdata
spec:
accessModes: ["ReadWriteMany"]
storageClassName: "nfs-client"
resources:
requests:
storage: 5Gi
保存退出
-
- mysql访问测试pod
[root@harbor java_project]# vim 06_create_pod_busybox.yaml
添加：
apiVersion: v1
kind: Pod
metadata:
name: busybox-pod
spec:
containers:
- name: busybox-container
image: busybox:1.28.4 #此镜像nslookup及ping命令都没有问题，不要下载最新版本和1.31。
imagePullPolicy: IfNotPresent
command:
- sleep
- "3600"
restartPolicy: Always
保存退出
注意：Metallb、ingress nginx负载均衡的yaml清单文件按前面课程准备
将提前准备好的清单文件metallb_ingress.tar.gz复制到本地
[root@harbor ~]# cd ~
[root@harbor ~]# tar xf metallb_ingress.tar.gz
[root@harbor ~]# mkdir /usr/local/nginx/html/metallb_ingress
[root@harbor ~]# cp /root/metallb_ingress/* /usr/local/nginx/html/metallb_ingress/
[root@harbor ~]# ls /usr/local/nginx/html/metallb_ingress
[root@harbor ~]# ls /usr/local/nginx/html/java_project
- 项目数据库 Mysql部署
- 创建命名空间
[root@master01 ~]# kubectl apply -f http://yaml.test.com:81/java_project/01_ns.yaml
[root@master01 ~]# kubectl get ns
提前导入镜像：
docker pull mysql:5.7
docker pull busybox:1.28.4
systemctl restart docker
- 部署数据库
[root@master01 ~]# kubectl apply -f http://yaml.test.com:81/java_project/05_mysql.yaml
[root@master01 ~]# kubectl get all -n javaproject
[root@harbor ~]# ls /data/nfs/
- 导入数据库
[root@harbor ~]# scp /root/javaproject/project-source/db/test.sql 192.168.10.11:/root
[root@master01 ~]# kubectl get pods -n javaproject
[root@master01 ~]# kubectl cp test.sql db-0:/ -n javaproject
[root@master01 ~]# kubectl exec -it db-0 -n javaproject -- bash
bash-4.2# ls
bash-4.2# mysql -uroot -p123456
mysql> use test;
mysql> source /test.sql;
mysql> show tables;
mysql> desc user;
- 验证数据库可用性
[root@master01 ~]# kubectl apply -f http://yaml.test.com:81/java_project/06_create_pod_busybox.yaml
[root@master01 ~]# kubectl get pods
[root@master01 ~]# kubectl exec -it busybox-pod -- sh
/ # nslookup db-0.mysql.javaproject
/ # nslookup db-0.mysql.javaproject.svc.cluster.local.
- 项目源码编译打包
- 修改项目连接数据库地址
[root@harbor ~]# cd /root/javaproject/project-source/pro-source/java-project/
[root@harbor java-project]# ls
Dockerfile LICENSE pom.xml README.md src target
[root@harbor java-project]# cd src/
[root@harbor src]# cd main/
[root@harbor main]# cd resources/
[root@harbor resources]# vim application.yml
添加：
server:
port: 8080
spring:
datasource:
url: jdbc:mysql://db-0.mysql.javaproject:3306/test?characterEncoding=utf-8
username: root
password: 123456
driver-class-name: com.mysql.jdbc.Driver
freemarker:
allow-request-override: false
cache: true
check-template-location: true
charset: UTF-8
content-type: text/html; charset=utf-8
expose-request-attributes: false
expose-session-attributes: false
expose-spring-macro-helpers: false
suffix: .ftl
template-loader-path:
- classpath:/templates/
保存退出
- 使用maven编译项目源码
[root@harbor java-project]# pwd
[root@harbor java-project]# mvn clean package
[root@harbor java-project]# ls
[root@harbor java-project]# ls target/
- 生成项目容器镜像制品
[root@harbor java-project]# pwd
[root@harbor java-project]# ls
[root@harbor java-project]# docker build -t harbor.test.com/java-project/java-project:v1 .
[root@harbor java-project]# docker login harbor.test.com
[root@harbor java-project]# docker push harbor.test.com/java-project/java-project:v1
- 部署metallb
- 修改kube-proxy代理模式
[root@master01 ~]# kubectl edit configmap kube-proxy -n kube-system
修改：
保存退出
[root@master01 ~]# kubectl rollout restart daemonset kube-proxy -n kube-system
四台主机提前导入镜像：metallb-controller.tar metallb_speaker.tar
docker load -i metallb-controller.tar
docker load -i metallb_speaker.tar
systemctl restart docker
- metallb部署
[root@master01 ~]# kubectl apply -f http://yaml.test.com:81/metallb_ingress/metallb-native.yaml
[root@master01 ~]# kubectl -n metallb-system get pod
（3）IP地址池准备
[root@master01 ~]# kubectl apply -f http://yaml.test.com:81/metallb_ingress/ippool.yaml
[root@master01 ~]# kubectl -n metallb-system get ipaddresspools
（4）开启二层通告
[root@master01 ~]# kubectl apply -f http://yaml.test.com:81/metallb_ingress/l2.yaml
- 服务代理ingress nginx部署
- 获取ingress nginx部署文件
[root@master01 ~]# wget http://yaml.test.com:81/metallb_ingress/deploy.yaml
删除所有的pod：
[root@master01 ~]# kubectl -n ingress-nginx delete all --all --force --grace-period=0
- 修改部署文件deploy.yaml
[root@master01 ~]# vim deploy.yaml
修改365此行 type: NodePort为 type: LoadBalancer
保存退出
可以提前准备镜像：复制deploy.yaml到master01主机；复制ingress-nginx.tar包到两台工作节点，并导入镜像
docker load -i ingress.tar
systemctl restart docker
（3）部署ingress nginx
[root@master01 ~]# kubectl apply -f deploy.yaml
[root@master01 ~]# kubectl -n ingress-nginx get pod
#如果状态异常，可手动下载镜像文件
kubectl -n ingress-nginx delete job ingress-nginx-admission-create
kubectl -n ingress-nginx delete job ingress-nginx-admission-patch
注：查看pod镜像文件
# kubectl -n ingress-nginx describe pod ingress-nginx-controller-c66f877dd-dxm45
# kubectl -n ingress-nginx describe pod ingress-nginx-admission-create-tkhgz
- 项目部署
[root@master01 ~]# kubectl apply -f http://yaml.test.com:81/java_project/02_deployment.yaml
[root@master01 ~]# kubectl get pods -n javaproject
[root@master01 ~]# kubectl apply -f http://yaml.test.com:81/java_project/03_service.yaml
[root@master01 ~]# kubectl get svc -n javaproject
[root@master01 ~]# kubectl apply -f http://yaml.test.com:81/java_project/04_ingress.yaml
[root@master01 ~]# kubectl get ingress -n javaproject
- 访问验证
[root@master01 ~]# kubectl get svc -n ingress-nginx
windows浏览器访问：javaweb.test.com
需要修改etc/hosts文件解析
记事本的形式打开并添加：192.168.10.240 javaweb.test.com
如果保存时需另存则：点击hosts右击——属性——安全——Users——编辑——勾选修改和写入权限——确定——重新修改hosts文件——
Windows访问：http://javaweb.test.com/
开源鸿蒙跨平台开发社区汇聚开发者与厂商，共建“一次开发，多端部署”的开源生态，致力于降低跨端开发门槛，推动万物智联创新。
更多推荐
4
0- 0
已为社区贡献2条内容
所有评论(0)