离线部署 k3s 集群和搭建管理环境
- 环境准备
- 安装部署 k3s
- 安装 Helm 部署工具
- 部署 Cert Manager
- 部署 Rancher 集群管理工具
- 部署网络存储
- 部署 Harbor 私有化仓库
- 部署 ChartMuseum 管理工具
- 部署 frps 反向代理
- 管理新的集群（业务集群）
环境准备
下载离线安装文件
本例子用 v1.30.4+k3s1
- 安装脚本 install.sh
wget https://github.com/k3s-io/k3s/raw/refs/tags/v1.30.4+k3s1/install.sh- 主程序 k3s
wget https://github.com/k3s-io/k3s/releases/download/v1.30.4%2Bk3s1/k3s- 镜像压缩包
wget https://github.com/k3s-io/k3s/releases/download/v1.30.4%2Bk3s1/k3s-airgap-images-amd64.tar准备数据库 PostgreSQL
- 数据库
| name | ip | notes |
|---|---|---|
| database | 192.168.100.100 | PostgreSQL |
- 创建 compose.yml
services:
postgres:
image: postgres:alpine
container_name: postgres
restart: unless-stopped
environment:
POSTGRES_USER: k3s
POSTGRES_PASSWORD: talent
POSTGRES_DB: k3s
ports:
- "5432:5432"
volumes:
- ./data:/var/lib/postgresql/data
- ./init-db.sh:/docker-entrypoint-initdb.d/init-db.sh - 创建脚本 init-db.sh
#!/bin/bash
set -e
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname=postgres <<-EOSQL
CREATE DATABASE harbor;
EOSQL- 启动 PostgreSQL 数据库容器
docker compose up -d- 验证数据库
docker exec -it postgres psql -U k3s -c '\l'
List of databases
Name | Owner | Encoding | Locale Provider | Collate | Ctype | Locale | ICU Rules | Access privileges
-----------+-------+----------+-----------------+------------+------------+--------+-----------+-------------------
harbor | k3s | UTF8 | libc | en_US.utf8 | en_US.utf8 | | |
k3s | k3s | UTF8 | libc | en_US.utf8 | en_US.utf8 | | |
...
(5 rows)安装部署 k3s
准备服务器
准备三台机器，都作为 master 运行，本节内容需要在三台都执行。
| name | ip | notes |
|---|---|---|
| master01 | 192.168.100.101 | 必须 |
| master02 | 192.168.100.102 | 可选 |
| master03 | 192.168.100.103 | 可选 |
cat <<EOF >> /etc/hosts
192.168.100.100 database
192.168.100.101 master01
192.168.100.102 master02
192.168.100.103 master03
EOF导入镜像包
- 离线安装重要步骤
mkdir -p /var/lib/rancher/k3s/agent/images/
cp k3s-airgap-images-amd64.tar /var/lib/rancher/k3s/agent/images/拷贝主程序 k3s
- 拷贝进执行目录，不需要下载主程序
chmod a+x k3s && cp k3s /usr/local/bin/安装
- 执行安装脚本，并连接数据库
INSTALL_K3S_SKIP_DOWNLOAD=true \
sh install.sh -s - server \
--datastore-endpoint="postgres://k3s:talent@database:5432/k3s?sslmode=disable" \
--token K10c71232f051944de58ab058871ac7a85b42cbdee5c6df94deb6bb493c79b15a92- 查看镜像状态
ctr i ls -q
docker.io/rancher/klipper-helm:v0.8.4-build20240523
docker.io/rancher/klipper-lb:v0.4.9
docker.io/rancher/local-path-provisioner:v0.0.28
docker.io/rancher/mirrored-coredns-coredns:1.10.1
docker.io/rancher/mirrored-library-busybox:1.36.1
docker.io/rancher/mirrored-library-traefik:2.10.7
docker.io/rancher/mirrored-metrics-server:v0.7.0
docker.io/rancher/mirrored-pause:3.6- 三台机器都执行完成，查看集群 node 状态
kubectl get nodes
NAME STATUS ROLES AGE VERSION
master01 Ready control-plane,master 4m49s v1.30.4+k3s1
master02 Ready control-plane,master 51s v1.30.4+k3s1
master03 Ready control-plane,master 45s v1.30.4+k3s1- 查看 pod 状态
kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-576bfc4dc7-f6bsb 1/1 Running 0 27m
kube-system helm-install-traefik-crd-5b2x4 0/1 Completed 0 27m
kube-system helm-install-traefik-pl6k7 0/1 Completed 1 27m
kube-system local-path-provisioner-5f9d8-lhntz 1/1 Running 0 27m
kube-system metrics-server-557ff575fb-77xzw 1/1 Running 0 27m
kube-system svclb-traefik-151f91a9-8tkdq 2/2 Running 0 23m
kube-system svclb-traefik-151f91a9-b7pwz 2/2 Running 0 23m
kube-system svclb-traefik-151f91a9-kwwp8 2/2 Running 0 25m
kube-system traefik-5fb479b77-r84mv 1/1 Running 0 25m安装 Helm 部署工具
- 下载 Helm 二进制，安装
wget https://get.helm.sh/helm-v3.16.2-linux-amd64.tar.gz
tar -zxvf helm-v3.16.2-linux-amd64.tar.gz
chmod a+x linux-amd64/helm && cp helm /usr/local/bin/- 设置环境变量
加入到～/.bashrc自启动
echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> ~/.bashrc && source ~/.bashrc部署 Cert Manager
- 在线下载 Cert Manager 镜像，并导出，复制到三台服务器
docker images --format "{{.Repository}}:{{.Tag}}" |grep cert-manager > cert-manager.txt
docker save $(awk '{printf "%s ", $0}' cert-manager.txt) | gzip > cert-manager.tar.gz- 导入 Cert Manager 镜像
gzip -d -c cert-manager-images.tar.gz | ctr i import -- 下载所需文件，复制到某一台服务器后
wget https://github.com/cert-manager/cert-manager/releases/download/v1.15.3/cert-manager.crds.yaml
wget https://charts.jetstack.io/charts/cert-manager-v1.15.3.tgz- 部署 cert-manager 配置
kubectl apply -f cert-manager.crds.yamlcustomresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created- 使用 Helm 部署 cert-manager 离线包
helm install cert-manager ./cert-manager-v1.15.3.tgz --namespace cert-manager --create-namespace
or
helm install cert-manager ./cert-manager-v1.15.3.tgz --namespace cert-manager --create-namespace --set crds.enabled=truekubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-9647b459d-dqd8b 1/1 Running 0 23m
cert-manager-cainjector-5d8798687c-l2kkf 1/1 Running 0 23m
cert-manager-startupapicheck-6gglh 0/1 Completed 1 23m
cert-manager-webhook-c77744d75-wd2kx 1/1 Running 0 23m部署 Rancher 集群管理工具
导入镜像
- 在线下载 Rancher 镜像，并导出，复制到三台服务器
docker images --format "{{.Repository}}:{{.Tag}}" |grep rancher > rancher.txt
docker save $(awk '{printf "%s ", $0}' rancher.txt) | gzip > rancher-images.tar.gz- 导入 Rancher 镜像
gzip -d -c rancher-images.tar.gz | ctr i import -部署 Rancher
- 下载 Rancher Helm 包
wget https://releases.rancher.com/server-charts/latest/rancher-2.9.2.tgz- 使用 Helm 部署 Rancher 离线包
helm install rancher ./rancher-2.9.2.tgz \
--namespace cattle-system \
--set hostname=manager.yiqisoft.cn \
--create-namespace- 查询 Rancher 安装结果
kubectl get pods -n cattle-system
NAME READY STATUS RESTARTS AGE
rancher-57c9747d96-p498t 1/1 Running 0 52m
rancher-57c9747d96-pdrtn 1/1 Running 1 (52m ago) 52m
rancher-57c9747d96-xm9sl 1/1 Running 0 52m
rancher-webhook-77b49b9ff9-fqdxf 1/1 Running 0 51m- 配置 dns 指向，访问 Rancher 页面
dns 指向到三台服务器，如果有条件，要设置 HAProxy 和 KeepAlived 做负载均衡。
address=/manager.yiqisoft.cn/192.168.100.101
address=/manager.yiqisoft.cn/192.168.100.102
address=/manager.yiqisoft.cn/192.168.100.103部署网络存储
使用 Ceph（推荐）
准备一个 Ceph 集群用于存储（这里省略，请自行补脑）。
使用 NFS 网络文件系统
简单实用，但是效率不高，安全性一般。
- 在一个文件服务器上部署 NFS，这里用 database 主机
apt update
apt install nfs-kernel-server -y
mkdir -p /opt/nfs_share
chown nobody:nogroup /opt/nfs_share
chmod 777 /opt/nfs_share
echo "/opt/nfs_share *(rw,sync,no_subtree_check)" >> /etc/exports
exportfs -ra
systemctl restart nfs-kernel-server部署 Harbor 私有化仓库
部署 pv 和 pvc
- 编辑 pv-pvc.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-pv
spec:
capacity:
storage: 30Gi
accessModes:
- ReadWriteMany
nfs:
path: /opt/nfs_share
server: database
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nfs-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
volumeName: nfs-pv
storageClassName: ""- 应用到 namespace harbor
kubectl create namespace harbor
kubectl apply -f pv-pvc.yaml -n harbor结果
persistentvolume/nfs-pv created
persistentvolumeclaim/nfs-pvc created- 查看结果，nfs-pvc 必须是
Bound
kubectl get pvc -n harbor结果
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
nfs-pvc Bound nfs-pv 30Gi RWX <unset> 92s- 查看 nfs-pv 必须是
Bound
kubectl get pv -n harbor结果
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
nfs-pv 30Gi RWX Retain Bound harbor/nfs-pvc <unset> 97s安装 NFS 依赖
所有的 Master 上需要安装 nfs-common，并验证（可选），可以访问 /tmp/nfs 目录
apt install nfs-common -y
mkdir /tmp/nfs
mount -t nfs database:/opt/nfs_share /tmp/nfs
ll /tmp/nfs部署 Harbor
- 在线下载 Harbor Helm 文件，并复制到某一台服务器
wget https://helm.goharbor.io/harbor-1.15.1.tgz- 在线方式，导出默认 values，修改
helm show values harbor/harbor > harbor-values.yml
# 内容修改如下
expose:
ingress:
hosts:
core: hub.yiqisoft.cn
externalURL: https://hub.yiqisoft.cn
persistentVolumeClaim:
registry:/exter
existingClaim: "nfs-pvc"
storageClass: "-"
database:
type: external
external:
host: "192.168.100.100"
port: "5432"
username: "k3s"
password: "talent"
coreDatabase: "harbor"
harborAdminPassword: "Harbor12345"- 在线下载 Harbor 镜像，导出
docker images --format "{{.Repository}}:{{.Tag}}" |grep goharbor > harbor.txt
docker save $(awk '{printf "%s ", $0}' harbor.txt) | gzip > harbor-images.tar.gz- 导入 Harbor 镜像
gzip -d -c harbor-images.tar.gz | ctr i import -- 安装 Harbor，需要通过互联网下载所有的镜像，时间根据网络带宽决定
helm install harbor ./harbor-1.15.1.tgz -f harbor-values.yml -n harbor- 确认 Harbor 是否安装成功
kubectl get pods -n harbor都显示 READY 即可
NAME READY STATUS RESTARTS AGE
harbor-core-666cf84bc6-vq8mp 1/1 Running 0 85s
harbor-jobservice-c68bc669-6twbp 1/1 Running 0 85s
harbor-portal-d6c99f896-cq9fx 1/1 Running 0 85s
harbor-redis-0 1/1 Running 0 84s
harbor-registry-5984c768c8-4xnrg 2/2 Running 0 84s
harbor-trivy-0 1/1 Running 0 84s- 配置 dns 指向，访问 Harbor 页面
dns 指向到三台服务器，如果有条件，要设置 HAProxy 和 KeepAlived 做负载均衡。
address=/hub.yiqisoft.cn/192.168.100.101
address=/hub.yiqisoft.cn/192.168.100.102
address=/hub.yiqisoft.cn/192.168.100.103- 验证 Harbor 工作
curl -k https://hub.yiqisoft.cn/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"unauthorized: unauthorized"}]}部署 ChartMuseum 管理工具
下载 ChartMuseum 镜像
- 在线下载 ChartMuseum 镜像，并导出，复制到三台服务器
docker save hub.yiqisoft.cn/yiqisoft/chartmuseum:v0.16.2 | gzip > chartmuseum-images.tar.gz- 上传到私有仓库
docker push hub.yiqisoft.cn/yiqisoft/chartmuseum:v0.16.2- 或导入 ChartMuseum 镜像
gzip -d -c chartmuseum-images.tar.gz | ctr i import -部署 ChartMuseum Helm Chart
- 下载 ChartMuseum chart
wget https://github.com/chartmuseum/charts/releases/download/chartmuseum-3.10.3/chartmuseum-3.10.3.tgz- 部署 Chart
helm install chartmuseum ./chartmuseum-3.10.3.tgz \
--namespace chartmuseum \
--create-namespace \
--set env.open.DISABLE_API=false \
--set service.type=LoadBalancer- 部署一个 Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
name: chartmuseum
namespace: chartmuseum
spec:
ingressClassName: traefik
rules:
- host: chart.yiqisoft.cn
http:
paths:
- backend:
service:
name: chartmuseum
port:
number: 8080
path: /
pathType: Prefix或在 Rancher UI 中增加一个类似配置。
上传 Charts 到 ChartMuseum
- 使用类似这样的命令即可上传，ChartMuseum 会自动索引文件到 index.yaml
curl --data-binary "@yiedge-base-3.1.0.tgz" http://chart.yiqisoft.cn/api/charts
- 检查上传结果
curl chart.yiqisoft.cn/index.yaml
apiVersion: v1
entries:
yiedge-base:
- apiVersion: v1
appVersion: Pnicom
created: "2024-11-11T10:11:55.473460704Z"
description: A Helm Chart for YiEDGE base
digest: 14b9a971a53a2482a6d4a930daefe2b0fee1cb1bcef909b023cc34959b4c142a
home: https://www.yiqisoft.cn
icon: https://www.yiqisoft.cn/images/yiqilogo.png
keywords:
- IoT
- Middleware
- YiEDGE
name: yiedge-base
sources:
- https://github.com/yiqisoft
urls:
- charts/yiedge-base-3.1.0.tgz
version: 3.1.0
generated: "2024-11-11T10:11:56Z"
serverInfo: {}部署 frps 反向代理
编译 frps
- clone 仓库，并编译 docker image
git clone https://github.com/fatedier/frp.git
git checkout v0.61.0
docker build -f dockerfiles/Dockerfile-for-frps . -t hub.yiqisoft.cn/yiqisoft/frps:v0.61.0- push 到私有仓库
docker push hub.yiqisoft.cn/yiqisoft/frps:v0.61.0开发 frps Helm Chart
给出范例，根据自己的需求完成。
- Helm Chart 目录结构
frp-server/
├── charts/ # 可选，依赖的子 Chart
├── templates/ # 存放模板文件
│ ├── deployment.yaml # 部署模板
│ ├── service.yaml # 服务模板
│ ├── configmap.yaml # 配置文件模板
│ ├── ingress.yaml # 可选的 Ingress 配置
├── values.yaml # 默认配置文件
├── Chart.yaml # Chart 描述文件- Chart.yaml
apiVersion: v1
name: frps
version: "0.0.1"
appVersion: frp Server
description: A Helm Chart for frp server
keywords:
- frp
- reverse proxy
home: https://www.yiqisoft.cn
icon: https://www.yiqisoft.cn/images/yiqilogo.png
sources:
- https://github.com/yiqisoft- values.yaml
根据需要修改domain和subDomainHost，还有一些对外开放的端口service.nodePorts.http/frp/dashboard
rreplicaCount: 1
fullnameOverride: "frp-server"
domain: "*.h.yiqisoft.cn" # dynamic domain
image:
repository: hub.yiqisoft.cn/yiqisoft/frps
tag: v0.61.0
pullPolicy: IfNotPresent
service:
type: NodePort
nodePorts:
http: 30080 # vhost http nodePort
frp: 30000 # frp service nodePort
dashboard: 30500 # dashboard nodePort
frpConfig:
bindPort: 7000
vhostHTTPPort: 8888
subDomainHost: "h.yiqisoft.cn"
webServer:
addr: "0.0.0.0"
port: 7500
user: "yiqisoft"
password: "talent123!"
pprofEnable: true
log:
to: "/tmp/frps.log"
level: "info"
maxDays: 7
auth:
token: "yiqisoft"- deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Values.fullnameOverride }}
labels:
app: {{ .Values.fullnameOverride }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: {{ .Values.fullnameOverride }}
template:
metadata:
labels:
app: {{ .Values.fullnameOverride }}
spec:
containers:
- name: frp-server
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
args:
- "-c"
- "/frps.toml"
ports:
- name: frp-vhost
containerPort: {{ .Values.frpConfig.vhostHTTPPort }}
- name: frp-port
containerPort: {{ .Values.frpConfig.bindPort }}
- name: frp-dashboard
containerPort: {{ .Values.frpConfig.webServer.port }}
volumeMounts:
- name: frp-config
mountPath: /frps.toml
subPath: frps.toml
volumes:
- name: frp-config
configMap:
name: {{ .Values.fullnameOverride }}-config- service.yaml
apiVersion: v1
kind: Service
metadata:
name: {{ .Values.fullnameOverride }}-service
labels:
app: {{ .Values.fullnameOverride }}
spec:
type: NodePort
ports:
- name: frp-vhost
port: {{ .Values.frpConfig.vhostHTTPPort }} # 配置的 HTTP 端口
targetPort: frp-vhost
protocol: TCP
nodePort: {{ .Values.service.nodePorts.http }} # 从 values.yaml 中读取对外暴露的端口
- name: frp-port
port: {{ .Values.frpConfig.bindPort }} # 配置的 bindPort
targetPort: frp-port
protocol: TCP
nodePort: {{ .Values.service.nodePorts.frp }} # 从 values.yaml 中读取对外暴露的端口
- name: frp-dashboard
port: {{ .Values.frpConfig.webServer.port }} # 配置的 dashboard 端口
targetPort: frp-dashboard
protocol: TCP
nodePort: {{ .Values.service.nodePorts.dashboard }} # 从 values.yaml 中读取对外暴露的端口
selector:
app: {{ .Values.fullnameOverride }}- ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ .Values.fullnameOverride }}-ingress
labels:
app: {{ .Values.fullnameOverride }}
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: "{{ .Values.domain }}"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: {{ .Values.fullnameOverride }}-service
port:
number: {{ .Values.frpConfig.vhostHTTPPort }}- configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Values.fullnameOverride }}-config
data:
frps.toml: |
bindPort = {{ .Values.frpConfig.bindPort }}
vhostHTTPPort = {{ .Values.frpConfig.vhostHTTPPort }}
subDomainHost = "{{ .Values.frpConfig.subDomainHost }}"
webServer.addr = "{{ .Values.frpConfig.webServer.addr }}"
webServer.port = {{ .Values.frpConfig.webServer.port }}
webServer.user = "{{ .Values.frpConfig.webServer.user }}"
webServer.password = "{{ .Values.frpConfig.webServer.password }}"
webServer.pprofEnable = {{ .Values.frpConfig.webServer.pprofEnable }}
log.to = "{{ .Values.frpConfig.log.to }}"
log.level = "{{ .Values.frpConfig.log.level }}"
log.maxDays = {{ .Values.frpConfig.log.maxDays }}
auth.token = "{{ .Values.frpConfig.auth.token }}"- 打包 Chart
helm package frps-server
Successfully packaged chart and saved it to: /opt/charts/frps-0.0.1.tgz- 上传到私有 ChartMuseum
curl --data-binary "@frps-0.0.1.tgz" http://chart.yiqisoft.cn/api/charts部署到 k3s 管理集群
- 通过手工方式将 frps-0.0.1.tgz 部署到集群
helm install frps-server ./frp-0.0.1.tgz -n frps --createnamespace- 通过 Rancher UI 部署更为简单一些。
验证结果
- 需要在私有 dns 解析地址，例如 dnsmasq
address=/h.yiqisoft.cn/192.168.100.101
address=/*.h.yiqisoft.cn/192.168.100.101- 访问
http://h.yiqisoft.cn:30500可以管理 frps - frpc 连接服务器
tcp://h.yiqisoft.cn:30000，即可访问client-id.h.yiqisoft.cn
管理新的集群（业务集群）
- 某些情况下，可能需要清理 CoreDNS 缓存
kubectl -n kube-system rollout restart deployment coredns