纯二进制安装高可用etcd+kubernetes
前提： 对dns，mTLS等了解
机器信息
| 操作系统 | ip | 作用 |
|---|---|---|
| debian12 | 192.168.0.89 | apiserver+kubelet+etcd |
| debian12 | 192.168.0.181 | apiserver+kubelet+etcd |
| debian12 | 192.168.0.135 | apiserver+kubelet+etcd |
| huawei | 121.37.44.236 | 备用 |
etcd
考虑到更灵活的扩缩容，所以使用服务发现代替静态配置
安装
安装没啥可多说的，就一个二进制文件，github自行下载即可 https://github.com/etcd-io/etcd/releases
配置
为了结构清晰，便于管理，在/data/目录下创建etcd目录及其子目录
root@node-01:/data/etcd# ls
cert data
证书签发
etcd 的 TLS 有两对，一对是 etcd 和 client 端的 TLS 配置。一对是 etcd 之间的 peer 的 TLS 配置。可以共用，也可以分别签，这里共用一个ca证书
签发ca
签发key
root@node-01:/data/etcd/cert# openssl genrsa -out etcd-ca.key 2048
root@node-01:/data/etcd/cert# ls
etcd-ca.key
签发一个10年过期的cert
root@node-01:/data/etcd/cert# openssl req -x509 -new -nodes -key etcd-ca.key -subj "/CN=etcd-ca" -days 3650 -out etcd-ca.crt
创建签发证书的相关内容
cat > ectd-ssl.conf << EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.thisjiayang.work
IP.1 = 127.0.0.1
IP.2 = 192.168.0.89
IP.3 = 192.168.0.181
IP.4 = 192.168.0.135
IP.5 = 121.37.44.236
EOF
签发节点间通讯的证书
签发key
root@node-01:/data/etcd/cert# openssl genrsa -out etcd-peer.key 2048
签发csr
root@node-01:/data/etcd/cert# openssl req -new -key etcd-peer.key -subj "/CN=kube-etcd-peer" -out etcd-peer.csr -config etcd-ssl.conf
签发cert
root@node-01:/data/etcd/cert# openssl x509 -req -in etcd-peer.csr -CA etcd-ca.crt -CAkey etcd-ca.key -CAcreateserial -out etcd-peer.crt -days 3650 -extensions v3_req -extfile etcd-ssl.conf
Certificate request self-signature ok
subject=CN = kube-etcd-peer
签发客户端和节点通讯的证书（如apiserver）
和peer如出一辙
签发key
root@node-01:/data/etcd/cert# openssl genrsa -out etcd-client.key 2048
签发csr
root@node-01:/data/etcd/cert# openssl req -new -key etcd-client.key -subj "/CN=kube-etcd" -out etcd-client.csr -config etcd-ssl.conf
签发cert
root@node-01:/data/etcd/cert# openssl x509 -req -in etcd-client.csr -CA etcd-ca.crt -CAkey etcd-ca.key -CAcreateserial -out etcd-client.crt -days 3650 -extensions v3_req -extfile etcd-ssl.conf
Certificate request self-signature ok
subject=CN = kube-etcd
配置dns
etcd用dns做服务发现，需要用到A和SRV https://etcd.io/docs/v3.5/op-guide/clustering/#dns-discovery
有点坑，如果是ssl的话srv是_etcd-server-ssl 文档都是按照非ssl的
A记录
root@node-01:/data/etcd# dig etcd-01.thisjiayang.work +short
192.168.0.89
SRV
root@node-01:/data/etcd# dig +noall +answer SRV _etcd-client._tcp.thisjiayang.work
_etcd-client._tcp.thisjiayang.work. 300 IN SRV 65535 65535 2379 etcd-01.thisjiayang.work.
dig +noall +answer SRV _etcd-server._tcp.thisjiayang.work
_etcd-server._tcp.thisjiayang.work. 300 IN SRV 65535 65535 2380 etcd-01.thisjiayang.work.
root@node-01:/data/etcd# dig +noall +answer SRV _etcd-server-ssl._tcp.thisjiayang.work
_etcd-server-ssl._tcp.thisjiayang.work. 184 IN SRV 65535 65535 2380 etcd-01.thisjiayang.work.
启动
启动参数就不用设置啥各个节点了，只要把自己的信息填写好就行了，设置下环境变量
export ETCD_NAME="etcd-01"
export ETCD_DISCOVERY_SRV=thisjiayang.work
export ETCD_DATA_DIR="/data/etcd/data/"
export ETCD_LISTEN_PEER_URLS="https://0.0.0.0:2380"
export ETCD_LISTEN_CLIENT_URLS="https://0.0.0.0:2379"
export ETCD_ADVERTISE_CLIENT_URLS="https://etcd-01.thisjiayang.work:2379"
export ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd-01.thisjiayang.work:2380"
export ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
export ETCD_INITIAL_CLUSTER_STATE="new"
export ETCD_PEER_CLIENT_CERT_AUTH=true
export ETCD_CLIENT_CERT_AUTH=true
export ETCD_TRUSTED_CA_FILE=/data/etcd/cert/etcd-ca.crt
export ETCD_CERT_FILE=/data/etcd/cert/etcd-client.crt
export ETCD_KEY_FILE=/data/etcd/cert/etcd-client.key
export ETCD_PEER_CERT_FILE=/data/etcd/cert/etcd-peer.crt
export ETCD_PEER_KEY_FILE=/data/etcd/cert/etcd-peer.key
export ETCD_PEER_TRUSTED_CA_FILE=/data/etcd/cert/etcd-ca.crt
这些环境变量设置好后，直接etcd启动即可，也可以不用环境变量使用 –追加，效果一样
etcd
换成systemd
接下来使用etcdctl客户端查看集群情况
etcdctl --endpoints https://etcd-01.thisjiayang.work:2379 --cacert=/data/etcd/cert/etcd-ca.crt --cert=/data/etcd/cert/etcd-client.crt --key=/data/etcd/cert/etcd-client.key endpoint status --cluster -w table
+---------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://etcd-01.thisjiayang.work:2379 | 90aa4004c004a1bf | 3.5.15 | 20 kB | true | false | 4 | 8 | 8 | |
+---------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
把证书全部拷贝到别的机器上去，然后dns设置下
最后三台srv记录全部设置好
dig SRV _etcd-client._tcp.thisjiayang.work +short
65535 65535 2379 etcd-03.thisjiayang.work.
65535 65535 2379 etcd-01.thisjiayang.work.
65535 65535 2379 etcd-02.thisjiayang.work.
dig SRV _etcd-server-ssl._tcp.thisjiayang.work +short
65535 65535 2380 etcd-01.thisjiayang.work.
65535 65535 2380 etcd-02.thisjiayang.work.
65535 65535 2380 etcd-03.thisjiayang.work.
直接看下etcd集群
etcdctl --endpoints https://etcd-02.thisjiayang.work:2379,https://etcd-01.thisjiayang.work:2379,https://etcd-03.thisjiayang.work:2379 --cacert=/data/etcd/cert/etcd-ca.crt --cert=/data/etcd/cert/etcd-client.crt --key=/data/etcd/cert/etcd-client.key endpoint status -w table
+---------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://etcd-02.thisjiayang.work:2379 | aafa318cf6d2538b | 3.5.15 | 20 kB | false | false | 4 | 40 | 40 | |
| https://etcd-01.thisjiayang.work:2379 | 90aa4004c004a1bf | 3.5.15 | 20 kB | true | false | 4 | 40 | 40 | |
| https://etcd-03.thisjiayang.work:2379 | fb81bfafd9e609b5 | 3.5.15 | 20 kB | false | false | 4 | 40 | 40 | |
+---------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
容器运行时containerd
安装containerd
containerd安装本身还是比较方便的 https://github.com/containerd/containerd/releases 选择带有cri和cni的
直接解压就行
tar xzvf cri-containerd-cni-1.7.21-linux-amd64.tar.gz -C /
使用命令生成一份模板文件
containerd config default > /etc/containerd/config.toml
安装包里自带systemd配置
systemctl daemon-reload && systemctl start containerd && systemctl enable containerd
另外一篇文章已经写过如何通过cloudfare代理镜像仓库 链接
ps: 原生ctr好像不是用cri接口的，xxx.cri可能不好使，所以ctr还是以查看为主
pps: 发现containerd又要换写法了 https://github.com/containerd/containerd/blob/release/1.7/docs/cri/registry.md
我们首先换一下config.toml
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/data/containerd/registry/"
然后看下目录结构
tree /data/containerd/registry/
/data/containerd/registry/
├── docker.io
│ └── hosts.toml
└── registry.k8s.io
└── hosts.toml
看下hosts.toml,server就是原始的镜像仓库 host.代理地址
[host."https://image-proxy.thisjiayang.work"]
capabilities = ["pull", "resolve"]
顺便也改下containerd的cgroup驱动
sed -i "s/SystemdCgroup = false/SystemdCgroup = true/g" /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
重启
systemctl daemon-reload && systemctl restart containerd
k8s
kubectl自动补全,安装bash-completion即可
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
kubectl completion bash > ~/.kube/completion.bash.inc
source '/root/.kube/completion.bash.inc'
source $HOME/.bash_profile
控制层
apiserver
证书
k8s和etcd一样，也是需要双向认证，所以，签发证书也是个体力活，天下文章一大抄，这里随便找个了个kubeadm搭建的apiserver，查看了下apiserver证书里有哪些信息
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 8212578102639259488 (0x71f8f045b1f6ef60)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=kubernetes
Validity
Not Before: Feb 18 14:30:11 2024 GMT
Not After : Feb 17 14:35:11 2025 GMT
Subject: CN=kube-apiserver
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
省略。。。不重要
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Authority Key Identifier:
keyid:39:33:E9:C8:D1:BE:F5:86:DD:0A:7C:33:DA:A9:6D:9F:5F:B5:DA:10
X509v3 Subject Alternative Name:
DNS:apiserver.test, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:192.168.0.1
Signature Algorithm: sha256WithRSAEncryption
省略。。。不重要
commonname写了kube-apiserver，san写了一些内部域名，apiserver的节点ip，一个kubernetes service的ip
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 192d
省流： apiserver的地址写到ssl证书san里去
和上面签发etcd证书差不多步骤，首先签发ca证书
openssl genrsa -out k8s-ca.key 2048
openssl req -x509 -new -nodes -key k8s-ca.key -subj "/CN=kubernetes" -days 3650 -out k8s-ca.crt
写一下证书配置,这里前期尽量规划好，coredns的后缀就是cluster.local，kubernetes这个service在default名称空间
cat > /data/k8s/pki/k8s-ssl.conf << EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = apiserver.thisjiayang.work
DNS.2 = kubernetes
DNS.3 = kubernetes.default
DNS.4 = kubernetes.default.svc
DNS.5 = kubernetes.default.svc.cluster.local
IP.1 = 127.0.0.1
IP.2 = 192.168.0.89
IP.3 = 192.168.0.181
IP.4 = 192.168.0.135
IP.5 = 121.37.44.236
EOF
签发
##key
openssl genrsa -out k8s-apiserver.key 2048
## csr
openssl req -new -key k8s-apiserver.key -subj "/CN=kube-apiserver" -out k8s-apiserver.csr -config k8s-ssl.conf
## crt
openssl x509 -req -in k8s-apiserver.csr -CA k8s-ca.crt -CAkey k8s-ca.key -CAcreateserial -out k8s-apiserver.crt -days 3650 -extensions v3_req -extfile k8s-ssl.conf
好了，有了etcd和apiserver证书，尝试启动,不整那些虚头巴脑的参数，只启动有用的，一眼就能看出干啥的
kube-apiserver \
--allow-privileged=true \
--enable-admission-plugins=NodeRestriction \
--enable-bootstrap-token-auth=true \
--etcd-cafile=/data/etcd/cert/etcd-ca.crt \
--etcd-certfile=/data/etcd/cert/etcd-client.crt \
--etcd-keyfile=/data/etcd/cert/etcd-client.key \
--etcd-servers=https://etcd-02.thisjiayang.work:2379,https://etcd-01.thisjiayang.work:2379,https://etcd-03.thisjiayang.work:2379 \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--secure-port=6443 \
--service-cluster-ip-range=10.96.0.0/16 \
--client-ca-file=/data/k8s/pki/k8s-ca.crt \
--kubelet-client-certificate=/data/k8s/pki/k8s-client.crt \
--kubelet-client-key=/data/k8s/pki/k8s-client.key \
--tls-cert-file=/data/k8s/pki/k8s-apiserver.crt \
--tls-private-key-file=/data/k8s/pki/k8s-apiserver.key \
--service-account-issuer=https://kubernetes.default.svc.cluster.local \
--service-account-signing-key-file=/data/k8s/pki/k8s-apiserver.key \
--service-account-key-file=/data/k8s/pki/k8s-apiserver.key \
--v=5
签发一个客户端证书，原始rbac是通过证书里的commonname认证的，偷懒一下，直接写admin了，之后所有组件也可以用这个客户端证书
kubernetes 将证书中的 CN 字段 作为 User O 字段作为 Group； “O”: “system:masters”, 必须是 system:masters
openssl genrsa -out k8s-client.key 2048
openssl req -new -key k8s-client.key -subj "/CN=admin/O=system:masters" -out k8s-client.csr -config k8s-ssl.conf
openssl x509 -req -in k8s-client.csr -CA k8s-ca.crt -CAkey k8s-ca.key -CAcreateserial -out k8s-client.crt -days 3650 -extensions v3_req -extfile k8s-ssl.conf
验证一下，可以先访问下apiserver的rest接口,如果权限不对证书错误都访问不了的,apiserver地址自己写host或者dns指定下即可
curl --cacert /data/k8s/pki/k8s-ca.crt --cert /data/k8s/pki/k8s-client.crt --key /data/k8s/pki/k8s-client.key https://apiserver.thisjiayang.work:6443
{
"paths": [
"/.well-known/openid-configuration",
"/api",
"/api/v1",
"/apis",
"/apis/",
"/apis/admissionregistration.k8s.io",
"/apis/admissionregistration.k8s.io/v1",
"/apis/apiextensions.k8s.io",
"/apis/apiextensions.k8s.io/v1",
"/apis/apiregistration.k8s.io",
....
}
制作下kubeconfig，kubeconfig由cluster，credentials和context组成
#设置集群
kubectl config set-cluster my-cluster --server=https://apiserver.thisjiayang.work:6443 --certificate-authority=/data/k8s/pki/k8s-ca.crt --embed-certs=true --kubeconfig=/data/k8s/kubeconfig
#设置用户凭证
kubectl config set-credentials admin --client-certificate=/data/k8s/pki/k8s-client.crt --client-key=/data/k8s/pki/k8s-client.key --embed-certs=true --kubeconfig=/data/k8s/kubeconfig
#设置上下文
kubectl config set-context main --cluster=my-cluster --user=admin --kubeconfig=/data/k8s/kubeconfig
# 拷贝给kubectl用
cp /data/k8s/kubeconfig /root/.kube/config
# 设置下上下文
kubectl config use-context main
现在看一下集群，就只有一个svc
kubectl get all -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 19h
controller-manager
之前就提到证书的复用，不用签那么多证书，所以原则上只要一个kubeconfig就行
kube-controller-manager \
--kubeconfig=/data/k8s/kubeconfig \
--service-cluster-ip-range=10.96.0.0/16 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/data/k8s/pki/k8s-ca.crt \
--cluster-signing-key-file=/data/k8s/pki/k8s-ca.key \
--cluster-cidr=10.100.0.0/16 \
--root-ca-file=/data/k8s/pki/k8s-ca.crt \
--service-account-private-key-file=/data/k8s/pki/k8s-ca.key \
--leader-elect=true \
--v=5
scheduler
和cm一样，基本上一条命令就起了
kube-scheduler \
--authentication-kubeconfig=/data/k8s/kubeconfig \
--authorization-kubeconfig=/data/k8s/kubeconfig \
--kubeconfig=/data/k8s/kubeconfig \
--leader-elect=true \
--v=5
我们检查一下控制面
kubectl get --raw='/readyz?verbose'
[+]ping ok
[+]log ok
[+]etcd ok
[+]etcd-readiness ok
[+]informer-sync ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/priority-and-fairness-config-consumer ok
[+]poststarthook/priority-and-fairness-filter ok
[+]poststarthook/storage-object-count-tracker-hook ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/start-service-ip-repair-controllers ok
[+]poststarthook/rbac/bootstrap-roles ok
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/priority-and-fairness-config-producer ok
[+]poststarthook/start-system-namespaces-controller ok
[+]poststarthook/bootstrap-controller ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/start-kube-apiserver-identity-lease-controller ok
[+]poststarthook/start-deprecated-kube-apiserver-identity-lease-garbage-collector ok
[+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok
[+]poststarthook/start-legacy-token-tracking-controller ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
[+]poststarthook/apiservice-openapiv3-controller ok
[+]poststarthook/apiservice-discovery-controller ok
[+]shutdown ok
readyz check passed
kubectl get componentstatuses
# 1.19废除了
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy ok
节点(不部署proxy，会用cilium替代)
kubelet
注意一下，这个集群没用bootstrap方式动态签发证书，所以apiserver侧启动的时候一定要加上kubelet的证书地址 https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/
创建kubelet配置文件
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 0s
enabled: true
x509:
clientCAFile: /data/k8s/pki/k8s-ca.crt
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 0s
cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: ""
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMaximumGCAge: 0s
imageMinimumGCAge: 0s
logging:
flushFrequency: 0
options:
json:
infoBufferSize: "0"
verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s
启动
kubelet --kubeconfig=/data/k8s/kubeconfig --config=/data/k8s/kubelet.conf --hostname-override=$HOSTNAME
好了查看节点
kubectl get no
NAME STATUS ROLES AGE VERSION
node-01 Ready <none> 50s v1.28.13
所有节点一样的操作，都起来了
kubectl get no -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node-01 Ready <none> 2d17h v1.28.13 192.168.0.89 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-22-amd64 containerd://1.7.21
node-02 Ready <none> 2d17h v1.28.13 192.168.0.181 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-22-amd64 containerd://1.7.21
node-03 Ready <none> 21m v1.28.13 192.168.0.135 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-22-amd64 containerd://1.7.21
运行
这边创建一个1个pod的dep
apiVersion: apps/v1
kind: Deployment
metadata:
name: busybox-deployment
spec:
replicas: 1
selector:
matchLabels:
app: busybox
template:
metadata:
labels:
app: busybox
spec:
containers:
- name: busybox
image: busybox
command: ["/bin/sh", "-c", "while true; do sleep 3600; done"]
resources:
requests:
cpu: "50m"
memory: "100Mi"
limits:
cpu: "100m"
memory: "200Mi"
pod ip并不是我们规定的cidr，是因为分配ip本质还是网络插件干的
kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox-deployment-55785fcdb8-2xdrz 1/1 Running 0 6m54s 10.88.0.36 node-03 <none> <none>
网络
安装cilium插件
https://github.com/cilium/charts
helm一把梭哈
helm install cilium \
--namespace kube-system \
--set operator.replicas=1 \
--set nodeinit.enabled=true \
--set nodeinit.restartPods=true \
--set externalIPs.enabled=true \
--set nodePort.enabled=true \
--set hostPort.enabled=true \
--set bpf.masquerade=true \
--set bpf.clockProbe=true \
--set bpf.waitForMount=true \
--set bpf.preallocateMaps=true \
--set bpf.tproxy=true \
--set bpf.hostRouting=true \
--set autoDirectNodeRoutes=false \
--set localRedirectPolicy=true \
--set enableCiliumEndpointSlice=true \
--set enableK8sEndpointSlice=true \
--set wellKnownIdentities.enabled=true \
--set sockops.enabled=true \
--set ipam.mode="cluster-pool" \
--set ipam.operator.clusterPoolIPv4PodCIDRList=10.100.0.0/16 \
--set ipv4NativeRoutingCIDR=10.100.0.0/16 \
--set nodePort.directRoutingDevice=eth0 \
--set devices=eth0 \
--set hubble.enabled=true \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true \
--set endpointRoutes.enabled=true \
--set kubeProxyReplacement=true \
--set hostServices.enabled=true \
--set k8sServiceHost=192.168.0.89 \
--set k8sServicePort=6443 .
这时候重启下刚才的busybox，看到ip已经正确了
kubectl get po -o wide
###
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox-deployment-55785fcdb8-fnrkd 1/1 Running 0 19h 10.100.0.166 node-01 <none> <none>
同时我们ping一下
ping 10.100.0.166
PING 10.100.0.166 (10.100.0.166) 56(84) bytes of data.
64 bytes from 10.100.0.166: icmp_seq=1 ttl=63 time=0.498 ms
64 bytes from 10.100.0.166: icmp_seq=2 ttl=63 time=0.414 ms
64 bytes from 10.100.0.166: icmp_seq=3 ttl=63 time=0.484 ms
我们再测试下service看看网络情况
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
resources:
limits:
cpu: "100m"
memory: "200Mi"
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
type: ClusterIP
selector:
app: nginx
ports:
- port: 80
protocol: TCP
直接curl pod的ip
kubectl get po -owide|grep nginx
nginx-deployment-7487687ff6-q4fn4 1/1 Running 0 48s 10.100.1.104 node-02 <none> <none>
curl 10.100.1.104
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
再curl一下service看看
kubectl get svc|grep nginx
nginx-service ClusterIP 10.96.133.229 <none> 80/TCP 2m
curl 10.96.133.229
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
随便进一个cilium容器，看下svc列表
kubectl exec -it -n kube-system cilium-24gbr -- cilium service list
ID Frontend Service Type Backend
1 10.96.0.1:443 ClusterIP 1 => 192.168.0.89:6443 (active)
2 => 192.168.0.181:6443 (active)
3 => 192.168.0.135:6443 (active)
2 10.96.81.122:443 ClusterIP 1 => 192.168.0.89:4244 (active)
3 10.96.117.32:80 ClusterIP
4 10.96.11.223:80 ClusterIP 1 => 10.100.2.195:8081 (active)
5 10.96.133.229:80 ClusterIP 1 => 10.100.1.104:80 (active)
coredns
同样的用helm可以一把直接起了,https://github.com/coredns/helm/tree/master 不过要注意下
- 如果要指定apiserver的话，corefile需要变动
- 自签证书或者kubeconfig需要挂载
所以生成corefile的模板要动一下
service:
clusterIP: "10.96.0.10"
# clusterIPs: []
# loadBalancerIP: ""
# externalIPs: []
# externalTrafficPolicy: ""
# ipFamilyPolicy: ""
# The name of the Service
# If not set, a name is generated using the fullname template
name: "kube-dns"
annotations: {}
# Pod selector
selector: {}
............................
podSecurityContext:
runAsUser: 0
fsGroup: 0
........................
- name: kubernetes
parameters: cluster.local in-addr.arpa ip6.arpa
configBlock: |-
kubeconfig /config main
endpoint https://192.168.0.89:6443
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
................
extraVolumes:
- name: kube-config-volume
hostPath:
path: /root/.kube/config
type: File
extraVolumeMounts:
- name: kube-config-volume
mountPath: /config
...
直接测试下
kubectl run -it --rm --restart=Never --image=infoblox/dnstools:latest dnstools
host kubernetes.default.svc.cluster.local
kubernetes.default.svc.cluster.local has address 10.96.0.1