高可用Kubernetes集群-12. 部署kubernetes-ingress

参考文档:

  1. Github:https://github.com/kubernetes/ingress-nginx
  2. Kubernetes ingress:https://kubernetes.io/docs/concepts/services-networking/ingress/
  3. Ingress:https://mritd.me/2017/03/04/how-to-use-nginx-ingress/
  4. 配置示例:https://www.cnblogs.com/iiiiher/p/8006801.html
  5. Github示例:https://github.com/kubernetes/ingress-nginx/tree/master/deploy
  6. Traefik示例:https://github.com/containous/traefik

Ingress是对外服务到集群内的Service之间规则的集合:允许进入集群的请求被转发至集群内的Service。

Ingress能把Service配置成外网能够访问的url,流量负载均衡,终止ssl,提供基于域名访问的虚拟主机等,用户通过访问url访问Service。

Ingress-controller负责处理所有Ingress的请求流量,它通常是一个负载均衡器。

一.环境

1.基础环境

组件

版本

Remark

kubernetes

v1.9.2

Ingress-nginx

0.11.0

default-backend

1.4

2.原理

高可用Kubernetes集群-12. 部署kubernetes-ingress
  1. ingress策略本质是转发的规则;
  2. ingress-controller基于ingress策略将客户端的请求转发到service对应的后端endpoint,即Pod上;实现了为所有后端service提供统一入口,基于不同的http url向后转发负载分发规则,并可以灵活设置7层的负载分发策略的功能;一般由nginx实现。

二.部署ingress-nginx

1.准备images

kubernetes部署Pod服务时,为避免部署时发生pull镜像超时的问题,建议提前将相关镜像pull到相关所有节点(实验),或搭建本地镜像系统。

  1. 基础环境已做了镜像加速,可参考:http://www.cnblogs.com/netonline/p/7420188.html
  2. 需要从gcr.io pull的镜像,已利用Docker Hub的"Create Auto-Build GitHub"功能(Docker Hub利用GitHub上的Dockerfile文件build镜像),在个人的Docker Hub build成功,可直接pull到本地使用。

# ingress-controller默认的backend,用于在客户端访问的url地址不存在时,能够返回一个正确的404应答

[root@kubenode1 ~]# docker pull netonline/defaultbackend:1.4

# ingress-nginx

[root@kubenode1 ~]# docker pull netonline/nginx-ingress-controller:0.11.0

2.下载ingress-nginx相关yaml范本

# 相关的yaml文件可在1个或多个master节点下载后修改

[root@kubenode1 ~]# mkdir -p /usr/local/src/yaml/ingress

[root@kubenode1 ~]# cd /usr/local/src/yaml/ingress/

#下载链接: https://github.com/kubernetes/ingress-nginx/tree/master/deploy

# namespace

[root@kubenode1 ingress]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/namespace.yaml

# configmap,此验证未使用

[root@kubenode1 ingress]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/configmap.yaml

# tcp-service-configmap

[root@kubenode1 ingress]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/tcp-services-configmap.yaml

# udp-service-configmap,此验证未使用

[root@kubenode1 ingress]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/udp-services-configmap.yaml

# rbac

[root@kubenode1 ingress]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/rbac.yaml

# default-backend

[root@kubenode1 ingress]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml

# with-rbac

[root@kubenode1 ingress]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/with-rbac.yaml

# without-rbac,此验证未使用

[root@kubenode1 ingress]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/without-rbac.yaml

# patch,此验证未使用

[root@kubenode1 ingress]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/publish-service-patch.yaml

3.namespace.yaml

# ingress-nginx github文档中将Namespace,ConfingMap,ServiceAccount,Deployment,default-backend,xxx-services-configmap等服务的yaml配置文件独立保存,以下章节分别针对各yaml文件修改,红色加粗字体即修改部分;

# 对Pod yaml文件的编写这里不做展开,可另参考资料,如《Kubernetes权威指南》;

# 修改后的ingress-nginx相关yaml文件请见:https://github.com/Netonline2016/kubernetes/tree/master/addons/ingress

# namespace.yaml不做修改,创建1个独立的namespace

[root@kubenode1 ingress]# cat namespace.yaml

apiVersion: v1

kind: Namespace

metadata:

name: ingress-nginx

4.tcp-services-configmap.yaml

# tcp-services-configmap.yaml不做修改

[root@kubenode1 ingress]# cat tcp-services-configmap.yaml

kind: ConfigMap

apiVersion: v1

metadata:

name: tcp-services

namespace: ingress-nginx

5.rbac.yaml

# ingress-controller需要监听apiserver,获取ingress定义,通过rbac授权;

# rbac.yaml文件不用修改

[root@kubenode1 ingress]# cat rbac.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

name: nginx-ingress-serviceaccount

namespace: ingress-nginx

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: ClusterRole

metadata:

name: nginx-ingress-clusterrole

rules:

- apiGroups:

- ""

resources:

- configmaps

- endpoints

- nodes

- pods

- secrets

verbs:

- list

- watch

- apiGroups:

- ""

resources:

- nodes

verbs:

- get

- apiGroups:

- ""

resources:

- services

verbs:

- get

- list

- watch

- apiGroups:

- "extensions"

resources:

- ingresses

verbs:

- get

- list

- watch

- apiGroups:

- ""

resources:

- events

verbs:

- create

- patch

- apiGroups:

- "extensions"

resources:

- ingresses/status

verbs:

- update

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: Role

metadata:

name: nginx-ingress-role

namespace: ingress-nginx

rules:

- apiGroups:

- ""

resources:

- configmaps

- pods

- secrets

- namespaces

verbs:

- get

- apiGroups:

- ""

resources:

- configmaps

resourceNames:

# Defaults to "<election-id>-<ingress-class>"

# Here: "<ingress-controller-leader>-<nginx>"

# This has to be adapted if you change either parameter

# when launching the nginx-ingress-controller.

- "ingress-controller-leader-nginx"

verbs:

- get

- update

- apiGroups:

- ""

resources:

- configmaps

verbs:

- create

- apiGroups:

- ""

resources:

- endpoints

verbs:

- get

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: RoleBinding

metadata:

name: nginx-ingress-role-nisa-binding

namespace: ingress-nginx

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: Role

name: nginx-ingress-role

subjects:

- kind: ServiceAccount

name: nginx-ingress-serviceaccount

namespace: ingress-nginx

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: ClusterRoleBinding

metadata:

name: nginx-ingress-clusterrole-nisa-binding

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

name: nginx-ingress-clusterrole

subjects:

- kind: ServiceAccount

name: nginx-ingress-serviceaccount

namespace: ingress-nginx

6.default-backend.yaml

# 提供1个默认的后台404错误页面与/healthz的健康检查页;

# 含1个Deployment与1个service;

# 只需要修改Pod启动调用的image文件名

[root@kubenode1 ingress]# vim default-backend.yaml

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

name: default-http-backend

labels:

app: default-http-backend

namespace: ingress-nginx

spec:

replicas: 1

selector:

matchLabels:

app: default-http-backend

template:

metadata:

labels:

app: default-http-backend

spec:

terminationGracePeriodSeconds: 60

containers:

- name: default-http-backend

# Any image is permissible as long as:

# 1. It serves a 404 page at /

# 2. It serves 200 on a /healthz endpoint

image: netonline/defaultbackend:1.4

livenessProbe:

httpGet:

path: /healthz

port: 8080

scheme: HTTP

initialDelaySeconds: 30

timeoutSeconds: 5

ports:

- containerPort: 8080

resources:

limits:

cpu: 10m

memory: 20Mi

requests:

cpu: 10m

memory: 20Mi

---

apiVersion: v1

kind: Service

metadata:

name: default-http-backend

namespace: ingress-nginx

labels:

app: default-http-backend

spec:

ports:

- port: 80

targetPort: 8080

selector:

app: default-http-backend

7.with-rbac.yaml

Ingress-Controller以Pod的形式运行,监控apiserver的/ingress接口后端的backend services,,如果service发生变化,则Ingress-Controller自动更新转发规则。

基本逻辑如下:

  1. 监听apiserver,获取全部ingress定义;
  2. 基于ingress定义,生成nginx的配置文件/etc/nginx/nginx.conf;
  3. 执行nginx -s reload,重新加载nginx.conf配置文件的内容。

# without-rbac.yaml与with-rbac.yaml的区别是没有调用rabc.yaml中定义的ServiceAccount: nginx-ingress-serviceaccount,这里访问apiserve需要认证;

# github文档给的kind: Deployment,replicas: 1,即在1个节点上启动1个ingress-nginx controller Pod,外部流量访问该节点,由该节点负载后端services;但Pod本身会因为故障而转移,节点ip会变更,DaemonSet会在多个节点(可以利用Pod的亲和性将指定Pod部署到指点节点)生成ingress-nginx controller Pod,则客户端可以访问任意节点;或者在前端部署负载,使用vip访问后端3个节点;

# hostNetwork: true,暴露ingress-nginx controller的相关业务端口到主机;

# 验证中暂时用不到的服务不启用,则相应的ingress-controller中的对应参数也注释不用

[root@kubenode1 ingress]# vim with-rbac.yaml

apiVersion: extensions/v1beta1

# kind: Deployment

# 变更kind

kind: DaemonSet

metadata:

name: nginx-ingress-controller

namespace: ingress-nginx

spec:

# 已变更kind,注释副本数

# replicas: 1

selector:

matchLabels:

app: ingress-nginx

template:

metadata:

labels:

app: ingress-nginx

annotations:

prometheus.io/port: '10254'

prometheus.io/scrape: 'true'

spec:

serviceAccountName: nginx-ingress-serviceaccount

# 暴露主机端口

hostNetwork: true

containers:

- name: nginx-ingress-controller

# 变更调用image名

image: netonline/nginx-ingress-controller:0.11.0

args:

- /nginx-ingress-controller

- --default-backend-service=$(POD_NAMESPACE)/default-http-backend

# 此验证未使用ConfigMap,注释

# - --configmap=$(POD_NAMESPACE)/nginx-configuration

- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services

# 此验证未使用udp-service-ConfigMap,注释

# - --udp-services-configmap=$(POD_NAMESPACE)/udp-services

- --annotations-prefix=nginx.ingress.kubernetes.io

env:

- name: POD_NAME

valueFrom:

fieldRef:

fieldPath: metadata.name

- name: POD_NAMESPACE

valueFrom:

fieldRef:

fieldPath: metadata.namespace

ports:

- name: http

containerPort: 80

# 暴露的主机端口

hostPort: 80

- name: https

containerPort: 443

hostPort: 443

livenessProbe:

failureThreshold: 3

httpGet:

path: /healthz

port: 10254

scheme: HTTP

initialDelaySeconds: 10

periodSeconds: 10

successThreshold: 1

timeoutSeconds: 1

readinessProbe:

failureThreshold: 3

httpGet:

path: /healthz

port: 10254

scheme: HTTP

periodSeconds: 10

successThreshold: 1

timeoutSeconds: 1

8.启动ingress-nginx

[root@kubenode1 ingress]# kubectl create -f namespace.yaml

[root@kubenode1 ingress]# kubectl create -f tcp-services-configmap.yaml

[root@kubenode1 ingress]# kubectl create -f rbac.yaml

[root@kubenode1 ingress]# kubectl create -f default-backend.yaml

[root@kubenode1 ingress]# kubectl create -f with-rbac.yaml

高可用Kubernetes集群-12. 部署kubernetes-ingress

9.设置iptables

# 3台master节点均设置,with-rbac.yaml/without-rbac.yaml会启用"hostNetwork",并且开放tcp80,443,18080(/nginx-status)端口;

# 因为这里docker服务已启动,采用直接在input链追加开放端口的方式;

# 建议在/etc/sysconfig/iptables配置文件中将相应端口打开;

# 如果采用"service iptables save"命令会将当前已有iptables规则(含docker服务相关规则)全部写入配置文件,慎用

[root@kubenode1 ~]# iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

[root@kubenode1 ~]# iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT

[root@kubenode1 ~]# iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 18080 -j ACCEPT

10.验证

# 1个default-htt-backend与3个ingress-nginx Pod已运行

[root@kubenode1 ~]# kubectl get pods -n ingress-nginx -o wide

高可用Kubernetes集群-12. 部署kubernetes-ingress

# 节点本机相应端口已被使用

[root@kubenode1 ~]# netstat -tunlp | grep nginx

高可用Kubernetes集群-12. 部署kubernetes-ingress

# 访问任意节点的80端口,返回404页面,ingress-nginx controller与default-http-backend生效

[root@kubenode1 ~]# curl http://172.30.200.21

[root@kubenode1 ~]# curl http://172.30.200.22

[root@kubenode1 ~]# curl http://172.30.200.23

高可用Kubernetes集群-12. 部署kubernetes-ingress

三.部署ingress

1.部署后端服务

# 在相关所有节点下载后端服务镜像,避免镜像下载超时;

# 后端服务使用nginx

[root@kubenode1 ~]# docker pull nginx

# 部署后端服务,这里将后端服务Pod(Deployment的形式下发Pod,直接创建的Pod与Service关联有问题,通过"kubectl get endpoints"可查看到Service的"ENDPOINTS"列关联不上后端Pod,原因未查明)与Service放在1个yaml文件中

[root@kubenode1 ~]# cd /usr/local/src/yaml/ingress/

[root@kubenode1 ingress]# touch nginx-svc.yaml

[root@kubenode1 ingress]# vim nginx-svc.yaml

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

name: nginx-01

spec:

replicas: 1

template:

metadata:

labels:

name: nginx-01

spec:

containers:

- name: nginx-01

image: nginx:latest

ports:

- containerPort: 80

---

apiVersion: v1

kind: Service

metadata:

# Service的全局唯一名称

name: nginx-svc

spec:

ports:

# Service服务监听的端口号

- port: 80

# 后端服务Pod提供的端口号

targetPort: 80

# 端口名称(非必须)

name: http

# Service关联定义了相应标签的Pod

selector:

name: nginx-01

# 启动后端服务

[root@kubenode1 ingress]# kubectl create -f nginx-svc.yaml

2.部署ingress

[root@kubenode1 ingress]# touch nginx-svc-ingress.yaml

[root@kubenode1 ingress]# vim nginx-svc-ingress.yaml

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: nginx-svc-ingress

spec:

rules:

# 主机域名,需要在本地绑定节点ip

- host: nginx-svc.me

http:

# 如果paths下有具体的路径,如/demo,需要与后端提供真实服务的path一致,这里即nginx下需要有/demo路径

paths:

- backend:

# 后端服务名

serviceName: nginx-svc

# 后端服务监听端口,区别于提供真实服务的容器监听端口

servicePort: 80

# 下发ingress

[root@kubenode1 ingress]# kubectl create -f nginx-svc-ingress.yaml

3.验证

# Service关联到后端Pod;

# 为主机"nginx-svc.me"定制了ingress策略;

# 理论上ingress的"ADDRESS"列显示ingress-nginx-controller Pod的ip地址则表示nginx已设置好后端Service的Endpoint,ingress此时可以正常工作;为空则有需要排错;但可能这里有个bug(在1.8.x与1.9.x版本都有此问题),在显示未空的状态下,ingress依然生效

[root@kubenode1 ingress]# cd ~

[root@kubenode1 ~]# kubectl get endpoints nginx-svc -o wide

[root@kubenode1 ~]# kubectl get ingress -o wide

高可用Kubernetes集群-12. 部署kubernetes-ingress

在本地浏览器访问host主机(注意提前绑定域名):http://nginx-svc.me

高可用Kubernetes集群-12. 部署kubernetes-ingress

# 或者采用--resolve参数模拟dns解析,目标地址为域名

[root@kubenode1 ~]# curl --resolve nginx-svc.me:80:172.30.200.21 http://nginx-svc.me

# 或者采用-H参数设置http头中需要访问的域名,目标地址为ip地址

[root@kubenode1 ~]# curl -H 'Host:nginx-svc.me' http://172.30.200.22

高可用Kubernetes集群-12. 部署kubernetes-ingress

4.ingress策略配置技巧

1)转发到单个后端服务

# 所有访问被转发到后端唯一的Service,此时可不定义rule

# 关注红色加粗字体

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: ingress

spec:

backend:

serviceName: nginx-svc

servicePort: 80

2)同一域名,不同的url路径被转发到不同的服务

# 相同域名下两个不同的路径对应不同的服务;

# 注意如果paths下有具体的路径,如/web,/api等,需要与后端提供真实服务的path一致,这里即nginx服务器下需要有/web,/api等路径

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: ingress

spec:

rules:

- host: nginx-svc.me

http:

paths:

- path: /web

backend:

serviceName: nginx-svc-web

servicePort: 80

- path: /api

backend:

serviceName: nginx-svc-api

servicePort: 8081

3)不同域名,被转发到不同的服务

# 不同域名对应不同的服务

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: ingress

spec:

rules:

- host: nginx-svc.me

http:

paths:

- backend:

serviceName: nginx-svc

servicePort: 80

- host: apache-svc.me

http:

paths:

- backend:

serviceName: apache-svc

servicePort: 80

4)不使用域名转发

# 使用无域名ingress规则时,默认禁用http,强制启用https;

# 此时客户端访问如下路径:curl http://172.30.200.21/demo 会返回301错误,但使用https访问则可以成功:curl -k https://172.30.200.21/demo ;

# 可以在ingress定义的metadata设置annotation "ingress.kubernetes.io/ssl-redirect=false"关闭强制启用https的设置,如下绿色加粗字体

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: ingress

# annotations:

# ingress.kubernetes.io/ssl-redirect: "false"

spec:

rules:

- http:

paths:

- path: /demo

backend:

serviceName: nginx-svc-demo

servicePort: 8080

四.ingress的tls设置

对ingress中的域名进行tls安全证书的设置步骤如下:

  1. 创建自签名的秘钥与ssl证书;
  2. 将证书保存到kubernetes集群的1个Secret资源对象上;
  3. 设置Secret资源对象到ingress中。

根据网站域名是1个还是多个,前两步的操作稍有不同,第3步操作相同,下面以多域名的操作为例:

1.生成ca证书

[root@kubenode1 ~]# mkdir -p /etc/kubernetes/ingress

[root@kubenode1 ~]# cd /etc/kubernetes/ingress/

[root@kubenode1 ingress]# openssl genrsa -out ca.key 2048

[root@kubenode1 ingress]# openssl req -x509 -new -nodes -key ca.key -days 3560 -out ca.crt -subj "/CN=ingress-ca"

高可用Kubernetes集群-12. 部署kubernetes-ingress

2.修改openssl.cnf文件

# 对于多域名,生成ssl证书需要使用额外的x509v3配置文件辅助;

# 在[alt_names]字段中设置多域名

[root@kubenode1 ingress]# cp /etc/pki/tls/openssl.cnf .

[root@kubenode1 ingress]# vim openssl.cnf

[ req ]

# 第126行,取消注释

req_extensions = v3_req # The extensions to add to a certificate request

[ v3_req ]

# Extensions to add to a certificate request

basicConstraints = CA:FALSE

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

# 第224行之后,新增部分

subjectAltName = @alt_names

[alt_names]

DNS.1 = nginx01-svc-tls.me

DNS.2 = nginx02-svc-tls.me

3.生成ingress ssl证书

# 基于修改的openssl.cnf与ca证书生成ingress ssl证书

# 生成秘钥

[root@kubenode1 ingress]# openssl genrsa -out ingress.key 2048

# 生成csr文件

[root@kubenode1 ingress]# openssl req -new -key ingress.key -out ingress.csr -subj "/CN=nginx-svc-tls" -config openssl.cnf

# 生成证书

[root@kubenode1 ingress]# openssl x509 -req -in ingress.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out ingress.crt -days 3650 -extensions v3_req -extfile openssl.cnf

高可用Kubernetes集群-12. 部署kubernetes-ingress

4.生成Secret资源对象

Secret对象的主要作用是保管私密数据,如:密码,OAuth Tokens,ssh Keys等信息。将私密信息存放在Secret对象中,比直接放在Pod或者Docker image中更安全,更便于使用与分发。

Secret对象创建完成之后,可通过3种方式调用:

  1. 在创建Pod时,通过为Pod指定Service Account来自动使用;
  2. 通过挂载Secret到Pod来使用;
  3. Docker image下载时使用,通过指定Pod的spc.ImagePullSecrets来引用。

# 编辑secret- ingress.yaml文件,将ingress.key与ingress.crt的内如复制到yaml文件中;

# 注意1:Secret的"data"域的各子域的值必须为BASE64编码;

# 注意2:复制key与crt的内容时去掉换行符,变成一行

[root@kubenode1 ingress]# cd /usr/local/src/yaml/ingress/

[root@kubenode1 ingress]# touch secret-ingress.yaml

[root@kubenode1 ingress]# vim secret-ingress.yaml

apiVersion: v1

kind: Secret

metadata:

name: secret-ingress

# 1.8.x之后使用kubernetes.io/tls替换Opaque

type: kubernetes.io/tls

data:

tls.crt: MIIC/TCCAeWgAwIBAgIJALGRacBg2fWIMA0GCSqGSIb3DQEBCwUAMBUxEzARBgNVBAMMCmluZ3Jlc3MtY2EwHhcNMTgwMjI4MDM0NDI1WhcNMjgwMjI2MDM0NDI1WjAYMRYwFAYDVQQDDA1uZ2lueC1zdmMtdGxzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuZvMBYF104JPtMZFFUxCpGGODFG4rGffN1FFC98CGt99QAwVMfABGDMU8zfa21twxON1v3WK8HdJH5KRdLOIRQnhuMHsC174sb/+FuOa0GhStgmNX0f2jGETuImPQ82faXACnUkUYuvYG5odbY+tS+LQBtIormpxWRlNNTVzT3jFD6JECVZzpMyCJutkwxJC083PS1VE9ki+7mgpPWbb9BqT0Tn672x4cHI8LZ5snr1fpR8I0sqADXY+KpFQeh7UJsWZjfr00wDBsg76aF3TNK+pecXnBNYPZ6o7sOGXvagAxU58xjjz75TwMQ7NnqF584fshvQLnzeTGhDbXx4GHQIDAQABo00wSzAJBgNVHRMEAjAAMAsGA1UdDwQEAwIF4DAxBgNVHREEKjAoghJuZ2lueDAxLXN2Yy10bHMubWWCEm5naW54MDItc3ZjLXRscy5tZTANBgkqhkiG9w0BAQsFAAOCAQEAaUQH21wct78wW1tAz1+3j9SGLLk7kd06PcnYH1pBGW3wlMFiusMOUdVfm2aPwkX7VFOfeo6LYtILisQ9+wJraQcGd31H5M/ILSH2bhd739CcySqm3aEYHplQCfepsRRcINp82N3GLjcT6sCeuvoC4l+rUDIKMEPt7Skwj1HjCj/NSpHguzmtmRG4PCvv/3nCrcntGLKxsKBD85llpRlT9/Q+9eMmQz7YRUjUEr/3cSmBfRjcBQhB7yXZyDLRbObAc1BMgxBRX/oexNIHCMNdRFSP6wAtlyajytOhcixBMu3RQ1g0lrFaT9vQWPXs6HV8xC6VvfhdSQN8B1T65klNNg==

tls.key: MIIEpAIBAAKCAQEAuZvMBYF104JPtMZFFUxCpGGODFG4rGffN1FFC98CGt99QAwVMfABGDMU8zfa21twxON1v3WK8HdJH5KRdLOIRQnhuMHsC174sb/+FuOa0GhStgmNX0f2jGETuImPQ82faXACnUkUYuvYG5odbY+tS+LQBtIormpxWRlNNTVzT3jFD6JECVZzpMyCJutkwxJC083PS1VE9ki+7mgpPWbb9BqT0Tn672x4cHI8LZ5snr1fpR8I0sqADXY+KpFQeh7UJsWZjfr00wDBsg76aF3TNK+pecXnBNYPZ6o7sOGXvagAxU58xjjz75TwMQ7NnqF584fshvQLnzeTGhDbXx4GHQIDAQABAoIBAFMGCI3R6eWRXZvsMEyljw2+gW6rQ2MDF4rD9JGp0GQ64ei7PuPWinbLqqxcqK4ESf4YDLx2lI6ZnQDda+j6wZK4J9qgC7jOY4oG6l5MsxxT/eNlhHJBW1xRtCOQjJ/0o0DjlJfMb60L99/o4Q73/Ll8HDdg3EegX1FOiwWpAgpipA+WyosAtrfR8DjOAVMavlhkCejmgupWU7syuVmVQ0Dz/z9zPESI1b6pHO0Js4Keb8vnUHPLNcq1HCdCMK+wrdUaW2YmuAr9uoF7Wqvp7MCog//cQX93mijJzW8GFPrSt2y4NHN6AnUw6PE3aoMgF1my7O1xLwOjCQz+eW8voyECgYEA8luy6iEDYkfq+tkxA9kl3CgXVk5WgiE/4mVaEjOIT2llgM+8K3TAn8EGATk5s79phn/MRfqi8YQ13Z9dzhp3R/ARynD+/TVRzMHe5830ysBScHaW4vxvPXEn2uBtB8TC8goxmoIu9My5H746ceyY2xBEn8HA0XZ7pQTrCRimcmUCgYEAxA5a3g/Ni/uwTUAsQJNUPyvjcYxq+E3S2VNsYZiOiogKqXeE0QtasNMh1L7Wv9aan5Xca7eKbHP4fZFxLif/YrwwcmktIX3u5vkGyq2VCAw5V8iGD3vdbDJvAc2+YVBoeWf4w4eDST2Ir6xrM3WCtXR35EM0Jhw+8PAdytIKrVkCgYAqEK5yIr7CnTb0ySPPxi3jE3ZRfZFYTssW0X6bsCQVnHaIsAW6CS6xy7/uEG+qeiuns6DR+Jm1j7wFtnaComdXrhx4ZbpsWofTIUc+NqopUs48ROkVhrkMEgrX26Iw+f7YIdrQNY5O4QW0s8DTKzywsRcoH2oHMShu0Pa2gnfJXQKBgQClzLn9t4GNk0EKY23JAo8piTUkbqp76Fyam5k5g+lvsBLMNB4nJyIADd07bFRyEcvbj8HDeolepEiN8HS1ou+wERQrfVTEURq7S/f5aQhysNvBp/vvlkGv4YrNDLCm3Xgsy8etm6lkQ9yXLAnQj90FFUTazhaI8DQuT/Hx9uU+qQKBgQCBEpc98YikgYmZk/6kyzUP3l+MIj5i3UK/7ZG3QOpTAeTbzBQQX0s31b2Lf9M+SN2+2XJb/0OUr3RKKkuf5KgedMll7hNaEaFu9z5qPepFUlKWZz2MkIRSljecbSJ8ZfGz2wCUhQoW8KLQY9ftEaz+27eEJ0FxHhuhe5+yQMpkKA==

# 生成Secret资源对象

[root@kubenode1 ingress]# kubectl create -f secret-ingress.yaml

# 以上编辑yaml文件,使用"kubectl create"命令生成Secret对象在步骤上更清晰;

# 但可以利用"kubectl create secret tls"命令直接创建Secret对象

[root@kubenode1 ingress]# kubectl create secret tls secret-ingress --key /etc/kubernetes/ingress/ingress.key --cert /etc/kubernetes/ingress/ingress.crt

高可用Kubernetes集群-12. 部署kubernetes-ingress

5.创建后端服务

# 编辑后端服务nginx01-svc-tls.yaml

[root@kubenode1 ingress]# touch nginx01-svc-tls.yaml

[root@kubenode1 ingress]# vim nginx01-svc-tls.yaml

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

name: nginx01-tls

spec:

replicas: 1

template:

metadata:

labels:

name: nginx01-tls

spec:

containers:

- name: nginx01-tls

image: nginx:latest

ports:

- containerPort: 80

---

apiVersion: v1

kind: Service

metadata:

name: nginx01-svc-tls

spec:

ports:

# Service服务监听的端口号

- port: 443

# 后端提供真实服务的Pod提供的端口号

targetPort: 80

name: https

selector:

name: nginx01-tls

# 编辑后端服务nginx02-svc-tls.yaml

[root@kubenode1 ingress]# cp nginx01-svc-tls.yaml nginx02-svc-tls.yaml

[root@kubenode1 ingress]# sed -i 's|nginx01|nginx02|g' nginx02-svc-tls.yaml

# 生成后端服务

[root@kubenode1 ingress]# kubectl create -f nginx01-svc-tls.yaml

[root@kubenode1 ingress]# kubectl create -f nginx02-svc-tls.yaml

高可用Kubernetes集群-12. 部署kubernetes-ingress

# 修改提供后端服务的nginx容器的html文件;

# 通过"kubectl exec -ti <pod-name> -c <container-name> /bin/bash"进入容器修改;pod-name可通过命令"kubectl get pods -o wide"获取;container-name即yaml文件中定义的名字

[root@kubenode1 ingress]# kubectl get pods -o wide

高可用Kubernetes集群-12. 部署kubernetes-ingress

# nginx官方容器的index.html文件在/usr/share/nginx/html/目录下

[root@kubenode1 ingress]# kubectl exec -ti nginx01-tls-59fbf6696c-qfq4k -c nginx01-tls /bin/bash

root@nginx01-tls-59fbf6696c-qfq4k:/# echo "<h1>Welcome to test site nginx01-svc-tls</h1>" > /usr/share/nginx/html/index.html

root@nginx01-tls-59fbf6696c-qfq4k:/# cat /usr/share/nginx/html/index.html

root@nginx01-tls-59fbf6696c-qfq4k:/# exit

[root@kubenode1 ingress]# kubectl exec -ti nginx02-tls-5559fd9bc7-dfbrp -c nginx02-tls /bin/bash

root@nginx02-tls-5559fd9bc7-dfbrp:/# echo "<h1>Welcome to test site nginx02-svc-tls</h1>" > /usr/share/nginx/html/index.html

root@nginx02-tls-5559fd9bc7-dfbrp:/# cat /usr/share/nginx/html/index.html

root@nginx02-tls-5559fd9bc7-dfbrp:/# exit

高可用Kubernetes集群-12. 部署kubernetes-ingress

6.创建ingress对象

# 编辑ingress对象yaml文件;

# 在"spec"域下新增"tls"子域,"hosts"字段加入多域名,"secretName"字段调用对应的Secret资源;

# 1个ingress对象只能使用1个Secret对象("secretName"字段value唯一),即只能使用1个证书,该正式需要支持"hosts"字段下所有域名;

# "secretName"字段一定要置于域名列表最后的位置;

# "hosts"字段的域名需要匹配"rules"字段域名;

# ingress默认情况下,当不配置证书或者证书配置错误时,会默认给出一个tls证书;如"secretName"字段配置了2个值,则所有域名采用默认证书;如"hosts"字段少配置一个域名,缺失的域名会采用默认证书;

# 更新ingress证书可能需要等待一段时间才能生效

[root@kubenode1 ingress]# touch nginx-svc-tls-ingress.yaml

[root@kubenode1 ingress]# vim nginx-svc-tls-ingress.yaml

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: ingress-tls

spec:

tls:

- hosts:

- nginx01-svc-tls.me

- nginx02-svc-tls.me

secretName: secret-ingress

rules:

- host: nginx01-svc-tls.me

http:

paths:

- backend:

serviceName: nginx01-svc-tls

# 后端服务监听端口,区别于提供真实服务的容器监听端口

servicePort: 443

- host: nginx02-svc-tls.me

http:

paths:

- backend:

serviceName: nginx02-svc-tls

servicePort: 443

# 生成ingress对象

[root@kubenode1 ingress]# kubectl create -f nginx-svc-tls-ingress.yaml

[root@kubenode1 ingress]# kubectl get ingress

高可用Kubernetes集群-12. 部署kubernetes-ingress

7.验证

# 采用--resolve参数模拟dns解析,目标地址为域名;

# http访问时被重定向,采用https访问正常

[root@kubenode1 ingress]# curl --resolve nginx01-svc-tls.me:80:172.30.200.21 http://nginx01-svc-tls.me

[root@kubenode1 ingress]# curl --resolve nginx01-svc-tls.me:443:172.30.200.21 -k https://nginx01-svc-tls.me

高可用Kubernetes集群-12. 部署kubernetes-ingress

# 或者采用-H参数设置http头中需要访问的域名,目标地址为ip地址

[root@kubenode1 ingress]# curl -H 'Host:nginx01-svc-tls.me' -k https://172.30.200.23

[root@kubenode1 ingress]# curl -H 'Host:nginx02-svc-tls.me' -k https://172.30.200.23

高可用Kubernetes集群-12. 部署kubernetes-ingress

在本地浏览器访问host主机(注意提前绑定域名):http://nginx01-svc-tls.me

采用http访问,重定向自动跳转为https访问,如下:

高可用Kubernetes集群-12. 部署kubernetes-ingress

站点:nginx01-svc-tls.me

高可用Kubernetes集群-12. 部署kubernetes-ingress

站点:nginx02-svc-tls.me

高可用Kubernetes集群-12. 部署kubernetes-ingress

相关推荐