Kubernetes 零停机部署 - Zero downtime deployments - Flagger

标签: | 发表时间:2022-01-15 16:37 | 作者:
出处:https://docs.flagger.app
This is a list of things you should consider when dealing with a high traffic production environment if you want to minimise the impact of rolling updates and downscaling.

Deployment strategy

Limit the number of unavailable pods during a rolling update:
apiVersion: apps/v1
kind: Deployment
spec:
  progressDeadlineSeconds: 120
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
Copied!
The default progress deadline for a deployment is ten minutes. You should consider adjusting this value to make the deployment process fail faster.

Liveness health check

You application should expose a HTTP endpoint that Kubernetes can call to determine if your app transitioned to a broken state from which it can't recover and needs to be restarted.
livenessProbe:
  exec:
    command:
    - wget
    - --quiet
    - --tries=1
    - --timeout=4
    - --spider
    - http://localhost:8080/healthz
  timeoutSeconds: 5
  initialDelaySeconds: 5
Copied!
If you've enabled mTLS, you'll have to use exec for liveness and readiness checks since kubelet is not part of the service mesh and doesn't have access to the TLS cert.

Readiness health check

You application should expose a HTTP endpoint that Kubernetes can call to determine if your app is ready to receive traffic.
readinessProbe:
  exec:
    command:
    - wget
    - --quiet
    - --tries=1
    - --timeout=4
    - --spider
    - http://localhost:8080/readyz
  timeoutSeconds: 5
  initialDelaySeconds: 5
  periodSeconds: 5
Copied!
If your app depends on external services, you should check if those services are available before allowing Kubernetes to route traffic to an app instance. Keep in mind that the Envoy sidecar can have a slower startup than your app. This means that on application start you should retry for at least a couple of seconds any external connection.

Graceful shutdown

Before a pod gets terminated, Kubernetes sends a SIGTERM signal to every container and waits for period of time (30s by default) for all containers to exit gracefully. If your app doesn't handle the SIGTERM signal or if it doesn't exit within the grace period, Kubernetes will kill the container and any inflight requests that your app is processing will fail.
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: app
        lifecycle:
          preStop:
            exec:
              command:
              - sleep
              - "10"
Copied!
Your app container should have a preStop hook that delays the container shutdown. This will allow the service mesh to drain the traffic and remove this pod from all other Envoy sidecars before your app becomes unavailable.

Delay Envoy shutdown

Even if your app reacts to SIGTERM and tries to complete the inflight requests before shutdown, that doesn't mean that the response will make it back to the caller. If the Envoy sidecar shuts down before your app, then the caller will receive a 503 error.
To mitigate this issue you can add a preStop hook to the Istio proxy and wait for the main app to exit before Envoy exits.
#!/bin/bash
set -e
if ! pidof envoy &>/dev/null; then
  exit 0
fi

if ! pidof pilot-agent &>/dev/null; then
  exit 0
fi

while [ $(netstat -plunt | grep tcp | grep -v envoy | wc -l | xargs) -ne 0 ]; do
  sleep 1;
done

exit 0
Copied!
You'll have to build your own Envoy docker image with the above script and modify the Istio injection webhook with the preStop directive.
Thanks to Stono for his excellent tips on minimising 503s.

Resource requests and limits

Setting CPU and memory requests/limits for all workloads is a mandatory step if you're running a production system. Without limits your nodes could run out of memory or become unresponsive due to CPU exhausting. Without CPU and memory requests, the Kubernetes scheduler will not be able to make decisions about which nodes to place pods on.
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: app
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 128Mi
Copied!
Note that without resource requests the horizontal pod autoscaler can't determine when to scale your app.

Autoscaling

A production environment should be able to handle traffic bursts without impacting the quality of service. This can be achieved with Kubernetes autoscaling capabilities. Autoscaling in Kubernetes has two dimensions: the Cluster Autoscaler that deals with node scaling operations and the Horizontal Pod Autoscaler that automatically scales the number of pods in a deployment.
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 2
  maxReplicas: 4
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageValue: 900m
  - type: Resource
    resource:
      name: memory
      targetAverageValue: 768Mi
Copied!
The above HPA ensures your app will be scaled up before the pods reach the CPU or memory limits.

Ingress retries

To minimise the impact of downscaling operations you can make use of Envoy retry capabilities.
apiVersion: flagger.app/v1beta1
kind: Canary
spec:
  service:
    port: 9898
    gateways:
    - public-gateway.istio-system.svc.cluster.local
    hosts:
    - app.example.com
    retries:
      attempts: 10
      perTryTimeout: 5s
      retryOn: "gateway-error,connect-failure,refused-stream"
Copied!
When the HPA scales down your app, your users could run into 503 errors. The above configuration will make Envoy retry the HTTP requests that failed due to gateway errors.

相关 [kubernetes zero downtime] 推荐:

Kubernetes 零停机部署 - Zero downtime deployments - Flagger

- -
This is a list of things you should consider when dealing with a high traffic production environment if you want to minimise the impact of rolling updates and downscaling..

如何用Kubernetes完成零停机部署 How to archive zero downtime deployments in Kubernetes | Cozi

- -
当您需要等到深夜才能在生产中部署您的应用程序时,情况会更糟,因为当部署后发生短暂的停机时间时,您害怕破坏用户的体验. 经过漫长的工作日后,您应该睡个好觉,所以让我们找出解决方案. 对于正常的应用,我们只需要做 3 个简单的步骤:. 添加容器生命周期 preStop 挂钩. kubelet 在您的容器运行后发送请求,但您的应用程序可能还没有准备好接收请求.

[原]Zero Copy 简介

- - 芒果先生Mango的专栏
最近在看Kafka的设计原理,kafka在数据传输时采用了zero copy技术:. 使用Zero copy方式在内核层直接将文件内容传送给网络Socket,避免应用层数据拷贝,减小IO开销. 关于Zero copy,更多请点击 Efficient data transfer through zero copy.

Kubernetes & Microservice

- - 午夜咖啡
这是前一段时间在一个微服务的 meetup 上的分享,整理成文章发布出来. 谈微服务之前,先澄清一下概念. 微服务这个词的准确定义很难,不同的人有不同的人的看法. 比如一个朋友是『微服务原教旨主义者』,坚持微服务一定是无状态的 http API 服务,其他的都是『邪魔歪道』,它和 SOA,RPC,分布式系统之间有明显的分界.

Fate/Zero壁纸第一弹

- 小班 - 和邪社
Fate/Zero自不用多说是目前新番中的佼佼者,可惜这部作品是分拆成了两季来播出的. Fate/Zero的魅力何在呢. 这个问题就很大了,简单点说它是一部集合很多题材优势的作品,穿越、魔幻、历史、爱情,这些你都可以在里面找到,而且最关键的是它融合的非常好. Fate/Zero自不用多说是目前新番中的佼佼者,可惜这部作品是分拆成了两季来播出的.

Zero Install Injector 1.4.1 发布

- ZeeJee - 新闻 - LUPA开源社区
Zero Install Injector是一款很神奇的软件,他的目标是终结Linux下软件安装困难的现象. 通过它你可以非常智能化、自动化的安装任何软件(包括各种打包格 式、源代码),作者自己声称这将会替代其他的软件管理软件:Y.

Kubernetes学习(Kubernetes踩坑记)

- - Z.S.K.'s Records
记录在使用Kubernetes中遇到的各种问题及解决方案, 好记性不如烂笔头. prometheus提示 /metrics/resource/v1alpha1 404. 原因: 这是因为[/metrics/resource/v1alpha1]是在v1.14中才新增的特性,而当前kubelet版本为1.13.

kubernetes移除Docker?

- -
两周前,Kubernetes在其最新的Changelog中宣布1.20之后将要弃用dockershime,也就说Kubernetes将不再使用Docker做为其容器运行时. 这一消息持续发酵,掀起了不小的波澜,毕竟Kubernetes+Docker的经典组合是被市场所认可的,大量企业都在使用. 看上去这个“弃用”的决定有点无厘头,那么为什么Kubernetes会做出这样的决定.

Spring Cloud 不停机发布服务(0-downtime Blue/Green deployments) | 帆的博客

- -
项目初期由于BUG和需求改动可能都会比较多,我们会很频繁的发布我们的应用. 但是如果不进行处理,在升级的过程中会导致用户服务中断. 实际上针对这两种情况,在传统的应用中我们是很容易做到不停机升级的. 例如nginx负载均衡2台tomcat实例,在升级的时候切断其中一台访问,升级完成以后切换流量,再升级另外一台.

Kubernetes 完全教程

- - 午夜咖啡
经过一个阶段的准备,视频版本的 《Kubernetes 完全教程》出炉了. 课程一共分为七节,另外有一节 Docker 预备课,每节课大约一个多小时. 目标是让从没接触过 Kubernetes 的同学也能通过这个课程掌握 Kubernetes. 为什么要学习 Kubernetes. 在介绍课程之前,先说说为什么要学习 Kubernetes 以及什么人需要学习 Kubernetes.