This page looks best with JavaScript enabled

Hey ! Kubernetes finalizers 不再煩惱

 ·  ☕ 3 min read

前言

透過原理來了解事情的因果關係可能會太複雜,但作為一個軟體工程師理解背後如何實踐以及為什麼會有這樣的東西出現(歷史緣由)是非常重要的。
本篇文章將會記錄 finalizers 的背後原理以及一些 source code ,這是使用者在操作 Kubernetes 常常會看到的一個欄位,好像有聽過但又不太了解的東西xD

觀察細節

finalizers 定義於 Kubernetes 的 metadata的欄位

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
...
// ObjectMeta is metadata that all persisted resources must have, which includes all objects
// users must create.
type ObjectMeta struct {
...
	// Populated by the system when a graceful deletion is requested.
	// Read-only.
	// More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
	// +optional
	DeletionTimestamp *Time `json:"deletionTimestamp,omitempty" protobuf:"bytes,9,opt,name=deletionTimestamp"`

	// Number of seconds allowed for this object to gracefully terminate before
	// it will be removed from the system. Only set when deletionTimestamp is also set.
	// May only be shortened.
	// Read-only.
	// +optional
	DeletionGracePeriodSeconds *int64 `json:"deletionGracePeriodSeconds,omitempty" protobuf:"varint,10,opt,name=deletionGracePeriodSeconds"`

	// Must be empty before the object is deleted from the registry. Each entry
	// is an identifier for the responsible component that will remove the entry
	// from the list. If the deletionTimestamp of the object is non-nil, entries
	// in this list can only be removed.
	// Finalizers may be processed and removed in any order.  Order is NOT enforced
	// because it introduces significant risk of stuck finalizers.
	// finalizers is a shared field, any actor with permission can reorder it.
	// If the finalizer list is processed in order, then this can lead to a situation
	// in which the component responsible for the first finalizer in the list is
	// waiting for a signal (field value, external system, or other) produced by a
	// component responsible for a finalizer later in the list, resulting in a deadlock.
	// Without enforced ordering finalizers are free to order amongst themselves and
	// are not vulnerable to ordering changes in the list.
	// +optional
	// +patchStrategy=merge
	Finalizers []string `json:"finalizers,omitempty" patchStrategy:"merge" protobuf:"bytes,14,rep,name=finalizers"`
    ...

從註解中我們大概可以了解這個欄位要表達的意義,我把它整理成比較容易閱讀的方式(可能只有我覺得godoc的 // 註解換行很煩xD

Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list.

大膽假設

我認為上述的文字簡單來說有三個重點(以下的順序不重要)

  1. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed.
    簡單來說當 deletionTimestamp 不是 nil 的時候需要先刪除 Finalizers 內的條目
  2. Finalizers may be processed and removed in any order
    Finalizers 條目的刪除順序不是固定的
  3. Must be empty before the object is deleted from the registry.
    再刪除這個物件前 Finalizers 條目必須是空的

從上述註解的我們大概可以推測幾件事情,第一 當使用者刪除 Kubernetes 物件時,GC 回收機制需要檢查 Finalizers是否為空,第二其他物件可以任意刪除 Finalizers 欄位,前提是 deletionTimestamp 欄位不為 nil。

大膽假設這個現象了,那就應該小心求證事實。

小心求證

我已最近我在玩耍的 argo cd 進行球證對象,大部分處理物件的邏輯都會落在 controller 的 reconcile 階段裡,只要在 reconcile 搜尋 Finalizer 應該可以發現點什麼。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
func (ctrl *ApplicationController) processProjectQueueItem() (processNext bool) {
    if origProj.DeletionTimestamp != nil && origProj.HasFinalizer() {
		if err := ctrl.finalizeProjectDeletion(origProj.DeepCopy()); err != nil {
			log.Warnf("Failed to finalize project deletion: %v", err)
		}
	}
	return
    ...
}

func (ctrl *ApplicationController) finalizeProjectDeletion(proj *appv1.AppProject) error {
	apps, err := ctrl.appLister.Applications(ctrl.namespace).List(labels.Everything())
	if err != nil {
		return err
	}
	appsCount := 0
	for i := range apps {
		if apps[i].Spec.GetProject() == proj.Name {
			appsCount++
			break
		}
	}
	if appsCount == 0 {
		return ctrl.removeProjectFinalizer(proj)
	} else {
		log.Infof("Cannot remove project '%s' finalizer as is referenced by %d applications", proj.Name, appsCount)
	}
	return nil
}

func (ctrl *ApplicationController) removeProjectFinalizer(proj *appv1.AppProject) error {
	proj.RemoveFinalizer()
	var patch []byte
	patch, _ = json.Marshal(map[string]interface{}{
		"metadata": map[string]interface{}{
			"finalizers": proj.Finalizers,
		},
	})
	_, err := ctrl.applicationClientset.ArgoprojV1alpha1().AppProjects(ctrl.namespace).Patch(context.Background(), proj.Name, types.MergePatchType, patch, metav1.PatchOptions{})
	return err
}

func (proj AppProject) HasFinalizer() bool {
	return getFinalizerIndex(proj.ObjectMeta, common.ResourcesFinalizerName) > -1
}

// getFinalizerIndex returns finalizer index in the list of object finalizers or -1 if finalizer does not exist
func getFinalizerIndex(meta metav1.ObjectMeta, name string) int {
	for i, finalizer := range meta.Finalizers {
		if finalizer == name {
			return i
		}
	}
	return -1
}

func (proj *AppProject) RemoveFinalizer() {
	setFinalizer(&proj.ObjectMeta, common.ResourcesFinalizerName, false)
}

// setFinalizer adds or removes finalizer with the specified name
func setFinalizer(meta *metav1.ObjectMeta, name string, exist bool) {
	index := getFinalizerIndex(*meta, name)
	if exist != (index > -1) {
		if index > -1 {
			meta.Finalizers[index] = meta.Finalizers[len(meta.Finalizers)-1]
			meta.Finalizers = meta.Finalizers[:len(meta.Finalizers)-1]
		} else {
			meta.Finalizers = append(meta.Finalizers, name)
		}
	}
}

我把主要的判斷邏輯抓出來,可以看到大致上的邏輯有幾項

  1. processProjectQueueItem function 裡面會判斷 Application 物件的 DeletionTimestamp 以及 Finalizers 內有沒有我們要關注的 key
  2. 檢查 Applications namespaces 下面的其他的相關物件,並且計算物件的總數。
    1. 如果>0
      • 直接回傳,因為相關的資源還沒情理乾淨
    2. 如果=0
      • 移除 Finalizers 我們要關注的 key

這邊可以看到幾個之前提過的重點

  1. 簡單來說當 deletionTimestamp 不是 nil 的時候需要先刪除 Finalizers 內的條目
  2. Finalizers may be processed and removed in any order
    Finalizers 條目的刪除順序不是固定的

結語

類似像 Finalizers 這種的小螺絲都能起到這麼大的作用,我們應該更靜下心來學時事務的本質,不只要會操作它更要理解背後的原理是什麼。


Meng Ze Li
WRITTEN BY
Meng Ze Li
Kubernetes / DevOps / Backend

What's on this Page