Helm Chart Repository

Helm Chart Repository

OCIR은 OCI(Open Container Initiative) Registry로 Helm v3.8.0에서 GA(General Availability)가 되어 공식적으로 지원하고 있습니다

컨테이너 이미지 레지스트리가 필요한데 Harbor를 도입한다면 Helm 레지스트리까지 동시에 사용할 수 있다는게 큰 장점
Helm은 차트 라는 패키징 형식을 사용
차트는 관련 Kubernetes 리소스 세트를 설명하는 파일 모음
Harbor의 Helm 차트에 대한 액세스는 역할 기반 액세스 제어(RBAC) 에 의해 제어되며 프로젝트에 의해 제한됨

 

 

HARBOR Install (helm)

환경
bastion
OS VERSION="20.04.6 LTS (Focal Fossa)"
NKS사용 v1.27.3
OS VERSION="20.04.6 LTS (Focal Fossa)"

전제 조건
Kubernetes 클러스터 1.10+
helm 2.8.0+
고가용성 수신 컨트롤러 (nginx ingress controller 사용)
고가용성 Redis
노드 또는 외부 객체 스토리지 간에 공유할 수 있는 PVC

ref.
https://goharbor.io/docs/2.3.0/install-config/harbor-ha-helm/#configuration
https://beer1.tistory.com/46


helm 설치
root 권한으로 설치
다운로드 helm3

$ cd /root/
$ curl -fsSL -o get_helm3.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
$ chmod 700 get_helm3.sh
$ ./get_helm3.sh

$ helm version
version.BuildInfo{Version:"v3.14.4", GitCommit:"81c902a123462fd4052bc5e9aa9c513c4c8fc142", GitTreeState:"clean", GoVersion:"go1.21.9"}

$ cd /root/
$ curl -fsSL -o get_helm3.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
$ chmod 700 get_helm3.sh
$ ./get_helm3.sh

$ helm version
version.BuildInfo{Version:"v3.14.4", GitCommit:"81c902a123462fd4052bc5e9aa9c513c4c8fc142", GitTreeState:"clean", GoVersion:"go1.21.9"}

root@helm:~/harbor# cat values.yaml
expose:
  ingress:
    hosts:
      core: happybirthdayfor.me # 원하는 도메인으로 설정
    className: "nginx"

externalURL: https://happy.me # 원하는 도메인으로 설정

persistence:
  persistentVolumeClaim:
    registry:
      existingClaim: "harbor-registry" 
      storageClass: "local-storage" 
    jobservice:
      jobLog:
        existingClaim: "harbor-job-service"
        storageClass: "local-storage" 
    database:
      existingClaim: "harbor-database" 
      storageClass: "local-storage"
    redis:
      existingClaim: "harbor-redis" 
      storageClass: "local-storage" 
    trivy:
      existingClaim: "harbor-trivy" 
      storageClass: "local-storage" 

harborAdminPassword: "Harbor12345"

로컬 스토리지를 사용하는 경우
로컬 스토리지를 사용하는 경우 따로 스토리지클래스와 PV, PVC를 생성

 

vi storageClass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

$  kubectl apply -f storageClass.yam
te-node-0 노드에 각 path 생성 해놔야함

vi harbor-pv.yaml

root@helm:~/harbor# cat harbor-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: harbor-registry
spec:
  capacity:
    storage: 5Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /home/beer1/pv/harbor-registry
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - te-node-0

---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: harbor-job-service
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /home/beer1/pv/harbor-job-service
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - te-node-0

---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: harbor-database
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /home/beer1/pv/harbor-database
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - -te--node-0

---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: harbor-redis
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /home/beer1/pv/harbor-redis
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - ka-te-ka1-node-0

---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: harbor-trivy
spec:
  capacity:
    storage: 5Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /home/beer1/pv/harbor-trivy
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - ka-te-ka1-node-0

kubectl apply -f harbor-pv.yaml

 vi harbor-pvc.yaml
root@helm:~/harbor# cat harbor-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: harbor-registry
  namespace: harbor
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: local-storage
  volumeName: harbor-registry
  resources:
    requests:
      storage: 5Gi

---

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: harbor-job-service
  namespace: harbor
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: local-storage
  volumeName: harbor-job-service
  resources:
    requests:
      storage: 1Gi

---

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: harbor-database
  namespace: harbor
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: local-storage
  volumeName: harbor-database
  resources:
    requests:
      storage: 1Gi

---

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: harbor-redis
  namespace: harbor
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: local-storage
  volumeName: harbor-redis
  resources:
    requests:
      storage: 1Gi

---

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: harbor-trivy
  namespace: harbor
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: local-storage
  volumeName: harbor-trivy
  resources:
    requests:
      storage: 5Gi

kubectl apply -f harbor-pvc.yaml

pv와 pvc가 바운드 되어있어야함

 

helm install harbor harbor/harbor -n harbor -f values.yaml

 

pods 확인
kubectl get pods -n harbor

외부에서 접속을 가능하게 하기 위해 nginx-ingress-controller 사용 

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

 helm install ingress-nginx ingress-nginx/ingress-nginx -n harbor

cat mynginx-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: mynginx-ingress
spec:
  rules:
  - host: happybirthdayfor.me
  - http:
      paths:
      - path: /nginx
        pathType: Prefix
        backend:
          service:
            name: nginxsvc
            port:
              number: 80

kubectl apply -f mynginx-ingress.yaml

vi harbor-ingress.yaml
root@helm:~/harbor# cat harbor-ingress.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: mynginx
  name: mynginx
spec:
  containers:
  - image: nginx:1.16
    name: mynginx
    resources: {}
  restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: nginxsvc
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: mynginx
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: mynginx-ingress
spec:
  rules:
  - host: happyr.me
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginxsvc
            port:
              number: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginxsvc-harbor-core
  namespace: harbor
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    component: core
kubectl apply -f harbor-ingress.yaml


vi cat nginxsvc-harbor-core.yml
root@helm:~/harbor# cat nginxsvc-harbor-core.yml
apiVersion: v1
kind: Service
metadata:
  name: nginxsvc-harbor-core
  namespace: harbor
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: harbor-core
kubectl apply -f nginxsvc-harbor-core.yml

 

os 수준에서 인증서를 신뢰하도록함.
-Ubuntu cp ca.crt /usr/local/share/ca-certificates/ca.crt
update-ca-certificates

vi /etc/docker/daemon.json

{
   "insecure-registries":[
      "happybirthdayfor.me"
   ]
}
systemctl restart docker
docker login https://happy.me -u admin -p Harbor12345