Kubernetes使用helm部署Mysql-Ha

# 1.首先检查一下 Helm repo 源中是否存在 incubator

```
helm repo list
#执行结果
NAME            URL                                                      
stable          https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts   
local           http://127.0.0.1:8879/charts                             
library         http://registry.jz-ins.com:8080/chartrepo/library        
azure           http://mirror.azure.cn/kubernetes/charts/                
google          https://kubernetes-charts.storage.googleapis.com         
incubator       http://storage.googleapis.com/kubernetes-charts-incubator

#如果不存在操作如下
# a. 添加新的远程仓库
helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator
#执行结果
"incubator" has been added to your repositories
#如上显示则为添加成功

#更新 helm 
helm repo update
#执行结果
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "stable" chart repository
...Successfully got an update from the "library" chart repository
...Successfully got an update from the "incubator" chart repository
...Successfully got an update from the "azure" chart repository
...Successfully got an update from the "google" chart repository
Update Complete.
```

# 2\. 查询 helm 库中的 chart

```

helm search mysqlha
#执行结果
NAME                    CHART VERSION   APP VERSION     DESCRIPTION                                                 
incubator/mysqlha       1.0.0           5.7.13          MySQL cluster with a single master and zero or more slave...

```

# 3.下载 mysqlha的helm包到本地, 以下命令将下载 redis-ha包到当前目录下，并创建一个 mysqlha 目录

```

helm fetch incubator/mysqlha --untar --untardir ./

#查看下载的目录结构
tree mysqlha/
#执行结果mysqlha/
├── Chart.yaml
├── OWNERS
├── README.md
├── templates
│   ├── configmap.yaml
│   ├── _helpers.tpl
│   ├── NOTES.txt
│   ├── secret.yaml
│   ├── statefulset.yaml
│   └── svc.yaml
└── values.yaml

#目录中的 values.yaml 是安装 redis-ha 的默认参数， 可参考官方 github 文档调整设置
cd mysqlha

```

###### 调整 values.yaml ，以下给出我的 values.yaml

```

## mysql image version
## ref: https://hub.docker.com/r/library/mysql/tags/
##
mysqlImage: mysql:5.7.13
xtraBackupImage: twoeo/gcr.io-google-samples-xtrabackup:latest

## Specify an imagePullPolicy (Required)
## It's recommended to change this to 'Always' if the image tag is 'latest'
## ref: http://kubernetes.io/docs/user-guide/images/#updating-images
##
imagePullPolicy: IfNotPresent

## String to partially override orangehrm.fullname template (will maintain the release name)
##
# nameOverride: ""
## String to fully override orangehrm.fullname template
##
# fullnameOverride: ""

mysqlha:
  replicaCount: 3

## Password for MySQL root user
  ##
  mysqlRootPassword: JZK8SUATP@ssw0rd ## Default: random 10 character string

## Username/password for MySQL replication user
  ##
  mysqlReplicationUser: repl
  mysqlReplicationPassword: JZK8SUATP@ssw0rd

## Create a database user
  ##
  # mysqlUser:
  # mysqlPassword: ## Default: random 10 character string

## Allow unauthenticated access, uncomment to enable
  ##
  # mysqlAllowEmptyPassword: true

## Create database with name and grant all permissions to user on startup, if needed
  # mysqlDatabase:

## Configuration files for the master and slaves
  ##
  configFiles:
    master.cnf: |
      # Apply this config only on the master.
      [mysqld]
      log-bin
      skip_name_resolve
    slave.cnf: |
      # Apply this config only on slaves.
      [mysqld]
      super-read-only
      skip_name_resolve

podAnnotations: {}

## Use an alternate scheduler, e.g. "stork".
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
# schedulerName:

## Enable persistence using Persistent Volume Claims
## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
##
persistence:
  enabled: true
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, azure-disk on
  ##   Azure, standard on GKE, AWS & OpenStack)
  ##
  storageClass: "managed-nfs-storage"
  accessModes:
  - ReadWriteOnce
  size: 10Gi
  annotations: {}

resources:
  requests:
    cpu: 100m
    memory: 128Mi

metrics:
  enabled: false
  image: prom/mysqld-exporter
  imageTag: v0.10.0
  annotations: {}

livenessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
  readinessProbe:
    initialDelaySeconds: 5
    timeoutSeconds: 1
  resources:
    requests:
      cpu: 100m
      memory: 100Mi

```

###### 此 yaml 主要修改了以下内容

1.  修改 xtraBackupImage 的镜像为 twoeo/gcr.io-google-samples-xtrabackup:latest， 由于默认配置为  
    gcr.io/google-samples/xtrabackup:1.0， 因为网络问题可能会导致镜像无法下载，而最终导致部署失败
2.  打开 # mysqlRootPassword: 代码的注释并设置 root 用户密码  
      
    注意：此处设置的密码尽量规范，我在初次部署的时候设置的密码结尾是 !v 导致被识别为了系统命令  
    导致部署失败  
    
3.  打开 # mysqlReplicationPassword: 代码的注释并设置 同步数据的密码
4.  打开 “ # storageClass: “-” ” 的注释，并修改 “-” 为 集群中的自动供给卷 “managed-nfs-storage”，  
    配置中 “size: 10Gi” 的大小为默认设置，可根据需要进行调整  
    [k8s 基于NFS部署storageclass pv自动供给可参考此博文](https://www.cnblogs.com/Smbands/p/11059843.html)

###### 部署 mysqlha, 此处最好指定一下 部署的 mysql 名称， 这里指定为 mysql-ha

```

helm install --name mysql-ha -f values.yaml incubator/mysqlha
#执行结果
NAME:   mysql-ha
LAST DEPLOYED: Thu Oct 17 17:04:37 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME              DATA  AGE
mysql-ha-mysqlha  4     0s

==> v1/Pod(related)
NAME                READY  STATUS   RESTARTS  AGE
mysql-ha-mysqlha-0  0/2    Pending  0         0s

==> v1/Secret
NAME              TYPE    DATA  AGE
mysql-ha-mysqlha  Opaque  2     0s

==> v1/Service
NAME                       TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)   AGE
mysql-ha-mysqlha           ClusterIP  None           <none>       3306/TCP  0s
mysql-ha-mysqlha-readonly  ClusterIP  10.108.155.15  <none>       3306/TCP  0s

==> v1beta1/StatefulSet
NAME              READY  AGE
mysql-ha-mysqlha  0/3    0s

NOTES:
The MySQL cluster is comprised of 3 MySQL pods: 1 master and 2 slaves. Each instance is accessible within the cluster through:

<pod-name>.mysql-ha-mysqlha

`mysql-ha-mysqlha-0.mysql-ha-mysqlha` is designated as the master and where all writes should be executed against. Read queries can be executed against the `mysql-ha-mysqlha-readonly` service which distributes connections across all MySQL pods.

To connect to your database:

1. Obtain the root password:

kubectl get secret --namespace default mysql-ha-mysqlha -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo

2. Run a pod to use as a client:

kubectl run mysql-client --image=mysql:5.7.13 -it --rm --restart='Never' --namespace default -- /bin/sh

3. To connect to Master service (read/write):

mysql -h mysql-ha-mysqlha-0.mysql-ha-mysqlha -u root -p

4. To connect to slave service (read-only):
   
   mysql -h mysql-ha-mysqlha-readonly -u root -p

#看到以上信息即为部署成功了
```

###### 查询部署状态

```

helm status mysql-ha
#执行结果
LAST DEPLOYED: Thu Oct 17 17:04:37 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME              DATA  AGE
mysql-ha-mysqlha  4     6m54s

==> v1/Pod(related)
NAME                READY  STATUS   RESTARTS  AGE
mysql-ha-mysqlha-0  2/2    Running  1         6m54s
mysql-ha-mysqlha-1  2/2    Running  0         6m31s
mysql-ha-mysqlha-2  2/2    Running  0         3m40s

==> v1/Secret
NAME              TYPE    DATA  AGE
mysql-ha-mysqlha  Opaque  2     6m54s

==> v1/Service
NAME                       TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)   AGE
mysql-ha-mysqlha           ClusterIP  None           <none>       3306/TCP  6m54s
mysql-ha-mysqlha-readonly  ClusterIP  10.108.155.15  <none>       3306/TCP  6m54s

==> v1beta1/StatefulSet
NAME              READY  AGE
mysql-ha-mysqlha  3/3    6m54s

NOTES:
The MySQL cluster is comprised of 3 MySQL pods: 1 master and 2 slaves. Each instance is accessible within the cluster through:

<pod-name>.mysql-ha-mysqlha

`mysql-ha-mysqlha-0.mysql-ha-mysqlha` is designated as the master and where all writes should be executed against. Read queries can be executed against the `mysql-ha-mysqlha-readonly` service which distributes connections across all MySQL pods.

To connect to your database:

1. Obtain the root password:

kubectl get secret --namespace default mysql-ha-mysqlha -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo

2. Run a pod to use as a client:

kubectl run mysql-client --image=mysql:5.7.13 -it --rm --restart='Never' --namespace default -- /bin/sh

3. To connect to Master service (read/write):

mysql -h mysql-ha-mysqlha-0.mysql-ha-mysqlha -u root -p

4. To connect to slave service (read-only):
   
   mysql -h mysql-ha-mysqlha-readonly -u root -p

```

###### 验证 mysql-ha

```

#先创建一个临时的 mysql-client
kubectl run mysql-client --image=mysql:5.7.13 -it --rm --restart='Never' --namespace default bash
#执行结果
If you don't see a command prompt, try pressing enter.
#开那单以上信息，敲击 回车即可进入 mysql-client 临时容器内
root@mysql-client:/# 
```

###### 1\. 进入 Master 库（读写）mysql-ha-mysqlha-0 ， 仅 mysql-ha-mysqlha-0 为 master 可读写

```

mysql -h mysql-ha-mysqlha-0.mysql-ha-mysqlha -u root -p JZK8SUATP@ssw0rd

#进入 Slave 库（只读）mysql-ha-mysqlha-1
mysql -h mysql-ha-mysqlha-1.mysql-ha-mysqlha -u root -p JZK8SUATP@ssw0rd

#进入 Slave 库（只读）mysql-ha-mysqlha-2
mysql -h mysql-ha-mysqlha-2.mysql-ha-mysqlha -u root -p JZK8SUATP@ssw0rd

#集群内部连接只读库， YOUR_PASSWORD 替换为你自己的密码
mysql -uroot -pYOUR_PASSWORD -h mysql-ha-mysqlha-readonly -P 3306

#集群内部连接读写库， YOUR_PASSWORD 替换为你自己的密码
mysql -uroot -pYOUR_PASSWORD -h mysql-ha-mysqlha-0.mysql-ha-mysqlha -P 3306

#在 Slave 库中执行写操作的时候会提示错误，如下：
ERROR 1290 (HY000): The MySQL server is running with the --super-read-only option so it cannot execute this statement

======================= 以下为主从复制验证过程，不需要的可以忽略 =====================
#进入 Master 库
mysql -uroot -pJZK8SUATP@ssw0rd -h mysql-ha-mysqlha-0.mysql-ha-mysqlha -P 3306
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1011
Server version: 5.7.13-log MySQL Community Server (GPL)

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type "help;" or "\h" for help. Type "\c" to clear the current input statement.

#创建 demo 数据库
mysql> CREATE DATABASE demo; 
Query OK, 1 row affected (0.04 sec)

#创建测试表 messages
mysql> CREATE TABLE demo.messages (message VARCHAR(250)); 
Query OK, 0 rows affected (0.05 sec)

#插入一条测试数据
mysql> INSERT INTO demo.messages VALUES (`hello`);#此处请将 “`”改为 “’”， 进行测试
Query OK, 1 row affected (0.01 sec)
```

###### 2\. 进入 Slave 库验证是否同步

```

mysql -uroot -pJZK8SUATP@ssw0rd -h mysql-ha-mysqlha-readonly -P 3306                            
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1032
Server version: 5.7.13 MySQL Community Server (GPL)

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use demo;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from messages;
+---------+
| message |
+---------+
| hello   |
+---------+
1 row in set (0.00 sec)

#在 master demo 数据库中插入的数据也已经同步

```

# 如果需要暴露给外部使用则需要再部署 NodePort Service

此处提供两个暴露给外部使用的 NodePort Service， 一个为读写服务， 一个为只读服务

###### 首先需要查询一下各个 pod 的特有标签

```
kubectl get pods --show-labels | grep mysql | awk '{print $6}' | awk -F, '{print $3}'
statefulset.kubernetes.io/pod-name=mysql-ha-mysqlha-0	# Master
statefulset.kubernetes.io/pod-name=mysql-ha-mysqlha-1	# Slave	 
statefulset.kubernetes.io/pod-name=mysql-ha-mysqlha-2	# Slave

# 读写服务的 selector 使用 Master 对外服务
# 只读服务的 selector 使用 任一 Slave 对外服务
# yaml 文件配置 selector 需将 “=” 改为 “: ” ,  注意： “:” 的后面要有一个空格
```

###### 编写读写服务

```
vi mysql-ha-wirteandread-service.yaml
```

```

apiVersion: v1
kind: Service
metadata:
  name: mysql-ha-wirteandread-service
  labels:
    app: mysql-ha-mysqlha
spec:
  ports:
  - name: mysql-ha-master
    protocol: "TCP"
    port: 3306
    targetPort: 3306
    nodePort: 30336		#此为外部连接k8s mysql-ha 服务的端口
  selector:
    statefulset.kubernetes.io/pod-name: mysql-ha-mysqlha-0
  type: NodePort

```

###### 部署读写服务

```

kubectl apply -f mysql-ha-wirteandread-service.yaml 
#执行结果
service/mysql-ha-wirteandread-service configured

```

###### 编写只读服务

```
vi mysql-ha-readonly-service.yaml
```

```

apiVersion: v1
kind: Service
metadata:
  name: mysql-ha-readonly-service
  labels:
    app: mysql-ha-mysqlha
spec:
  ports:
  - name: mysql-ha-master
    protocol: "TCP"
    port: 3306
    targetPort: 3306
    nodePort: 30336		#此为外部连接k8s mysql-ha 服务的端口
  selector:
    statefulset.kubernetes.io/pod-name: mysql-ha-mysqlha-1
  type: NodePort

```

###### 部署只读服务

```

kubectl apply -f mysql-ha-readonly-service.yaml
#执行结果
service/mysql-ha-readonly-service configured

```

###### 查看服务部署结果

```
kubectl get svc
#执行结果
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                       AGE
kubernetes                  ClusterIP   10.96.0.1        <none>        443/TCP                       3d1h
mysql-ha-mysqlha            ClusterIP   None             <none>        3306/TCP                      3h5m
mysql-ha-mysqlha-readonly   ClusterIP   10.108.155.15    <none>        3306/TCP                      3h5m
mysql-ha-service            NodePort    10.110.186.230   <none>        3306:30336/TCP                167m
mysql-ha-service-readonly   NodePort    10.97.9.165      <none>        3306:30337/TCP                121m
redis-ha                    ClusterIP   None             <none>        6379/TCP,26379/TCP,9121/TCP   10h
redis-ha-announce-0         ClusterIP   10.99.18.87      <none>        6379/TCP,26379/TCP,9121/TCP   10h
redis-ha-announce-1         ClusterIP   10.100.130.186   <none>        6379/TCP,26379/TCP,9121/TCP   10h
redis-ha-service            NodePort    10.109.28.12     <none>        26379:30379/TCP               8h
```

###### 此时可以通过 k8s 任意 master 节点 IP:30336 端口进行连接读写服务

###### 此时可以通过 k8s 任意 master 节点 IP:30337 端口进行连接只读服务# 1.首先检查一下 Helm repo 源中是否存在 incubator