docker run -p 9030:9030 \
-p 8030:8030 -p 8040:8040 \
-itd \
-v `pwd`/start.sh:/data/deploy/start_fe_be.sh \
--name StarRocks-AllinOne \
starrocks.docker.scarf.sh/starrocks/allin1-ubuntu
Docker - StarRocks部署（支持集群）
StarRocks介绍
StarRocks 是一款高性能分析型数据仓库，使用向量化、MPP 架构、可实时更新的列式存储引擎等技术实现多维、实时、高并发的数据分析。StarRocks 既支持从各类实时和离线的数据源高效导入数据，也支持直接分析数据湖上各种格式的数据。StarRocks 兼容 MySQL 协议，可使用 MySQL 客户端和常用 BI 工具对接。同时 StarRocks 具备水平扩展，高可用、高可靠、易运维等特性。广泛应用于实时数仓、OLAP 报表、数据湖分析等场景。
-
StarRocks是新一代极速全场景MPP（Massively Parallel Processing）数据库，致力于构建极速和统一分析体验。
-
StarRocks兼容MySQL协议，可使用MySQL客户端和常用BI工具对接StarRocks来分析数据。
-
StarRocks采用分布式架构：
-
对数据表进行水平划分并以多副本存储。
-
实例规模可以灵活伸缩，支持10 PB级别的数据分析。
-
支持MPP框架，并行加速计算。
-
支持多副本，具有弹性容错能力。
-
StarRocks特性
架构精简
- StarRocks内部通过MPP计算框架完成SQL的具体执行工作。MPP框架能够充分的利用多节点的计算能力，整个查询可以并行执行，从而实现良好的交互式分析体验。
- StarRocks实例不需要依赖任何其他组件，易部署、易维护和极简的架构设计，降低了StarRocks系统的复杂度和维护成本，同时也提升了系统的可靠性和扩展性。管理员只需要专注于StarRocks系统，无需学习和管理任何其他外部系统。
全面向量化引擎
StarRocks的计算层全面采用了向量化技术，将所有算子、函数、扫描过滤和导入导出模块进行了系统性优化。通过列式的内存布局、适配CPU的SIMD指令集等手段，充分发挥了现代CPU的并行计算能力，从而实现亚秒级别的多维分析能力。
智能查询优化
StarRocks通过CBO优化器（Cost Based Optimizer）可以对复杂查询自动优化。无需人工干预，就可以通过统计信息合理估算执行成本，生成更优的执行计划，大大提高了AdHoc和ETL场景的数据分析效率。
联邦查询
StarRocks支持使用外表的方式进行联邦查询，当前可以支持Hive、MySQL、Elasticsearch、Iceberg和Hudi类型的外表，您无需通过数据导入，可以直接进行数据查询加速。
高效更新
StarRocks支持明细模型、聚合模型、主键模型和更新模型，其中主键模型可以按照主键进行Upsert或Delete操作，通过存储和索引的优化可以在并发更新的同时实现高效的查询优化，更好的服务实时数仓的场景。
智能物化视图
- StarRocks支持智能的物化视图。您可以通过创建物化视图，预先计算生成预聚合表用于加速聚合类查询请求。
- StarRocks的物化视图能够在数据导入时自动完成汇聚，与原始表数据保持一致。
- 查询的时候，您无需指定物化视图，StarRocks能够自动选择最优的物化视图来满足查询请求。
标准SQL
- StarRocks支持标准的SQL语法，包括聚合、JOIN、排序、窗口函数和自定义函数等功能。
- StarRocks可以完整支持TPC-H的22个SQL和TPC-DS的99个SQL。
- StarRocks兼容MySQL协议语法，可以使用现有的各种客户端工具、BI软件访问StarRocks，对StarRocks中的数据进行拖拽式分析。
流批一体
- StarRocks支持实时和批量两种数据导入方式。
- StarRocks支持的数据源有Kafka、HDFS和本地文件。
- StarRocks支持的数据格式有ORC、Parquet和CSV等。
- StarRocks可以实时消费Kafka数据来完成数据导入，保证数据不丢不重（exactly once）。
- StarRocks也可以从本地或者远程（HDFS）批量导入数据。
高可用易扩展
- StarRocks的元数据和数据都是多副本存储，并且实例中服务有热备，多实例部署，避免了单点故障。
- 实例具有自愈能力，可弹性恢复，节点的宕机、下线和异常都不会影响StarRocks实例服务的整体稳定性。
- StarRocks采用分布式架构，存储容量和计算能力可近乎线性水平扩展。StarRocks单实例的节点规模可扩展到数百节点，数据规模可达到10 PB级别。
- 扩缩容期间无需停服，可以正常提供查询服务。
- StarRocks中表模式热变更，可通过一条简单SQL命令动态地修改表的定义，例如增加列、减少列和新建物化视图等。同时，处于模式变更中的表也可以正常导入和查询数据。
数据湖分析
StarRocks不仅能高效的分析本地存储的数据，也可以作为计算引擎直接分析数据湖中的数据，支持Apache Hive、Apache Iceberg、Apache Hudi等数据组织结构；支持Parquet、ORC、CSV等文件格式；也支持HDFS、S3、OSS等存储方式。
在数据湖分析场景中，StarRocks主要负责数据的计算分析，而数据湖则主要负责数据的存储、组织和维护。使用数据湖的优势在于可以使用开放的存储格式和灵活多变的Schema定义方式，可以让BI、AI、AdHoc、报表等业务维持在统一的单点身份可信（Single Source of Truth），而StarRocks作为数据湖的计算引擎，可以充分发挥向量化引擎和CBO（Cost Based Optimizer）的优势，大大提升了数据湖分析的性能。
-----------------
官方文档：
硬件要求：部署BE服务的机器 CPU 需支持 AVX2 指令集，查看命令cat /proc/cpuinfo | grep avx2 有结果输出即表明CPU支持。如果无输出，建议更换支持AVX2的机器。
概念介绍
StarRocks架构的核心为FE（Frontend）和BE（Backend）两类进程，不依赖任何外部组件，方便部署与维护。FE和BE都可以在线水平扩展，元数据和业务数据都有副本机制，确保整个系统无单点。StarRocks提供MySQL协议接口，支持标准的SQL语法，可以通过MySQL客户端方便地查询和分析StarRocks中的数据。
- FE、BE、Broker、CN
| 名词 | 描述 |
|---|---|
| FE | FrontEnd简称FE，是StarRocks的前端节点，负责管理元数据，管理客户端连接，进行查询规划，查询调度等工作。 |
| BE | BackEnd简称BE，是StarRocks的后端节点，负责数据存储，计算执行，以及compaction，副本管理等工作。 |
| Broker | Broker，是StarRocks中和外部HDFS/对象存储等外部数据对接的中转服务，辅助提供导入导出功能。 |
| CN【公测】 | CN 节点（Compute Node）是一种无状态的计算服务，自身不维护数据，可以承担部分 SQL 计算。 |
- 分区、分桶、副本数
| 名词 | 描述 |
|---|---|
| PARTITION BY RANGE() | 分区，⽤于将数据划分成不同的区间。分区的主要作⽤是将⼀张表按照分区键拆分成不同的管理单元。当前分区键仅支持日期类型和整数类型 |
| DISTRIBUTED BY HASH() | 分区的下⼀级是分桶，StarRocks 采⽤ Hash 算法作为分桶算法。在同⼀分区内，分桶键哈希值相同的数据形成 Tablet。分桶数量（建议）= BE节点数量 * CPU 核数 |
| replication_num | 副本数，就是同一个Tablet保存的份数。 |
FE
FE是StarRocks的前端节点，负责管理元数据、管理客户端连接、进行查询规划、查询调度等工作。每个FE节点都会在内存保留一份完整的元数据，这样每个FE节点都能够提供无差别的服务。
FE根据配置有Follower和Observer两种角色。Follower会通过类Paxos的Berkeley DB Java Edition（BDBJE）协议选举出一个Leader。三者详细介绍如下：
- Leader
- 提供元数据读写服务。仅Leader节点会对元数据进行写操作，Follower和Observer只有读取权限。Follower和Observer将元数据写入请求路由到Leader节点，Leader更新完数据后，会通过BDB JE同步给Follower和Observer。必须有半数以上的Follower节点同步成功才算元数据写入成功。
- Leader从Follower中选出，进行选主需要实例中有半数以上的Follower节点存活。如果Leader节点失败，则Follower会发起新一轮选举。
- Follower
- 只有元数据读取权限，无写入权限。通过回放Leader的元数据日志来异步同步数据。
- 参与Leader选举，必须有半数以上的Follower节点存活才能进行选主。
- Observer
- 主要用于扩展实例的查询并发能力，可选部署。
- 不参与选主，不会增加实例选主压力。
- 通过回放Leader的元数据日志来异步同步数据。
BE
BE是StarRocks的后端节点，负责数据存储、SQL执行等工作。
-
数据存储方面，StarRocks的BE节点都是完全对等的，FE按照一定策略将数据分配到对应的BE节点。BE负责将导入数据写成对应的格式存储，并生成相关索引。
在进行Stream Load导入数据时，FE会选定一个BE节点作为Coordinator BE，负责将数据分发到其他BE节点，导入的最终结果由Coordinator BE返回。
-
在执行SQL计算时，一条SQL语句首先会按照具体的语义规划成逻辑执行单元，然后再按照数据的分布情况拆分成具体的物理执行单元。物理执行单元会在对应的数据存储节点上执行，实现本地计算，避免数据的传输与拷贝，从而极大的提升了查询性能。
一、拉取镜像
- 前端节点，负责管理元数据，管理客户端连接，进行查询规划，查询调度等工作
docker pull renzicu/starrocks-fe:2.5.0
docker pull starrocks/fe-ubuntu:3.0.0
- 后端节点，负责数据存储，计算执行，以及compaction，副本管理等工作.
docker pull renzicu/starrocks-be:2.5.0
docker pull starrocks/be-ubuntu:3.0.0
# 机器 CPU 不支持 AVX2 指令集，下面的BE命令都需要换成此镜像
docker pull renzicu/starrocks-be-noavx:2.5.0
# 和外部HDFS/对象存储等外部数据对接的中转服务，辅助提供导入导出功能
docker pull renzicu/starrocks-broker:2.5.0
二、准备工作
将容器中配置文件映射到本地，方便日后调整配置参数
# 本地存储StarRocks根目录
export StarRocks_Home='/mnt/Docker/017.Temp_StarRocks/'
export StarRocks_Ver='3.0.0'
# 提前创建需要保存配置文件的目录，示自己情况创建
mkdir -p $StarRocks_Home/fe-v3/conf $StarRocks_Home/be-v3/conf $StarRocks_Home/broker/conf
# 创建临时容器，导出FE配置文件
docker create --name Starrocks-Fe-V3 starrocks/fe-ubuntu:${StarRocks_Ver} \
&& docker cp Starrocks-Fe-V3:/starrocks/conf/fe.conf $StarRocks_Home/fe-v3/conf/fe.conf \
&& docker rm Starrocks-Fe-V3
# 创建临时容器，导出BE配置文件
docker create --name Starrocks-Be-V3 starrocks/fe-ubuntu:${StarRocks_Ver} \
&& docker cp Starrocks-Be-V3:/starrocks/conf/be.conf $StarRocks_Home/be-v3/conf/be.conf \
&& docker rm Starrocks-Be-V3
# 创建临时容器，导出Broker配置文件
docker create --name starrocks-broker renzicu/starrocks-broker:${StarRocks_Ver} \
&& docker cp starrocks-broker:/starrocks/conf/apache_hdfs_broker.conf $StarRocks_Home/broker/conf/apache_hdfs_broker.conf \
&& docker rm starrocks-broker
三、FE启动和配置
注意
当新的FE节点首次被添加到已有集群并启动时，您需要指定集群中现有的一个节点作为 helper 节点，并从该节点获得集群的所有 FE 节点的配置信息，才能建立通信连接。因此，在首次启动新 FE 节点时候，您需要通过命令行指定 –helper 参数。详见官方文档：部署 FE 高可用集群
当一台机器拥有多个非回环 IP 地址时，需要在 FE 配置文件 conf/fe.conf 中设置 priority_networks，为该节点设定唯一 IP。
添加节点到现有集群步骤（顺序不能乱，否则会加入集群失败）
- 先执行SQL语句添加节点
ALTER SYSTEM ADD FOLLOWER/OBSERVER "host:edit_log_port";- 然后再启动程序并指定 –helper 参数。
加入集群失败后，可使用如下方法重新加入
- 执行SQL语句删除节点
ALTER SYSTEM DROP FOLLOWER/OBSERVER "host:edit_log_port";- 清除容器内
/starrocks/meta/*目录和/starrocks/HELPER_READY文件。- 执行SQL语句添加节点
ALTER SYSTEM ADD FOLLOWER/OBSERVER "host:edit_log_port";- 重启先前创建的容器
作者封装的镜像对参数做了包装，见示例中-e OPTS=参数
预期网络创建
docker network create --subnet=172.19.0.0/24 Flink_Network
docker run -itd \
# http_port、query_port
-p 8030:8030 -p 9030:9030 \
# edit_log_port、rpc_port
-p 9010:9010 -p 9020:9020 \
--restart unless-stopped \
--name starrocks-fe1 \
# -e OPTS="--helper 172.17.0.1:9010" \
-v /etc/localtime:/etc/localtime:ro \
-v /mnt/Docker/017.StarRocks-V2/fe/conf/fe.conf:/starrocks/conf/fe.conf \
-v /mnt/Docker/017.StarRocks-V2/fe/meta:/starrocks/meta \
-v /mnt/Docker/017.StarRocks-V2/fe/log:/starrocks/log \
renzicu/starrocks-fe:2.5.0
定制版启动
docker run -itd \
-p 8030:8030 -p 9030:9030 \
-p 9010:9010 -p 9020:9020 \
--restart unless-stopped \
--name Starrocks-Fe-V2-1 \
-v /etc/localtime:/etc/localtime:ro \
-v /mnt/000.Docker/017.StarRocks-V2/fe/conf/fe.conf:/starrocks/conf/fe.conf \
-v /mnt/000.Docker/017.StarRocks-V2/fe/meta:/starrocks/meta \
-v /mnt/000.Docker/017.StarRocks-V2/fe/log:/starrocks/log \
--network 017starrocks-v2_Flink_Network \
renzicu/starrocks-fe:2.5.0
docker run -itd \
-p 8030:8030 -p 9030:9030 \
-p 9010:9010 -p 9020:9020 \
--restart unless-stopped \
--name Starrocks-Fe-V2 \
-v /etc/localtime:/etc/localtime:ro \
-v ./fe/conf/fe.conf:/starrocks/conf/fe.conf \
-v ./fe/meta:/starrocks/meta \
-v ./fe/log:/starrocks/log \
--network 017starrocks-v2_Flink_Network \
renzicu/starrocks-fe:2.5.0
docker run -itd \
-p 8032:8030 -p 9032:9030 \
-p 9012:9010 -p 9022:9020 \
--restart unless-stopped \
--name Starrocks-Fe-V2-2 \
-v /etc/localtime:/etc/localtime:ro \
renzicu/starrocks-fe:2.5.0
docker run -itd \
-p 8033:8030 -p 9033:9030 \
-p 9013:9010 -p 9023:9020 \
--restart unless-stopped \
--name Starrocks-Fe-V2-3 \
-v /etc/localtime:/etc/localtime:ro \
renzicu/starrocks-fe:2.5.0
docker run -itd \
-p 8034:8030 -p 9034:9030 \
-p 9014:9010 -p 9024:9020 \
--restart unless-stopped \
--name Starrocks-Fe-V2-4 \
-v /etc/localtime:/etc/localtime:ro \
renzicu/starrocks-fe:2.5.0
docker run -itd \
-p 8035:8030 -p 9035:9030 \
-p 9015:9010 -p 9025:9020 \
--restart unless-stopped \
--name Starrocks-Fe-V2-5 \
-v /etc/localtime:/etc/localtime:ro \
renzicu/starrocks-fe:2.5.0
预期网络创建
docker network create --subnet=172.19.0.0/24 Flink_Network
V 3.0.0启动方式
#!/bin/bash
# author:WinJayX
# date:2020-05-07
# Maintainer WinJayX <WinJayX@Gmail.com>
# func:Docker Demo
# !!! -p 参数说明 -p ：前的为宿主机的端口，后的为容器业务所需的端口!!!
# --privileged=true 特权模式 --restart unless-stopped 重启策略，除非停止的状态
docker stop Starrocks-Fe-V3-1
docker rm -f Starrocks-Fe-V3-1
docker container run -itd \
--user root \
-p 8030:8030 \
-p 9030:9030 \
-p 9010:9010 \
-p 9020:9020 \
--ip 172.19.0.11 \
--net Flink_Network \
--restart unless-stopped \
--name Starrocks-Fe-V3-1 \
--volume /etc/localtime:/etc/localtime:ro \
--volume `pwd`/SR_Data/fe_data/log-1/:/opt/starrocks/fe/log/\
--volume `pwd`/SR_Data/fe_data/meta-1/:/opt/starrocks/fe/meta/ \
starrocks/fe-ubuntu:3.0.1 /bin/bash -c /opt/starrocks/fe/bin/start_fe.sh --daemon
docker stop Starrocks-Fe-V3-2
docker rm -f Starrocks-Fe-V3-2
docker container run -itd \
--user root \
-p 8032:8030 \
-p 9032:9030 \
-p 9012:9010 \
-p 9022:9020 \
--ip 172.19.0.12 \
--net Flink_Network \
--restart unless-stopped \
--name Starrocks-Fe-V3-2 \
--volume /etc/localtime:/etc/localtime:ro \
--volume `pwd`/SR_Data/fe_data/log-2/:/opt/starrocks/fe/log/\
--volume `pwd`/SR_Data/fe_data/meta-2/:/opt/starrocks/fe/meta/ \
starrocks/fe-ubuntu:3.0.1 /bin/bash -c /opt/starrocks/fe/bin/start_fe.sh --helper Starrocks-Fe-V3-1:9010 --daemon
docker stop Starrocks-Fe-V3-3
docker rm -f Starrocks-Fe-V3-3
docker container run -itd \
--user root \
-p 8033:8030 \
-p 9033:9030 \
-p 9013:9010 \
-p 9023:9020 \
--ip 172.19.0.13 \
--net Flink_Network \
--restart unless-stopped \
--name Starrocks-Fe-V3-3 \
--volume /etc/localtime:/etc/localtime:ro \
--volume `pwd`/SR_Data/fe_data/log-3/:/opt/starrocks/fe/log/\
--volume `pwd`/SR_Data/fe_data/meta-3/:/opt/starrocks/fe/meta/ \
starrocks/fe-ubuntu:3.0.1 /bin/bash -c /opt/starrocks/fe/bin/start_fe.sh --helper Starrocks-Fe-V3-1:9010 --daemon
-v /etc/localtime:/etc/localtime:ro \
-v /mnt/Docker/017.StarRocks-V2/fe/conf/fe.conf:/starrocks/conf/fe.conf \
-v /mnt/Docker/017.StarRocks-V2/fe/meta:/starrocks/meta \
-v /mnt/Docker/017.StarRocks-V2/fe/log:/starrocks/log \
renzicu/starrocks-fe:2.5.0
当 Docker 容器成功启动后，运行以下命令连接 StarRocks。详细权限配置见官网：用户权限及认证
- 容器内执行
-- 容器内已集成mysql，可在容器内执行。默认用户root，密码为空
mysql -uroot -h127.0.0.1 -P9030
-- 修改root密码
SET PASSWORD FOR 'root' = PASSWORD('xxxxxxxxxxxxxxx');
SET PASSWORD FOR 'root' = PASSWORD('P@88W0rd');
-- 创建新用户lit_sr
CREATE USER lit_sr@'%' IDENTIFIED WITH mysql_native_password BY 'P@88W0rd';
-- 删除用户
DROP USER 'user_identity';
-- 授予所有权限
GRANT SELECT_PRIV ON *.* TO 'lit_sr'@'%';
-- -----------------------添加/删除 FE、BE、Broker-------------------
-- 添加 Follower FE 节点
ALTER SYSTEM ADD FOLLOWER "fe_host:edit_log_port";
ALTER SYSTEM ADD FOLLOWER "starrocks-fe-2:9010";
ALTER SYSTEM ADD FOLLOWER "starrocks-fe-3:9010";
# V3-单独Docker部署的启动后添加
ALTER SYSTEM ADD FOLLOWER "Starrocks-Fe-V3-2:9010";
ALTER SYSTEM ADD FOLLOWER "Starrocks-Fe-V3-3:9010";
-- 删除 Follower FE 节点
ALTER SYSTEM DROP FOLLOWER "Starrocks-Fe-V3-2:9010";
ALTER SYSTEM DROP FOLLOWER "Starrocks-Fe-V3-3:9010";
-- 添加 Observer FE 节点
ALTER SYSTEM ADD OBSERVER "fe_host:edit_log_port";
ALTER SYSTEM ADD OBSERVER "172.18.0.19:9010";
ALTER SYSTEM ADD OBSERVER "172.18.0.20:9010";
-- 删除 Observer FE 节点
ALTER SYSTEM DROP OBSERVER "fe_host:edit_log_port";
-- 查看 FE 节点状态
SHOW PROC '/frontends'\G;
-- 添加 BE 节点，先将下面的BE容器启动完成后再操作。
ALTER SYSTEM ADD BACKEND "be_host:heartbeat_port";
ALTER SYSTEM ADD BACKEND "172.22.0.7:9050";
ALTER SYSTEM ADD BACKEND "172.22.0.4:9050";
ALTER SYSTEM ADD BACKEND "172.22.0.3:9050";
ALTER SYSTEM ADD BACKEND "172.22.0.5:9050";
ALTER SYSTEM ADD BACKEND "172.22.0.8:9050";
ALTER SYSTEM ADD BACKEND "starrocks-be-1:9050";
ALTER SYSTEM ADD BACKEND "starrocks-be-2:9050";
ALTER SYSTEM ADD BACKEND "starrocks-be-3:9050";
ALTER SYSTEM ADD BACKEND "starrocks-be-1:9050";
ALTER SYSTEM ADD BACKEND "starrocks-be-1:9050";
# V3-单独Docker部署的启动后添加
ALTER SYSTEM ADD BACKEND "Starrocks-Be-V3-1:9050";
ALTER SYSTEM ADD BACKEND "Starrocks-Be-V3-2:9050";
ALTER SYSTEM ADD BACKEND "Starrocks-Be-V3-3:9050";
-- 删除 BE 节点
ALTER SYSTEM DROP BACKEND "be_host:heartbeat_port";
-- 查看 BE 节点状态
SHOW PROC '/backends'\G;
-- 添加 Broker 节点
ALTER SYSTEM ADD BROKER broker_name "broker_host:port";
-- 删除 Broker 节点
ALTER SYSTEM DROP BROKER broker_name "broker_host:port";
-- 查看 Broker 节点状态
SHOW PROC '/brokers'\G;
四、BE启动和配置
注意
当一台机器拥有多个非回环 IP 地址时，需要在 BE 配置文件 conf/be.conf 中设置 priority_networks，为该节点设定唯一 IP。 特别说明：CPU 不支持 AVX2，需使用
renzicu/starrocks-be-noavx相对应版本镜像。
docker run -d \
# heartbeat_port、be_port
-p 9050:9050 -p 9060:9060 \
# webserver_port、brpc_port
-p 8040:8040 -p 8060:8060 \
--restart unless-stopped \
--name starrocks-be1 \
-v /etc/localtime:/etc/localtime:ro \
-v /mnt/Docker/017.StarRocks-V2/be/conf/be.conf:/starrocks/conf/be.conf \
-v /mnt/Docker/017.StarRocks-V2/be/storage:/starrocks/storage \
-v /mnt/Docker/017.StarRocks-V2/be/log:/starrocks/log \
renzicu/starrocks-be:2.5.0
定制化启动方式
docker run -d \
-p 9050:9050 -p 9060:9060 \
-p 8040:8040 -p 8060:8060 \
--restart unless-stopped \
--name Starrocks-Be-V2 \
-v /etc/localtime:/etc/localtime:ro \
-v /mnt/Docker/017.StarRocks-V2/be/conf/be.conf:/starrocks/conf/be.conf \
-v /mnt/Docker/017.StarRocks-V2/be/storage:/starrocks/storage \
-v /mnt/Docker/017.StarRocks-V2/be/log:/starrocks/log \
renzicu/starrocks-be:2.5.0
docker run -d \
-p 9051:9050 -p 9061:9060 \
-p 8041:8040 -p 8061:8060 \
--restart unless-stopped \
--name Starrocks-Be-V2-2 \
-v /etc/localtime:/etc/localtime:ro \
renzicu/starrocks-be:2.5.0
docker run -d \
-p 9053:9050 -p 9063:9060 \
-p 8043:8040 -p 8063:8060 \
--restart unless-stopped \
--name Starrocks-Be-V2-3 \
-v /etc/localtime:/etc/localtime:ro \
renzicu/starrocks-be:2.5.0
docker run -d \
-p 9054:9050 -p 9064:9060 \
-p 8044:8040 -p 8064:8060 \
--restart unless-stopped \
--name Starrocks-Be-V2-4 \
-v /etc/localtime:/etc/localtime:ro \
renzicu/starrocks-be:2.5.0
docker run -d \
-p 9055:9050 -p 9065:9060 \
-p 8045:8040 -p 8065:8060 \
--restart unless-stopped \
--name Starrocks-Be-V2-5 \
-v /etc/localtime:/etc/localtime:ro \
renzicu/starrocks-be:2.5.0
V3.0.0启动方式
#!/bin/bash
# author:WinJayX
# date:2020-05-07
# Maintainer WinJayX <WinJayX@Gmail.com>
# func:Docker Demo
# !!! -p 参数说明 -p ：前的为宿主机的端口，后的为容器业务所需的端口!!!
# --privileged=true 特权模式 --restart unless-stopped 重启策略，除非停止的状态
# --volume /etc/localtime:/etc/localtime:ro \
docker stop Starrocks-Be-V3-1
docker rm -f Starrocks-Be-V3-1
docker container run -itd \
--user root \
-p 9050:9050 \
-p 9060:9060 \
-p 8040:8040 \
-p 8060:8060 \
--ip 172.19.0.21 \
--net Flink_Network \
--restart unless-stopped \
--name Starrocks-Be-V3-1 \
--volume /etc/localtime:/etc/localtime:ro \
--volume `pwd`/SR_Data/be_data/log-1/:/opt/starrocks/be/log/ \
--volume `pwd`/SR_Data/be_data/storage-1/:/opt/starrocks/be/storage/ \
starrocks/be-ubuntu:3.0.1 /bin/bash -c /opt/starrocks/be/bin/start_be.sh --daemon
docker stop Starrocks-Be-V3-2
docker rm -f Starrocks-Be-V3-2
docker container run -itd \
--user root \
-p 9052:9050 \
-p 9062:9060 \
-p 8042:8040 \
-p 8062:8060 \
--ip 172.19.0.22 \
--net Flink_Network \
--restart always \
--name Starrocks-Be-V3-2 \
--volume /etc/localtime:/etc/localtime:ro \
--volume `pwd`/SR_Data/be_data/log-2/:/opt/starrocks/be/log/ \
--volume `pwd`/SR_Data/be_data/storage-2/:/opt/starrocks/be/storage/ \
starrocks/be-ubuntu:3.0.1 /bin/bash -c /opt/starrocks/be/bin/start_be.sh --daemon
docker stop Starrocks-Be-V3-3
docker rm -f Starrocks-Be-V3-3
docker container run -itd \
--user root \
-p 9053:9050 \
-p 9063:9060 \
-p 8043:8040 \
-p 8063:8060 \
--ip 172.19.0.23 \
--net Flink_Network \
--restart always \
--name Starrocks-Be-V3-3 \
--volume /etc/localtime:/etc/localtime:ro \
--volume `pwd`/SR_Data/be_data/log-3/:/opt/starrocks/be/log/ \
--volume `pwd`/SR_Data/be_data/storage-3/:/opt/starrocks/be/storage/ \
starrocks/be-ubuntu:3.0.1 /bin/bash -c /opt/starrocks/be/bin/start_be.sh --daemon
五、Broker启动和配置
docker run -d \
-p 8003:8000 \
--restart unless-stopped \
--name Starrocks-Broker-1 \
-v /etc/localtime:/etc/localtime:ro \
-v /mnt/Docker/017.StarRocks-V2/broker/conf/apache_hdfs_broker.conf:/starrocks/conf/apache_hdfs_broker.conf \
renzicu/starrocks-broker:2.5.0
六、访问测试
注：访问需要开放相应防火墙端口。
- 通过StarRocks自带WebUI，8030端口访问
- 使用Mysql连接工具，9030端口访问。
附录一、相关端口介绍
| 实例对象 | 端口名称 | 默认端口 | 通讯方向 | 说明 |
|---|---|---|---|---|
| FE | http_port | 8030 | FE <–> FE FE <–> 用户 | FE 节点上 HTTP 服务器的端口。 |
| FE | query_port | 9030 | FE <–> 用户 | FE 节点上 MySQL 服务器的端口。 |
| FE | edit_log_port | 9010 | FE <–> FE | FE 所在 StarRocks 集群中各 Leader FE、Follower FE、Observer FE 之间通信端口。 |
| FE | rpc_port | 9020 | BE <–> FE FE <–> FE | FE 节点上 Thrift 服务器的端口。 |
| BE | heartbeat_port | 9050 | FE <–> BE | BE上心跳服务端口（thrift），用于接收来自FE的心跳。 |
| BE | be_port | 9060 | FE <–> BE | BE 上 thrift server 的端口，用于接收来自 FE 的请求。 |
| BE | webserver_port | 8040 | BE <–> BE | BE上的HTTP Server 端口。 |
| BE | brpc_port | 8060 | FE <–> BE BE <–> BE | BE上的 BRPC 端口，可以查看 BRPC 的一些网络统计信息。 |
| Broker | broker_ipc_port | 8000 | FE <–> Broker BE <–> Broker | Broker上的thrift server，用于接收请求。 |
mysql -h172.17.16.66 -P9300 -uroot -pP@88W0rd < smt/result/flink-create.all.sql
负载
Error
20230602
- CDC V2.2.0
2023-06-02 07:50:20,525 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - --------------------------------------------------------------------------------
2023-06-02 07:50:20,527 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Preconfiguration:
2023-06-02 07:50:20,528 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] -
RESOURCE_PARAMS extraction logs:
jvm_params: -Xmx5007998883 -Xms5007998883 -XX:MaxDirectMemorySize=4429185024 -XX:MaxMetaspaceSize=268435456
dynamic_configs: -D taskmanager.memory.network.min=4294967296b -D taskmanager.cpu.cores=999.0 -D taskmanager.memory.task.off-heap.size=0b -D taskmanager.memory.jvm-metaspace.size=268435456b -D external-resources=none -D taskmanager.memory.jvm-overhead.min=1073741824b -D taskmanager.memory.framework.off-heap.size=134217728b -D taskmanager.memory.network.max=4294967296b -D taskmanager.memory.framework.heap.size=134217728b -D taskmanager.memory.managed.size=6291456093b -D taskmanager.memory.task.heap.size=4873781155b -D taskmanager.numberOfTaskSlots=999 -D taskmanager.memory.jvm-overhead.max=1073741824b
logs: WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
INFO [] - Loading configuration property: jobmanager.rpc.address, 92cd35079d84
INFO [] - Loading configuration property: jobmanager.rpc.port, 6123
INFO [] - Loading configuration property: jobmanager.bind-host, 0.0.0.0
INFO [] - Loading configuration property: jobmanager.memory.process.size, 1600m
INFO [] - Loading configuration property: taskmanager.bind-host, 0.0.0.0
INFO [] - Loading configuration property: taskmanager.memory.process.size, 1728m
INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
INFO [] - Loading configuration property: parallelism.default, 1
INFO [] - Loading configuration property: jobmanager.execution.failover-strategy, region
INFO [] - Loading configuration property: rest.address, 0.0.0.0
INFO [] - Loading configuration property: rest.bind-address, 0.0.0.0
INFO [] - Loading configuration property: blob.server.port, 6124
INFO [] - Loading configuration property: query.server.port, 6125
INFO [] - Loading configuration property: jobmanager.rpc.address, jobmanager
INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 999
INFO [] - Loading configuration property: taskmanager.memory.process.size, 16280m
INFO [] - Loading configuration property: taskmanager.memory.network.min, 4096mb
INFO [] - Loading configuration property: taskmanager.memory.network.max, 8gb
INFO [] - The derived from fraction jvm overhead memory (1.590gb (1707081753 bytes)) is greater than its max value 1024.000mb (1073741824 bytes), max value will be used instead
INFO [] - The derived from fraction network memory (1.465gb (1572864023 bytes)) is less than its min value 4.000gb (4294967296 bytes), min value will be used instead
INFO [] - Final TaskExecutor Memory configuration:
INFO [] - Total Process Memory: 15.898gb (17070817280 bytes)
INFO [] - Total Flink Memory: 14.648gb (15728640000 bytes)
INFO [] - Total JVM Heap Memory: 4.664gb (5007998883 bytes)
INFO [] - Framework: 128.000mb (134217728 bytes)
INFO [] - Task: 4.539gb (4873781155 bytes)
INFO [] - Total Off-heap Memory: 9.984gb (10720641117 bytes)
INFO [] - Managed: 5.859gb (6291456093 bytes)
INFO [] - Total JVM Direct Memory: 4.125gb (4429185024 bytes)
INFO [] - Framework: 128.000mb (134217728 bytes)
INFO [] - Task: 0 bytes
INFO [] - Network: 4.000gb (4294967296 bytes)
INFO [] - JVM Metaspace: 256.000mb (268435456 bytes)
INFO [] - JVM Overhead: 1024.000mb (1073741824 bytes)
2023-06-02 07:50:20,529 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - --------------------------------------------------------------------------------
2023-06-02 07:50:20,529 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Starting TaskManager (Version: 1.15.4, Scala: 2.12, Rev:fc679b6, Date:2023-03-08T18:49:43+01:00)
2023-06-02 07:50:20,529 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - OS current user: flink
2023-06-02 07:50:20,529 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Current Hadoop/Kerberos user: <no hadoop dependency found>
2023-06-02 07:50:20,530 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - JVM: OpenJDK 64-Bit Server VM - Eclipse Adoptium - 11/11.0.18+10
2023-06-02 07:50:20,530 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Arch: amd64
2023-06-02 07:50:20,530 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Maximum heap size: 4776 MiBytes
2023-06-02 07:50:20,530 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - JAVA_HOME: /opt/java/openjdk
2023-06-02 07:50:20,530 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - No Hadoop Dependency available
2023-06-02 07:50:20,531 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - JVM Options:
2023-06-02 07:50:20,531 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -XX:+UseG1GC
2023-06-02 07:50:20,531 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -Xmx5007998883
2023-06-02 07:50:20,531 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -Xms5007998883
2023-06-02 07:50:20,531 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -XX:MaxDirectMemorySize=4429185024
2023-06-02 07:50:20,531 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -XX:MaxMetaspaceSize=268435456
2023-06-02 07:50:20,532 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -Dlog.file=/opt/flink/log/flink--taskexecutor-0-92cd35079d84.log
2023-06-02 07:50:20,532 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2023-06-02 07:50:20,532 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
2023-06-02 07:50:20,532 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2023-06-02 07:50:20,532 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Program Arguments:
2023-06-02 07:50:20,534 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - --configDir
2023-06-02 07:50:20,534 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - /opt/flink/conf
2023-06-02 07:50:20,534 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -D
2023-06-02 07:50:20,535 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - taskmanager.memory.network.min=4294967296b
2023-06-02 07:50:20,535 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -D
2023-06-02 07:50:20,535 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - taskmanager.cpu.cores=999.0
2023-06-02 07:50:20,535 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -D
2023-06-02 07:50:20,535 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - taskmanager.memory.task.off-heap.size=0b
2023-06-02 07:50:20,536 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -D
2023-06-02 07:50:20,536 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - taskmanager.memory.jvm-metaspace.size=268435456b
2023-06-02 07:50:20,536 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -D
2023-06-02 07:50:20,536 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - external-resources=none
2023-06-02 07:50:20,536 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -D
2023-06-02 07:50:20,536 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - taskmanager.memory.jvm-overhead.min=1073741824b
2023-06-02 07:50:20,537 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -D
2023-06-02 07:50:20,537 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - taskmanager.memory.framework.off-heap.size=134217728b
2023-06-02 07:50:20,537 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -D
2023-06-02 07:50:20,537 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - taskmanager.memory.network.max=4294967296b
2023-06-02 07:50:20,537 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -D
2023-06-02 07:50:20,537 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - taskmanager.memory.framework.heap.size=134217728b
2023-06-02 07:50:20,537 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -D
2023-06-02 07:50:20,538 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - taskmanager.memory.managed.size=6291456093b
2023-06-02 07:50:20,538 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -D
2023-06-02 07:50:20,538 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - taskmanager.memory.task.heap.size=4873781155b
2023-06-02 07:50:20,538 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -D
2023-06-02 07:50:20,538 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - taskmanager.numberOfTaskSlots=999
2023-06-02 07:50:20,538 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -D
2023-06-02 07:50:20,538 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - taskmanager.memory.jvm-overhead.max=1073741824b
2023-06-02 07:50:20,539 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Classpath: /opt/flink/lib/flink-cep-1.15.4.jar:/opt/flink/lib/flink-connector-files-1.15.4.jar:/opt/flink/lib/flink-connector-starrocks-1.2.6_flink-1.15.jar:/opt/flink/lib/flink-csv-1.15.4.jar:/opt/flink/lib/flink-json-1.15.4.jar:/opt/flink/lib/flink-scala_2.12-1.15.4.jar:/opt/flink/lib/flink-shaded-zookeeper-3.5.9.jar:/opt/flink/lib/flink-sql-connector-mysql-cdc-2.2.0.jar:/opt/flink/lib/flink-table-api-java-uber-1.15.4.jar:/opt/flink/lib/flink-table-planner-loader-1.15.4.jar:/opt/flink/lib/flink-table-runtime-1.15.4.jar:/opt/flink/lib/log4j-1.2-api-2.17.1.jar:/opt/flink/lib/log4j-api-2.17.1.jar:/opt/flink/lib/log4j-core-2.17.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.17.1.jar:/opt/flink/lib/flink-dist-1.15.4.jar:::
2023-06-02 07:50:20,539 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - --------------------------------------------------------------------------------
2023-06-02 07:50:20,541 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Registered UNIX signal handlers for [TERM, HUP, INT]
2023-06-02 07:50:20,544 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Maximum number of open file descriptors is 1048576.
2023-06-02 07:50:20,556 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, 92cd35079d84
2023-06-02 07:50:20,556 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123
2023-06-02 07:50:20,556 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.bind-host, 0.0.0.0
2023-06-02 07:50:20,557 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m
2023-06-02 07:50:20,557 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.bind-host, 0.0.0.0
2023-06-02 07:50:20,557 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 1728m
2023-06-02 07:50:20,558 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2023-06-02 07:50:20,558 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1
2023-06-02 07:50:20,559 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region
2023-06-02 07:50:20,560 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: rest.address, 0.0.0.0
2023-06-02 07:50:20,560 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: rest.bind-address, 0.0.0.0
2023-06-02 07:50:20,561 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: blob.server.port, 6124
2023-06-02 07:50:20,561 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: query.server.port, 6125
2023-06-02 07:50:20,561 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, jobmanager
2023-06-02 07:50:20,561 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 999
2023-06-02 07:50:20,562 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 16280m
2023-06-02 07:50:20,562 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.network.min, 4096mb
2023-06-02 07:50:20,562 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.network.max, 8gb
2023-06-02 07:50:20,617 INFO org.apache.flink.core.fs.FileSystem [] - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
2023-06-02 07:50:20,674 INFO org.apache.flink.runtime.state.changelog.StateChangelogStorageLoader [] - StateChangelogStorageLoader initialized with shortcut names {memory,filesystem}.
2023-06-02 07:50:20,679 INFO org.apache.flink.runtime.state.changelog.StateChangelogStorageLoader [] - StateChangelogStorageLoader initialized with shortcut names {memory,filesystem}.
2023-06-02 07:50:20,694 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory [] - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2023-06-02 07:50:20,707 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-11013723736024675138.conf.
2023-06-02 07:50:20,717 INFO org.apache.flink.runtime.security.contexts.HadoopSecurityContextFactory [] - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.
2023-06-02 07:50:21,205 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils [] - Trying to select the network interface and address to use by connecting to the leading JobManager.
2023-06-02 07:50:21,205 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils [] - TaskManager will try to connect for PT10S before falling back to heuristics
2023-06-02 07:50:21,648 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Trying to connect to address jobmanager/172.18.0.8:6123
2023-06-02 07:50:21,649 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [jobmanager/172.18.0.8:6123] from local address [92cd35079d84/172.18.0.9] with timeout [200] due to: Connection refused (Connection refused)
2023-06-02 07:50:21,650 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [jobmanager/172.18.0.8:6123] from local address [/172.18.0.9] with timeout [50] due to: Connection refused (Connection refused)
2023-06-02 07:50:21,652 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [jobmanager/172.18.0.8:6123] from local address [/172.18.0.9] with timeout [50] due to: Connection refused (Connection refused)
2023-06-02 07:50:21,652 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [jobmanager/172.18.0.8:6123] from local address [/127.0.0.1] with timeout [50] due to: Invalid argument (connect failed)
2023-06-02 07:50:21,653 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [jobmanager/172.18.0.8:6123] from local address [/172.18.0.9] with timeout [1000] due to: Connection refused (Connection refused)
2023-06-02 07:50:21,654 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [jobmanager/172.18.0.8:6123] from local address [/127.0.0.1] with timeout [1000] due to: Invalid argument (connect failed)
2023-06-02 07:50:21,755 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Trying to connect to address jobmanager/172.18.0.8:6123
2023-06-02 07:50:21,756 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [jobmanager/172.18.0.8:6123] from local address [92cd35079d84/172.18.0.9] with timeout [200] due to: Connection refused (Connection refused)
2023-06-02 07:50:21,758 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [jobmanager/172.18.0.8:6123] from local address [/172.18.0.9] with timeout [50] due to: Connection refused (Connection refused)
2023-06-02 07:50:21,759 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [jobmanager/172.18.0.8:6123] from local address [/172.18.0.9] with timeout [50] due to: Connection refused (Connection refused)
2023-06-02 07:50:21,760 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [jobmanager/172.18.0.8:6123] from local address [/127.0.0.1] with timeout [50] due to: Invalid argument (connect failed)
2023-06-02 07:50:21,761 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [jobmanager/172.18.0.8:6123] from local address [/172.18.0.9] with timeout [1000] due to: Connection refused (Connection refused)
2023-06-02 07:50:21,762 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [jobmanager/172.18.0.8:6123] from local address [/127.0.0.1] with timeout [1000] due to: Invalid argument (connect failed)
2023-06-02 07:50:21,962 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Trying to connect to address jobmanager/172.18.0.8:6123
2023-06-02 07:50:21,963 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - TaskManager will use hostname/address '92cd35079d84' (172.18.0.9) for communication.
2023-06-02 07:50:22,022 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start actor system, external address 172.18.0.9:0, bind address 0.0.0.0:0.
2023-06-02 07:50:22,839 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started
2023-06-02 07:50:22,901 INFO akka.remote.RemoteActorRefProvider [] - Akka Cluster not in use - enabling unsafe features anyway because `akka.remote.use-unsafe-remote-features-outside-cluster` has been enabled.
2023-06-02 07:50:22,902 INFO akka.remote.Remoting [] - Starting remoting
2023-06-02 07:50:23,140 INFO akka.remote.Remoting [] - Remoting started; listening on addresses :[akka.tcp://flink@172.18.0.9:35157]
2023-06-02 07:50:23,325 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started at akka.tcp://flink@172.18.0.9:35157
2023-06-02 07:50:23,350 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Using working directory: WorkingDirectory(/tmp/tm_172.18.0.9:35157-8c33e7)
2023-06-02 07:50:23,361 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl [] - No metrics reporter configured, no metrics will be exposed/reported.
2023-06-02 07:50:23,367 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start actor system, external address 172.18.0.9:0, bind address 0.0.0.0:0.
2023-06-02 07:50:23,400 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started
2023-06-02 07:50:23,406 INFO akka.remote.RemoteActorRefProvider [] - Akka Cluster not in use - enabling unsafe features anyway because `akka.remote.use-unsafe-remote-features-outside-cluster` has been enabled.
2023-06-02 07:50:23,406 INFO akka.remote.Remoting [] - Starting remoting
2023-06-02 07:50:23,443 INFO akka.remote.Remoting [] - Remoting started; listening on addresses :[akka.tcp://flink-metrics@172.18.0.9:35858]
2023-06-02 07:50:23,456 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started at akka.tcp://flink-metrics@172.18.0.9:35858
2023-06-02 07:50:23,476 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/rpc/MetricQueryService_172.18.0.9:35157-8c33e7 .
2023-06-02 07:50:23,500 INFO org.apache.flink.runtime.blob.PermanentBlobCache [] - Created BLOB cache storage directory /tmp/tm_172.18.0.9:35157-8c33e7/blobStorage
2023-06-02 07:50:23,510 INFO org.apache.flink.runtime.blob.TransientBlobCache [] - Created BLOB cache storage directory /tmp/tm_172.18.0.9:35157-8c33e7/blobStorage
2023-06-02 07:50:23,518 INFO org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Enabled external resources: []
2023-06-02 07:50:23,519 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Starting TaskManager with ResourceID: 172.18.0.9:35157-8c33e7
2023-06-02 07:50:23,544 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices [] - Temporary file directory '/tmp': total 199 GB, usable 48 GB (24.12% usable)
2023-06-02 07:50:23,548 INFO org.apache.flink.runtime.io.disk.iomanager.IOManager [] - Created a new FileChannelManager for spilling of task related data to disk (joins, sorting, ...). Used directories:
/tmp/flink-io-2b91090e-8361-483a-8bf5-6e9c6f1aa4b2
2023-06-02 07:50:23,562 INFO org.apache.flink.runtime.io.network.netty.NettyConfig [] - NettyConfig [server address: /0.0.0.0, server port: 0, ssl enabled: false, memory segment size (bytes): 32768, transport type: AUTO, number of server threads: 999 (manual), number of client threads: 999 (manual), server connect backlog: 0 (use Netty's default), client connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)]
2023-06-02 07:50:23,667 INFO org.apache.flink.runtime.io.network.NettyShuffleServiceFactory [] - Created a new FileChannelManager for storing result partitions of BLOCKING shuffles. Used directories:
/tmp/flink-netty-shuffle-c7ce3290-f0c2-4415-b85d-a6cd6cc66353
2023-06-02 07:50:25,998 INFO org.apache.flink.runtime.io.network.buffer.NetworkBufferPool [] - Allocated 4096 MB for network buffer pool (number of memory segments: 131072, bytes per segment: 32768).
2023-06-02 07:50:26,017 INFO org.apache.flink.runtime.io.network.NettyShuffleEnvironment [] - Starting the network environment and its components.
2023-06-02 07:50:26,228 INFO org.apache.flink.runtime.io.network.netty.NettyClient [] - Transport type 'auto': using EPOLL.
2023-06-02 07:50:26,231 INFO org.apache.flink.runtime.io.network.netty.NettyClient [] - Successful initialization (took 213 ms).
2023-06-02 07:50:26,461 INFO org.apache.flink.runtime.io.network.netty.NettyServer [] - Transport type 'auto': using EPOLL.
2023-06-02 07:50:26,521 INFO org.apache.flink.runtime.io.network.netty.NettyServer [] - Successful initialization (took 288 ms). Listening on SocketAddress /0.0.0.0:40618.
2023-06-02 07:50:26,523 INFO org.apache.flink.runtime.taskexecutor.KvStateService [] - Starting the kvState service and its components.
2023-06-02 07:50:26,557 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.taskexecutor.TaskExecutor at akka://flink/user/rpc/taskmanager_0 .
2023-06-02 07:50:26,581 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Start job leader service.
2023-06-02 07:50:26,585 INFO org.apache.flink.runtime.filecache.FileCache [] - User file cache uses directory /tmp/flink-dist-cache-0650c23c-1586-4e9f-a861-b1471e4a78f2
2023-06-02 07:50:26,588 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Connecting to ResourceManager akka.tcp://flink@jobmanager:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000).
2023-06-02 07:50:26,894 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Resolved ResourceManager address, beginning registration
2023-06-02 07:50:27,057 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Successful registration at resource manager akka.tcp://flink@jobmanager:6123/user/rpc/resourcemanager_* under registration id c4a599469947cfdd4bfb39a6a0a1dcc0.
2023-06-02 08:44:50,580 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Receive slot request dc26e0fe0de1e153d5d029c1c4646e81 for job 27aca1c93e4339b22baaacbbb36d1152 from resource manager with leader id 00000000000000000000000000000000.
2023-06-02 08:44:50,594 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Allocated slot for dc26e0fe0de1e153d5d029c1c4646e81.
2023-06-02 08:44:50,596 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Add job 27aca1c93e4339b22baaacbbb36d1152 for job leader monitoring.
2023-06-02 08:44:50,597 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Try to register at job manager akka.tcp://flink@jobmanager:6123/user/rpc/jobmanager_2 with leader id 00000000-0000-0000-0000-000000000000.
2023-06-02 08:44:50,632 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Resolved JobManager address, beginning registration
2023-06-02 08:44:50,664 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Successful registration at job manager akka.tcp://flink@jobmanager:6123/user/rpc/jobmanager_2 for job 27aca1c93e4339b22baaacbbb36d1152.
2023-06-02 08:44:50,665 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Establish JobManager connection for job 27aca1c93e4339b22baaacbbb36d1152.
2023-06-02 08:44:50,671 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Offer reserved slots to the leader of job 27aca1c93e4339b22baaacbbb36d1152.
2023-06-02 08:44:50,716 INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl [] - Activate slot dc26e0fe0de1e153d5d029c1c4646e81.
2023-06-02 08:44:50,744 INFO org.apache.flink.runtime.state.changelog.StateChangelogStorageLoader [] - Creating a changelog storage with name 'memory'.
2023-06-02 08:44:50,760 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Received task Source: as_person_assets_allot_src[1] -> ConstraintEnforcer[2] -> Sink: as_person_assets_allot_sink[2] (1/1)#0 (7c41006a36141aa60bb873b092c0d98a), deploy into slot with allocation id dc26e0fe0de1e153d5d029c1c4646e81.
2023-06-02 08:44:50,761 INFO org.apache.flink.runtime.taskmanager.Task [] - Source: as_person_assets_allot_src[1] -> ConstraintEnforcer[2] -> Sink: as_person_assets_allot_sink[2] (1/1)#0 (7c41006a36141aa60bb873b092c0d98a) switched from CREATED to DEPLOYING.
2023-06-02 08:44:50,764 INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl [] - Activate slot dc26e0fe0de1e153d5d029c1c4646e81.
2023-06-02 08:44:50,769 INFO org.apache.flink.runtime.taskmanager.Task [] - Loading JAR files for task Source: as_person_assets_allot_src[1] -> ConstraintEnforcer[2] -> Sink: as_person_assets_allot_sink[2] (1/1)#0 (7c41006a36141aa60bb873b092c0d98a) [DEPLOYING].
2023-06-02 08:44:50,773 INFO org.apache.flink.runtime.blob.BlobClient [] - Downloading 27aca1c93e4339b22baaacbbb36d1152/p-7ae43ac11b505732642e76168fb8966c68a7a6e0-e81f4d2081892c6cabb9f46d61b3e734 from jobmanager/172.18.0.8:6124
2023-06-02 08:44:50,804 INFO org.apache.flink.runtime.blob.BlobClient [] - Downloading 27aca1c93e4339b22baaacbbb36d1152/p-027ab2f212c11e0f1cb7a73827d21ca9d71679a8-c443612920f5d3e27453618fc8e2c195 from jobmanager/172.18.0.8:6124
2023-06-02 08:44:51,341 INFO org.apache.flink.streaming.runtime.tasks.StreamTask [] - No state backend has been configured, using default (HashMap) org.apache.flink.runtime.state.hashmap.HashMapStateBackend@53dcf3e7
2023-06-02 08:44:51,341 INFO org.apache.flink.runtime.state.StateBackendLoader [] - State backend loader loads the state backend as HashMapStateBackend
2023-06-02 08:44:51,342 INFO org.apache.flink.streaming.runtime.tasks.StreamTask [] - Checkpoint storage is set to 'jobmanager'
2023-06-02 08:44:51,356 INFO org.apache.flink.runtime.taskmanager.Task [] - Source: as_person_assets_allot_src[1] -> ConstraintEnforcer[2] -> Sink: as_person_assets_allot_sink[2] (1/1)#0 (7c41006a36141aa60bb873b092c0d98a) switched from DEPLOYING to INITIALIZING.
2023-06-02 08:44:51,851 WARN org.apache.flink.metrics.MetricGroup [] - Name collision: Group already contains a Metric with the name 'currentEmitEventTimeLag'. Metric will not be reported.[172.18.0.9, taskmanager, 172.18.0.9:35157-8c33e7, insert-into_default_catalog.flink_source_db.as_person_assets_allot_sink, Source: as_person_assets_allot_src[1], 0]
2023-06-02 08:44:51,851 WARN org.apache.flink.metrics.MetricGroup [] - Name collision: Group already contains a Metric with the name 'sourceIdleTime'. Metric will not be reported.[172.18.0.9, taskmanager, 172.18.0.9:35157-8c33e7, insert-into_default_catalog.flink_source_db.as_person_assets_allot_sink, Source: as_person_assets_allot_src[1], 0]
2023-06-02 08:44:51,902 INFO com.starrocks.connector.flink.manager.StarRocksSinkManagerV2 [] - Flink-StarRocks-Sink-Manager start, enableAutoCommit: true, streamLoader: com.starrocks.data.load.stream.TransactionStreamLoader
2023-06-02 08:44:51,902 INFO com.starrocks.connector.flink.manager.StarRocksSinkManagerV2 [] - manager running, scanningFrequency : 50
2023-06-02 08:44:52,070 INFO com.starrocks.data.load.stream.DefaultStreamLoader [] - Default Stream Loader start, properties : {"connectTimeout":1000,"defaultTableProperties":{"chunkLimit":3221225472,"dataFormat":{},"database":"flink_source_db","enableUpsertDelete":true,"properties":{"columns":"`id`,`create_user_name`,`assets_ids`,`user_name`,`user_itcode`,`user_job_num`,`user_dept`,`user_dept_num`,`user_project`,`user_project_num`,`start_use_date`,`use_location`,`allot_approval_status`,`allot_process_instance_id`,`allot_revoke_node_id`,`status`,`create_by`,`create_time`,`update_by`,`update_time`,`__op`","db":"flink_source_db","table":"as_person_assets_allot"},"table":"as_person_assets_allot","uniqueKey":"flink_source_db-as_person_assets_allot"},"enableTransaction":true,"expectDelayTime":1000,"headers":{"format":"csv","strip_outer_array":"true"},"ioThreadCount":2,"jdbcUrl":"jdbc:mysql://10.17.0.28:9030","loadUrls":["http://10.17.0.28:8030"],"maxCacheBytes":94371840,"oldThreshold":0.9,"opAutoProjectionInJson":true,"regionBufferRatio":0.6,"scanningFrequency":50,"socketTimeout":0,"starRocksVersion":{"major":3,"minor":0,"patch":0},"tablePropertiesMap":{},"username":"root","version":"3.0.0 48f4d81d7b","writingThreshold":50,"youngThreshold":0.1}, defaultHeaders : [{"elements":[{"name":"csv","parameterCount":0,"parameters":[]}],"name":"format","value":"csv"},{"elements":[{"name":"Basic cm9vdDpQQDg4VzByZA","parameterCount":0,"parameters":[],"value":"="}],"name":"Authorization","value":"Basic cm9vdDpQQDg4VzByZA=="},{"elements":[{"name":"true","parameterCount":0,"parameters":[]}],"name":"strip_outer_array","value":"true"},{"elements":[{"name":"100-continue","parameterCount":0,"parameters":[]}],"name":"Expect","value":"100-continue"},{"elements":[{"name":"600","parameterCount":0,"parameters":[]}],"name":"timeout","value":"600"}]
2023-06-02 08:44:52,129 INFO com.starrocks.connector.flink.table.sink.StarRocksDynamicSinkFunctionV2 [] - Open sink function v2
2023-06-02 08:44:52,147 INFO org.apache.flink.runtime.taskmanager.Task [] - Source: as_person_assets_allot_src[1] -> ConstraintEnforcer[2] -> Sink: as_person_assets_allot_sink[2] (1/1)#0 (7c41006a36141aa60bb873b092c0d98a) switched from INITIALIZING to RUNNING.
2023-06-02 08:44:52,248 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Add Split: MySqlSnapshotSplit{tableId=flink_source_db.as_person_assets_allot, splitId='flink_source_db.as_person_assets_allot:0', splitKeyType=[`id` BIGINT NOT NULL], splitStart=null, splitEnd=null, highWatermark=null}
2023-06-02 08:44:52,249 INFO org.apache.flink.connector.base.source.reader.SourceReaderBase [] - Adding split(s) to reader: [MySqlSnapshotSplit{tableId=flink_source_db.as_person_assets_allot, splitId='flink_source_db.as_person_assets_allot:0', splitKeyType=[`id` BIGINT NOT NULL], splitStart=null, splitEnd=null, highWatermark=null}]
2023-06-02 08:44:52,259 INFO org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher [] - Starting split fetcher 0
2023-06-02 08:44:52,638 INFO io.debezium.relational.history.DatabaseHistoryMetrics [] - Started database history recovery
2023-06-02 08:44:52,641 INFO io.debezium.relational.history.DatabaseHistoryMetrics [] - Finished database history recovery of 0 change(s) in 2 ms
2023-06-02 08:44:52,702 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Snapshot step 1 - Determining low watermark {ts_sec=0, file=binlog.000006, pos=1264722, gtids=, row=0, event=0} for split MySqlSnapshotSplit{tableId=flink_source_db.as_person_assets_allot, splitId='flink_source_db.as_person_assets_allot:0', splitKeyType=[`id` BIGINT NOT NULL], splitStart=null, splitEnd=null, highWatermark=null}
2023-06-02 08:44:52,706 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Snapshot step 2 - Snapshotting data
2023-06-02 08:44:52,707 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exporting data from split 'flink_source_db.as_person_assets_allot:0' of table flink_source_db.as_person_assets_allot
2023-06-02 08:44:52,708 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - For split 'flink_source_db.as_person_assets_allot:0' of table flink_source_db.as_person_assets_allot using select statement: 'SELECT * FROM `flink_source_db`.`as_person_assets_allot`'
2023-06-02 08:44:52,827 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Finished exporting 115 records for split 'flink_source_db.as_person_assets_allot:0', total duration '00:00:00.119'
2023-06-02 08:44:52,829 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Snapshot step 3 - Determining high watermark {ts_sec=0, file=binlog.000006, pos=1264722, gtids=, row=0, event=0} for split MySqlSnapshotSplit{tableId=flink_source_db.as_person_assets_allot, splitId='flink_source_db.as_person_assets_allot:0', splitKeyType=[`id` BIGINT NOT NULL], splitStart=null, splitEnd=null, highWatermark=null}
2023-06-02 08:44:53,173 INFO org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher [] - Finished reading from splits [flink_source_db.as_person_assets_allot:0]
2023-06-02 08:44:53,210 INFO org.apache.flink.connector.base.source.reader.SourceReaderBase [] - Finished reading split(s) [flink_source_db.as_person_assets_allot:0]
2023-06-02 08:44:53,213 INFO org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager [] - Closing splitFetcher 0 because it is idle.
2023-06-02 08:44:53,213 INFO org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher [] - Shutting down split fetcher 0
2023-06-02 08:44:53,214 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSplitReader [] - Close current debezium reader com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader
2023-06-02 08:44:53,231 INFO io.debezium.jdbc.JdbcConnection [] - Connection gracefully closed
2023-06-02 08:44:53,233 INFO org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher [] - Split fetcher 0 exited.
2023-06-02 08:44:53,433 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Add Split: MySqlBinlogSplit{splitId='binlog-split', offset={ts_sec=0, file=binlog.000006, pos=1264722, gtids=, row=0, event=0}, endOffset={ts_sec=0, file=, pos=-9223372036854775808, row=0, event=0}, isSuspended=false}
2023-06-02 08:44:53,434 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - Read list of available databases
2023-06-02 08:44:53,444 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - list of available databases is: [flink_source_db, information_schema, mysql, performance_schema, sys]
2023-06-02 08:44:53,444 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - Read list of available tables in each database
2023-06-02 08:44:53,447 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_assets_apply_base' is filtered out of capturing
2023-06-02 08:44:53,447 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_assets_apply_person' is filtered out of capturing
2023-06-02 08:44:53,448 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_assets_apply_return' is filtered out of capturing
2023-06-02 08:44:53,448 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_assets_apply_use' is filtered out of capturing
2023-06-02 08:44:53,448 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_assets_cust_type' is filtered out of capturing
2023-06-02 08:44:53,448 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_assets_manage' is filtered out of capturing
2023-06-02 08:44:53,448 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_assets_receiver' is filtered out of capturing
2023-06-02 08:44:53,449 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_assets_sf_cargo_detail' is filtered out of capturing
2023-06-02 08:44:53,449 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_assets_sf_order' is filtered out of capturing
2023-06-02 08:44:53,449 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_assets_sf_waybill_no_info' is filtered out of capturing
2023-06-02 08:44:53,449 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_assets_sr_apply_base' is filtered out of capturing
2023-06-02 08:44:53,450 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_assets_sr_apply_detail' is filtered out of capturing
2023-06-02 08:44:53,450 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_assets_use_record' is filtered out of capturing
2023-06-02 08:44:53,450 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_assets_use_record_shengchan' is filtered out of capturing
2023-06-02 08:44:53,450 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_birthday_gift_apply' is filtered out of capturing
2023-06-02 08:44:53,450 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_cost_allocation_info' is filtered out of capturing
2023-06-02 08:44:53,450 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_cost_share_detail' is filtered out of capturing
2023-06-02 08:44:53,451 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_cost_share_detail_base' is filtered out of capturing
2023-06-02 08:44:53,451 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_cost_share_info' is filtered out of capturing
2023-06-02 08:44:53,451 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_expressage_record' is filtered out of capturing
2023-06-02 08:44:53,451 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'flink_source_db.as_gift' is filtered out of capturing
2023-06-02 08:44:53,451 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including 'flink_source_db.as_person_assets_allot' for further processing
2023-06-02 08:44:53,456 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.columns_priv' is filtered out of capturing
2023-06-02 08:44:53,456 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.component' is filtered out of capturing
2023-06-02 08:44:53,457 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.db' is filtered out of capturing
2023-06-02 08:44:53,457 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.default_roles' is filtered out of capturing
2023-06-02 08:44:53,457 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.engine_cost' is filtered out of capturing
2023-06-02 08:44:53,457 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.func' is filtered out of capturing
2023-06-02 08:44:53,457 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.general_log' is filtered out of capturing
2023-06-02 08:44:53,458 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.global_grants' is filtered out of capturing
2023-06-02 08:44:53,458 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.gtid_executed' is filtered out of capturing
2023-06-02 08:44:53,458 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.help_category' is filtered out of capturing
2023-06-02 08:44:53,458 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.help_keyword' is filtered out of capturing
2023-06-02 08:44:53,459 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.help_relation' is filtered out of capturing
2023-06-02 08:44:53,459 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.help_topic' is filtered out of capturing
2023-06-02 08:44:53,459 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.innodb_index_stats' is filtered out of capturing
2023-06-02 08:44:53,459 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.innodb_table_stats' is filtered out of capturing
2023-06-02 08:44:53,459 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.ndb_binlog_index' is filtered out of capturing
2023-06-02 08:44:53,460 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.password_history' is filtered out of capturing
2023-06-02 08:44:53,460 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.plugin' is filtered out of capturing
2023-06-02 08:44:53,460 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.procs_priv' is filtered out of capturing
2023-06-02 08:44:53,460 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.proxies_priv' is filtered out of capturing
2023-06-02 08:44:53,460 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.replication_asynchronous_connection_failover' is filtered out of capturing
2023-06-02 08:44:53,460 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.replication_asynchronous_connection_failover_managed' is filtered out of capturing
2023-06-02 08:44:53,461 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.replication_group_configuration_version' is filtered out of capturing
2023-06-02 08:44:53,461 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.replication_group_member_actions' is filtered out of capturing
2023-06-02 08:44:53,461 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.role_edges' is filtered out of capturing
2023-06-02 08:44:53,461 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.server_cost' is filtered out of capturing
2023-06-02 08:44:53,461 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.servers' is filtered out of capturing
2023-06-02 08:44:53,461 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.slave_master_info' is filtered out of capturing
2023-06-02 08:44:53,462 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.slave_relay_log_info' is filtered out of capturing
2023-06-02 08:44:53,462 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.slave_worker_info' is filtered out of capturing
2023-06-02 08:44:53,462 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.slow_log' is filtered out of capturing
2023-06-02 08:44:53,462 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.tables_priv' is filtered out of capturing
2023-06-02 08:44:53,462 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.time_zone' is filtered out of capturing
2023-06-02 08:44:53,462 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.time_zone_leap_second' is filtered out of capturing
2023-06-02 08:44:53,463 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.time_zone_name' is filtered out of capturing
2023-06-02 08:44:53,463 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.time_zone_transition' is filtered out of capturing
2023-06-02 08:44:53,463 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.time_zone_transition_type' is filtered out of capturing
2023-06-02 08:44:53,463 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'mysql.user' is filtered out of capturing
2023-06-02 08:44:53,466 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.accounts' is filtered out of capturing
2023-06-02 08:44:53,466 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.binary_log_transaction_compression_stats' is filtered out of capturing
2023-06-02 08:44:53,466 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.cond_instances' is filtered out of capturing
2023-06-02 08:44:53,466 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.data_lock_waits' is filtered out of capturing
2023-06-02 08:44:53,467 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.data_locks' is filtered out of capturing
2023-06-02 08:44:53,467 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.error_log' is filtered out of capturing
2023-06-02 08:44:53,467 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_errors_summary_by_account_by_error' is filtered out of capturing
2023-06-02 08:44:53,467 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_errors_summary_by_host_by_error' is filtered out of capturing
2023-06-02 08:44:53,467 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_errors_summary_by_thread_by_error' is filtered out of capturing
2023-06-02 08:44:53,467 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_errors_summary_by_user_by_error' is filtered out of capturing
2023-06-02 08:44:53,467 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_errors_summary_global_by_error' is filtered out of capturing
2023-06-02 08:44:53,467 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_stages_current' is filtered out of capturing
2023-06-02 08:44:53,468 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_stages_history' is filtered out of capturing
2023-06-02 08:44:53,468 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_stages_history_long' is filtered out of capturing
2023-06-02 08:44:53,468 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_stages_summary_by_account_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,469 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_stages_summary_by_host_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,469 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_stages_summary_by_thread_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,469 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_stages_summary_by_user_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,469 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_stages_summary_global_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,469 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_statements_current' is filtered out of capturing
2023-06-02 08:44:53,469 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_statements_histogram_by_digest' is filtered out of capturing
2023-06-02 08:44:53,469 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_statements_histogram_global' is filtered out of capturing
2023-06-02 08:44:53,469 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_statements_history' is filtered out of capturing
2023-06-02 08:44:53,469 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_statements_history_long' is filtered out of capturing
2023-06-02 08:44:53,470 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_statements_summary_by_account_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,470 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_statements_summary_by_digest' is filtered out of capturing
2023-06-02 08:44:53,470 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_statements_summary_by_host_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,470 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_statements_summary_by_program' is filtered out of capturing
2023-06-02 08:44:53,470 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_statements_summary_by_thread_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,470 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_statements_summary_by_user_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,470 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_statements_summary_global_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,470 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_transactions_current' is filtered out of capturing
2023-06-02 08:44:53,471 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_transactions_history' is filtered out of capturing
2023-06-02 08:44:53,471 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_transactions_history_long' is filtered out of capturing
2023-06-02 08:44:53,471 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_transactions_summary_by_account_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,471 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_transactions_summary_by_host_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,471 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_transactions_summary_by_thread_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,471 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_transactions_summary_by_user_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,471 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_transactions_summary_global_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,471 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_waits_current' is filtered out of capturing
2023-06-02 08:44:53,471 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_waits_history' is filtered out of capturing
2023-06-02 08:44:53,472 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_waits_history_long' is filtered out of capturing
2023-06-02 08:44:53,472 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_waits_summary_by_account_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,472 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_waits_summary_by_host_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,472 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_waits_summary_by_instance' is filtered out of capturing
2023-06-02 08:44:53,472 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_waits_summary_by_thread_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,472 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_waits_summary_by_user_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,472 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.events_waits_summary_global_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,472 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.file_instances' is filtered out of capturing
2023-06-02 08:44:53,472 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.file_summary_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,472 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.file_summary_by_instance' is filtered out of capturing
2023-06-02 08:44:53,472 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.global_status' is filtered out of capturing
2023-06-02 08:44:53,473 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.global_variables' is filtered out of capturing
2023-06-02 08:44:53,473 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.host_cache' is filtered out of capturing
2023-06-02 08:44:53,473 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.hosts' is filtered out of capturing
2023-06-02 08:44:53,473 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.innodb_redo_log_files' is filtered out of capturing
2023-06-02 08:44:53,473 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.keyring_component_status' is filtered out of capturing
2023-06-02 08:44:53,473 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.keyring_keys' is filtered out of capturing
2023-06-02 08:44:53,473 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.log_status' is filtered out of capturing
2023-06-02 08:44:53,473 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.memory_summary_by_account_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,473 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.memory_summary_by_host_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,474 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.memory_summary_by_thread_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,474 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.memory_summary_by_user_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,474 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.memory_summary_global_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,474 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.metadata_locks' is filtered out of capturing
2023-06-02 08:44:53,474 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.mutex_instances' is filtered out of capturing
2023-06-02 08:44:53,474 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.objects_summary_global_by_type' is filtered out of capturing
2023-06-02 08:44:53,474 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.performance_timers' is filtered out of capturing
2023-06-02 08:44:53,474 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.persisted_variables' is filtered out of capturing
2023-06-02 08:44:53,474 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.prepared_statements_instances' is filtered out of capturing
2023-06-02 08:44:53,475 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.processlist' is filtered out of capturing
2023-06-02 08:44:53,475 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.replication_applier_configuration' is filtered out of capturing
2023-06-02 08:44:53,475 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.replication_applier_filters' is filtered out of capturing
2023-06-02 08:44:53,475 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.replication_applier_global_filters' is filtered out of capturing
2023-06-02 08:44:53,475 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.replication_applier_status' is filtered out of capturing
2023-06-02 08:44:53,475 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.replication_applier_status_by_coordinator' is filtered out of capturing
2023-06-02 08:44:53,475 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.replication_applier_status_by_worker' is filtered out of capturing
2023-06-02 08:44:53,475 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.replication_asynchronous_connection_failover' is filtered out of capturing
2023-06-02 08:44:53,475 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.replication_asynchronous_connection_failover_managed' is filtered out of capturing
2023-06-02 08:44:53,475 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.replication_connection_configuration' is filtered out of capturing
2023-06-02 08:44:53,476 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.replication_connection_status' is filtered out of capturing
2023-06-02 08:44:53,476 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.replication_group_member_stats' is filtered out of capturing
2023-06-02 08:44:53,476 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.replication_group_members' is filtered out of capturing
2023-06-02 08:44:53,476 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.rwlock_instances' is filtered out of capturing
2023-06-02 08:44:53,476 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.session_account_connect_attrs' is filtered out of capturing
2023-06-02 08:44:53,476 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.session_connect_attrs' is filtered out of capturing
2023-06-02 08:44:53,476 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.session_status' is filtered out of capturing
2023-06-02 08:44:53,476 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.session_variables' is filtered out of capturing
2023-06-02 08:44:53,476 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.setup_actors' is filtered out of capturing
2023-06-02 08:44:53,476 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.setup_consumers' is filtered out of capturing
2023-06-02 08:44:53,476 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.setup_instruments' is filtered out of capturing
2023-06-02 08:44:53,477 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.setup_objects' is filtered out of capturing
2023-06-02 08:44:53,477 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.setup_threads' is filtered out of capturing
2023-06-02 08:44:53,477 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.socket_instances' is filtered out of capturing
2023-06-02 08:44:53,477 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.socket_summary_by_event_name' is filtered out of capturing
2023-06-02 08:44:53,477 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.socket_summary_by_instance' is filtered out of capturing
2023-06-02 08:44:53,477 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.status_by_account' is filtered out of capturing
2023-06-02 08:44:53,477 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.status_by_host' is filtered out of capturing
2023-06-02 08:44:53,477 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.status_by_thread' is filtered out of capturing
2023-06-02 08:44:53,477 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.status_by_user' is filtered out of capturing
2023-06-02 08:44:53,477 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.table_handles' is filtered out of capturing
2023-06-02 08:44:53,477 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.table_io_waits_summary_by_index_usage' is filtered out of capturing
2023-06-02 08:44:53,477 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.table_io_waits_summary_by_table' is filtered out of capturing
2023-06-02 08:44:53,478 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.table_lock_waits_summary_by_table' is filtered out of capturing
2023-06-02 08:44:53,478 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.threads' is filtered out of capturing
2023-06-02 08:44:53,478 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.tls_channel_status' is filtered out of capturing
2023-06-02 08:44:53,478 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.user_defined_functions' is filtered out of capturing
2023-06-02 08:44:53,478 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.user_variables_by_thread' is filtered out of capturing
2023-06-02 08:44:53,478 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.users' is filtered out of capturing
2023-06-02 08:44:53,478 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.variables_by_thread' is filtered out of capturing
2023-06-02 08:44:53,478 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'performance_schema.variables_info' is filtered out of capturing
2023-06-02 08:44:53,481 INFO com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'sys.sys_config' is filtered out of capturing
2023-06-02 08:44:53,495 WARN io.debezium.relational.history.DatabaseHistoryMetrics [] - Unable to register the MBean 'debezium.mysql:type=connector-metrics,context=schema-history,server=mysql_binlog_source': debezium.mysql:type=connector-metrics,context=schema-history,server=mysql_binlog_source
2023-06-02 08:44:54,092 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - The table schema discovery for binlog split binlog-split success
2023-06-02 08:44:54,099 INFO io.debezium.jdbc.JdbcConnection [] - Connection gracefully closed
2023-06-02 08:44:54,100 INFO org.apache.flink.connector.base.source.reader.SourceReaderBase [] - Adding split(s) to reader: [MySqlBinlogSplit{splitId='binlog-split', offset={ts_sec=0, file=binlog.000006, pos=1264722, gtids=, row=0, event=0}, endOffset={ts_sec=0, file=, pos=-9223372036854775808, row=0, event=0}, isSuspended=false}]
2023-06-02 08:44:54,101 INFO org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher [] - Starting split fetcher 1
2023-06-02 08:44:54,105 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSplitReader [] - BinlogSplitReader is created.
2023-06-02 08:44:54,121 WARN io.debezium.relational.history.DatabaseHistoryMetrics [] - Unable to register the MBean 'debezium.mysql:type=connector-metrics,context=schema-history,server=mysql_binlog_source': debezium.mysql:type=connector-metrics,context=schema-history,server=mysql_binlog_source
2023-06-02 08:44:54,121 INFO io.debezium.connector.mysql.MySqlConnection [] - Get all known binlogs from MySQL
2023-06-02 08:44:54,149 INFO com.ververica.cdc.connectors.mysql.debezium.task.context.StatefulTaskContext [] - MySQL has the binlog file 'binlog.000006' required by the connector
2023-06-02 08:44:54,149 INFO io.debezium.relational.history.DatabaseHistoryMetrics [] - Started database history recovery
2023-06-02 08:44:54,150 INFO io.debezium.relational.history.DatabaseHistoryMetrics [] - Finished database history recovery of 0 change(s) in 0 ms
2023-06-02 08:44:54,158 INFO io.debezium.util.Threads [] - Requested thread factory for connector MySqlConnector, id = mysql_binlog_source named = binlog-client
2023-06-02 08:44:54,165 INFO com.starrocks.connector.flink.manager.TransactionTableRegion [] - Flush uniqueKey : flink_source_db-as_person_assets_allot, label : null, bytes : 22033
2023-06-02 08:44:54,168 INFO com.starrocks.connector.flink.manager.TransactionTableRegion [] - Generate entity meta, db: flink_source_db, table: as_person_assets_allot, total rows : 115, entity rows : 115, entity bytes : 22148
2023-06-02 08:44:54,186 INFO com.starrocks.data.load.stream.TransactionStreamLoader [] - Transaction start, label : 713ed52f-36ca-4fd0-841b-44ed36fc63ee
2023-06-02 08:44:54,189 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Skip 0 events on streaming start
2023-06-02 08:44:54,189 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Skip 0 rows on streaming start
2023-06-02 08:44:54,190 INFO io.debezium.util.Threads [] - Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
2023-06-02 08:44:54,194 INFO com.starrocks.data.load.stream.TransactionStreamLoader [] - Transaction start, request : POST http://10.17.0.28:8030/api/transaction/begin HTTP/1.1
2023-06-02 08:44:54,194 INFO io.debezium.util.Threads [] - Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
2023-06-02 08:44:54,221 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Connected to MySQL binlog at 10.17.0.28:3306, starting at MySqlOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.mysql.Source:STRUCT}, sourceInfo=SourceInfo [currentGtid=null, currentBinlogFilename=binlog.000006, currentBinlogPosition=1264722, currentRowNumber=0, serverId=0, sourceTime=null, threadId=-1, currentQuery=null, tableIds=[], databaseName=null], partition={server=mysql_binlog_source}, snapshotCompleted=false, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], restartGtidSet=null, currentGtidSet=null, restartBinlogFilename=binlog.000006, restartBinlogPosition=1264722, restartRowsToSkip=0, restartEventsToSkip=0, currentEventLengthInBytes=0, inTransaction=false, transactionId=null]
2023-06-02 08:44:54,223 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Waiting for keepalive thread to start
2023-06-02 08:44:54,223 INFO io.debezium.util.Threads [] - Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
2023-06-02 08:44:54,324 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Keepalive thread is running
2023-06-02 08:44:54,598 INFO com.starrocks.data.load.stream.TransactionStreamLoader [] - Transaction started, body : {
"Status": "OK",
"Message": "",
"Label": "713ed52f-36ca-4fd0-841b-44ed36fc63ee",
"TxnId": 2,
"BeginTxnTimeMs": 70
}
2023-06-02 08:44:54,629 INFO com.starrocks.data.load.stream.DefaultStreamLoader [] - Stream loading, label : 713ed52f-36ca-4fd0-841b-44ed36fc63ee, region : flink_source_db-as_person_assets_allot, request : PUT http://10.17.0.28:8030/api/transaction/load HTTP/1.1
2023-06-02 08:44:54,785 INFO com.starrocks.data.load.stream.http.StreamLoadEntity [] - Entity write end, contentLength : 22148, total : 22148
2023-06-02 08:44:54,787 INFO com.starrocks.data.load.stream.DefaultStreamLoader [] - Stream load completed, label : 713ed52f-36ca-4fd0-841b-44ed36fc63ee, database : flink_source_db, table : as_person_assets_allot, body : {
"Status": "OK",
"Message": "",
"Label": "713ed52f-36ca-4fd0-841b-44ed36fc63ee",
"TxnId": 2,
"LoadBytes": 22148,
"StreamLoadPlanTimeMs": 129,
"ReceivedDataTimeMs": 0
}
2023-06-02 08:44:54,819 INFO com.starrocks.connector.flink.manager.StarRocksSinkManagerV2 [] - Receive load response, cacheByteBeforeFlush: 22033, currentCacheBytes: 0, totalFlushRows : 115
2023-06-02 08:44:54,821 INFO com.starrocks.connector.flink.manager.TransactionTableRegion [] - Stream load flushed, db: flink_source_db, table: as_person_assets_allot, label : 713ed52f-36ca-4fd0-841b-44ed36fc63ee
2023-06-02 08:44:54,823 INFO com.starrocks.connector.flink.manager.TransactionTableRegion [] - Stream load completed, db: flink_source_db, table: as_person_assets_allot, label : 713ed52f-36ca-4fd0-841b-44ed36fc63ee
2023-06-02 08:44:54,828 INFO com.starrocks.data.load.stream.TransactionStreamLoader [] - Transaction prepare, label : 713ed52f-36ca-4fd0-841b-44ed36fc63ee, request : POST http://10.17.0.28:8030/api/transaction/prepare HTTP/1.1
2023-06-02 08:44:54,940 INFO com.starrocks.data.load.stream.TransactionStreamLoader [] - Transaction prepared, body : {
"Status": "INTERNAL_ERROR",
"Message": "too many filtered rows",
"ErrorURL": "http://172.18.0.7:8040/api/_load_error_log?file=error_log_2d47645c7c4c0130_942ef90b057f5bb8"
}
2023-06-02 08:44:54,947 ERROR com.starrocks.data.load.stream.TransactionStreamLoader [] - Transaction prepare failed, db: flink_source_db, table: as_person_assets_allot, label: 713ed52f-36ca-4fd0-841b-44ed36fc63ee,
responseBody: {
"Status": "INTERNAL_ERROR",
"Message": "too many filtered rows",
"ErrorURL": "http://172.18.0.7:8040/api/_load_error_log?file=error_log_2d47645c7c4c0130_942ef90b057f5bb8"
}
errorLog: Error: Value count does not match column count. Expect 21, but got 12. Row: 44 汪利君 10079 季平 jiping 10654 业务一处 210411 联想-百应 22030067 2022-12-20T00:00 联想全球总部西区
Error: Value count does not match column count. Expect 21, but got 10. Row: 4 5467820 \N 1 10305 2022-12-20T17:12:31 10305 2022-12-20T17:12:31 0
Error: Value count does not match column count. Expect 21, but got 12. Row: 43 汪利君 10074 冉磊 ranlei 83280 业务一处 210411 联想-中国区 22030068 2022-12-20T00:00 联想-天府 成都
Error: Value count does not match column count. Expect 21, but got 10. Row: 4 5465360 \N 1 10305 2022-12-20T17:11:14 10305 2022-12-20T17:11:14
2023-06-02 08:44:54,948 ERROR com.starrocks.connector.flink.manager.TransactionTableRegion [] - TransactionTableRegion commit failed, db: flink_source_db, table: as_person_assets_allot, label: 713ed52f-36ca-4fd0-841b-44ed36fc63ee
com.starrocks.data.load.stream.exception.StreamLoadFailException: Failed to prepare transaction, please check taskmanager log for details, Transaction{database='flink_source_db', table='as_person_assets_allot', label='713ed52f-36ca-4fd0-841b-44ed36fc63ee', finish=false}
at com.starrocks.connector.flink.manager.TransactionTableRegion.commit(TransactionTableRegion.java:253) ~[flink-connector-starrocks-1.2.6_flink-1.15.jar:?]
at com.starrocks.connector.flink.manager.StarRocksSinkManagerV2.lambda$init$0(StarRocksSinkManagerV2.java:161) ~[flink-connector-starrocks-1.2.6_flink-1.15.jar:?]
at java.lang.Thread.run(Unknown Source) [?:?]
2023-06-02 08:44:54,956 ERROR com.starrocks.connector.flink.manager.StarRocksSinkManagerV2 [] - Stream load failed
com.starrocks.data.load.stream.exception.StreamLoadFailException: Failed to prepare transaction, please check taskmanager log for details, Transaction{database='flink_source_db', table='as_person_assets_allot', label='713ed52f-36ca-4fd0-841b-44ed36fc63ee', finish=false}
at com.starrocks.connector.flink.manager.TransactionTableRegion.commit(TransactionTableRegion.java:253) ~[flink-connector-starrocks-1.2.6_flink-1.15.jar:?]
at com.starrocks.connector.flink.manager.StarRocksSinkManagerV2.lambda$init$0(StarRocksSinkManagerV2.java:161) ~[flink-connector-starrocks-1.2.6_flink-1.15.jar:?]
at java.lang.Thread.run(Unknown Source) [?:?]
# 练习随记--
mysql -P9030 -h127.0.0.1 -uroot --prompt="StarRocks > "
172.18.0.12
mysql -P9030 -h10.17.0.23 -uroot --prompt="StarRocks > "
mysql -P9030 -h10.17.0.28 -uroot -pP@88W0rd --prompt="StarRocks > "
mysql -P9033 -h<YourServerIPaddr> -uroot --prompt="StarRocks > "
mysql -h172.17.16.89 -P9030 -uroot -pP@88W0rd < ./result/starrocks-create.all.sql
mysql -h10.17.0.28 -P9030 -uroot -pP@88W0rd < ./result/starrocks-create.all.sql
mysql -P9030 -hfe-c-49dc31d44d5bf0f1-internal.starrocks.aliyuncs.com -plitSoft@2022 -uadmin --prompt="StarRocks > "
mysql -P9030 -hfe-c-49dc31d44d5bf0f1-internal.starrocks.aliyuncs.com -plitSoft@2022 -uadmin < starrocks-create.1.sql
as_assets_apply_base
mysql -P9030 -h<YourServerIPaddr> -pP@88W0rd -uroot --prompt="StarRocks > "
SET PASSWORD for root = PASSWORD('P@88W0rd');
CREATE USER lit_sr@'%' IDENTIFIED WITH mysql_native_password BY 'P@88W0rd';
/opt/flink/smt/result/flink-create.all.sql
mysql -h172.17.16.66 -P9030 -uroot -pP@88W0rd < starrocks-create.all.sql
mysql -h10.17.0.28 -P9030 -uroot -pP@88W0rd < starrocks-create.all.sql
底层数据库扩充，增加数据查询效率
helm install -n starrocks -f lit-values.yaml starrocks starrocks-community/kube-starrocks
评论区