03 模拟数据

模拟数据

使用说明

创建mock-edu项目目录

[zhangsan@node1 ~]$ mkdir /opt/bigdata/mock/edu/
[zhangsan@node1 ~]$ mkdir /opt/bigdata/mock/edu/log

上传文件

将edu-mock-data-1.0-SNAPSHOT.jar上传到/opt/bigdata/mock/edu目录下

[zhangsan@node1 edu]$ ll
-rw-rw-r--. 1 zhangsan zhangsan 33186121 Mar 18 08:25 edu-mock-data-1.0-SNAPSHOT.jar

springboot

创建application.yml文件

可以根据需求生成对应日期的用户行为日志。

# 外部配置打开
#logging.config: "classpath:logback.xml"
logging.config: "./logback.xml"
#业务日期
mock.date: "2026-03-17"

  #模拟数据发送模式
mock.type: "log"
#mock.type: "http"
#mock.type: "kafka"
  #http模式下，发送的地址
mock.url: "http://localhost:8090/applog"

spring:
  datasource:
    type: com.alibaba.druid.pool.DruidDataSource
    druid:
      url: jdbc:mysql://node1:3306/edu?characterEncoding=utf-8&useSSL=false&serverTimezone=GMT%2B8
      username: root
      password: "123456"
      driver-class-name: com.mysql.jdbc.Driver
      max-active: 200
      test-on-borrow: true

mybatis-plus.global-config.db-config.field-strategy: not_null
mybatis-plus:
  mapper-locations: classpath:mapper/*.xml

mybatis:
   mapper-locations: classpath:mapper/*.xml

#mock:
#  kafka-server: "node1:9092,node2:9092,node3:9092"
#  kafka-topic: "ODS_BASE_LOG"

# 清空
mock.clear.busi: 0
# 清空用户
mock.clear.user: 0
#是否初始化试卷
mock.if-init-paper: 0
# 生成新用户
mock.new.user: 1000
  #启动次数
mock.user-session.count: 1000
  #设备最大值
mock.max.mid: 1000

mock.if-realtime: 0
#访问时间分布权重
mock.start-time-weight: "10:5:0:0:0:0:5:5:5:10:10:15:20:10:10:10:10:10:20:25:30:35:30:20"

#支付类型占比 支付宝 ：微信 :银联
mock.payment_type_weight: "40:50:10"
  #课程最大值
mock.max.course-id: 10
  #页面平均访问时间
mock.page.during-time-ms: 20000
  #错误概率 百分比
mock.error.rate: 3
  #每条日志发送延迟 ms
mock.log.sleep: 100
  #课程详情来源  用户查询，商品推广，智能推荐, 促销活动
mock.detail.source-type-rate: "40:25:15:20"

#领取购物券概率
mock.if_get_coupon_rate: 75
#优惠券最大id
mock.max.coupon-id: 3
  #搜索关键词
mock.search.keyword: "java,python,多线程,前端,数据库,大数据,hadoop,flink"
  #男性比例
mock.user.male-rate: 20
  #用户数据变化概率
mock.user.update-rate: 20

# 男女浏览商品比重（35sku)
mock.course-weight.male: "10:10:10:10:10:10:10:5:5:5:5:5:10:10:10:10:12:12:12:12:12:5:5:5:5:3:3:3:3:3:3:3:3:10:10"
mock.course-weight.female: "1:1:1:1:1:1:1:5:5:5:5:5:1:1:1:1:2:2:2:2:2:8:8:8:8:15:15:15:15:15:15:15:15:1:1"

创建path.json文件

path.json文件用来配置访问路径，根据需求，可以灵活配置用户点击路径。

{ "visit_path" : [
  {"path":["start_app","home","course_list","course_detail" ,"order","payment" ,"end" ],"rate":20},
  {"path":["start_app","home","course_list","course_detail" ,"order" ,"end" ],"rate":20},
  {"path":["start_app","home","course_list","course_detail" ,"course_list","course_detail" ,"order" ,"end" ],"rate":20},
  {"path":["start_app","home","course_list","course_detail","cart" ,"order" ,"payment" ,"end" ],"rate":20}
],

  "study_path": [
    {"path":["start_app","mine","course_detail","chapter_video","end" ],"rate":10},
    {"path":["start_app","mine","course_detail","exam","end" ],"rate":10}
  ]
}

创建logback配置文件

可配置日志生成路径，修改内容如下

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <property name="LOG_HOME" value="/opt/bigdata/mock/edu/log" />
    <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%msg%n</pattern>
        </encoder>
    </appender>

    <appender name="rollingFile" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOG_HOME}/app.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LOG_HOME}/app.%d{yyyy-MM-dd}.log</fileNamePattern>
        </rollingPolicy>
        <encoder>
            <pattern>%msg%n</pattern>
        </encoder>
    </appender>

    <!-- 将某一个包下日志单独打印日志 -->
    <logger name="com.studybigdata.mock.util.LogUtil"
            level="INFO" additivity="false">
         <appender-ref ref="rollingFile" />
         <appender-ref ref="console" />
    </logger>

    <root level="error"  >
        <appender-ref ref="console" />
        <!-- <appender-ref ref="async-rollingFile" />  -->
    </root>
</configuration>

集群日志及业务数据生成脚本

在node1的/home/zhangsan目录下创建bin目录，并将该目录添加到PATH环境变量，这样脚本可以在服务器的任何目录执行。

mock.sh

在/home/zhangsan/bin目录下创建脚本mock.sh，在脚本中编写如下内容。

#!/bin/bash
DATA_HOME=/opt/bigdata/mock/edu
#MAXWELL_HOME=/opt/bigdata/maxwell

function mock_data() {
  if [ $1 ]
  then
    sed -i "/mock.date/s/.*/mock.date: \"$1\"/" $DATA_HOME/application.yml
    echo "正在生成 $1 当日的数据"
  fi
  cd $DATA_HOME
      nohup java -jar "edu-mock-data-1.0-SNAPSHOT.jar" >/dev/null 2>&1
}

case $1 in
"init")
  [ $2 ] && do_date=$2 || do_date='2026-03-17'
  sed -i "/mock.clear.busi/s/.*/mock.clear.busi: 1/" $DATA_HOME/application.yml
  sed -i "/mock.clear.user/s/.*/mock.clear.user: 1/" $DATA_HOME/application.yml
  mock_data $(date -d "$do_date -5 days" +%F)
  sed -i "/mock.clear.busi/s/.*/mock.clear.busi: 0/" $DATA_HOME/application.yml
  sed -i "/mock.clear.user/s/.*/mock.clear.user: 0/" $DATA_HOME/application.yml
  for ((i=4;i>=0;i--));
  do
    mock_data $(date -d "$do_date -$i days" +%F)
  done
  ;;
[0-9][0-9][0-9][0-9]-[0-1][0-9]-[0-3][0-9])

    # sed -i "/mock_date/s/.*/mock_date=$1/" $MAXWELL_HOME/config.properties
    # mxw.sh restart
    sleep 1
    mock_data $1
    ;;
esac

注：

• DATA_HOME变量为jar包及配置文件所在路径

• /dev/null代表linux的空设备文件，所有往这个文件里面写入的内容都会丢失，俗称“黑洞”。

• 标准输入0：从键盘获得输入 /proc/self/fd/0

• 标准输出1：输出到屏幕（即控制台） /proc/self/fd/1

• 错误输出2：输出到屏幕（即控制台） /proc/self/fd/2

修改脚本执行权限

[zhangsan@node1 bin]$ chmod +x mock.sh

测试脚本

生成指定日期及前5天的数据。

[zhangsan@node1 edu]$ mock.sh init 2026-03-22
正在生成 2026-03-17 当日的数据
正在生成 2026-03-18 当日的数据
正在生成 2026-03-19 当日的数据
正在生成 2026-03-20 当日的数据
正在生成 2026-03-21 当日的数据
正在生成 2026-03-22 当日的数据

查看生成的数据

日志数据

在node1的/opt/bigdata/mock/edu/log目录上查看生成的数据

[zhangsan@node1 log]$ ls
app.log

业务数据

image-20260318092536643