Kafka 性能测试

发布于 2024-06-08 21:16:46 字数 27700 浏览 0 评论 0 收藏 0

性能测试工具是 Kafka 本身提供的用于生产者性能测试kafka-producer-perf-test.sh和用于消费者性能测试的kafka-consumer-perf-test.sh

kafka-producer-perf-test.sh命令详解

usage: producer-performance \
          [-h] \
          --topic TOPIC \
          --num-records NUM-RECORDS \
          [--payload-delimiter PAYLOAD-DELIMITER] \
          --throughput THROUGHPUT \
          [--producer-props PROP-NAME=PROP-VALUE [PROP-NAME=PROP-VALUE ...]] \
          [--producer.config CONFIG-FILE]
          [--print-metrics] \
          [--transactional-id TRANSACTIONAL-ID] 
          [--transaction-duration-ms TRANSACTION-DURATION] 
          (--record-size RECORD-SIZE | --payload-file PAYLOAD-FILE)

This tool is used to verify the producer performance.

optional arguments:
  -h, --help                 show this help message and exit
  --topic TOPIC              produce messages to this topic
  --num-records NUM-RECORDS  number of messages to produce
  --payload-delimiter PAYLOAD-DELIMITER  provides delimiter to be used when --payload-file is provided. Defaults to new line. Note that this parameter will be ignored if --payload-file is not provided. (default: \n)
  --throughput THROUGHPUT    throttle maximum message throughput to *approximately* THROUGHPUT messages/sec. Set this to -1 to disable throttling.
  --producer-props PROP-NAME=PROP-VALUE [PROP-NAME=PROP-VALUE ...]
                         kafka producer related configuration properties like bootstrap.servers,client.id etc. These configs take precedence over those passed via --producer.config.
  --producer.config CONFIG-FILE    producer config properties file.
  --print-metrics        print out metrics at the end of the test. (default: false)
  --transactional-id TRANSACTIONAL-ID
                         The transactionalId to use if transaction-duration-ms is > 0. Useful when testing the performance of concurrent transactions. (default: performance-producer-default-transactional-id)
  --transaction-duration-ms TRANSACTION-DURATION   The max age of each transaction. The commitTransaction will be called after this time has elapsed. Transactions are only enabled if this value is positive. (default: 0) either --record-size or --payload-file must be specified but not both.
  --record-size RECORD-SIZE    message size in bytes. Note that you must provide exactly one of --record-size or --payload-file.
  --payload-file PAYLOAD-FILE  file to read the message payloads from. This works only for UTF-8 encoded text files. Payloads will be read from  this file and a payload will be randomly selected when sending messages. Note that you must provide exactly one                         of --record-size or --payload-file.

kafka-consumer-perf-test.sh命令详解

This tool helps in performance test for the full zookeeper consumer
Option                                   Description
------                                   -----------
--bootstrap-server <String: server to    REQUIRED unless --broker-list
  connect to>                              (deprecated) is specified. The server
                                           (s) to connect to.
--broker-list <String: broker-list>      DEPRECATED, use --bootstrap-server
                                           instead; ignored if --bootstrap-
                                           server is specified.  The broker
                                           list string in the form HOST1:PORT1,
                                           HOST2:PORT2.
--consumer.config <String: config file>  Consumer config properties file.
--date-format <String: date format>      The date format to use for formatting
                                           the time field. See java.text.
                                           SimpleDateFormat for options.
                                           (default: yyyy-MM-dd HH:mm:ss:SSS)
--fetch-size <Integer: size>             The amount of data to fetch in a
                                           single request. (default: 1048576)
--from-latest                            If the consumer does not already have
                                           an established offset to consume
                                           from, start with the latest message
                                           present in the log rather than the
                                           earliest message.
--group <String: gid>                    The group id to consume on. (default:
                                           perf-consumer-50295)
--help                                   Print usage information.
--hide-header                            If set, skips printing the header for
                                           the stats
--messages <Long: count>                 REQUIRED: The number of messages to
                                           send or consume
--num-fetch-threads <Integer: count>     Number of fetcher threads. (default: 1)
--print-metrics                          Print out the metrics.
--reporting-interval <Integer:           Interval in milliseconds at which to
  interval_ms>                             print progress info. (default: 5000)
--show-detailed-stats                    If set, stats are reported for each
                                           reporting interval as configured by
                                           reporting-interval
--socket-buffer-size <Integer: size>     The size of the tcp RECV size.
                                           (default: 2097152)
--threads <Integer: count>               Number of processing threads.
                                           (default: 10)
--timeout [Long: milliseconds]           The maximum allowed time in
                                           milliseconds between returned
                                           records. (default: 10000)
--topic <String: topic>                  REQUIRED: The topic to consume from.
--version                                Display Kafka version.

向一个只有1个分区和1个副本的Topic主题perf-producer-test 中发送 100 万条消息，并且每条消息大小为 1024B 生产者，对应的 acks 参数为1

kafka-producer-perf-test.sh \
  --topic perf-producer-test \
  --num-records 1000000 \
  --record-size 1024 \
  --throughput -1 \
  --print-metrics \
  --producer-props bootstrap.servers=localhost:9092 acks=1

# --nurn-records：用来指定发送消息的总条数 
# --record-size：用来设置每条消息的字节数
# --throughput：用来进行限流控制，当设定的值小于0不限流，当设定的值大于0时，当发送的吞吐量大于该值时就会被阻塞一段时间。
# --print-metrics：测试完成后的Metrics信息
# --producer-props：参数用来指定生产者的配置，可同时指定多组配置，各组配置之间以空格分隔，与producer-props 参数对应的还有一个producer-config参数，它用来指定生产者的配置文件

测试结果

67164 records sent, 13430.1 records/sec (13.12 MB/sec), 1461.5 ms avg latency, 2014.0 ms max latency.
92910 records sent, 18430.9 records/sec (18.00 MB/sec), 1718.4 ms avg latency, 2038.0 ms max latency.
120060 records sent, 23741.3 records/sec (23.18 MB/sec), 1304.2 ms avg latency, 1688.0 ms max latency.
110055 records sent, 21853.7 records/sec (21.34 MB/sec), 1428.9 ms avg latency, 1660.0 ms max latency.
119745 records sent, 23949.0 records/sec (23.39 MB/sec), 1281.9 ms avg latency, 1692.0 ms max latency.
110910 records sent, 22182.0 records/sec (21.66 MB/sec), 1383.2 ms avg latency, 1516.0 ms max latency.
135255 records sent, 27051.0 records/sec (26.42 MB/sec), 1191.1 ms avg latency, 1580.0 ms max latency.
122490 records sent, 24498.0 records/sec (23.92 MB/sec), 1214.0 ms avg latency, 1527.0 ms max latency.

1000000 records sent, 22281.639929 records/sec (21.76 MB/sec), 1332.82 ms avg latency, 2038.00 ms max latency, 1357 ms 50th, 1822 ms 95th, 1955 ms 99th, 2010 ms 99.9th.

#  records sent：表示测试时发送的消息总数
#  records sec：表示以每秒发迭的消息数来统计吞吐量，括号中 MB/sec 表示以每秒发送的消息大小来统计吞吐量，
#  avg latency: 表示消息处理的平均耗时
#  max latency：表示消息处理的最大耗时 
#  50th,95th,99t,99.9th： 分别表示 50% 95 99% 99.9% 的消息处理耗时。

# 以下为测试完成后的Metrics信息
Metric Name                                                                                   Value
app-info:commit-id:{client-id=producer-1}                                                   : 66563e712b0b9f84
app-info:start-time-ms:{client-id=producer-1}                                               : 1608628521735
app-info:version:{client-id=producer-1}                                                     : 2.5.0
kafka-metrics-count:count:{client-id=producer-1}                                            : 102.000
producer-metrics:batch-size-avg:{client-id=producer-1}                                      : 15555.699
producer-metrics:batch-size-max:{client-id=producer-1}                                      : 15570.000
producer-metrics:batch-split-rate:{client-id=producer-1}                                    : 0.000
producer-metrics:batch-split-total:{client-id=producer-1}                                   : 0.000
producer-metrics:buffer-available-bytes:{client-id=producer-1}                              : 33554432.000
producer-metrics:buffer-exhausted-rate:{client-id=producer-1}                               : 0.000
producer-metrics:buffer-exhausted-total:{client-id=producer-1}                              : 0.000
producer-metrics:buffer-total-bytes:{client-id=producer-1}                                  : 33554432.000
producer-metrics:bufferpool-wait-ratio:{client-id=producer-1}                               : 0.861
producer-metrics:bufferpool-wait-time-total:{client-id=producer-1}                          : 37854962935.000
producer-metrics:compression-rate-avg:{client-id=producer-1}                                : 1.000
producer-metrics:connection-close-rate:{client-id=producer-1}                               : 0.000
producer-metrics:connection-close-total:{client-id=producer-1}                              : 0.000
producer-metrics:connection-count:{client-id=producer-1}                                    : 2.000
producer-metrics:connection-creation-rate:{client-id=producer-1}                            : 0.044
producer-metrics:connection-creation-total:{client-id=producer-1}                           : 2.000
producer-metrics:failed-authentication-rate:{client-id=producer-1}                          : 0.000
producer-metrics:failed-authentication-total:{client-id=producer-1}                         : 0.000
producer-metrics:failed-reauthentication-rate:{client-id=producer-1}                        : 0.000
producer-metrics:failed-reauthentication-total:{client-id=producer-1}                       : 0.000
producer-metrics:incoming-byte-rate:{client-id=producer-1}                                  : 114570.421
producer-metrics:incoming-byte-total:{client-id=producer-1}                                 : 5134588.000
producer-metrics:io-ratio:{client-id=producer-1}                                            : 0.224
producer-metrics:io-time-ns-avg:{client-id=producer-1}                                      : 70608.839
producer-metrics:io-wait-ratio:{client-id=producer-1}                                       : 0.561
producer-metrics:io-wait-time-ns-avg:{client-id=producer-1}                                 : 177047.069
producer-metrics:io-waittime-total:{client-id=producer-1}                                   : 25274885534.000
producer-metrics:iotime-total:{client-id=producer-1}                                        : 10079976609.000
producer-metrics:metadata-age:{client-id=producer-1}                                        : 44.621
producer-metrics:network-io-rate:{client-id=producer-1}                                     : 2975.167
producer-metrics:network-io-total:{client-id=producer-1}                                    : 133344.000
producer-metrics:outgoing-byte-rate:{client-id=producer-1}                                  : 23246321.000
producer-metrics:outgoing-byte-total:{client-id=producer-1}                                 : 1041667644.000
producer-metrics:produce-throttle-time-avg:{client-id=producer-1}                           : 0.000
producer-metrics:produce-throttle-time-max:{client-id=producer-1}                           : 0.000
producer-metrics:reauthentication-latency-avg:{client-id=producer-1}                        : NaN
producer-metrics:reauthentication-latency-max:{client-id=producer-1}                        : NaN
producer-metrics:record-error-rate:{client-id=producer-1}                                   : 0.000
producer-metrics:record-error-total:{client-id=producer-1}                                  : 0.000
producer-metrics:record-queue-time-avg:{client-id=producer-1}                               : 1329.508
producer-metrics:record-queue-time-max:{client-id=producer-1}                               : 2034.000
producer-metrics:record-retry-rate:{client-id=producer-1}                                   : 0.000
producer-metrics:record-retry-total:{client-id=producer-1}                                  : 0.000
producer-metrics:record-send-rate:{client-id=producer-1}                                    : 22429.068
producer-metrics:record-send-total:{client-id=producer-1}                                   : 1000000.000
producer-metrics:record-size-avg:{client-id=producer-1}                                     : 1110.000
producer-metrics:record-size-max:{client-id=producer-1}                                     : 1110.000
producer-metrics:records-per-request-avg:{client-id=producer-1}                             : 15.000
producer-metrics:request-latency-avg:{client-id=producer-1}                                 : 3.289
producer-metrics:request-latency-max:{client-id=producer-1}                                 : 305.000
producer-metrics:request-rate:{client-id=producer-1}                                        : 1487.617
producer-metrics:request-size-avg:{client-id=producer-1}                                    : 15623.765
producer-metrics:request-size-max:{client-id=producer-1}                                    : 15639.000
producer-metrics:request-total:{client-id=producer-1}                                       : 66672.000
producer-metrics:requests-in-flight:{client-id=producer-1}                                  : 0.000
producer-metrics:response-rate:{client-id=producer-1}                                       : 1487.650
producer-metrics:response-total:{client-id=producer-1}                                      : 66672.000
producer-metrics:select-rate:{client-id=producer-1}                                         : 3169.020
producer-metrics:select-total:{client-id=producer-1}                                        : 142758.000
producer-metrics:successful-authentication-no-reauth-total:{client-id=producer-1}           : 0.000
producer-metrics:successful-authentication-rate:{client-id=producer-1}                      : 0.000
producer-metrics:successful-authentication-total:{client-id=producer-1}                     : 0.000
producer-metrics:successful-reauthentication-rate:{client-id=producer-1}                    : 0.000
producer-metrics:successful-reauthentication-total:{client-id=producer-1}                   : 0.000
producer-metrics:waiting-threads:{client-id=producer-1}                                     : 0.000
producer-node-metrics:incoming-byte-rate:{client-id=producer-1, node-id=node--1}            : 12.562
producer-node-metrics:incoming-byte-rate:{client-id=producer-1, node-id=node-2}             : 115053.336
producer-node-metrics:incoming-byte-total:{client-id=producer-1, node-id=node--1}           : 563.000
producer-node-metrics:incoming-byte-total:{client-id=producer-1, node-id=node-2}            : 5134025.000
producer-node-metrics:outgoing-byte-rate:{client-id=producer-1, node-id=node--1}            : 2.254
producer-node-metrics:outgoing-byte-rate:{client-id=producer-1, node-id=node-2}             : 23345828.974
producer-node-metrics:outgoing-byte-total:{client-id=producer-1, node-id=node--1}           : 101.000
producer-node-metrics:outgoing-byte-total:{client-id=producer-1, node-id=node-2}            : 1041667543.000
producer-node-metrics:request-latency-avg:{client-id=producer-1, node-id=node--1}           : NaN
producer-node-metrics:request-latency-avg:{client-id=producer-1, node-id=node-2}            : 3.289
producer-node-metrics:request-latency-max:{client-id=producer-1, node-id=node--1}           : NaN
producer-node-metrics:request-latency-max:{client-id=producer-1, node-id=node-2}            : 305.000
producer-node-metrics:request-rate:{client-id=producer-1, node-id=node--1}                  : 0.045
producer-node-metrics:request-rate:{client-id=producer-1, node-id=node-2}                   : 1494.039
producer-node-metrics:request-size-avg:{client-id=producer-1, node-id=node--1}              : 50.500
producer-node-metrics:request-size-avg:{client-id=producer-1, node-id=node-2}               : 15624.232
producer-node-metrics:request-size-max:{client-id=producer-1, node-id=node--1}              : 51.000
producer-node-metrics:request-size-max:{client-id=producer-1, node-id=node-2}               : 15639.000
producer-node-metrics:request-total:{client-id=producer-1, node-id=node--1}                 : 2.000
producer-node-metrics:request-total:{client-id=producer-1, node-id=node-2}                  : 66670.000
producer-node-metrics:response-rate:{client-id=producer-1, node-id=node--1}                 : 0.045
producer-node-metrics:response-rate:{client-id=producer-1, node-id=node-2}                  : 1494.073
producer-node-metrics:response-total:{client-id=producer-1, node-id=node--1}                : 2.000
producer-node-metrics:response-total:{client-id=producer-1, node-id=node-2}                 : 66670.000
producer-topic-metrics:byte-rate:{client-id=producer-1, topic=perf-producer-test2}          : 23259932.490
producer-topic-metrics:byte-total:{client-id=producer-1, topic=perf-producer-test2}         : 1037067350.000
producer-topic-metrics:compression-rate:{client-id=producer-1, topic=perf-producer-test2}   : 1.000
producer-topic-metrics:record-error-rate:{client-id=producer-1, topic=perf-producer-test2}  : 0.000
producer-topic-metrics:record-error-total:{client-id=producer-1, topic=perf-producer-test2} : 0.000
producer-topic-metrics:record-retry-rate:{client-id=producer-1, topic=perf-producer-test2}  : 0.000
producer-topic-metrics:record-retry-total:{client-id=producer-1, topic=perf-producer-test2} : 0.000
producer-topic-metrics:record-send-rate:{client-id=producer-1, topic=perf-producer-test2}   : 22428.565
producer-topic-metrics:record-send-total:{client-id=producer-1, topic=perf-producer-test2}  : 1000000.000

消费主题perf-producer-test中的 100 万条消息

kafka-consumer-perf-test.sh \
  --topic perf-producer-test \
  --messages 1000000 \
  --print-metrics \
  --broker-list localhost:9092

测试结果

| start.time | end.time | data.consumed.in.MB | MB.sec | data.consumed.in.nMsg | nMsg.sec | rebalance.time.ms | fetch.time.ms | fetch.MB.sec | fetch.nMsg.sec
==================================================================================================
| 2020-12-22 09:28:31:384 | 2020-12-22 09:28:47:769 | 976.5625 | 59.6010 |  1000000 | 61031.4312 | 1608629311854 |  -1608629295469 | -0.0000 | -0.0006

# start.time: 起始运行时间 
# end.time: 结束运行时
# data.consumed.in.MB: 消息总量（ 单位为 MB ）
# MB.sec: 按字节大小计算的消费吞吐量（ MB单位为 MB ）、
# data.consumed.in.nMsg :消费的消息总数
# nMsg.sec: 按消息个数计算的吞吐量
# rebalance.time.ms：再平衡的时间,单位为ms 
# fetch.time.ms：拉取消息的持续时间，单位为ms 
# fetch.MB.sec: 每秒拉取消息的字节大小，单位MB 
# fetch.nMsg.sec: 每秒拉取消息的个数 ,其中 fetch.time. s= end.time - start.time - rebalance.time.ms a


Metric Name                                                                                  Value
consumer-coordinator-metrics:assigned-partitions:{client-id=consumer-perf-consumer-89119-1} : 0.000
consumer-coordinator-metrics:commit-latency-avg:{client-id=consumer-perf-consumer-89119-1} : 11.750
consumer-coordinator-metrics:commit-latency-max:{client-id=consumer-perf-consumer-89119-1} : 30.000
consumer-coordinator-metrics:commit-rate:{client-id=consumer-perf-consumer-89119-1} : 0.097
consumer-coordinator-metrics:commit-total:{client-id=consumer-perf-consumer-89119-1} : 4.000
consumer-coordinator-metrics:failed-rebalance-rate-per-hour:{client-id=consumer-perf-consumer-89119-1} : 77.983
consumer-coordinator-metrics:failed-rebalance-total:{client-id=consumer-perf-consumer-89119-1} : 1.000
consumer-coordinator-metrics:heartbeat-rate:{client-id=consumer-perf-consumer-89119-1} : 0.116
consumer-coordinator-metrics:heartbeat-response-time-max:{client-id=consumer-perf-consumer-89119-1} : 9.000
consumer-coordinator-metrics:heartbeat-total:{client-id=consumer-perf-consumer-89119-1} : 5.000
consumer-coordinator-metrics:join-rate:{client-id=consumer-perf-consumer-89119-1} : 0.022
consumer-coordinator-metrics:join-time-avg:{client-id=consumer-perf-consumer-89119-1} : 32.000
consumer-coordinator-metrics:join-time-max:{client-id=consumer-perf-consumer-89119-1} : 32.000
consumer-coordinator-metrics:join-total:{client-id=consumer-perf-consumer-89119-1} : 1.000
consumer-coordinator-metrics:last-heartbeat-seconds-ago:{client-id=consumer-perf-consumer-89119-1} : 1.000
consumer-coordinator-metrics:last-rebalance-seconds-ago:{client-id=consumer-perf-consumer-89119-1} : 16.000
consumer-coordinator-metrics:partition-assigned-latency-avg:{client-id=consumer-perf-consumer-89119-1} : 0.000
consumer-coordinator-metrics:partition-assigned-latency-max:{client-id=consumer-perf-consumer-89119-1} : 0.000
consumer-coordinator-metrics:partition-lost-latency-avg:{client-id=consumer-perf-consumer-89119-1} : NaN
consumer-coordinator-metrics:partition-lost-latency-max:{client-id=consumer-perf-consumer-89119-1} : NaN
consumer-coordinator-metrics:partition-revoked-latency-avg:{client-id=consumer-perf-consumer-89119-1} : 0.000
consumer-coordinator-metrics:partition-revoked-latency-max:{client-id=consumer-perf-consumer-89119-1} : 0.000
consumer-coordinator-metrics:rebalance-latency-avg:{client-id=consumer-perf-consumer-89119-1} : 101.000
consumer-coordinator-metrics:rebalance-latency-max:{client-id=consumer-perf-consumer-89119-1} : 101.000
consumer-coordinator-metrics:rebalance-latency-total:{client-id=consumer-perf-consumer-89119-1} : 101.000
consumer-coordinator-metrics:rebalance-rate-per-hour:{client-id=consumer-perf-consumer-89119-1} : 78.086
consumer-coordinator-metrics:rebalance-total:{client-id=consumer-perf-consumer-89119-1} : 1.000
consumer-coordinator-metrics:sync-rate:{client-id=consumer-perf-consumer-89119-1} : 0.022
consumer-coordinator-metrics:sync-time-avg:{client-id=consumer-perf-consumer-89119-1} : 24.000
consumer-coordinator-metrics:sync-time-max:{client-id=consumer-perf-consumer-89119-1} : 24.000
consumer-coordinator-metrics:sync-total:{client-id=consumer-perf-consumer-89119-1} : 1.000
consumer-fetch-manager-metrics:bytes-consumed-rate:{client-id=consumer-perf-consumer-89119-1, topic=perf-producer-test2} : 22454094.6
consumer-fetch-manager-metrics:bytes-consumed-rate:{client-id=consumer-perf-consumer-89119-1} : 22453606.095
consumer-fetch-manager-metrics:bytes-consumed-total:{client-id=consumer-perf-consumer-89119-1, topic=perf-producer-test2}：100602.000
consumer-fetch-manager-metrics:bytes-consumed-total:{client-id=consumer-perf-consumer-89119-1} : 1033000602.000
consumer-fetch-manager-metrics:fetch-latency-avg:{client-id=consumer-perf-consumer-89119-1} : 13.353
consumer-fetch-manager-metrics:fetch-latency-max:{client-id=consumer-perf-consumer-89119-1} : 288.000
consumer-fetch-manager-metrics:fetch-rate:{client-id=consumer-perf-consumer-89119-1} : 21.633
consumer-fetch-manager-metrics:fetch-size-avg:{client-id=consumer-perf-consumer-89119-1, topic=perf-producer-test2} : 1037149.199
consumer-fetch-manager-metrics:fetch-size-avg:{client-id=consumer-perf-consumer-89119-1} : 1037149.199
consumer-fetch-manager-metrics:fetch-size-max:{client-id=consumer-perf-consumer-89119-1, topic=perf-producer-test2} : 1038179.000
consumer-fetch-manager-metrics:fetch-size-max:{client-id=consumer-perf-consumer-89119-1} : 1038179.000
consumer-fetch-manager-metrics:fetch-throttle-time-avg:{client-id=consumer-perf-consumer-89119-1} : 0.000
consumer-fetch-manager-metrics:fetch-throttle-time-max:{client-id=consumer-perf-consumer-89119-1} : 0.000
consumer-fetch-manager-metrics:fetch-total:{client-id=consumer-perf-consumer-89119-1} : 996.000

分享到QQ

分享到微博