DB (SQL) 自动压力/负载工具?

发布于 2024-07-22 12:55:49 字数 820 浏览 2 评论 0原文

我想测量数据库应用程序的性能和可扩展性。 我正在寻找一种工具,允许我对数据库运行许多 SQL 语句,将数据库和脚本 (SQL) 文件作为参数(+必要的详细信息,例如主机名、端口、登录名...)。

理想情况下,它应该让我控制参数,例如模拟客户端的数量、测试持续时间、随机化变量或从列表中选择(例如 SELECT FROM ... WHERE value = @var,其中 var 从命令行读取或每次执行随机化) 。 我想将测试结果保存为 CSV 或 XML 文件,以便我可以分析和绘制它们。 当然,就定价而言,我更喜欢“免费”或“演示”:-)

令人惊讶的是(至少对我来说),虽然有数十种用于 Web 应用程序负载测试的此类工具,但我找不到任何用于数据库测试的工具!? 我确实看到过,例如 pgbench,使用基于某些 TPC 场景的内置数据库,因此它们有助于测试 DBMS 配置和 H/W,但我无法测试我的数据库! 有什么建议么?

具体来说,我在 Linux 上使用 Postgres 8.3,尽管我可以使用任何满足这些要求的 DB 通用工具。 硬件具有 32GB RAM,而主表和索引的大小约为 120GB。 因此,冷缓存与热缓存运行(I/O 与 RAM)之间的响应时间比可能为 1:10。 实际上,我希望请求能够均匀分布,因此针对数据库的不同部分测试查询对我来说很重要。

请随时通过电子邮件与我联系。 谢谢!

-- Shaul Dar ([电子邮件受保护])

I want to measure the performance and scalability of my DB application. I am looking for a tool that would allow me to run many SQL statements against my DB, taking the DB and script (SQL) file as arguments (+necessary details, e.g. host name, port, login...).

Ideally it should let me control parameters such as number of simulated clients, duration of test, randomize variables or select from a list (e.g. SELECT FROM ... WHERE value = @var, where var is read from command line or randomized per execution). I would like to test results to be saved as CSV or XML file that I can analyze and plot them. And of course in terms of pricing I prefer "free" or "demo" :-)

Surprisingly (for me at least) while there are dozens of such tools for web application load testing, I couldn't find any for DB testing!? The ones I did see, such as pgbench, use a built-in DB based on some TPC scenario, so they help test the DBMS configuration and H/W but I cannot test MY DB! Any suggestions?

Specifically I use Postgres 8.3 on Linux, though I could use any DB-generic tool that meets these requirements. The H/W has 32GB of RAM while the size of the main tables and indexes is ~120GB. Hence there can be a 1:10 response time ratio between cold vs warm cache runs (I/O vs RAM). Realistically I expect requests to be spread evenly, so it's important for me to test queries against different pieces of the DB.

Feel free to also contact me via email.
Thanks!

-- Shaul Dar ([email protected])

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(5

捎一片雪花 2024-07-29 12:55:49

Apache 的 JMeter 可以处理不同的服务器类型。 我使用它对 Web 应用程序进行负载测试,团队中的其他人使用它进行数据库调用。 它可以通过多种方式进行配置以获得您需要的负载。 它可以在控制台模式下运行,甚至可以使用不同的客户端进行集群,以最大限度地减少客户端开销(从而伪造结果)。

这是一个java应用程序,乍一看有点复杂。 但我们仍然喜欢它。 :-)

JMeter from Apache can handle different server types. I use it for load tests against web applications, others in the team use it for DB calls. It can be configured in many ways to get the load you need. It can be run in console mode and even be clustered using different clients to minimize client overhead ( and so falsifying the results).

It's a java application and a bit complex at first sight. But still we love it. :-)

终止放荡 2024-07-29 12:55:49

k6.io 可以使用 xk6-sql 扩展

作为参考,测试脚本可能类似于:

import sql from 'k6/x/sql';

const db = sql.open("sqlite3", "./test.db");

export function setup() {
  db.exec(`CREATE TABLE IF NOT EXISTS keyvalues (
           id integer PRIMARY KEY AUTOINCREMENT,
           key varchar NOT NULL,
           value varchar);`);
}

export function teardown() {
  db.close();
}

export default function () {
  db.exec("INSERT INTO keyvalues (key, value) VALUES('plugin-name', 'k6-plugin-sql');");

  let results = sql.query(db, "SELECT * FROM keyvalues;");
  for (const row of results) {
    console.log(`key: ${row.key}, value: ${row.value}`);
  }
}

阅读更多内容 简短教程

k6.io can stress test a few relational databases with the xk6-sql extension.

For reference, a test script could be something like:

import sql from 'k6/x/sql';

const db = sql.open("sqlite3", "./test.db");

export function setup() {
  db.exec(`CREATE TABLE IF NOT EXISTS keyvalues (
           id integer PRIMARY KEY AUTOINCREMENT,
           key varchar NOT NULL,
           value varchar);`);
}

export function teardown() {
  db.close();
}

export default function () {
  db.exec("INSERT INTO keyvalues (key, value) VALUES('plugin-name', 'k6-plugin-sql');");

  let results = sql.query(db, "SELECT * FROM keyvalues;");
  for (const row of results) {
    console.log(`key: ${row.key}, value: ${row.value}`);
  }
}

Read more on this short tutorial.

暗恋未遂 2024-07-29 12:55:49

SQL Load Generator 是另一个这样的工具:

http://sqlloadgenerator.codeplex.com/

我喜欢它,但是它还没有保存测试设置的选项。

The SQL Load Generator is another such tool:

http://sqlloadgenerator.codeplex.com/

I like it, but it doesn't yet have the option to save test setup.

黑白记忆 2024-07-29 12:55:49

我们从未真正找到合适的解决方案来对大型机 DB2 数据库进行压力测试,因此我们最终推出了自己的解决方案。 它实际上仅由一组 30 台运行 Linux 并安装了 DB2 Connect 的 PC 组成。

其中 29 个盒子运行一个脚本,该脚本只需等待起始文件出现在 NFS 挂载上,然后开始根据数据执行固定查询。 事实上,这些查询(以及数据库中的数据)是固定的,这意味着我们可以轻松地与以前的成功运行进行比较。

第 30 个盒子连续运行两个脚本(第二个与所有其他盒子相同)。 第一个清空,然后用我们已知的数据填充数据库表,然后创建启动文件以允许所有其他机器(及其本身)继续。

这一切都是通过 bash 和 DB2 Connect 完成的,因此相当容易维护(并且免费)。

我们还有另一种变体,可以根据对数月收集的生产信息的分析进行随机查询。 对照已知的成功基线检查输出比较困难,但在这种情况下,我们只是寻找功能和性能问题(因此我们检查错误和耗时过长的查询)。

我们目前正在研究是否可以将所有这些物理服务器整合到虚拟机中,在运行 zLinux 的大型机(它将使用 TCP/IP 的共享内存 HyperSockets,基本上消除网络延迟)和带有 VMWare 的 Intel 平台上,释放一些硬件。

如果您不介意预先做一些工作,那么您应该考虑一下这个选项,因为它可以让您在后续过程中获得很大的控制权。

We never really found an adequate solution for stress testing our mainframe DB2 database so we ended up rolling our own. It actually just consists of a bank of 30 PCs running Linux with DB2 Connect installed.

29 of the boxes run a script which simply wait for a starter file to appear on an NFS mount then start executing fixed queries based on the data. The fact that these queries (and the data in the database) are fixed means we can easily compare against previous successful runs.

The 30th box runs two scripts in succession (the second is the same as all the other boxes). The first empties then populates the database tables with our known data and then creates the starter file to allow all the other machines (and itself) to continue.

This is all done with bash and DB2 Connect so is fairly easily maintainable (and free).

We also have another variant to do random queries based on analysis of production information collected over many months. It's harder to check the output against a known successful baseline but, in that circumstance, we're only looking for functional and performance problems (so we check for errors and queries that take too long).

We're currently examining whether we can consolidate all those physical servers into virtual machines, on both the mainframe running zLinux (which will use the shared-memory HyperSockets for TCP/IP, basically removing the network delays) and Intel platforms with VMWare, to free up some of that hardware.

It's an option you should examine if you don't mind a little bit of work up front since it gives you a great deal of control down the track.

紫南 2024-07-29 12:55:49

您检查过来自Continentt 的开源工具Bristlecone 吗? 我不使用它,但它适用于 Postgres,并且似乎能够完成您要求的操作。 (抱歉,作为新用户,我无法为您提供工具页面的直接链接,但 Google 会将您带到那里;o])

Did you check Bristlecone an open source tool from Continuent? I don't use it, but it works for Postgres and seems to be able to do the things that your request. (sorry as a new user, I cannot give you the direct link to the tool page, but Google will get you there ;o])

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文