当没有重复条目时mysql重复条目错误(通过php批量加载)

发布于 2024-07-07 06:06:08 字数 651 浏览 11 评论 0原文

我正在使用 mysql (5.0.32-Debian_7etch6-log) 并且我有一个每晚运行的批量加载 php (5.2.6) 脚本(使用 Zend_DB (1.5.1) 通过 PDO),它执行以下操作:

  1. 截断一组 4 个“导入”表
  2. 将数据批量插入到这 4 个“导入”表中(也重新使用之前在表中的 id ,但我截断了整个表,所以这不应该是一个问题,对吧?)
  3. 如果一切顺利,将“live”表重命名为“temp”,将“import”表重命名为“live”,然后将“temp” ”(旧的“实时”)表“导入”

这几周以来效果很好。 现在,在整个批量加载过程中间的某个地方,我偶尔会遇到以下情况:

SQLSTATE[23000]: Integrityconstraint Violation: 1062 Duplicate entry '911' for key 1

请注意,这不是截断之前已经存在于表中的第一个 id。 当我再次手动启动脚本时,它就像一个魅力。

有任何想法吗? 剩余索引,可能与重命名有关?

另外,当我事后检查表中是否有 id 为 911 的条目时,它甚至不在那里。

I am using mysql (5.0.32-Debian_7etch6-log) and i've got a nightly running bulk load php (5.2.6) script (using Zend_DB (1.5.1) via PDO) which does the following:

  1. truncating a set of 4 'import' tables
  2. bulk inserting data into these 4 'import' tables (re-using ids that have previously been in the tables as well, but i truncated the whole table, so that shouldn't be an issue, right?)
  3. if everything goes well, rename the 'live' tables to 'temp', the 'import' tables to 'live' and then the 'temp' (old 'live') tables to 'import'

This worked great for weeks. Now I am occassionally getting this, somewhere in the middle of the whole bulk loading process:

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '911' for key 1

Mind you that, this is not the first id that has been in the table before the truncation already. When I just start the script manually again, it works like a charm.

Any ideas? leftover indexes, something to do with the renaming maybe?

In addition, when I check the table for an entry with the id 911 afterwards, it is not even in there.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(6

写下不归期 2024-07-14 06:06:09

您正在创建一条省略“id”字段(或 NULL)的新记录,
但之前您已经更新了另一条记录并将其“id”更改为“911”。
换句话说,如果采用表的 AUTO_INCRMENT 值,则无法创建另一条记录。

You are creating a new record with 'id' field omitted (or NULL),
BUT previously you have updated another record and changed it's 'id' to '911'.
In other words, you can't create another record if your table's AUTO_INCREMENT value is taken.

凡尘雨 2024-07-14 06:06:08

当 MyISAM 表损坏时,可能会发生此类错误。 通常只需在有问题的表上运行修复命令即可修复它:

> repair table mytablename;

更好的解决方案是不要对数据不断变化的表使用 MyISAM - InnoDB 更加安全,正如 Paul 正确指出的那样,您可以在 InnoDB 表上使用事务,但不能在 MyISAM 上使用事务。

顺便说一句,我会避免动态重命名表 - 这是定期执行的相当笨重的事情,并且如果在重命名过程中系统上有其他用户,可能会导致一些非常意外的结果。 为什么不这样做:

> truncate table temptable;
> truncate table importtable;

> #bulk insert new data
> insert into importtable(col1,col2,col3) 
> values(1,2,3),(4,5,6),(7,8,9);

> #now archive the live data
> insert into temptable(col1,col2,col3)
> select col1,col2,col3 from livetable;

> #finally copy the new data to live
> truncate table livetable;
> insert into livetable(col1,col2,col3)
> select col1,col2,col3 from importtable;

当然,如果您要插入大量行,那么风险是,只要插入完成,所有实时数据都不可用,但总体而言,这种方法要少得多对索引、触发器或可能链接到相关表的任何其他内容具有破坏性。

Errors like this can occur when a MyISAM table becomes corrupt. Running the repair command on the table in question is usually all that's required to fix it:

> repair table mytablename;

A better solution is not to use MyISAM for tables where the data is constantly changing - InnoDB is much more bulletproof, and as Paul correctly points out, you can use transactions on InnoDB tables, but not on MyISAM.

By the way, I would avoid renaming tables on the fly - that's a fairly clunky thing to be doing on a regular basis, and could cause some very unexpected results if you ever have other users on the system while the renaming is going on. Why not just do something like this:

> truncate table temptable;
> truncate table importtable;

> #bulk insert new data
> insert into importtable(col1,col2,col3) 
> values(1,2,3),(4,5,6),(7,8,9);

> #now archive the live data
> insert into temptable(col1,col2,col3)
> select col1,col2,col3 from livetable;

> #finally copy the new data to live
> truncate table livetable;
> insert into livetable(col1,col2,col3)
> select col1,col2,col3 from importtable;

Of course if you are inserting a very large number of rows then the risk would be that all of your live data is unavailable for as long as the insert takes to complete, but overall this approach is far less destructive to indexes, triggers or anything else that may be linked to the tables in question.

徒留西风 2024-07-14 06:06:08

显然存在一些锁定问题或其他问题,我能够通过在并行连接中向受影响的表和相关表发送“SELECT”语句来重现该行为。

现在我使用 DELETE FROM 而不是 TRUNCATE 并将 RENAME TABLE 语句(其中我一次进行了 3 次重命名)更改为一堆单个ALTER TABLE xxx RENAME TO zzz 语句并且无法再重现该错误。

所以这可能会得到解决。 也许其他人可以从我花在研究和大量试错上的时间中获益。

Apparently there were some lock issues or something, I was able to reproduce the behavior by shooting 'SELECT' statements to the affected and related tables in a parallel connection.

now i used DELETE FROM instead of TRUNCATE and changed the RENAME TABLE statements (where i did 3 renames at once each) to a bunch of single ALTER TABLE xxx RENAME TO zzz statements and can't reproduce the error any more.

so this might be solved. maybe someone else can profit from my day spent with research and a lot of try-and-error.

淡淡離愁欲言轉身 2024-07-14 06:06:08

当您的导入脚本运行时,是否可能有其他脚本插入到数据库中?

Could some other script be inserting into the database while your import script is running?

薄凉少年不暖心 2024-07-14 06:06:08

您是否尝试过启用查询日志来查看是否确实插入了重复项?

你能在你的测试环境中重现它吗? 不要在生产中启用查询日志。

如果问题确实存在,则该表可能已损坏; 这可能是由多种原因引起的,但有可能是硬件故障或电源故障。

检查mysql日志,看看最近或者期间是否出现过问题(或者崩溃)。

同样,我所能建议的就是尝试在您的测试环境中重现它。 创建大量测试数据并重复加载它们。

Have you tried enabling the query log to see if you really ARE inserting a duplicate?

Can you reproduce it in your test environment? Do not enable the query log in production.

It is possible that the table has been corrupted if the problem is genuine; this could be caused by a number of things, but dodgy hardware or power failure are possibilities.

Check the mysql log to see if it has had any problems (or crashed) recently or during the period.

Again, all I can suggest is to try to reproduce it in your test environment. Create very large loads of test data and repeatedly load them.

鹿! 2024-07-14 06:06:08

您使用交易吗? 您可以消除许多此类事务问题,特别是如果可以锁定表或将事务隔离模式设置为可序列化的话。 我不太熟悉 MySQL 上的事务,但我相信事务仅适用于 InnoDB 表(或者这可能是过时的知识)。

Are you using transactions? You can eliminate a lot of these sorts of problems with transactions, especially if it's possible to either lock the tables or set the transaction isolation mode to serializable. I'm not really familiar with those on MySQL, but I believe that transactions only work on InnoDB tables (or that could be obsolete knowledge).

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文