处理许多记录的一般事务问题
我正在使用一个对我来说很新的代码库，它使用 iBatis。 我需要更新或添加现有的表，可能涉及20,000+条记录。 该过程每天运行一次，并在半夜运行。
我正在从网络服务调用中获取数据。我计划获取数据，然后为每条记录填充一个模型类型对象，并将每个模型类型对象传递给某种方法，该方法将读取对象中的数据，并将数据更新/插入到表中。
示例：
ArrayList records= new ArrayList();
Foo foo= new Foo();
foo.setFirstName("Homer");
foo.setLastName("Simpson");
records.add(foo);
//make more Foo objects, and put in ArrayList.
updateOrInsert(records); //this method then iterates over the list and calls some method that does the updating/inserting
我的主要问题是如何将所有更新/插入作为事务处理。如果系统在读取用于更新/插入表的所有记录之前出现故障，我需要知道，因此我可能会返回到 Web 服务调用，并在系统正常时重试。
我使用的是Java 1.4，数据库是Oracle。
I'm working with a code base that is new to me, and it uses iBatis.
I need to update or add to an existing table, and it may involve 20,000+ records.
The process will run once per day, and run in the middle of the night.
I'm getting the data from a web services call. I plan to get the data, then populate one model type object per record, and pass each model type object to some method that will read the data in the object, and update/insert the data into the table.
Example:
ArrayList records= new ArrayList();
Foo foo= new Foo();
foo.setFirstName("Homer");
foo.setLastName("Simpson");
records.add(foo);
//make more Foo objects, and put in ArrayList.
updateOrInsert(records); //this method then iterates over the list and calls some method that does the updating/inserting
My main question is how to handle all of the updating/inserting as a transaction. If the system goes down before all of the records are read as used to update/insert the table, I need to know, so I may go back to the web services call and try again when the system is ok.
I am using Java 1.4, and the db is Oracle.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
我强烈建议您考虑使用 spring Batch - http://static.springsource.org/spring-batch/该
框架提供了批处理所需的许多基本功能 - 错误报告、事务管理、多线程、扩展、输入验证。
该框架设计得非常好并且非常易于使用。
您列出的方法可能性能不佳，因为您正在等待读取所有对象，将它们全部存储在内存中，然后插入数据库。
您可能需要考虑按如下方式设计流程：
SpringBatch 将允许您执行批量提交、控制批量提交的大小、在读取输入时执行错误处理 - 在您的情况下重试请求，在将数据写入数据库时执行错误处理。
看看吧。
I would highly recommend you consider using spring batch - http://static.springsource.org/spring-batch/
The framework provides lot of the essential features required for batch processing - error reporting, transaction management, multi-threading, scaling, input validation.
The framework is very well designed and very easy to use.
The approach you have listed might not perform very well, since you are waiting to read all objects, storing them all in memory and then inserting in the database.
You might want to consider designing the process as follows:
SpringBatch will allow you to perform batch commits, control the size of the batch commits, perform error handling when reading input - in your case retry the request, perform error handling while writing the data to the database.
Have a look at it.