将单个 txt 文件中的数千条记录导入 MySQL 数据库的更快方法?
我有大约 50 个 txt 文件,每个文件大约有 8 列和 80,000 条记录。我在 Matlab 中编写了一个脚本,它逐一读取所有文件,然后对每个文件使用单个 INSERT 语句将其放入 MySQL。然而这样做会花费大量时间(即使对于单个文件!)。我也尝试过使用 PHPmyAdmin,它说文件太大而无法上传(大约 8 MB)。因此,请建议一种解决方法以及通常导入大型 txt 文件的理想方法是什么。另外,导入 1 个此类文件通常需要多长时间?
I have like 50 txt files each with around 8 columns and 80,000 records. I have written a script in Matlab that reads all the files one by one and then puts it into MySQL using a single INSERT statement for each file. However doing so is taking a huge amount of time(even for a single file!). I have also tried using PHPmyAdmin which says the file is too big to upload (around 8 MB). So please suggest a workaround and what is the ideal way to usually import large txt files. Also how much time should it normally take to import 1 single such file?
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(4)
您需要的一切都在本手册中:http://dev。 mysql.com/doc/refman/5.0/en/insert-speed.html
All you need is in this manual: http://dev.mysql.com/doc/refman/5.0/en/insert-speed.html
使用
LOAD DATA INFILE
语法。http://dev.mysql.com/doc/refman/ 5.1/en/load-data.html
Use
LOAD DATA INFILE
syntax.http://dev.mysql.com/doc/refman/5.1/en/load-data.html
尝试 mysqlimport
另外,myisam 的表类型会导入得更快,具体取决于如果您需要事务支持(innodb),则打开。
Try mysqlimport
Also table type of myisam will import faster, depends on if you need transactional support (innodb).
使用多个插入。它并没有那么昂贵,因为你仍然只连接到 mysql 一次。然后,您可以使用循环来更新您的值(->您的sql)并执行(我认为它是数学实验室的“获取”)查询。
(参见http://spx.arizona.edu/Projects/例如数据库/访问%20mySQL%20through%20Matlab.pdf)
Use multible INSERTS. It isn't that expensive, since you still connect to mysql only once. You can then use a loop to update your values ( -> your sql) and execute (i think it's "fetch" for mathlab) the query.
(see http://spx.arizona.edu/Projects/Database/Accessing%20mySQL%20through%20Matlab.pdf for example)