从文件中快速删除带有索引的行
我有一个10G的巨大文件。我想从此文件中删除第 188888 行。
我按如下方式使用 sed：
sed -i '188888d' file
问题是它真的很慢。我知道这是因为文件的大小，但是有什么方法可以更快地做到这一点。
谢谢
I have a HUGE file of 10G. I want to remove line 188888 from this file.
I use sed as follows:
sed -i '188888d' file
The problem is it is really slow. I understand it is because of the size of the file, but is there any way that I can do that faster.
Thanks
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
尝试
您可能需要尝试保留上面的分号中的哪一个，
{d;q} ...是第二个要尝试的事情。这将在删除该行后停止搜索文件，但您仍然需要花时间重写文件。还值得测试
备用安装点位于单独的磁盘驱动器上的位置。
最终编辑
啊，另一种选择是在通过管道写入文件时对其进行编辑
但这假设您知道要删除的数据始终位于“188888”行，这可能吗？
我希望这有帮助。
Try
You may need to experiment with which of the above semi-colons you keep,
{d;q} ...being the 2nd thing to try.This will stop searching the file after it deletes that one line, but you'll still have to spend the time re-writing the file. It would also be worth testing
where the alternate mountpoint is on a separate disk drive.
final edit
Ah, one other option would be to edit the file while it is being written through a pipe
But this assumes that you know that the data your want to delete is always at line '188888, is that possible?
I hope this helps.
文件行是通过计算 \n 字符来确定的，如果行大小是可变的，那么您无法计算给定行的位置的偏移量，但必须计算换行符的数量。
这始终是 O(n)，其中 n 是文件中的字节数。
并行算法也无济于事，因为此操作受磁盘 IO 限制，分而治之会更慢。
如果您要对同一个文件多次执行此操作，可以通过多种方法来预处理该文件并使其速度更快。
一个简单的方法是建立一个索引
，当你想查找一行时，在索引中进行二分搜索（Log n）查找你想要的行号，并使用偏移量在原始文件中找到该行。
The file lines are determined by counting the \n character, if the line size are variable then you cannot calculate the offset to the location given a line but have to count the number of newlines.
This will always be O(n) where n is the number of bytes in the file.
Parallel algorithms does not help either because this operation is disk IO limited, divide and conquer will be even slower.
If you will do this a lot on a same file, there are ways to preprocess the file and make it faster.
A easy way is to build a index with
And when you want to find a line, do binary search (Log n) in index for the line number you want, and use the offset to locate the line in the original file.