shell脚本 - 遍历文本文件中的每一行，并重命名HDFS文件

发布于 2025-01-17 23:34:44 字数 605 浏览 2 评论 0原文

我在HDFS中有一个文本文件，该文件将具有以下记录。文件中的行数可能每次都会有所不同。 hdfs：//myfile.txt

file_name_1
file_name_2
file_name_3

I具有以下HDFS目录和文件结构。

hdfs://myfolder/
hdfs://myfolder/file1.csv
hdfs://myfolder/file2.csv
hdfs://myfolder/file3.csv

使用Shell脚本，我可以计算HDFS目录中的文件数和HDFS文本文件中可用的行数。只有当目录中的文件数量与文本文件中的记录数之间匹配时，我才能进一步继续该过程。

现在，我尝试使用我的文本文件中的第一个记录来重命名hdfs：//myfolder/file1.csv至hdfs：//myfolder/file_name_1.csv。第二个文件应重命名为hdfs：//myfolder/file_name_2.csv，第三个文件和第三个文件将其重命名为hdfs：//myfolder/file_name_3.csv，

我难以通过文本文件以及HDFS目录中的文件循环。是否有一种使用Shell脚本实现此目的的最佳方法。

原文

I have a text file in HDFS which would have records like below. The number of lines in file may vary every time.
hdfs://myfile.txt

file_name_1
file_name_2
file_name_3

I have the below hdfs directory and file structure like below.

hdfs://myfolder/
hdfs://myfolder/file1.csv
hdfs://myfolder/file2.csv
hdfs://myfolder/file3.csv

Using shell script I am able to count the number of files in HDFS directory and number of lines available in my HDFS text file. Only if the count matches between the number of files in directory and number of records in my text file, I am going to proceed further with the process.

Now, i am trying to rename hdfs://myfolder/file1.csv to hdfs://myfolder/file_name_1.csv using the first record from my text file.
Second file should be renamed to hdfs://myfolder/file_name_2.csv and third file to hdfs://myfolder/file_name_3.csv

I have difficulty in looping through both the text file and also the files in HDFS directory.
Is there an optimal way to achieve this using shell script.

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

最舍不得你 2025-01-24 23:34:44

您无法直接从 HDFS 执行此操作，您需要流式传输文件内容，然后发出单独的移动命令。

例如

#!/bin/sh

COUNTER = 0
for file in $(hdfs dfs -cat file.txt)
do 
  NAME = $(sed $file ...)  # replace text, as needed. TODO: extract the extension
  hdfs dfs -mv file "$NAME_${COUNTER}.csv"  # 'csv' for example - make sure the extension isn't duplicated!!
  COUNTER = $((COUNTER + 1)
done

You cannot do this directly from HDFS, you'd need to stream the file contents, then issue individual move commands.

e.g.

#!/bin/sh

COUNTER = 0
for file in $(hdfs dfs -cat file.txt)
do 
  NAME = $(sed $file ...)  # replace text, as needed. TODO: extract the extension
  hdfs dfs -mv file "$NAME_${COUNTER}.csv"  # 'csv' for example - make sure the extension isn't duplicated!!
  COUNTER = $((COUNTER + 1)
done

回复收藏 0 原文

~没有更多了~