我有2个文件夹，在每个文件夹上，每个文件夹都有70个CSV文件，每个文件的大小为3MB至5MB，因此一般而言，数据大约是20亿行，每个列有5列。
我使用Amazon Wrangler S3.Read_CSV将所有70 CSV的文件夹加载到数据框架上，不确定这是否是一种很好的方法，因为数据确实很大。
我想知道如何使用AWS Wrangler S3.ReadCSV从这两个文件夹中加载整个CSV文件，还是应该使用Pyspark？
另一个问题是，是否可以在本地使用Amazon Sagemaker依据进行本地工作？我不确定是否使用萨吉式笔记本进行管道开发可能会给我的客户花费很多。
I have 2 folder, on each folder I have 70 csv files each one with a size of 3mb to 5mb, so in general the data is like 20 millions rows with 5 columns each.
I used amazon wrangler s3.read_csv to load just one folder with all the 70 csv to a dataframe, not sure if this is a good approach due to the fact the data is really large.
I want to know how can I load the entire csv files from those 2 folders with aws wrangler s3.readcsv, or should I use pyspark?
Also another question is, is it possible to work locally using amazon sagemaker depenencies? I am not sure if using sagemaker notebook for the pipeline development might cost a lot for my client.
发布评论
评论（1）
您也可以使用pyspark将数据也加载到笔记本中，请参见此repo 用于说明。
至于SageMaker，您可以使用 sagemaker python sdk 或：//boto3.amazonaws.com/v1/documentation/api/latest/reference/services/services/sagemaker.html“ rel =“ nofollow noreferrer”> boto3 以从您本地计算机运行作业。您还可以创建一个具有较小实例大小的笔记本实例，在数据子集上进行实验，然后触发a
You can use PySpark to load data into your notebook as well, see this repo for instructions.
As for SageMaker, you can use the SageMaker Python SDK, or Boto3 to run jobs from your local machine. You can also create a notebook instance with a small instance size, experiment on a subset of your data, and trigger a Processing job to keep your notebook costs low. You only pay for the duration your processing job runs, and you can scale up for preparing the entire dataset.