如何定义create_tf_dataset_from_all_all_clients（）函数

发布于 2025-02-13 06:50:21 字数 1021 浏览 4 评论 0原文

我正在尝试从联合的数据集中制作集中式数据集。数据包含路径，client_id和标签，

因此首先，我使用接受客户端ID创建客户端data的函数

def extract_file_paths(dataset):
  return [item["path"] for item in dataset]

@tf.function
def create_dataset(client_id):
    new_datset = tf.data.Dataset.from_tensor_slices(dict(df_aml))
    client_id = int(client_id)
    client_id = tf.cast(client_id, dtype=tf.int64)

    files = new_datset.filter(lambda x: x['client_id'] == client_id)


    list_ds = tf.data.Dataset.list_files(tf.py_function(func=extract_file_paths,inp=[files], Tout = tf.string ))
    
    
    images_ds = list_ds.map(parse_image)

        
    return images_ds

创建客户端对象：

client_ids = ['0', '1', '2']


client_data = tff.simulation.datasets.ClientData.from_clients_and_tf_fn(client_ids,
create_dataset)

centralized = client_data.create_tf_dataset_from_all_clients()

我期望一个具有不同标签和文件的数据集，但是此代码正在生成一个只有一个文件的数据集。是因为我试图实现图形执行方法吗？

我尝试使用类似的函数来创建适用于联合设置并生成预期数据集的客户端数据集，但是当我尝试产生集中式数据集时，使用相同的功能会给我一个错误

原文

I am trying to make a centralized dataset from a federated one. Data contains path, client_id and label

So first I create a clientdata object using a function that accepts the client's id

def extract_file_paths(dataset):
  return [item["path"] for item in dataset]

@tf.function
def create_dataset(client_id):
    new_datset = tf.data.Dataset.from_tensor_slices(dict(df_aml))
    client_id = int(client_id)
    client_id = tf.cast(client_id, dtype=tf.int64)

    files = new_datset.filter(lambda x: x['client_id'] == client_id)


    list_ds = tf.data.Dataset.list_files(tf.py_function(func=extract_file_paths,inp=[files], Tout = tf.string ))
    
    
    images_ds = list_ds.map(parse_image)

        
    return images_ds

creating clientdata:

client_ids = ['0', '1', '2']


client_data = tff.simulation.datasets.ClientData.from_clients_and_tf_fn(client_ids,
create_dataset)

centralized = client_data.create_tf_dataset_from_all_clients()

I expected a dataset with different labels and files but this code is producing a dataset with only one file in it. Is it because I am trying to implement the graph execution method?

I tried using a similar function for creating a clientdata object that works for federated settings and produces the expected dataset, but using the same function gives me an error when I try to produce a centralized dataset

分享到QQ

分享到微博