展平嵌套字典,压缩键
假设您有一本类似的字典:
{'a': 1,
'c': {'a': 2,
'b': {'x': 5,
'y' : 10}},
'd': [1, 2, 3]}
您如何将其扁平化为:
{'a': 1,
'c_a': 2,
'c_b_x': 5,
'c_b_y': 10,
'd': [1, 2, 3]}
Suppose you have a dictionary like:
{'a': 1,
'c': {'a': 2,
'b': {'x': 5,
'y' : 10}},
'd': [1, 2, 3]}
How would you go about flattening that into something like:
{'a': 1,
'c_a': 2,
'c_b_x': 5,
'c_b_y': 10,
'd': [1, 2, 3]}
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(30)
基本上与展平嵌套列表的方式相同,您只需要做额外的工作即可按键/值迭代字典,为新字典创建新键并在最后一步创建字典。
Basically the same way you would flatten a nested list, you just have to do the extra work for iterating the dict by key/value, creating new keys for your new dictionary and creating the dictionary at final step.
如果您已经在使用 pandas,则可以使用
json_normalize 来完成()
像这样:输出:
If you are already using pandas, you can do it with
json_normalize()
like so:Output:
原始发布者需要考虑两个重要因素:
{'a_b':{'c':1}, 'a':{'b_c':2}}
将导致{'a_b_c':???}
。下面的解决方案通过返回可迭代的对来避免这个问题。joinedKey = '_'.join(*keys)
,这将花费您 O(N^2) 的运行时间。但是,如果您愿意说nextKey = previousKey+'_'+thisKey
,那么您将获得 O(N) 时间。下面的解决方案允许您同时执行这两项操作(因为您只需连接所有键,然后对它们进行后处理)。(性能不太可能是一个问题,但我将详细说明第二点,以防其他人关心:在实现这一点时,有许多危险的选择。如果您递归地执行此操作并产生并重新产生,或者任何东西 相当于多次接触节点(这很容易意外发生),您正在做潜在的 O(N^2) 工作,而不是 O(N) 这是因为您可能正在计算一个键
。 >a
然后a_1
然后a_1_i
...,然后计算a
然后a_1
然后a_1_ii
...,但实际上您不必再次计算a_1
即使您没有重新计算它,重新生成它(“逐级”方法)也一样。一个很好的例子就是思考一下。{1:{1:{1:{1:...(N times)...{1:SOME_LARGE_DICTIONARY_OF_SIZE_N}...}}}}
上的性能编写了
flattenDict(d, join=..., lift=...)
,它可以适应多种用途,并且可以做您想做的事情。遗憾的是,要在不产生上述性能损失的情况下制作这个函数的惰性版本是相当困难的(许多像 chain.from_iterable 这样的 python 内置函数实际上并不高效,我只是在对这个代码的三个不同版本进行了广泛测试之后才意识到这一点,然后才决定使用这个)。为了更好地理解发生了什么,下面为那些不熟悉
reduce
(左)(也称为“向左折叠”)的人提供了一个图表。有时它会用初始值代替 k0(不是列表的一部分,传递到函数中)来绘制。这里,J
是我们的join
函数。我们用lift(k)
预处理每个 kn。这实际上与 functools.reduce 相同,但我们的函数对树的所有关键路径执行此操作。
演示(否则我会放入文档字符串中):
性能:
...叹息,不要认为这是我的错...
[由于审核问题而产生的不重要的历史注释]
关于所谓的重复 压平字典的字典(2层深)列表
该问题的解决方案可以通过执行
sorted( sum(flatten(...),[]) )
来实现。相反的情况是不可能的:虽然确实可以通过映射高阶累加器从所谓的重复项中恢复flatten(...)
的值,但无法恢复密钥。 (编辑:此外,事实证明,所谓的重复所有者的问题完全不同,因为它只处理恰好 2 层深度的字典,尽管该页面上的答案之一给出了通用解决方案。)There are two big considerations that the original poster needs to consider:
{'a_b':{'c':1}, 'a':{'b_c':2}}
would result in{'a_b_c':???}
. The below solution evades the problem by returning an iterable of pairs.joinedKey = '_'.join(*keys)
, that will cost you O(N^2) running time. However if you're willing to saynextKey = previousKey+'_'+thisKey
, that gets you O(N) time. The solution below lets you do both (since you could merely concatenate all the keys, then postprocess them).(Performance is not likely an issue, but I'll elaborate on the second point in case anyone else cares: In implementing this, there are numerous dangerous choices. If you do this recursively and yield and re-yield, or anything equivalent which touches nodes more than once (which is quite easy to accidentally do), you are doing potentially O(N^2) work rather than O(N). This is because maybe you are calculating a key
a
thena_1
thena_1_i
..., and then calculatinga
thena_1
thena_1_ii
..., but really you shouldn't have to calculatea_1
again. Even if you aren't recalculating it, re-yielding it (a 'level-by-level' approach) is just as bad. A good example is to think about the performance on{1:{1:{1:{1:...(N times)...{1:SOME_LARGE_DICTIONARY_OF_SIZE_N}...}}}}
)Below is a function I wrote
flattenDict(d, join=..., lift=...)
which can be adapted to many purposes and can do what you want. Sadly it is fairly hard to make a lazy version of this function without incurring the above performance penalties (many python builtins like chain.from_iterable aren't actually efficient, which I only realized after extensive testing of three different versions of this code before settling on this one).To better understand what's going on, below is a diagram for those unfamiliar with
reduce
(left), otherwise known as "fold left". Sometimes it is drawn with an initial value in place of k0 (not part of the list, passed into the function). Here,J
is ourjoin
function. We preprocess each kn withlift(k)
.This is in fact the same as
functools.reduce
, but where our function does this to all key-paths of the tree.Demonstration (which I'd otherwise put in docstring):
Performance:
... sigh, don't think that one is my fault...
[unimportant historical note due to moderation issues]
Regarding the alleged duplicate of Flatten a dictionary of dictionaries (2 levels deep) of lists
That question's solution can be implemented in terms of this one by doing
sorted( sum(flatten(...),[]) )
. The reverse is not possible: while it is true that the values offlatten(...)
can be recovered from the alleged duplicate by mapping a higher-order accumulator, one cannot recover the keys. (edit: Also it turns out that the alleged duplicate owner's question is completely different, in that it only deals with dictionaries exactly 2-level deep, though one of the answers on that page gives a general solution.)如果您使用的是
pandas
,pandas.io.json._normalize
1 中隐藏着一个名为nested_to_record
的函数这正是这样做的。1 在 pandas 版本
0.24.x
及更早版本中使用pandas.io.json.normalize
(不带_
)If you're using
pandas
there is a function hidden inpandas.io.json._normalize
1 callednested_to_record
which does this exactly.1 In pandas versions
0.24.x
and older usepandas.io.json.normalize
(without the_
)不完全是OP所要求的,但很多人来这里寻找方法来展平现实世界的嵌套JSON数据,这些数据可以嵌套键值json对象和数组以及数组内的json对象等等。 JSON 不包含元组,因此我们不必担心这些。
我找到了列表包含 @roneo 对 < href="https://stackoverflow.com/a/6027615/4355695">发布者的答案@Imran:
https://github.com/ ScriptSmith/socialreaper/blob/master/socialreaper/tools.py#L8
测试一下:
这完成了我需要完成的工作:我将任何复杂的 json 扔到这里,它会为我将其展平。
所有功劳均归功于 https://github.com/ScriptSmith 。
2023-06-14 python 更新 >= 3.10
自 Python 3.10 起,
collections.MutableMapping
已更改为collections.abc.MutableMapping
。因此,上面的代码经过编辑以反映相同的情况。如果您的Python版本是3.10之前,请将其改回您身边的collections.MutableMapping
。参考:https://stackoverflow.com/a/71902541/4355695
Not exactly what the OP asked, but lots of folks are coming here looking for ways to flatten real-world nested JSON data which can have nested key-value json objects and arrays and json objects inside the arrays and so on. JSON doesn't include tuples, so we don't have to fret over those.
I found an implementation of the list-inclusion comment by @roneo to the answer posted by @Imran :
https://github.com/ScriptSmith/socialreaper/blob/master/socialreaper/tools.py#L8
Test it:
Annd that does the job I need done: I throw any complicated json at this and it flattens it out for me.
All credits to https://github.com/ScriptSmith .
2023-06-14 Update for python >= 3.10
Since Python 3.10,
collections.MutableMapping
has changed tocollections.abc.MutableMapping
. Hence code above is edited to reflect the same. If your python version is before 3.10, please change it back tocollections.MutableMapping
at your side.Ref: https://stackoverflow.com/a/71902541/4355695
这是一种“功能性”、“单行”实现。它是递归的,基于条件表达式和字典理解。
测试:
Here is a kind of a "functional", "one-liner" implementation. It is recursive, and based on a conditional expression and a dict comprehension.
Test:
代码:
结果:
我使用的是python3.2,更新为你的python版本。
Code:
Results:
I am using python3.2, update for your version of python.
这不仅限于字典,还包括实现 .items() 的每个映射类型。进一步更快,因为它避免了 if 条件。尽管如此,功劳归于伊姆兰:
This is not restricted to dictionaries, but every mapping type that implements .items(). Further ist faster as it avoides an if condition. Nevertheless credits go to Imran:
Python3.5 中的功能性和高性能解决方案怎么样?
这甚至更加高效:
使用中:
How about a functional and performant solution in Python3.5?
This is even more performant:
In use:
如果您是pythonic oneliners的粉丝:
返回:
如果您有一个字典列表而不仅仅是一个字典,则可以从末尾保留
[0]
。If you are a fan of pythonic oneliners:
returns:
You can leave the
[0]
from the end, if you have a list of dictionaries and not just a single dictionary.我使用生成器的 Python 3.3 解决方案:
My Python 3.3 Solution using generators:
利用递归,保持简单和人类可读:
调用很简单:
或者
如果我们想更改默认分隔符。
一点小细节:
首次调用该函数时,仅传递我们想要展平的
字典
。 Accumulator 参数在这里支持递归,我们稍后会看到。因此,我们将累加器实例化为一个空字典,将原始字典中的所有嵌套值放入其中。当我们迭代字典的值时,我们为每个值构造一个键。对于第一次调用,
parent_key
参数将为None
,而对于每个嵌套字典,它将包含指向它的键,因此我们在前面添加该键。如果值
v
的键k
指向的是一个字典,则该函数调用自身,传递嵌套字典、累加器
(其中通过引用传递,因此对它所做的所有更改都是在同一个实例上完成的)和键k
,以便我们可以构造串联键。请注意continue
语句。我们希望跳过if
块之外的下一行,以便嵌套字典不会最终出现在键k
accumulator 中>。那么,如果值
v
不是字典,我们该怎么办?只需将其原封不动地放入累加器中即可。完成后,我们只需返回累加器,而原始字典参数保持不变。
注意
这仅适用于以字符串作为键的字典。它将与实现 __repr__ 方法的可哈希对象一起使用,但会产生不需要的结果。
Utilizing recursion, keeping it simple and human readable:
Call is simple:
or
if we want to change the default separator.
A little breakdown:
When the function is first called, it is called only passing the
dictionary
we want to flatten. Theaccumulator
parameter is here to support recursion, which we see later. So, we instantiateaccumulator
to an empty dictionary where we will put all of the nested values from the originaldictionary
.As we iterate over the dictionary's values, we construct a key for every value. The
parent_key
argument will beNone
for the first call, while for every nested dictionary, it will contain the key pointing to it, so we prepend that key.In case the value
v
the keyk
is pointing to is a dictionary, the function calls itself, passing the nested dictionary, theaccumulator
(which is passed by reference, so all changes done to it are done on the same instance) and the keyk
so that we can construct the concatenated key. Notice thecontinue
statement. We want to skip the next line, outside of theif
block, so that the nested dictionary doesn't end up in theaccumulator
under keyk
.So, what do we do in case the value
v
is not a dictionary? Just put it unchanged inside theaccumulator
.Once we're done we just return the
accumulator
, leaving the originaldictionary
argument untouched.NOTE
This will work only with dictionaries that have strings as keys. It will work with hashable objects implementing the
__repr__
method, but will yield unwanted results.这是使用堆栈的解决方案。没有递归。
here's a solution using a stack. No recursion.
用于展平嵌套字典的简单函数。对于 Python 3,将
.iteritems()
替换为.items()
想法/要求是:
获取扁平字典,无需保留父密钥。
使用示例:
保留父键也很简单。
Simple function to flatten nested dictionaries. For Python 3, replace
.iteritems()
with.items()
The idea/requirement was:
Get flat dictionaries with no keeping parent keys.
Example of usage:
Keeping parent keys is simple as well.
我正在考虑 UserDict 的子类来自动平放按键。
可以即时添加键或使用标准字典实例化的优点,毫不奇怪:
I was thinking of a subclass of UserDict to automagically flat the keys.
The advantages it that keys can be added on the fly, or using standard dict instanciation, without surprise:
这与伊姆兰和拉鲁的回答相似。它不使用生成器,而是使用带有闭包的递归:
This is similar to both imran's and ralu's answer. It does not use a generator, but instead employs recursion with a closure:
上面的答案确实很好用。只是想我会添加我编写的 unflatten 函数:
注意:这并不考虑键中已经存在的“_”,就像展平对应项一样。
The answers above work really well. Just thought I'd add the unflatten function that I wrote:
Note: This doesn't account for '_' already present in keys, much like the flatten counterparts.
Davoud 的解决方案非常好,但当嵌套字典还包含字典列表时,不会给出令人满意的结果,但他的代码适合这种情况:
Davoud's solution is very nice but doesn't give satisfactory results when the nested dict also contains lists of dicts, but his code be adapted for that case:
事实上,我最近写了一个名为cherrypicker的包来处理这类事情,因为我不得不经常这样做!
我认为以下代码将为您提供所需的信息:
您可以使用以下命令安装软件包:
...并且在 https://cherrypicker.readthedocs.io。
其他方法可能会更快,但此包的首要任务是使此类任务变得简单。如果您确实有大量对象需要展平,您也可以告诉 CherryPicker 使用并行处理来加快速度。
I actually wrote a package called cherrypicker recently to deal with this exact sort of thing since I had to do it so often!
I think the following code would give you exactly what you're after:
You can install the package with:
...and there's more docs and guidance at https://cherrypicker.readthedocs.io.
Other methods may be faster, but the priority of this package is to make such tasks easy. If you do have a large list of objects to flatten though, you can also tell CherryPicker to use parallel processing to speed things up.
使用生成器:
Using generators:
使用 Flatdict 库:
Using flatdict library:
这是一种优雅的就地替换算法。使用 Python 2.7 和 Python 3.5 进行测试。使用点字符作为分隔符。
示例:
输出:
我在此处发布了此代码以及匹配的
unflatten_json
功能。Here's an algorithm for elegant, in-place replacement. Tested with Python 2.7 and Python 3.5. Using the dot character as a separator.
Example:
Output:
I published this code here along with the matching
unflatten_json
function.如果您想要平面嵌套字典并想要所有唯一键列表,那么这里是解决方案:
If you want to flat nested dictionary and want all unique keys list then here is the solution:
我总是更喜欢通过
.items()
访问dict
对象,因此为了扁平化 dict,我使用以下递归生成器flat_items(d)
。如果您想再次使用dict
,只需将其包装如下:flat = dict(flat_items(d))
I always prefer access
dict
objects via.items()
, so for flattening dicts I use the following recursive generatorflat_items(d)
. If you like to havedict
again, simply wrap it like this:flat = dict(flat_items(d))
在简单的类似嵌套列表的递归中使用 dict.popitem() :
Using dict.popitem() in straightforward nested-list-like recursion:
如果您不介意递归函数,这里有一个解决方案。我还冒昧地添加了一个排除参数,以防您希望保留一个或多个值。
代码:
用法:
输出:
If you do not mind recursive functions, here is a solution. I have also taken the liberty to include an exclusion-parameter in case there are one or more values you wish to maintain.
Code:
Usage:
Output:
我尝试了此页面上的一些解决方案 - 虽然不是全部 - 但我尝试的解决方案未能处理字典的嵌套列表。
考虑这样的字典:
这是我的临时解决方案:
它产生:
一个临时解决方案,它并不完美。
注意:
它不会保留空字典,例如
address: {}
k/v 对。它不会展平嵌套元组中的字典 - 尽管使用Python元组的行为类似于列表这一事实很容易添加。
I tried some of the solutions on this page - though not all - but those I tried failed to handle the nested list of dict.
Consider a dict like this:
Here's my makeshift solution:
which produces:
A makeshift solution and it's not perfect.
NOTE:
it doesn't keep empty dicts such as the
address: {}
k/v pair.it won't flatten dicts in nested tuples - though it would be easy to add using the fact that python tuples act similar to lists.
此使用 max_level 和自定义化简器压平嵌套字典、压缩键的变体。
Variation of this Flatten nested dictionaries, compressing keys with max_level and custom reducer.