无法从 apache Pig 中的地图中提取值

Question

我在 Apache Pig 中有一个简单的关系 v：

dump v;

(151364,[ 'ref'#'R813','highway'#'secondary', 'name:ga'#'Lána Chairdif', 'name'#'Cardiff Lane'],(31015271, 31053762))
(151368,[ 'ref'#'N1', 'oneway'#'yes','designation'#'Buses Only', 'highway'#'trunk', 'motor_vehicle'#'designated', 'name:ga'#'Cearnóg Pharnell Thoir', 'maxspeed'#'30', 'name'#'Parnell Square East'],(389365, 540403072))
(151596,[ 'name:en'#'Liffey', 'boundary'#'administrative', 'name:ga'#'An Life','admin_level'#'8', 'name'#'Liffey', 'waterway'#'river'],(1347749, 1426049020, 1347745, 1426049019, 1347742, 900075612))
(367947,[ 'maxspeed'#'80', 'ref'#'L2223','highway'#'tertiary'],(13259933, 2384217, 335978958))
(367952,['created_by'#'YahooApplet 1.0', 'name'#'Charnwood Avenue', 'highway'#'residential'],(2384386, 25963471, 14949594, 2384385, 6146344, 2384254))
(508603,[ 'ref'#'L3018','highway'#'tertiary', 'maxspeed'#'50', 'name'#'Shelerin Road'],(2854184, 2854168, 335978984, 2853307, 2384254, 335978978, 335978975, 2655735, 2655703, 392675957, 11676198, 920037194, 244531387, 2655952, 11675077))
(727153,[ 'ref'#'N8','highway'#'trunk', 'name'#'Merchants' Quay'],(354153, 453344873))
(727157,['highway'#'unclassified', 'oneway'#'yes', 'maxspeed'#'30', 'name'#'Kyle Street'],(354168, 354167))
(727159,['highway'#'unclassified', 'oneway'#'yes', 'maxspeed'#'30', 'name'#'North Main Street'],(354178, 465226768, 354167, 413995429, 72219131, 685537307, 1232381779, 354164))
(727161,[ 'maxspeed'#'30','highway'#'pedestrian', 'name'#'Maylor Street'],(1486492976, 1515360721, 1515360722, 1515345383, 1515344226, 1515344227, 1515344228, 1515344231))

根据 @orangeoctopus 的建议，我尝试使用键名称中的任何 ' 重新生成数据，并且我拥有此数据：

(151364,[ ref#'R813', name:ga#'Lána Chairdif', name#'Cardiff Lane',highway#'secondary'],(31015271, 31053762))
(151368,[ motor_vehicle#'designated', name#'Parnell Square East', highway#'trunk', oneway#'yes',designation#'Buses Only', maxspeed#'30', name:ga#'Cearnóg Pharnell Thoir', ref#'N1'],(389365, 540403072))
(151596,[ name:en#'Liffey', boundary#'administrative', waterway#'river', name:ga#'An Life',admin_level#'8', name#'Liffey'],(1347749, 1426049020, 1347745, 1426049019, 1347742, 900075612))
(367947,[highway#'tertiary', maxspeed#'80', ref#'L2223'],(13259933, 2384217, 335978958))
(367952,[ name#'Charnwood Avenue',created_by#'YahooApplet 1.0', highway#'residential'],(2384386, 25963471, 14949594, 2384385, 6146344, 2384254))
(508603,[ maxspeed#'50', ref#'L3018', name#'Shelerin Road',highway#'tertiary'],(2854184, 2854168, 335978984, 2853307, 2384254, 335978978, 335978975, 2655735, 2655703, 392675957, 11676198, 920037194, 244531387, 2655952, 11675077))
(727153,[highway#'trunk', name#'Merchants' Quay', ref#'N8'],(354153, 453344873))
(727157,[ oneway#'yes', maxspeed#'30', name#'Kyle Street',highway#'unclassified'],(354168, 354167))
(727159,[ oneway#'yes', maxspeed#'30', name#'North Main Street',highway#'unclassified' (354178, 465226768, 354167, 413995429, 72219131, 685537307, 1232381779, 354164))
(727161,[highway#'pedestrian', name#'Maylor Street', maxspeed#'30'],(1486492976, 1515360721, 1515360722, 1515345383, 1515344226, 1515344227, 1515344228, 1515344231))

在这两种情况下 v 都有相同的模式/结构：

grunt> describe v;
2012-01-09 22:55:34,271 [main] WARN  org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_CHARARRAY 1 time(s).
v: {id: int,tags: map[ ],nodes: (null)}

然后我尝试从 tags 映射中仅提取一个值：

grunt> w = foreach v generate tags#'ref';    
dump w;

但它只给我空数据，即使某些元素在这里有数据。

()
()
()
()
()
()
()
()
()
()

使用旧的“引用”键，我尝试了（按照@orangeoctopus'解决方案）

w = foreach v generate tags#'\'ref\''; 

，这给了我相同的“空”数据，但不起作用。 （我还尝试了 ' 和 " 的其他组合，例如 "'ref'"/'"ref"'/etc.，但除 '\'ref\'' 之外的所有内容都是无效的 Pig Latin 语法）

如果我尝试根据标签值进行过滤（例如 filter v by Tags#'highway' != '')，我明白了什么都没有，这与上述无法从地图中提取数据的问题一致，我做错了什么吗？

Answer 1

非常棘手！

您的问题是您的文字数据包含单引号。您的字符串不是 ref （3 个字符长），而是 'ref' （5 个字符长）。我意识到这一点是因为包含字符串的映射的转储通常没有引号。

因此，您需要键入包括这些引号的键（您必须使用 \ 转义它们）：

grunt> w = foreach v generate tags#'\'ref\'';    

您的另一个选择是更改数据加载的方式，使其不包含单引号在字符串本身中，并将它们剥离出来。 PigStorage 不会免费执行此操作，但您可以使用类似 REPLACE 或您自己的 UDF 来执行此操作。

Answer 2

您是否也正确加载数据？奇怪的是，当您转储地图时，[ 后面和 ] 之前有一个空格。

此外，删除 输入数据。例如：

输入文件

151364  [ref#R813,highway#secondary]

Pig

a = LOAD 'data.txt' AS (id:INT, m:MAP[]);
DUMP a;
b = FOREACH a GENERATE m#'ref';
DUMP b;

输出

(151364,[highway#secondary,ref#R813])

(R813)