I have a situation where I want to have call function in beam pipeline with side inputs which will be the lists with some key and it all should resolve to the return list. I started like this:
transformed_output = (
parsed_input_data
| "Merge Data"
>> beam.FlatMap(
process_records,
beam.pvalue.AsList(auxiliary_data),
beam.pvalue.AsDict(
{
"lookup_table_1": beam.pvalue.AsList(lookup_table_data_1),
"lookup_table_2": beam.pvalue.AsList(lookup_table_data_2),
}
),
)
)
And here lookup_table_data
is the result of previous pipeline. I am getting such error:
self._window_mapping_fn = sideinputs.default_window_mapping_fn( pcoll.windowing.windowfn) E AttributeError: 'dict' object has no attribute 'windowing'
Is it even possible? I want to have as an input anything which will store both list and keys, because I need to use that key later. Also this function will be called mulstiple time with different side inputs parameter.
发布者:admin,转转请注明出处:http://www.yc00.com/questions/1745275319a4620003.html
评论列表(0条)