python - Combining Beam FlatMap pvalue.AsDict and pvalue.AsList - Stack Overflow

I have a situation where I want to have call function in beam pipeline with side inputs which will be t

I have a situation where I want to have call function in beam pipeline with side inputs which will be the lists with some key and it all should resolve to the return list. I started like this:

transformed_output = (
    parsed_input_data
    | "Merge Data" 
    >> beam.FlatMap(
        process_records,
        beam.pvalue.AsList(auxiliary_data),
        beam.pvalue.AsDict(
            {
                "lookup_table_1": beam.pvalue.AsList(lookup_table_data_1),
                "lookup_table_2": beam.pvalue.AsList(lookup_table_data_2),
            }
        ),
    )
)

And here lookup_table_data is the result of previous pipeline. I am getting such error:

self._window_mapping_fn = sideinputs.default_window_mapping_fn( pcoll.windowing.windowfn) E AttributeError: 'dict' object has no attribute 'windowing'

Is it even possible? I want to have as an input anything which will store both list and keys, because I need to use that key later. Also this function will be called mulstiple time with different side inputs parameter.

发布者:admin,转转请注明出处:http://www.yc00.com/questions/1745275319a4620003.html

相关推荐

发表回复

评论列表(0条)

  • 暂无评论

联系我们

400-800-8888

在线咨询: QQ交谈

邮件:admin@example.com

工作时间:周一至周五,9:30-18:30,节假日休息

关注微信