John Mertens

Lessons From Our First Trillion Messages with Flow

For many Elixir converts, the attraction of Elixir was rooted in the promise of the BEAM concurrency model. The Flow library has made it easy to build concurrent data pipelines utilising the BEAM. The problem is, that while the docs are great, there are not many resources on running Flow-based systems in production. In this talk, John will share the lessons his team learned from processing their first trillion messages through Flow. Attendees will leave with practical knowledge they can apply directly to their applications.

Objectives

Fill in some of the gaps between watching José's introductory Flow talk and using it to power a highly-concurrent data processing pipeline.

Audience

Developers interested in using Elixir for data pipelines but want to know more about what it is like in a real-world system.