Lambda and Kinesis - beware of hot streams

AWS has introduced enhanced fan-out for Kinesis streams, which allows each function to do five reads per second per shard, up to 2MB per second. However, this improvement does not completely eliminate the problems of hot streams with many Lambda subscribers. In fact, it can lead to a new issue where it becomes almost impossible to deploy subsequent functions after a certain number of initial subscriber functions are deployed, resulting in ProvisionThroughputExceededException errors during deployment. To work around this limitation, design considerations such as using multiple write streams or pushing events to a centralized stream and then fanning out to multiple reader streams can be employed. These approaches offer flexibility but also come with additional costs. Ultimately, the goal is to find a balance between scalability and cost when dealing with hot streams in Kinesis.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.