Efficient Processing of Large JSONL Bulk Exports from Shopify
I'm trying to come up with an efficient way to process large JSONL bulk exports of order data from Shopify. The structure of the JSONL is:
- Each row is either an
-
- An
I have created a stream
I'm thinking it's some combination of
- Each row is either an
Order, a LineItem or a Metafield-
LineItems and Metafields belong to a parent Order (identified by the __parentId key), but appear as individual lines immediately after their parent- An
Order can have an arbitrary number of LineItems and MetafieldsI have created a stream
Stream<never, E1, Order | LineItem | Metafield> which just reads and parses each row. Now I want to transform that into a stream Stream<never,E2, OrderWihLineItemsAndMetafields>, which means statefully taking elements until some condition is met, then mapping them into an OrderWihLineItemsAndMetafields and repeating until the original stream is empty. What's the best way to do this?I'm thinking it's some combination of
takeUntil and then folding (code example in thread), but haven't been able to figure out how to repeatedly takeUntil from the same stream.
