Aggregating Nested Data: Unleash the Power Of Your Arrays

As your product evolves, so do your analytics events and so do your aggregation needs. That’s why we’re always adding new power to your real-time metrics toolbelt. Today that means you can flatten and aggregate arrays nested inside your JSON payloads.

Array Calculations

First off, you can now run your calculations over either nested elements in arrays of objects or on array values directly.

Let’s say you’re running a live-service game and your game server emits a heartbeat every 30s. Heartbeats are amazing and powerful, but are usually emitted from clients where the context is a single user’s context. When you’re working with a server heartbeat, there is likely information about many users.

So if we want to extract CCU using a server heartbeat, we need to count distinct user_ids, that may be in an array, or nested in an array of objects.

This is now possible! Set your aggregation_field to @.users[*] (assuming an array of strings) or @.users[*].id for an array of nested objects.

You can also work with arrays of arrays or arrays on objects inside arrays as well. Maybe your server is hosting multiple sessions with different lists of users? @.sessions[*].users[*].id enables you to maintain unique counting without having to transform your payloads or send extraneous events.

Array Flattening

If you need to actually group by or filter within individual array elements, then you’re essentially treating each array element like an individual event. But you don’t want to send each array element as a separate event - that’s wasteful! In the past few years, data warehouses like Snowflake and ClickHouse have made working with arrays trivial, but most platforms where you’re abstracted from the raw data make it nearly impossible. With the power of JSON and the ethos of “no transformations necessary” - we’ve decided to enable this feature for all our users.

How it works

Example: Impression Tracking 👀

A common use case for Array flattening is Impression Tracking. You don’t want your client to send events every time they see something. You may want to count a duration, hover time, number of scroll-bys, etc. You also don’t want to emit maybe 100 events when they navigate away, per impression… So what do you do? You design a “summary” event that includes the relevant details.

Let’s imagine an event like this:

{
    "event_name": "Impressions",
    "ts": 1721491167001,
    "user_id": "user123456",
    "device": {
        "type": "web"
    },
    "impressions": [
        {
            "item_id": "abc123",
            "duration_ms": 4744,
            "detail_clicks": 1
        },
        {
            "item_id": "xyz456",
            "duration_ms": 5500,
            "detail_clicks": 10
        },
        {
            "item_id": "zzz555",
            "duration_ms": 1199,
            "detail_clicks": 4
        }
    ]
}

There’s so much awesome information locked inside that @.impressions array, just waiting to be unleashed.

You can use this now for both monitoring and feeding back into your systems to decide what has been “seen enough” and what needs more views.

Some useful metrics might be:

• Total impression duration: aggregate @.duration_ms, grouped by $.device.type and @.item_id
• Unique users per item: aggregate $.user_id, grouping by @.item_id
• Click through rate: aggregate @.detail_clicks and a Count, grouping by @.item_id (the calculation can be done via the API or grafana).

Here’s this example from our Docs

Debugger Support

With the super handy Debug Mode we can see the results are validated and what values they will use when performing calculations.