JSON Path Extraction Algorithm

This document explains how Traces extracts multiple values from JSON responses when iterating over arrays. This is the algorithm used by "iterate" mode to run an endpoint once per extracted value.

Overview

When an endpoint needs to run multiple times (iterate mode), the system extracts ALL matching values from the source response, not just the first one.

Source file: apps/web/src/components/dependency-mapper/EndpointTestPanel.tsx — the extractAllValuesFromPath() function (and its inner recursive traverse()).

Single vs Multiple Value Extraction

Single Value (First Mode)

Used for testing and "first" mode mappings:

Data: { items: [{ id: 1 }, { id: 2 }, { id: 3 }] }
Path: "items[*].id"
Result: 1 (first value only)

Multiple Values (Iterate Mode)

Used for iteration:

Data: { items: [{ id: 1 }, { id: 2 }, { id: 3 }] }
Path: "items[*].id"
Result: [1, 2, 3] (all values)

Algorithm Walk-Through

Step 1: Parse the Path

The path is split into parts, removing brackets:

Path: "results[*].children[*].id"
Parts: ["results", "*", "children", "*", "id"]

Step 2: Count Wildcards

The algorithm counts total wildcards to know which is the "last" one (where filters apply):

Parts: ["results", "*", "children", "*", "id"]
Wildcard positions: 1, 3
Total wildcards: 2
Last wildcard index: 3

Step 3: Recursive Traversal

The extractAllValuesFromPath() function traverses recursively, collecting ALL values at wildcards instead of just the first:

Data: {
  results: [
    { children: [{ id: "a" }, { id: "b" }] },
    { children: [{ id: "c" }, { id: "d" }] }
  ]
}

Path: "results[*].children[*].id"

Traversal tree:
  results[0]
    children[0] -> id: "a" (collected)
    children[1] -> id: "b" (collected)
  results[1]
    children[0] -> id: "c" (collected)
    children[1] -> id: "d" (collected)

Final result: ["a", "b", "c", "d"]

Step 4: Apply Filter at Last Wildcard

If a filter condition exists, it's applied at the last wildcard only:

Data: {
  results: [
    { children: [{ id: "a", type: "page" }, { id: "b", type: "db" }] },
    { children: [{ id: "c", type: "page" }, { id: "d", type: "db" }] }
  ]
}

Path: "results[*].children[*].id"
Filter: type = 'page'

Traversal (filter at second [*]):
  results[0]
    children filtered to: [{ id: "a", type: "page" }]
    -> id: "a" (collected)
  results[1]
    children filtered to: [{ id: "c", type: "page" }]
    -> id: "c" (collected)

Final result: ["a", "c"]

Traversal Function Logic

The recursive function tracks:

• Current position in the data
• Current index in the path parts
• Current wildcard count (per traversal branch)

function traverse(value, pathIndex, wildcardCount):
  if pathIndex >= parts.length:
    collect(value)  // Reached end, save value
    return

  part = parts[pathIndex]

  if part == "*":
    newWildcardCount = wildcardCount + 1
    if newWildcardCount == lastWildcardIndex AND hasFilter:
      value = applyFilter(value)

    for each item in value:
      traverse(item, pathIndex + 1, newWildcardCount)

  else if part is numeric:
    traverse(value[part], pathIndex + 1, wildcardCount)

  else:
    traverse(value[part], pathIndex + 1, wildcardCount)

Key insight: Each branch of traversal maintains its own wildcard count. This is critical for nested arrays where we traverse multiple parent items.

Path Syntax Reference

Syntax	Meaning	Example
field	Access object property	user
.	Property separator	user.name
[*]	All array elements	items[*]
[0]	First element	items[0]
[-1]	Last element	items[-1]
[n]	Specific index	items[3]

Complex Examples

Deeply Nested Arrays

Data: {
  pages: [
    {
      blocks: [
        { items: [{ id: 1 }, { id: 2 }] },
        { items: [{ id: 3 }] }
      ]
    },
    {
      blocks: [
        { items: [{ id: 4 }, { id: 5 }] }
      ]
    }
  ]
}

Path: "pages[*].blocks[*].items[*].id"
Result: [1, 2, 3, 4, 5]

Mixed Fixed and Wildcard Indices

Data: {
  sections: [
    { rows: [{ cells: ["a", "b"] }, { cells: ["c", "d"] }] },
    { rows: [{ cells: ["e", "f"] }] }
  ]
}

Path: "sections[*].rows[0].cells[*]"
Result: ["a", "b", "e", "f"]
(Only first row of each section)

Filter on Nested Property

Data: {
  items: [
    { meta: { status: "active" }, id: 1 },
    { meta: { status: "deleted" }, id: 2 },
    { meta: { status: "active" }, id: 3 }
  ]
}

Path: "items[*].id"
Filter: meta.status = 'active'

Result: [1, 3]

Key Code References

All within EndpointTestPanel.tsx (search by name — line numbers drift as the file is edited):

Function / region	Purpose
extractAllValuesFromPath()	Entry point; normalizes single-object data and merges across response arrays
wildcard counting (totalWildcards, countWildcardsInPath)	Determine which wildcard level the filter applies at
inner traverse()	Recursive collection of all values, tracking a per-branch wildcard count
* / -1 handling in traverse()	Wildcard iteration with filter application at the last (or filterAtPath) wildcard

Performance Considerations

Large Arrays

Extracting from very large arrays (1000+ items) can be slow. The UI shows a preview count before running.

Deep Nesting

Each level of nesting multiplies the number of values. A path like [*][*][*][*] with 10 items at each level produces 10,000 values.

Filter Early

When possible, configure the API to return filtered data rather than filtering on the client. This reduces data transfer and extraction time.

Integration with Iterate Testing

When "Iterate Test" runs:

1. Extract all values using this algorithm
2. For each value, substitute into the target endpoint's path/query/body
3. Execute the endpoint
4. Collect all responses into an array

Source values: ["id1", "id2", "id3"]
Target path: /users/{userId}/profile

Executes:
  GET /users/id1/profile
  GET /users/id2/profile
  GET /users/id3/profile

Result: Array of 3 profile objects

Debugging Tips

"Not all values extracted"

Check:

1. Path uses [*] not [0] for arrays you want to iterate
2. Filter isn't excluding items unexpectedly
3. Source data has the expected structure

"Too many values"

Verify:

1. Filter condition is correctly configured
2. You're extracting from the right array level
3. Consider using a fixed index for outer arrays if needed

"Duplicate values"

This can happen with:

1. Source data containing duplicates
2. Multiple paths to the same value (unusual)

Related Documentation

• Parameter Extraction - Single value extraction
• Conditional Filtering - Filter application
• Test Modes - Using iterate mode in the UI