Conditional Filtering Algorithm

This document explains how Traces filters array data using conditions. Conditional filtering allows users to select only specific items from an API response (e.g., only blocks where type = 'page').

Overview

Conditional filtering is applied when:

1. A reusable variable has a filter condition defined
2. That variable is used in a dependency mapping
3. The source data is an array

Source file: apps/web/src/lib/filter-condition.ts

Filter Condition Structure

A filter condition has these parts:

interface VariableFilterCondition {
  field: string;     // Path to field in each item (e.g., "type", "status.active")
  operator: string;  // Comparison operator
  value?: string;    // Value to compare against (not needed for "exists")
  orConditions?: Array<{ field; operator; value? }>;  // OR-chained alternatives
}

A condition matches if the primary clause matches OR any of the orConditions match (matchesCondition() → matchesSingleCondition()). The UI displays this as type = 'page' OR type = 'database'.

Outer-field conditions (nested iteration)

When filtering an inner array based on a parent item's field, prefix the field with outer. (e.g. outer.status = 'active'). matchesConditionWithContext(innerItem, outerItem, condition) strips the outer. prefix and evaluates that clause against the outer item; non-prefixed clauses evaluate against the inner item. isOuterFieldCondition() reports whether a condition references an outer field.

Supported Operators

Operator	Description	Example
equals	Exact string match	type = 'page'
not_equals	Does not match	status != 'deleted'
contains	Substring search	name contains 'test'
starts_with	Prefix match	id starts with 'user_'
ends_with	Suffix match	email ends with '@company.com'
exists	Field is not null/undefined	profile exists

Algorithm Walk-Through

Step 1: Get Field Value

For each item in the array, the algorithm extracts the field value using dot notation:

Item: { user: { type: "admin", status: { active: true } } }
Field: "user.type"

Traversal:
  1. Start at item
  2. Access "user" -> { type: "admin", status: {...} }
  3. Access "type" -> "admin"

Result: "admin"

Key code: getNestedValue()

Step 2: Apply Operator

The operator comparison happens in matchesCondition():

Field value: "admin"
Condition: { field: "user.type", operator: "equals", value: "admin" }

Comparison: String("admin") === "admin"
Result: true

Type coercion: For equals / not_equals, the field value is stringified (String(fieldValue)) before comparison; contains / starts_with / ends_with stringify and treat null/undefined as the empty string. This keeps behavior consistent regardless of the original data type.

Step 3: Filter Array

Items that match the condition are kept; others are removed:

Input: [
  { id: 1, type: "page" },
  { id: 2, type: "database" },
  { id: 3, type: "page" },
  { id: 4, type: "text" }
]

Condition: type = 'page'

Output: [
  { id: 1, type: "page" },
  { id: 3, type: "page" }
]

Operator Details

equals

Exact string match after type coercion.

Field value: true (boolean)
Condition value: "true" (string)
Comparison: String(true) === "true"
Result: true (matches)

not_equals

Inverse of equals.

Field value: "active"
Condition value: "deleted"
Comparison: "active" !== "deleted"
Result: true (matches)

contains

Checks if field value includes the condition value as a substring.

Field value: "hello world"
Condition value: "world"
Check: "hello world".includes("world")
Result: true (matches)

starts_with / ends_with

Checks prefix or suffix of the string value.

Field value: "user_12345"
starts_with "user_": true
ends_with "_12345": true

exists

Checks that the field is present and not null/undefined. No value comparison.

Item: { name: "Alice", email: null }
Field: "name" -> exists: true
Field: "email" -> exists: false (null)
Field: "phone" -> exists: false (undefined)

Nested Field Access

The getNestedValue() function supports dot notation for accessing nested properties:

Item: {
  metadata: {
    properties: {
      type: { value: "page" }
    }
  }
}

Field: "metadata.properties.type.value"

Traversal:
  1. metadata -> { properties: {...} }
  2. properties -> { type: {...} }
  3. type -> { value: "page" }
  4. value -> "page"

Result: "page"

Depth limit: The field selector UI (getAvailableFields()) only shows fields up to 2 levels deep, but the matching algorithm has no depth limit.

Helper Functions

countMatching

Shows how many items pass the filter:

Items: [{type: "page"}, {type: "db"}, {type: "page"}]
Condition: type = 'page'

Result: { matching: 2, total: 3 }

Used in the UI to show "2 of 3 items match" before saving.

getUniqueFieldValues

Lists all distinct values for a field:

Items: [{type: "page"}, {type: "db"}, {type: "page"}]
Field: "type"

Result: ["db", "page"] (sorted)

Used for autocomplete suggestions in the condition value input.

formatCondition

Converts a condition to human-readable text:

{ field: "type", operator: "equals", value: "page" }
-> "type = 'page'"

{ field: "status", operator: "exists" }
-> "status exists"

Key Code References

Functions in filter-condition.ts (search by name — line numbers drift):

Function	Purpose
getNestedValue()	Dot-notation field access
matchesSingleCondition() / matchesCondition()	Single-clause comparison, plus OR-chaining
matchesConditionWithContext() / isOuterFieldCondition()	outer.-prefixed (parent-item) conditions
filterByCondition()	Array filtering
countMatching()	Preview counts
getUniqueFieldValues() / getAvailableFields()	Autocomplete + field discovery (≤2 levels deep)
formatCondition() / formatSingleCondition()	Display formatting (incl. OR clauses)

Integration with Parameter Extraction

When extracting parameter values, filtering is applied at the last array wildcard in the JSON path:

Path: "results[*].children[*].id"
Filter: type = 'page'

The filter applies to the "children" array (last [*]),
not the "results" array.

This ensures filtering happens at the most granular level, matching user intent when they select a nested array path.

Debugging Tips

"Filter excludes everything"

Check:

1. Field name matches exactly (case-sensitive)
2. Value matches data format (e.g., "true" vs true)
3. Operator is appropriate for the comparison

"Filter has no effect"

Verify:

1. Filter condition is saved on the variable
2. Variable is used in the mapping (not a direct mapping)
3. Source data is actually an array

"Nested field not found"

Ensure:

1. Full path uses dots (e.g., user.profile.type)
2. All intermediate objects exist
3. Field names are spelled correctly

Related Documentation

• Parameter Extraction - How filters integrate with extraction
• Conditional Variables - User guide for creating filters