Pagination Reference

This document provides comprehensive reference for configuring pagination in Traces connectors.

Overview

Pagination ensures complete data collection from APIs that return results in pages. Without proper pagination configuration, you'll only collect the first page of data.

Most list endpoints return paginated results. Your task is to:

1. Identify the pagination pattern used by the API
2. Configure the correct parameters and paths
3. Set an appropriate stop condition
4. Test to verify all pages are retrieved

Decision Tree

Use this flowchart to determine which pagination type to use:

Does the API response contain a cursor/token for the next page?
├── YES → Does the cursor contain a full URL path?
│         ├── YES → Use CURSOR with cursorIsFullPath: true
│         └── NO → Use CURSOR (standard)
└── NO → Does the API accept page numbers (page=1, page=2)?
         ├── YES → Use PAGE
         └── NO → Does the API accept offset values (offset=0, offset=100)?
                  ├── YES → Use OFFSET
                  └── NO → API may not support pagination

Note: APIs that only signal the next page via an RFC 5988 `Link` header
(GitHub/GitLab style) are not driven by header parsing today. Use PAGE with
the API's `page`/`per_page` query parameters instead.

Pagination Types

Type	When to Use
Cursor-based	API returns cursor/token for next page
URI-style Cursor	Cursor is a complete URL path
Offset-based	API uses offset and limit parameters
Page-based	API uses page numbers

Configuration Structure

{
  enabled: boolean;
  type: "cursor" | "offset" | "page";
  paramsLocation: "query" | "body";

  params: {
    cursorParam?: string;      // Parameter name for cursor
    limitParam?: string;       // Parameter name for page size
    offsetParam?: string;      // Parameter name for offset
    pageParam?: string;        // Parameter name for page number
    defaultLimit?: number;     // Default page size (default 100)
  };

  response: {
    nextCursorPath?: string;   // Path to next page cursor
    hasMorePath?: string;      // Path to has_more boolean
    dataArrayPath?: string;    // Path to data array (used to count items)
    totalCountPath?: string;   // Path to total count
    totalPagesPath?: string;   // Path to total page count (for __totalPages__)
  };

  cursorIsFullPath?: boolean;  // Cursor is a complete URL/path

  stopCondition: {
    type: "cursor_empty" | "boolean_flag" | "comparison";
    flagPath?: string;
    flagValue?: boolean;
    comparePath?: string;      // JSON path, "...length", or "__page__"
    compareOperator?: "lt" | "lte" | "eq" | "neq" | "gte" | "gt";
    compareValue?: number | string;  // number, "{limit}", or "__totalPages__"
  };
}

The config schema also accepts type: "link_header" | "custom" and paramsLocation: "path", but only the cursor / offset / page types and the query / body locations are executed by the pagination engine. The dropdowns expose the extra values, but selecting them will not drive additional requests. Stick to the documented types/locations.

Quick Reference Tables

Request Parameters

Field	Type	Description	Example
cursorParam	string	Query param name for cursor	"cursor", "starting_after"
limitParam	string	Query param name for page size	"limit", "per_page"
offsetParam	string	Query param name for offset	"offset", "skip"
pageParam	string	Query param name for page number	"page"
defaultLimit	number	Default page size	100, 50

Response Paths

Field	Type	Description	Example
nextCursorPath	string	JSON path to next cursor	"meta.next_cursor"
hasMorePath	string	JSON path to has_more flag	"has_more"
dataArrayPath	string	JSON path to data array	"data", "items"
totalCountPath	string	JSON path to total count	"meta.total"

Stop Conditions

Type	Use When
cursor_empty	Cursor becomes null/empty on last page
boolean_flag	API returns has_more: false
comparison	Compare value (e.g., items < limit)

Common Patterns

Stripe-style

{
  "data": [...],
  "has_more": true
}

→ Cursor-based with starting_after parameter, has_more stop condition

Slack-style

{
  "members": [...],
  "response_metadata": {
    "next_cursor": "dXNlcjpVMDYxTkZUVDI="
  }
}

→ Cursor-based with cursor parameter, empty cursor stop condition

Twilio-style

{
  "messages": [...],
  "next_page_uri": "/2010-04-01/Messages.json?Page=1"
}

→ Cursor-based with cursorIsFullPath: true

GitHub-style

[...]

With Link header: <url?page=2>; rel="next"

→ Page-based pagination with page/per_page query parameters (the Link header itself is not consumed)

Next Steps

• Cursor-based Pagination
• URI-style Cursor
• Offset-based Pagination
• Page-based Pagination
• Stop Conditions