Offset-based Pagination

Uses offset and limit parameters to fetch pages. The offset increments by the limit for each page.

When to Use

• API accepts offset, skip, or start parameters
• Results are accessed by numeric position
• Total count is often provided

Configuration

{
  enabled: true,
  type: "offset",
  paramsLocation: "query",
  params: {
    offsetParam: "offset",
    limitParam: "limit",
    defaultLimit: 100
  },
  response: {
    dataArrayPath: "results",
    totalCountPath: "total"
  },
  stopCondition: {
    type: "comparison",
    comparePath: "results.length",
    compareOperator: "lt",
    compareValue: "{limit}"
  }
}

Example: Generic REST API

API Behavior

Requests:

GET /api/items?offset=0&limit=100
GET /api/items?offset=100&limit=100
GET /api/items?offset=200&limit=100

Response:

{
  "results": [
    { "id": 1, "name": "Item 1" },
    { "id": 2, "name": "Item 2" }
  ],
  "total": 250,
  "offset": 0,
  "limit": 100
}

Configuration

{
  type: "offset",
  params: {
    offsetParam: "offset",
    limitParam: "limit",
    defaultLimit: 100
  },
  response: {
    dataArrayPath: "results",
    totalCountPath: "total"
  },
  stopCondition: {
    type: "comparison",
    comparePath: "results.length",
    compareOperator: "lt",
    compareValue: "{limit}"
  }
}

How It Works

1. First request: offset=0, returns first 100 items
2. Second request: offset=100, returns items 101-200
3. Third request: offset=200, returns items 201-250
4. Stop: When returned items < limit (only 50 items)

Request Flow

Request 1: GET /items?offset=0&limit=100    → 100 items
Request 2: GET /items?offset=100&limit=100  → 100 items
Request 3: GET /items?offset=200&limit=100  → 50 items (STOP)

Common Offset Parameter Names

API	Offset Parameter	Limit Parameter
Generic	offset	limit
Elasticsearch	from	size
SQL-style	skip	take
Legacy	start	count

Stop Conditions for Offset

Results Less Than Limit (Recommended)

Stop when returned items fewer than requested:

{
  stopCondition: {
    type: "comparison",
    comparePath: "results.length",
    compareOperator: "lt",
    compareValue: "{limit}"
  }
}

Empty Results

Stop when no items returned:

{
  stopCondition: {
    type: "comparison",
    comparePath: "results.length",
    compareOperator: "eq",
    compareValue: 0
  }
}

Note: There is no __offset__ stop-condition token. The engine tracks the offset internally (it advances by the number of items returned each page and injects it into the next request), but you cannot reference it from a stop condition. Use the "results less than limit" or "empty results" comparison above, or a boolean_flag / cursor_empty condition if the API provides one. A totalCountPath can still be configured for reporting, but it does not feed a stop condition — only response.totalPagesPath (via __totalPages__) does.

Limitations of Offset Pagination

Data Consistency Issues

If data changes during pagination:

• New items may be skipped
• Items may be duplicated
• Results may be inconsistent

Mitigation:

• Use cursor-based pagination when available
• Collect during low-activity periods
• Accept potential minor inconsistencies

Performance at Scale

Large offsets can be slow:

• offset=10000 requires scanning 10000 rows
• Gets slower as offset increases

Mitigation:

• Use smaller page sizes
• Prefer cursor-based pagination for large datasets

Troubleshooting

Missing Data

Symptoms: Fewer items than total indicates

Check:

• Stop condition not triggering early
• Offset incrementing correctly
• API returning correct counts

Duplicate Data

Symptoms: Same items appearing twice

Check:

• Data may have changed during collection
• Offset not incrementing by limit
• Consider using cursor-based if available

Performance Issues

Symptoms: Later pages very slow

Causes:

• Database scanning for large offsets
• API rate limiting

Solutions:

• Reduce page size
• Accept longer collection times
• Use cursor-based if available