Skip to main content

Page-based Pagination

Uses page numbers to fetch pages. Simple and intuitive.

When to Use​

  • API accepts page or pageNumber parameter
  • Pages are numbered starting from 1 (or 0)
  • Total pages is often provided

Configuration​

{
  enabled: true,
  type: "page",
  paramsLocation: "query",
  params: {
    pageParam: "page",
    limitParam: "per_page",
    defaultLimit: 100
  },
  response: {
    dataArrayPath: "items",
    totalPagesPath: "total_pages"
  },
  stopCondition: {
    type: "comparison",
    comparePath: "items.length",
    compareOperator: "lt",
    compareValue: "{limit}"
  }
}

Example: GitHub API​

API Behavior​

Requests:

GET /repos/owner/repo/issues?page=1&per_page=100
GET /repos/owner/repo/issues?page=2&per_page=100

Response:

[
  { "id": 1, "title": "Bug report" },
  { "id": 2, "title": "Feature request" }
]

Configuration​

{
  type: "page",
  params: {
    pageParam: "page",
    limitParam: "per_page",
    defaultLimit: 100
  },
  response: {
    dataArrayPath: "$"
  },
  stopCondition: {
    type: "comparison",
    comparePath: "$.length",
    compareOperator: "lt",
    compareValue: "{limit}"
  }
}

Note: GitHub returns array at root, so dataArrayPath is $.

Example: Generic REST API​

API Behavior​

Response:

{
  "items": [...],
  "page": 1,
  "per_page": 100,
  "total_pages": 5,
  "total_items": 450
}

Configuration​

{
  type: "page",
  params: {
    pageParam: "page",
    limitParam: "per_page",
    defaultLimit: 100
  },
  response: {
    dataArrayPath: "items",
    totalPagesPath: "total_pages"
  },
  stopCondition: {
    type: "comparison",
    comparePath: "items.length",
    compareOperator: "lt",
    compareValue: "{limit}"
  }
}

Starting Page Number​

The engine always starts at page 1 and increments by one each request (page 1, 2, 3, …) — the page number sent is the internal loop counter.

Page 1 → items 1-100
Page 2 → items 101-200

There is currently no setting for a 0-based starting page. If an API numbers pages from 0, page-number injection will be off by one; prefer offset-based or cursor-based pagination for those APIs, or use the API's own next-page token.

Stop Conditions for Page-based​

{
  stopCondition: {
    type: "comparison",
    comparePath: "items.length",
    compareOperator: "lt",
    compareValue: "{limit}"
  }
}

Using Total Pages​

Requires response.totalPagesPath to be set so __totalPages__ resolves to the page count from the response:

{
  response: {
    totalPagesPath: "total_pages"
  },
  stopCondition: {
    type: "comparison",
    comparePath: "__page__",
    compareOperator: "gte",
    compareValue: "__totalPages__"
  }
}

Empty Results​

{
  stopCondition: {
    type: "comparison",
    comparePath: "items.length",
    compareOperator: "eq",
    compareValue: 0
  }
}

Common Parameter Names​

APIPage ParameterLimit Parameter
GitHubpageper_page
Genericpagelimit
WordPresspageper_page
LegacypageNumberpageSize

Limitations​

Same as offset-based:

  • Data consistency issues during collection
  • Cannot efficiently skip to arbitrary page
  • Changes during pagination may cause issues

Troubleshooting​

Wrong Page Requested​

Symptoms: Starting at wrong page number

Check:

  • API starts at page 0 or 1
  • First request has correct page number
  • Page incrementing correctly

Missing Last Page​

Symptoms: Last items not collected

Check:

  • Stop condition not triggering early
  • Total pages calculation correct
  • Empty page handling

Duplicate Items​

Symptoms: Same items on multiple pages

Causes:

  • Data changed during collection
  • API issue

Note: Similar issues to offset-based pagination