Many services have limits on the number of records they will return in one http response. Typically services describe this under a “Pagination” section of their “API Basics” section. Previously we didn’t really need to worry about this, but now that we are implementing more actions which return collections, it is becoming common that connector developers need to make paged http calls in order to return all necessary records.
For example, our Zendesk connector has a “Get Organization Tickets” method which is intended to return all active tickets. However, the implementation just called a connector method normally, which means that at most 100 records would ever be returned. Our test organization already has over 300 active tickets so this method was busted.
Happily there is a solution: the http.paginate brick. This allows connector developers to basically implement a repeat/until loop, call an http service repeatedly to request each page of data until all records are received.
Overview of the http.paginate brick:
- It takes 3 inputs: an object, a path, and a Flow.
- The child Flow receives the object as a variant input, the same way that a list processing Flow does. The child Flow should also output an object.
- After each each execution of the child Flow, the system will test the key in the output object identified by path.
- If it doesn’t exist, then the pagination is finished and the object is returned as the output of the http.pagination brick
- If it does exist, then the pagination continues and the child Flow is invoked again with the new object.
Logically, it behaves like this:
function paginate(flow, object, path) {
if (object[path] !== undefined) return paginate( flow, flow(object), path );
else return object;
}
where flow(object) means “call flow with object as the input”.
Example from Zendesk
The Zendesk API paging system is very convenient: it returns a next_page key in its response which is a full URL to the next page, or else is null. Other APIs return tokens, page numbers, record ranges, etc - usually it is a simple math or string operation to generate the next URL to call. Here’s an overview of how the zendesk method works:
- Set up a seed object for use in the initial paginate call. This has a url key which is the first URL to call, and a list key which is where the output list will be stored, set to [].
- Pass that into http.paginate, with path set to the url key.
- The child Flow calls http.get with the url in the object, then generates a new output object. The url key of the output object is set to the next_page value from the http response, which is filtered out (i.e. will be undefined) if null. The list key collects the the output list by appending the new items to the input list from the input object - that’s why the seed object starts off as [].
- After paginate completes, the full output list is in the output.list output.
Here’s a snippet from the Zendesk “Get Organization Tickets” action:
{
"brick": "object.construct",
"id": "pageSeed",
"inputs": {
"url": {
"_type": "string",
"_array": false,
"_value": "https://{{auth.subdomain}}.zendesk.com/api/v2/search.json?query=type:ticket organization:{{input.Get by.Organization ID}}"
},
"list": {
"_type": "object",
"_array": true,
"_value": []
}
},
"outputs": {
"output": {
"_type": "object",
"_array": false
}
}
},
{
"brick": "http.paginate",
"id": "allPages",
"item": "item",
"inputs": {
"object": {
"_type": "object",
"_array": false,
"_value": "{{pageSeed.output}}"
},
"path": {
"_type": "string",
"_array": false,
"_value": "url"
},
"flo": {
"_type": "flo",
"_array": false,
"_value": "pagedGetOrganizationTickets"
}
},
"outputs": {
"output": {
"_type": "object",
"_array": false
}
}
}… and the next brick refers to the output list as “{{allPages.output.list}}” And here’s the metadata method “pagedGetOrganizationTickets” that paginate calls:
{
"name": "pagedGetOrganizationTickets",
"description": "Inner Flow for the paginate brick of action Get Organization Tickets",
"kind": "metadata",
"variant": {
"_type": "object",
"_key": "page",
"_array": false,
"_defaultValue": {}
},
"zebricks": [
{
"brick": "http.get",
"id": "tickets",
"inputs": {
"url": "{{page.url}}",
"auth": {
"username": "{{{auth.username}}}",
"password": "{{{auth.password}}}"
}
},
"outputs": {
"body": "string",
"statusCode": "number",
"headers": "object"
}
},
{
"brick": "list.union",
"id": "union",
"inputs": {
"list1": {
"_type": "object",
"_array": true,
"_value": "{{page.list}}"
},
"list2": {
"_type": "object",
"_array": true,
"_value": "{{tickets.body.results}}"
}
},
"outputs": {
"union": {
"_type": "object",
"_array": true
}
}
},
{
"brick": "object.construct",
"id": "out",
"inputs": {
"url": {
"_type": "string",
"_array": false,
"_value": "{{tickets.body.next_page}}"
},
"list": {
"_type": "object",
"_array": true,
"_value": "{{union.union}}"
}
},
"outputs": {
"output": {
"_type": "object",
"_array": false
}
}
},
{
"brick": "object.filter",
"id": "SJbOE",
"inputs": {
"object": {
"_type": "object",
"_array": false,
"_value": "{{out.output}}"
}
},
"outputs": {
"output": {
"_type": "object",
"_array": false
}
}
}
]
}