I'm using a function which returns data in a paginated form. So it'll return max 100 items and a key to retrieve the next 100 items. I want to retrieve all the items available.

How do I recursively achieve this? Is recursion a good choice here? Can I do it any other way without recursion?

I'm using Bluebird 3x as the promises library.

Here is a snippet of what I'm trying to achieve:

getEndpoints(null, platformApplication)
  .then(function(allEndpoints) {
    // process on allEndpoints
  });


function getEndpoints(nextToken, platformApplication) {
  var params = {
    PlatformApplicationArn: platformApplication
  };

  if (nextToken) {
    params.NextToken = nextToken;
  }

  return sns.listEndpointsByPlatformApplicationAsync(params)
    .then(function(data) {
      if (data.NextToken) {
        // There is more data available that I want to retrieve.
        // But the problem here is that getEndpoints return a promise
        // and not the array. How do I chain this here so that 
        // in the end I get an array of all the endpoints concatenated.
        var moreEndpoints = getEndpoints(data.NextToken, platformApplication);
        moreEndpoints.push.apply(data.Endpoints, moreEndpoints);
      }

      return data.Endpoints;
    });
}

But the problem is that if there is more data to be retrieved (see if (data.NextToken) { ... }), how do I chain the promises up so that in the end I get the list of all endpoints etc.

I'm using a function which returns data in a paginated form. So it'll return max 100 items and a key to retrieve the next 100 items. I want to retrieve all the items available.

How do I recursively achieve this? Is recursion a good choice here? Can I do it any other way without recursion?

I'm using Bluebird 3x as the promises library.

Here is a snippet of what I'm trying to achieve:

getEndpoints(null, platformApplication)
  .then(function(allEndpoints) {
    // process on allEndpoints
  });


function getEndpoints(nextToken, platformApplication) {
  var params = {
    PlatformApplicationArn: platformApplication
  };

  if (nextToken) {
    params.NextToken = nextToken;
  }

  return sns.listEndpointsByPlatformApplicationAsync(params)
    .then(function(data) {
      if (data.NextToken) {
        // There is more data available that I want to retrieve.
        // But the problem here is that getEndpoints return a promise
        // and not the array. How do I chain this here so that 
        // in the end I get an array of all the endpoints concatenated.
        var moreEndpoints = getEndpoints(data.NextToken, platformApplication);
        moreEndpoints.push.apply(data.Endpoints, moreEndpoints);
      }

      return data.Endpoints;
    });
}

But the problem is that if there is more data to be retrieved (see if (data.NextToken) { ... }), how do I chain the promises up so that in the end I get the list of all endpoints etc.

• javascript
• node.js
• pagination
• promise
• bluebird

Share Improve this question Follow edited Feb 1, 2016 at 20:00 Shikasta_Kashti asked Feb 1, 2016 at 19:55 Shikasta_KashtiShikasta_Kashti 79122 gold badges1515 silver badges2929 bronze badges 4

• When do you want to get the next 100? What triggers that? – Tad Donaghe Commented Feb 1, 2016 at 19:56
• When there is more data available. If data.NextToken is available, that means more data is available so I have to get it. – Shikasta_Kashti Commented Feb 1, 2016 at 19:59
• So you're trying to get all endpoints? – CrimsonChris Commented Feb 1, 2016 at 20:00
• Yes. Sorry this wasn't clear in the question, updated it. – Shikasta_Kashti Commented Feb 1, 2016 at 20:01

Add a ment  | 

2 Answers 2

Sorted by: Reset to default 5

Recursion is probably the easiest way to get all the endpoints.

function getAllEndpoints(platformApplication) {
    return getEndpoints(null, platformApplication);
}

function getEndpoints(nextToken, platformApplication, endpoints = []) {
  var params = {
    PlatformApplicationArn: platformApplication
  };

  if (nextToken) {
    params.NextToken = nextToken;
  }

  return sns.listEndpointsByPlatformApplicationAsync(params)
    .then(function(data) {
      endpoints.push.apply(endpoints, data.Endpoints);
      if (data.NextToken) {
          return getEndpoints(data.NextToken, platformApplication, endpoints);
      } else {
          return endpoints;
      }
    });
}

For a more general purpose example of recursively getting and returning data from a paginated endpoint, here is what I came up with:

getData(page, dataArray) {
  return new Promise((resolve, reject) => {
    getEndpointHere(
      {         
        page,
        pageSize: 50,
      },     
      (err, result) => {
        if (err)
          return console.error("there was a problem retrieving your data");          

        dataArray = dataArray.concat(result);

        if (result.length < 50) {
          resolve(dataArray);
        } else {
          resolve(getData(page + 1, dataArray));
        }
      }
    );
  });
}

getData(1, [])
.then((res) => {      
  console.log("SEVERAL PAGES OF DATA", res);
})

This example is employing a callback -- (err, result) -- however, that logic could be extracted out to handle the response from the endpoint. The resource I was consuming does not return a cursor or "next" token indicating if there are more records, so I used the logic of if the response has less than 50 records as the basis for continuing to request more data.