Thursday, 12 October 2023

Using Promise to get paginated response using axios

I am trying to figure out how one can use Promise and async/ await to fetch the data from a rest API that returns the data with some pagination logic.

Specifically I am trying to fetch Azure Users data using MS Graph API. Since azure does not send all the data related to Users in a single API, we have to make multiple calls to build the data for a single Azure User.

To give some insights into my code/logic, here is how I am fetching the Users' basic data using the pagination logic:

let nextPageLink: string| null = null;
do {
      let pagedResult: AggregationPageResult = await this.fetchNextPage(nextPageLink);
      let usersMap: Map<string, any>[] = pagedResult.data!;
      nextPageLink = pagedResult.nextPageLink!;
      this.sendPageResult(usersMap, res);
} while(nextPageLink) 

The pagination logic here is simple and doesn't bother me. Let us say the first call to the Users Graph API returns 100 users. Now, for each of these 100 Users, I have to make separate calls to get their Group Membership data, Directory Roles Data, Last Login Activity and other 'N' number of details.

Here is the code that I am using to get the Group Membership details for each User in current page:

for(let user of usersToSearch) {
    let groupMembershipAPI = "https://graph.microsoft.com/v1.0/users/" + user + "/memberOf/microsoft.graph.group";
    
     let httpConfig: HttpRequestConfig = {
             baseURL: groupMembershipAPI, 
             method: 'GET',
             queryFilter: filters
         }
    
      executions.push(httpClient.executeRequest(httpConfig, true));
}    

let executionResults: HttpResponse [] = await Promise.all(executions);

Now this code works fine and get Group Membership details for all the Users, iff the memberships are less, or in other words the membership data fits to the default page size or any page size for that matter. What if the data is more, what if any particular User has 1000 group memberships and the default page size is only 100. In that case, my current logic will only bring 100 memberships since I am not executing it for next page

For all the requests that I am pushing to executions I also want to take care if the API is sending a valid nextPageLink with it, so that the code goes to fetch the membership data for that User again till the nextPageLink is empty.

Now this is just one scenario for Group Memberships, consider what will happen if I want to bring all the other data like Directory Roles, PIM data, Last Login Activity and more!

The original pagination logic used to bring User Page can be employed here as well, but I am looking for a solution where I can recursively or dynamically add Promise chains or is there any other mechanism that can help me to solve it in a better way? I also want to make sure regarding performance and resource utilisation


Edit 1: My intention to use a different approach for pagination while fetching user properties such as its group memberships/ roles etc is because:

  1. I want to process User pages sequentially and synchronously
  2. On the contrary, while fetching these these properties, they can run in parallel.
  3. Meaning, what I want to achieve is that, fetching User group membership properties can run in parallel with fetching User Roles...
  4. I don't want to wait to start Role fetch because User Group fetch is in progress. Both can run in parallel
  5. If I use the original approach, then that will be sequential.

Pseudo code to explain my above points:

 async public fetchNextPage(nextPageLink: string): AggregationPageResult {
    let propertyFetchResults = []
     1. await fetch the next user page
     2. for every extra property to be fetched with pagination
          2.a propertyFetchResults.push (...result)
     3. await Promise.all(propertyFetchResults)
     4. merge the above property results with users of current page
    
 }


from Using Promise to get paginated response using axios

No comments:

Post a Comment