Sunday 25 October 2020

RxJS: Parallel http call to paginated API using NestJS HttpService

Im using NestJS, and this is my current implementation to make parallel http request:

@Injectable()
export class AppService {
  constructor(private readonly http: HttpService) {}

  private fetchData(index: number) {
    Logger.log(`Collect index: ${index}`);

    return this.http
      .get<Passenger>(
        `https://api.instantwebtools.net/v1/passenger?page=${index}&size=100`,
        { validateStatus: null },
      )
      .pipe(concatMap(response => of(response.data)));
  }

  async getAllData() {
    let a = 0;
    const collect: Passenger[] = [];
    const $subject = new BehaviorSubject(a);

    await $subject
      .pipe(
        flatMap(index =>
          forkJoin([
            this.fetchData(index),
            this.fetchData(index + 1),
            this.fetchData(index + 2),
          ]).pipe(mergeAll()),
        ),
        tap(data => {
          collect.push(data);

          if (data?.data?.length === 0) {
            $subject.complete();     // stop the stream
          } else {
            a += 3;     // increment by 3, because the request is 3 times at a time
            $subject.next(a);
          }
        }),
      )
      .toPromise();

    return collect;
  }
}

This service is to collect the 3rd party data. As for now, the fetchData() function is called multiple times according to how many parallel requests I want at a time. I use a dummy API for the test, but in the real scenario the API endpoint size limit is 100 and it doesn't return the meta information about how much the totalPage. It just returns the empty data when the last page is reached.

The goal is to make a parallel request and combine the result at the end. I'm doing this to keep the request time as minimum as possible and because the API itself has a rate limit of 50 requests per second. How to optimize this code?



from RxJS: Parallel http call to paginated API using NestJS HttpService

No comments:

Post a Comment