Implement iterator to lazily go through the paged response

[Fokko]

Feature Request / Improvement

Recently the support for paged response was added to the list-view operationr: #3349. Right now it will just iterate over all the pages, instead it would be good to see if we can call the next page lazily to avoid many network calls: #3349 (comment)

[harishkesavarao]

Is this open for me to work on? I would like to work on it if so.

[ebyhr]

I suggest discussing and reaching a consensus before beginning your work. This change may become unnecessary once we implement the page-size option in #3377.

[kevinjqliu]

#3377 is merged, thanks! An iterator might still be useful here to lazily fetch pages on the client side. page-size only controls the size of each individual response, not when or whether subsequent pages are fetched. Without an iterator, the client eagerly requests all pages upfront, even if the caller only needs the first few results.

[Fokko]

I think we need to revive #2172. This is a breaking change when it comes to the returned types, but I think that's it. We might want to double check for len if that works the same way.

[rambleraptor]

Iterators don't have a len method, so it would throw an error.

What would we think about creating our own list? That way, we can simply add in pagination ourselves while keeping the list syntax. Here's some quick example code (I think this would work)

class PaginationList(list):
     # The list stores a token + callback to fetch new results.
     self._pagination_token = ""
     self._pagination_callback: Callable = None

     # Calls the next page
     # If pagination has no results, list will be set to empty.
     def next_page() -> None:


# Example
tables: PaginationList = catalog.list_tables()
while len(tables) > 0:
     for table in tables:
         print(table)
     tables.next_page()
          

[Fokko]

Yes, I've also stumbled upon it in #2172 (comment)

I think #3454 is the right approach, this way we don't have to introduce any breaking changes.