BGPKIT Broker v0.9.0: Better Pagination and New Collectors

We are excited to announce the release of BGPKIT Broker v0.9.0. This release introduces total count support for efficient pagination, making it easier to build data exploration interfaces and manage large result sets. We also expand our collector coverage with two new Internet Exchange points.

Previously on BGPKIT Broker

Since the major v0.7 release that unified the architecture into a single SQLite-backed CLI application, BGPKIT Broker has been serving the community with stable uptime and continuous data coverage. The v0.8 series brought several enhancements including automated backup systems, configuration validation, and convenient query shortcuts.

However, one common challenge remained for developers building interfaces on top of Broker: pagination. When querying large time ranges, users needed to fetch all results to know the total count, making it inefficient to build proper pagination controls or display result statistics.

V0.9: Efficient Pagination Support

Version 0.9.0 addresses pagination needs with total count support at both the SDK and API levels. This allows applications to fetch result counts independently from the data itself, enabling responsive user interfaces without over-fetching data.

SDK: query_total_count() Method

The SDK now provides a dedicated query_total_count() method that returns the total number of matching items without retrieving the actual data. This is useful when you need to know how many results exist before deciding how to paginate through them.

use bgpkit_broker::BgpkitBroker;

#[tokio::main]
async fn main() {
    let broker = BgpkitBroker::new()
        .ts_start("2024-10-01")
        .ts_end("2024-10-02")
        .project("route-views");

    // Get total count without fetching items
    let total = broker.query_total_count().await.unwrap();
    println!("Total matching items: {}", total);

    // Now fetch paginated results
    let result = broker
        .page_size(100)
        .page(1)
        .query()
        .await
        .unwrap();

    println!("Retrieved {} items out of {} total", 
             result.data.len(), 
             result.total.unwrap_or(0));
}

This method executes a COUNT(*) query against the database with the same filters as your main query, returning just the number rather than all the data. For large time ranges with thousands of results, this can save significant bandwidth and processing time.

API: Total Field in Search Results

The /v3/search endpoint now includes a total field in every response, providing the total count of matching items alongside the paginated results. This enhancement makes it straightforward to build pagination controls in web interfaces or CLI tools.

curl "https://api.bgpkit.com/v3/broker/search?ts_start=2024-10-01&ts_end=2024-10-02&page_size=10&page=1"

Response:

{
  "total": 1234,
  "data": [
    {
      "ts_start": 1727740800,
      "ts_end": 1727740800,
      "collector": "route-views2",
      "project": "route-views",
      "url": "...",
      ...
    }
  ]
}

With the total field, client applications can:

• Display "showing X of Y results" to users

• Calculate the number of pages needed for pagination

• Decide whether to fetch all results or paginate

• Provide accurate progress indicators for data processing

New RouteViews Collectors

This release adds support for two new RouteViews collectors:

• hkix.hkg: Hong Kong Internet Exchange (HKIX) collector in Hong Kong

• ix-br.gru: IX.br (PTT.br) in São Paulo

These additions enhance BGPKIT Broker's global coverage, providing more diverse vantage points into internet routing behavior. Users who update to v0.9.0 can run bgpkit-broker update to automatically bootstrap data for these new collectors.

Behind the Scenes Improvements

Beyond the user-facing features, we also made several code improvements:

• Refactored bootstrap download logging to centralize progress reporting and eliminate redundant code paths

• Updated the oneio dependency to v0.20.0 with optimized feature flags for better handling rustls providers

• Enhanced test coverage overall

Upgrading to v0.9.0

For SDK Users

Update your Cargo.toml to use the latest v0.9 release:

[dependencies]
bgpkit-broker = "0.9"

Then run:

cargo update

The new query_total_count() method is immediately available for use. The total field in BrokerQueryResult is backward compatible as an optional field.

For Self-Hosted Instances

If you run your own BGPKIT Broker instance, update to the latest version:

cargo install --force bgpkit-broker --version 0.9.0 --features cli

Or pull the latest Docker image:

docker pull bgpkit/bgpkit-broker:latest

After upgrading, run the update command to add the new collectors:

bgpkit-broker update --db-path your-database.sqlite3

The update process will automatically detect and bootstrap historical data for the two new collectors.

Looking Forward

BGPKIT Broker continues to serve as a fundamental component for BGP data processing pipelines. With v0.9.0, we focused on making the developer experience better for building applications that need pagination and result statistics. We continue to maintain the public Broker instance at https://api.bgpkit.com/v3/broker with 99.996% uptime, and encourage users to deploy on-premise instances for production pipelines to reduce external dependencies.

For the full v0.9.0 release notes, please check out our GitHub release page. If you have any comments, please drop us a message on Twitter, Mastodon, Bluesky or email.

Get Started

Ready to try out the new features? Here are some resources:

• GitHub: github.com/bgpkit/bgpkit-broker

• Examples: Browse example code demonstrating common usage patterns

• Public API: Try queries at https://api.bgpkit.com

Have questions or feedback? Open an issue on GitHub or join discussions in our Discord Channel.

💖

If you find our libraries and services useful, we would highly appreciate if you consider sponsoring us on GitHub.