Introducing BGPKIT Broker

BGPKIT Broker is a data API service that focuses on building a BGP data file index to enable searching for public/private BGP data files with custom filters. It is one of the building block components designed by BGPKIT to facilitate BGP data processing with ease.

The first step to investigate a BGP event.

Imagine this scenario: a malicious player just attempted to hijack a IP prefix using BGP announcements, and you are interested in learning what exactly happened during that half-hour down time of the victim network, how would you start investigating?

Collecting evidences. Luckily for us, the actors on the Internet, good and bad, always leave traces behind if they use BGP as their method. There are a number of reputable public BGP route collectors operating for years collecting all BGP messages received from their connected router peers and dump them into regular dump files. The most used projects are RouteViews and RIPE RIS Data.

Files are all over the places. There are more than 60 different data collectors from the two projects alone, and each publishing their data under separate sites. The two projects also have different data file structure and compression algorithms for their data dump files. It is not hard to go find one data file during the interested event time from one collector, but it will be a real hassle to gather URLs toward all data files that include information within a specified time range.

Collector’s data is published as compressed MRT files at regular frequency.
Collector’s data is published as compressed MRT files at regular frequency.

BGPKIT Broker — A BGP Data File Index API Service

We designed the BGPKIT Broker to resolve one and only one problem: quickly collect links to the BGP data files that matches a filtering criteria.

You can filter BGP data using multiple criteria:

  • start_ts : UNIX timestamp that all files must be dumped after
  • end_ts : UNIX timestamp that all files must be dumped before
  • data_type : the type of the data file, can be update or rib
  • collector: the collector ID that the files are generated from
  • project: the data collection project, can be route-views or riperis
  • page and page_size: the pagination control for collecting a large number of files

Here is an example REST API call: https://api.broker.bgpkit.com/v1/search?data_type=update&start_ts=1633046400&end_ts=1633132800&collector=rrc00&project=riperis&page%20=2&page_size=3

It asks for all data updates files dumped between 1633046400 and 1633132800, from RIPE RIS’s collector rrc00. It also requested for the second page of the results and each page contains 3 items.

BGPKIT Broker API service is freely available to use, hosted at https://api.broker.bgpkit.com/v1/. The documentation is available at https://docs.broker.bgpkit.com/

BGPKIT Broker Rust API

The BGPKIT Broker API service is built entirely using Rust, and of course we also developed native Rust API to access the broker data with ease using Rust.

Example BGPKIT Broker Rust API call

The Rust API is open source under MIT license, and with the free API, you can already build your own workflow today!

GitHub - bgpkit/bgpkit-broker: A library to access BGPKIT Broker API and enable searching for BGP data archive files over time from public available data sources.
A library to access BGPKIT Broker API and enable searching for BGP data archive files over time from public available data sources. - GitHub - bgpkit/bgpkit-broker: A library to access BGPKIT Broke...

Easy On-premise Deployment

For API service like this, especially that it also collects and indexes data of over 10 years span, one might imagine the deployment could be complex and slow.

For BGPKIT Broker, we spent extra efforts to make the API deployment process as quickly as possible, and also efficient on resource consumption. For references, we bootstrapped the entirety of the database in under 5 minutes, and cost less than 1 GB storage to store the information in PostgreSQL database. The whole database and API run fluently with less than 500MB of RAM usage. This allows use to deployment extra instances when under heavy load without costing a fortune.

Users who need dedicated resource allocation for query performance can contact us for private API hosting that are not shared by others. Enterprise option is also available for on-premise deployment with customization consultation available. If you are interested in testing it out, feel free to shoot us an email at [email protected]

For more information, checkout our website!

BGPKIT Broker
API service for searching BGP archive data files

Subscribe to BGPKIT

Don’t miss out on the latest issues. Sign up now to get access to the library of members-only issues.
[email protected]
Subscribe