fetcher.core

Implements Core that is used in other modules.

class web3cat.fetcher.core.Core(rpc: Optional[str] = None, cache_path: Optional[str] = None, block_grid_step: int = 1000, w3: Optional[Web3] = None, conn: Optional[Connection] = None)

Bases: object

A base class for any class that wants to use an Ethereum RPC or Sqlite3 cache database.

When deriving this class, you’re providing arguments like rpc url or OS path to the database. The resources are instantiated on demand though. It means that if you’re just using the Ethereum RPC it’s sufficient to supply only the rpc endpoint and skip OS path to the database in the constructor.

So this class lightweight and safe to derive from any other class.

Caching

The web3 instance and chain_id are cached by the rpc url key. The sqlite3 connection is cached by the OS path of the database.

While this might not work well in a multi-threaded scenario, for single-threaded there’s no overhead like making new connections and, for example, querying chain_id each time it’s accessed.

Block grid

It’s often desirable to convert block number to timestamp and vice versa. In a way, blocks are blockchain-readable, and timestamps are human-readable.

However, fetching every single block is impractical in many cases.

That’s why the following algorithm is used for timestamp estimation:

  1. We make a block number grid with a width specified by the block_grid_step parameter.

  2. For each block number, we take the two closest grid blocks (below and above).

  3. Fetch the grid blocks

  4. Assume \(a_n\) and \(a_t\) is a number and a timestamp for the block above

  5. Assume \(b_n\) and \(b_t\) is a number and a timestamp for the block below

  6. Assume \(c_n\) and \(c_t\) is a number and a timestamp for the block we’re looking for

  7. \(w = (c_n - b_n) / (a_n - b_n)\)

  8. Then \(c_t = b_t \cdot (1-w) + a_t * w\)

This algorithm gives a reasonably good approximation for the block timestamp and considerably reduces the number of block fetches. For example, if we have 500 events happening in the 1000 - 2000 block range, then we fetch only two blocks (1000, 2000) instead of 500.

If you still want the exact precision, use block_grid_step = 1.

Warning

It’s highly advisable to use a single block_grid_step for all data. Otherwise (in theory) the happens-before relationship might be violated for the data points.

Parameters:

  • rpc – An https Ethereum RPC endpoint uri

  • cache_path – OS path to the cache database

  • block_grid_step – Distance between two adjacent grid blocks

  • w3 – an instance of web3 (overrides rpc)

  • conn – an instance of database connection (overrides cache_path)

cache: str | None

OS path to the cache database. Can be None if sqlite3.Connection is injected directly.

property block_grid_step: int

Distance between two adjacent grid blocks

property chain_id

Chain id for the current web3 connection

property rpc: str

Ethereum RPC endpoint.

Note

Can be None if web3.Web3 is injected directly.

property w3: Web3

web3.Web3 instance for working with Ethereum RPC

property conn: Connection

sqlite3.Connection to a database cache

web3cat.fetcher.core.connection_from_path(path: str) → Connection

Creates a connection to a database at path. If the file at path doesn’t exist, creates a new one and initializes a database schema.

Parameters:

path – The absolute path to the database

Returns:

An instance of sqlite3 Connection

Note

The schema migrations are currently not supported.