Indexing Guide: Build Once, Query Faster

Indexing Guide: Build Once, Query Faster#

GFQL runs without any indexes: every query is a vectorized scan over your dataframes. When your workload is seeded — “expand from these 50 accounts”, “look up this id and hop out” — you can opt into resident indexes: build them once with one call, and seeded queries reuse them automatically after that. This page is the user guide to that lifecycle: what the indexes are, what engages them, when they go stale, and what they cost. For the planner policy knobs and competitive benchmarks, see Adjacency Index.

g = g.gfql_index_all()   # pay once ...
g.gfql(...)              # ... every later seeded lookup on g rides the index

What a resident index is#

gfql_index_all() builds up to three sidecar structures and returns a new g carrying them:

Index

What it accelerates

edge_out_adj

CSR adjacency over outgoing edges: a forward hop becomes an O(degree) positional gather instead of an O(E) scan over every edge.

edge_in_adj

The same for incoming edges (reverse hops; undirected needs both).

node_id

Sorted node-id lookup: seed-row and endpoint materialization become positional gathers instead of O(N) scans. Requires unique node ids — gfql_index_all() silently skips it otherwise (adjacency is still built).

node_prop

Sorted lookup on a node property column (a secondary index): a seed predicate like MATCH (m {id: 42}) on a column that is not the node-id binding becomes a positional gather instead of an O(N) scan. Duplicate values are fine (all matching rows are gathered). Integer columns without nulls only — anything else declines to the scan. Opt-in per column.

They are sidecars over row positions: your .edges / .nodes frames are never reordered or copied, and the resident footprint is visible per index via g.show_indexes() (the nbytes column). The model is pay-as-you-go: one O(E log E) build, amortized over every seeded query afterward. Nothing is built unless you ask.

Quick start#

A complete, runnable example:

import pandas as pd
import graphistry
from graphistry import n, e_forward, is_in

# A small synthetic graph: 6 accounts, 8 transfers
edges_df = pd.DataFrame({
    "src": [0, 0, 1, 1, 2, 3, 4, 5],
    "dst": [1, 2, 2, 3, 4, 4, 5, 0],
    "amount": [10, 20, 30, 40, 50, 60, 70, 80],
})
nodes_df = pd.DataFrame({
    "id": [0, 1, 2, 3, 4, 5],
    "risk": ["low", "high", "low", "high", "low", "low"],
})
g = graphistry.edges(edges_df, "src", "dst").nodes(nodes_df, "id")

# Pay once: build out+in adjacency + node-id indexes (resident on the returned g)
g_indexed = g.gfql_index_all()
print(g_indexed.show_indexes()[["name", "kind", "key_col", "n_keys", "valid"]])

# Seeded 1-hop: who did accounts 0 and 3 transfer to?
out = g_indexed.gfql([n({"id": is_in([0, 3])}), e_forward(), n()])
print(sorted(out._nodes["id"].tolist()))          # [0, 1, 2, 3, 4]

# Decline safety: the same query without any index gives the SAME answer
out_scan = g.gfql([n({"id": is_in([0, 3])}), e_forward(), n()])
assert sorted(out._nodes["id"].tolist()) == sorted(out_scan._nodes["id"].tolist())

# Direct hop() uses the index too
hop_out = g_indexed.hop(nodes=pd.DataFrame({"id": [0]}), hops=2, direction="forward")
print(sorted(hop_out._nodes["id"].tolist()))      # [0, 1, 2, 3, 4]

The lifecycle calls, all returning a new g (functional style, like the rest of the API):

g = g.gfql_index_all()               # out+in adjacency + node_id (the one-liner)
g = g.gfql_index_edges("forward")    # or just one direction: 'forward'|'reverse'|'both'
g = g.create_index("edge_out_adj")   # or one kind: 'edge_out_adj'|'edge_in_adj'|'node_id'
g = g.gfql_index_node_props(["id"])  # secondary indexes on node property columns
g.show_indexes()                     # pandas DataFrame: kind, engine, ..., valid, usable, reason
g = g.drop_index()                   # drop all (or drop_index("edge_out_adj"))

Unlike gfql_index_all(), an explicit create_index("node_id") raises on non-unique node ids rather than skipping.

Seeding on a property (secondary index)#

The node_id index covers the column bound as the node id. A query that seeds on a different column — a business key such as MATCH (m {id: 42}) when the graph is keyed by something else — otherwise scans the whole node table to find its seed. node_prop indexes that column instead:

g = g.gfql_index_all().gfql_index_node_props(["id"])   # skips unindexable columns
g = g.create_index("node_prop", column="id")           # or one column, raising if it cannot

# equivalently over the Cypher DDL / JSON surfaces
g.gfql('CREATE GFQL INDEX FOR node_prop ON id')
g = g.drop_index("node_prop", column="id")             # or drop_index("node_prop") for all

When several indexed columns appear in one seed predicate, the planner gathers on the most selective one (estimated for free from the index’s own offsets) and applies the remaining predicates to those candidates, so results never depend on which index happens to be resident. As with every kind, a missing, stale, or cost-gated-out index falls back to the scan.

What uses the index today#

On 0.58.0, a resident index is consumed automatically by:

  • Seeded typed-hop fast paths (native chain or Cypher): a seeded typed 1-hop — [n({"id": is_in([...])}), e_forward(), n(...)] or MATCH (m {id: $x})-[:T]->(p) RETURN p — including the single-alias property RETURN form (RETURN p.a AS x, p.b). The seed lookup, frontier expansion, and endpoint materialization all become positional index gathers, so the lookup stops paying graph-size costs.

  • Property-seeded lookups: the seed filter may hit a property column (e.g. MATCH (m {id: $x}) when the graph is bound on a different key column). The seed row falls back to a property scan, but the adjacency and endpoint gathers still engage — the common pattern of a synthetic key binding plus an id property filter is covered.

  • Direct g.hop(nodes=..., hops=..., direction=...) — the O(degree) gather path.

Not yet covered: the general Polars chain traversal — multi-hop and multi-alias queries executed by the Polars chain engine take their scan/join path even with an index resident. Coverage is decline-gated: anything the index path does not handle falls back to the scan, so the worst case is the speed you already had.

Staleness and safety#

The validity contract is simple: an index serves only while the frames it was built over are unchanged (checked by object identity plus a structural fingerprint at use time). Consequences:

  • Rebinding .edges(...) invalidates the edge adjacency indexes; rebinding .nodes(...) invalidates the node-id index. A stale index is treated as absent — skipped, never consulted.

  • g.show_indexes() reports liveness in the valid column, so you can see at a glance whether a rebind knocked an index out.

  • valid alone is not “this index will serve your query”: indexes are also engine-specific. The usable column is True only when the index is fresh AND built for the resolved query engine (shown in query_engine); otherwise reason explains the decline — e.g. a polars-built index on a graph whose default queries resolve to pandas shows usable=False with an engine-mismatch reason. Pass show_indexes(engine=...) to preview an explicit engine choice.

  • Rebuild by calling gfql_index_all() again on the rebound g.

new_edges_df = edges_df.assign(amount=edges_df["amount"] + 1)
g2 = g_indexed.edges(new_edges_df, "src", "dst")
g2.show_indexes()          # edge_out_adj / edge_in_adj now valid=False; node_id still True
g2 = g2.gfql_index_all()   # pay again for the new frame; all valid=True

Declines are always safe. Whether an index is missing, stale, or the query shape is uncovered, results are identical either way — indexes only ever change speed, never answers (index-vs-scan parity is differentially tested across engines).

Note

Stability. The index kinds, sidecar layout, and show_indexes() columns describe the current implementation and may evolve between releases; the stable contract is the lifecycle and decline-safety guarantees on this page — pay once, automatic reuse, staleness on rebind, and identical results with or without an index.

Engines#

  • pandas and cuDF: build with the default gfql_index_all() (AUTO resolves to the frames’ engine — numpy sidecars for pandas, on-device cupy for cuDF).

  • Polars: on 0.58.0, pass the engine explicitly — gfql_index_all(engine='polars'). With AUTO, an index build on Polars frames swaps them to pandas (the same AUTO behavior described in Choosing a GFQL Engine: pandas, Polars, cuDF, Polars-GPU; a fix is tracked in PR #1767).

  • Polars-GPU: rides the Polars-tagged index — an index built with engine='polars' (or 'polars-gpu') serves both.

What it costs, what it buys#

Build (the “pay” side): one-time and O(E log E) — a sort over the edge frame, amortized across every subsequent seeded query. index_policy='auto' only pays it when the planner predicts a selective query will earn it back.

Seeded lookup (the “go” side): on a covered shape, the seeded lookup drops from the general path to the fast path, and again with the index resident, on both CPU engines.

Flat in graph size: a direct seeded g.hop() with the index resident turns the O(E) scan into an O(degree) gather, so its cost tracks the seeds’ neighborhood rather than the graph.

Measured figures are published on GFQL Performance: Measured Against Graph Databases and Adjacency Index: Fast Lookups from Known Nodes only, and only when they trace to a committed benchmark artifact.

See also#