.. _gfql-indexing: Indexing Guide: Build Once, Query Faster ======================================== GFQL runs without any indexes: every query is a vectorized scan over your dataframes. When your workload is **seeded** — "expand from these 50 accounts", "look up this id and hop out" — you can opt into **resident indexes**: build them once with one call, and seeded queries reuse them automatically after that. This page is the user guide to that lifecycle: what the indexes are, what engages them, when they go stale, and what they cost. For the planner policy knobs and competitive benchmarks, see :doc:`Adjacency Index `. .. doc-test: skip .. code-block:: python g = g.gfql_index_all() # pay once ... g.gfql(...) # ... every later seeded lookup on g rides the index What a resident index is ------------------------ ``gfql_index_all()`` builds up to three sidecar structures and returns a new ``g`` carrying them: .. list-table:: :header-rows: 1 :widths: 22 78 * - Index - What it accelerates * - ``edge_out_adj`` - CSR adjacency over outgoing edges: a forward hop becomes an ``O(degree)`` positional gather instead of an ``O(E)`` scan over every edge. * - ``edge_in_adj`` - The same for incoming edges (reverse hops; undirected needs both). * - ``node_id`` - Sorted node-id lookup: seed-row and endpoint materialization become positional gathers instead of ``O(N)`` scans. Requires unique node ids — ``gfql_index_all()`` silently skips it otherwise (adjacency is still built). * - ``node_prop`` - Sorted lookup on a node **property** column (a secondary index): a seed predicate like ``MATCH (m {id: 42})`` on a column that is not the node-id binding becomes a positional gather instead of an ``O(N)`` scan. Duplicate values are fine (all matching rows are gathered). Integer columns without nulls only — anything else declines to the scan. Opt-in per column. They are **sidecars over row positions**: your ``.edges`` / ``.nodes`` frames are never reordered or copied, and the resident footprint is visible per index via ``g.show_indexes()`` (the ``nbytes`` column). The model is **pay-as-you-go**: one ``O(E log E)`` build, amortized over every seeded query afterward. Nothing is built unless you ask. Quick start ----------- A complete, runnable example: .. code-block:: python import pandas as pd import graphistry from graphistry import n, e_forward, is_in # A small synthetic graph: 6 accounts, 8 transfers edges_df = pd.DataFrame({ "src": [0, 0, 1, 1, 2, 3, 4, 5], "dst": [1, 2, 2, 3, 4, 4, 5, 0], "amount": [10, 20, 30, 40, 50, 60, 70, 80], }) nodes_df = pd.DataFrame({ "id": [0, 1, 2, 3, 4, 5], "risk": ["low", "high", "low", "high", "low", "low"], }) g = graphistry.edges(edges_df, "src", "dst").nodes(nodes_df, "id") # Pay once: build out+in adjacency + node-id indexes (resident on the returned g) g_indexed = g.gfql_index_all() print(g_indexed.show_indexes()[["name", "kind", "key_col", "n_keys", "valid"]]) # Seeded 1-hop: who did accounts 0 and 3 transfer to? out = g_indexed.gfql([n({"id": is_in([0, 3])}), e_forward(), n()]) print(sorted(out._nodes["id"].tolist())) # [0, 1, 2, 3, 4] # Decline safety: the same query without any index gives the SAME answer out_scan = g.gfql([n({"id": is_in([0, 3])}), e_forward(), n()]) assert sorted(out._nodes["id"].tolist()) == sorted(out_scan._nodes["id"].tolist()) # Direct hop() uses the index too hop_out = g_indexed.hop(nodes=pd.DataFrame({"id": [0]}), hops=2, direction="forward") print(sorted(hop_out._nodes["id"].tolist())) # [0, 1, 2, 3, 4] The lifecycle calls, all returning a new ``g`` (functional style, like the rest of the API): .. code-block:: python g = g.gfql_index_all() # out+in adjacency + node_id (the one-liner) g = g.gfql_index_edges("forward") # or just one direction: 'forward'|'reverse'|'both' g = g.create_index("edge_out_adj") # or one kind: 'edge_out_adj'|'edge_in_adj'|'node_id' g = g.gfql_index_node_props(["id"]) # secondary indexes on node property columns g.show_indexes() # pandas DataFrame: kind, engine, ..., valid, usable, reason g = g.drop_index() # drop all (or drop_index("edge_out_adj")) Unlike ``gfql_index_all()``, an explicit ``create_index("node_id")`` **raises** on non-unique node ids rather than skipping. Seeding on a property (secondary index) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``node_id`` index covers the column bound as the node id. A query that seeds on a *different* column — a business key such as ``MATCH (m {id: 42})`` when the graph is keyed by something else — otherwise scans the whole node table to find its seed. ``node_prop`` indexes that column instead: .. code-block:: python g = g.gfql_index_all().gfql_index_node_props(["id"]) # skips unindexable columns g = g.create_index("node_prop", column="id") # or one column, raising if it cannot # equivalently over the Cypher DDL / JSON surfaces g.gfql('CREATE GFQL INDEX FOR node_prop ON id') g = g.drop_index("node_prop", column="id") # or drop_index("node_prop") for all When several indexed columns appear in one seed predicate, the planner gathers on the **most selective** one (estimated for free from the index's own offsets) and applies the remaining predicates to those candidates, so results never depend on which index happens to be resident. As with every kind, a missing, stale, or cost-gated-out index falls back to the scan. What uses the index today ------------------------- On 0.58.0, a resident index is consumed automatically by: - **Seeded typed-hop fast paths** (native chain or Cypher): a seeded typed 1-hop — ``[n({"id": is_in([...])}), e_forward(), n(...)]`` or ``MATCH (m {id: $x})-[:T]->(p) RETURN p`` — including the single-alias **property RETURN** form (``RETURN p.a AS x, p.b``). The seed lookup, frontier expansion, and endpoint materialization all become positional index gathers, so the lookup stops paying graph-size costs. - **Property-seeded lookups**: the seed filter may hit a *property* column (e.g. ``MATCH (m {id: $x})`` when the graph is bound on a different key column). The seed row falls back to a property scan, but the adjacency and endpoint gathers still engage — the common pattern of a synthetic key binding plus an ``id`` property filter is covered. - **Direct** ``g.hop(nodes=..., hops=..., direction=...)`` — the ``O(degree)`` gather path. **Not yet covered**: the general Polars chain traversal — multi-hop and multi-alias queries executed by the Polars chain engine take their scan/join path even with an index resident. Coverage is decline-gated: anything the index path does not handle falls back to the scan, so the worst case is the speed you already had. Staleness and safety -------------------- The validity contract is simple: **an index serves only while the frames it was built over are unchanged** (checked by object identity plus a structural fingerprint at use time). Consequences: - Rebinding ``.edges(...)`` invalidates the edge adjacency indexes; rebinding ``.nodes(...)`` invalidates the node-id index. A stale index is treated as *absent* — skipped, never consulted. - ``g.show_indexes()`` reports liveness in the ``valid`` column, so you can see at a glance whether a rebind knocked an index out. - ``valid`` alone is not "this index will serve your query": indexes are also **engine-specific**. The ``usable`` column is True only when the index is fresh AND built for the resolved query engine (shown in ``query_engine``); otherwise ``reason`` explains the decline — e.g. a polars-built index on a graph whose default queries resolve to pandas shows ``usable=False`` with an engine-mismatch reason. Pass ``show_indexes(engine=...)`` to preview an explicit engine choice. - Rebuild by calling ``gfql_index_all()`` again on the rebound ``g``. .. doc-test: skip .. code-block:: python new_edges_df = edges_df.assign(amount=edges_df["amount"] + 1) g2 = g_indexed.edges(new_edges_df, "src", "dst") g2.show_indexes() # edge_out_adj / edge_in_adj now valid=False; node_id still True g2 = g2.gfql_index_all() # pay again for the new frame; all valid=True **Declines are always safe.** Whether an index is missing, stale, or the query shape is uncovered, results are identical either way — indexes only ever change speed, never answers (index-vs-scan parity is differentially tested across engines). .. note:: **Stability.** The index kinds, sidecar layout, and ``show_indexes()`` columns describe the current implementation and may evolve between releases; the stable contract is the lifecycle and decline-safety guarantees on this page — pay once, automatic reuse, staleness on rebind, and identical results with or without an index. Engines ------- - **pandas and cuDF**: build with the default ``gfql_index_all()`` (AUTO resolves to the frames' engine — numpy sidecars for pandas, on-device cupy for cuDF). - **Polars**: on 0.58.0, pass the engine explicitly — ``gfql_index_all(engine='polars')``. With AUTO, an index build on Polars frames swaps them to pandas (the same AUTO behavior described in :doc:`engines`; a fix is tracked in PR `#1767 `_). - **Polars-GPU**: rides the Polars-tagged index — an index built with ``engine='polars'`` (or ``'polars-gpu'``) serves both. What it costs, what it buys --------------------------- **Build (the "pay" side)**: one-time and ``O(E log E)`` — a sort over the edge frame, amortized across every subsequent seeded query. ``index_policy='auto'`` only pays it when the planner predicts a selective query will earn it back. **Seeded lookup (the "go" side)**: on a covered shape, the seeded lookup drops from the general path to the fast path, and again with the index resident, on both CPU engines. **Flat in graph size**: a direct seeded ``g.hop()`` with the index resident turns the ``O(E)`` scan into an ``O(degree)`` gather, so its cost tracks the seeds' neighborhood rather than the graph. Measured figures are published on :doc:`performance` and :doc:`index_adjacency` only, and only when they trace to a committed benchmark artifact. See also -------- - :doc:`Adjacency Index ` — the planner (``index_policy``), Cypher DDL / wire protocol forms, and the index cost model. - :doc:`engines` — choosing pandas / Polars / cuDF / Polars-GPU. - :doc:`performance` — the vectorization + GPU design behind GFQL.