GFQL vs Apache Spark GraphFrames on One Node#

This page compares GFQL with Apache Spark GraphFrames on one machine. GFQL is Graphistry’s open-source graph query language: Cypher and Python chains that run in-process on dataframes, with no database or cluster. GraphFrames is Spark’s graph library, run here on local[*], a single-node JVM using all cores. The workload is four tasks on two SNAP graphs, LiveJournal and Orkut, with Friendster as the larger-than-memory size measured last. Every number below renders from a committed pyg-bench receipt; the Measurement block at the end names the runs, hosts, and commits.

Where it stands. The single-server ceiling measured here is Friendster: 1,806,067,135 edges bound from a lazy Polars scan, a degree filter in 3111.7 ms, a 1-hop from 50 hub seeds in 6123.4 ms, and a 2-hop in 164890.2 ms on the CPU streaming path, with resident memory peaking at 103.6 GiB of the 119 GiB host; the GPU path stops at the 1-hop, PageRank does not fit on either path, and GraphFrames on local[*] did not load the graph at all. Below that ceiling the picture is mixed and both sides are printed: on whole-graph PageRank GFQL on the GPU is 18.3x faster than GraphFrames on LiveJournal and 12.5x on Orkut, while GFQL on the CPU loses PageRank on both (0.3x and 0.2x of GraphFrames’ speed): on both paths the solver is a small part of the time (the shaded bars) and the rest is the conversion into the solver’s graph and the join of scores back onto the nodes; on degree filters and 1-hop the CPU engine is faster on both graphs; on 2-hop GraphFrames wins on both (0.5x and 0.4x of its speed). The GFQL filter and hop rows were measured at the head of the fix for #2023 (#2024, measured at that pull request’s head; it has since landed on master and the ladder was not re-run); the released code’s LiveJournal 2-hop was 64273.1 ms (diagnostic), the before-state the disclosures keep.

LiveJournal task times: GFQL and GraphFrames for filter, 1-hop, 2-hop, and PageRank, with the PageRank solver time shaded inside the GFQL bar

Orkut task times: GFQL and GraphFrames for filter, 1-hop, 2-hop, and PageRank, with the PageRank solver time shaded inside the GFQL bar

Friendster task times: GFQL CPU streaming filter, 1-hop, and 2-hop; PageRank and GraphFrames not measured

GFQL binds each graph from a lazy Polars scan of the edge parquet and runs the filter and hop tasks with engine="polars" under the Polars CPU streaming collect, or with engine="polars-gpu" under the cudf-polars streaming executor. PageRank re-binds an eager copy outside the timer and calls cuGraph on the GPU or igraph on the CPU. The streaming collect is not a tax: with the same commit and protocol the eager collect matched it on filter and 2-hop and was slower on 1-hop (the receipts are named in the Measurement block). Every cell is the median of 5 timed runs after 2 warmups, and every task returns the same result size on every system that ran it. Times are milliseconds unless marked; lower is better.

LiveJournal#

Task	GFQL polars (CPU)	GFQL polars-gpu (GPU)	GraphFrames (local[*])	GFQL CPU vs GraphFrames
filter (degree >= 42)	`26.1 ms`	`27.1 ms`	`80.4 ms`	`3.1x`
1-hop (50 seeds)	`219.7 ms`	`1159.6 ms`	`1460.2 ms`	`6.6x`
2-hop (50 seeds)	`6478.3 ms`	`7162.5 ms`	`3475.8 ms`	`0.5x` (GraphFrames wins)
PageRank (full graph)	`52622.4 ms`; solver `2834.8 ms (diagnostic)`	`890.9 ms`; solver `59.3 ms (diagnostic)`	`16311.2 ms`	GPU: `18.3x`; CPU: `0.3x` (GraphFrames wins)

The GPU streaming executor is slower than the CPU streaming collect on both hops here (1159.6 ms against 219.7 ms); at these result sizes the work is data movement, and the GPU column is a loss for traversal.

Orkut#

Task	GFQL polars (CPU)	GFQL polars-gpu (GPU)	GraphFrames (local[*])	GFQL CPU vs GraphFrames
filter (degree >= 162)	`59.4 ms`	`62.6 ms`	`66.1 ms`	`1.1x`
1-hop (50 seeds)	`514.0 ms`	`4037.1 ms`	`3632.2 ms`	`7.1x`
2-hop (50 seeds)	`27348.3 ms`	`33852.9 ms`	`10885.8 ms`	`0.4x` (GraphFrames wins)
PageRank (full graph)	`170865.5 ms`; solver `7834.5 ms (diagnostic)`	`2795.7 ms`; solver `151.4 ms (diagnostic)`	`34860.7 ms`	GPU: `12.5x`; CPU: `0.2x` (GraphFrames wins)

Friendster#

On the Polars CPU streaming path; the GPU path stopped at the 1-hop and no other system ran (see Friendster (1.8B edges): the ceiling).

Task	GFQL polars (CPU)	Result
filter (degree >= 148, the 90th percentile)	`3111.7 ms`	6,585,312 nodes
1-hop (50 seeds)	`6123.4 ms`	166,615 nodes
2-hop (50 seeds)	`164890.2 ms`	15,878,312 nodes
PageRank	not attempted	see Friendster (1.8B edges): the ceiling

Result sizes agree across the systems that ran each task, as recorded in the receipts:

Graph	filter	1-hop	2-hop	PageRank
LiveJournal	403,561	119,877	1,378,430	3,997,962
Orkut	308,666	434,973	1,991,366	3,072,441
Friendster (GFQL only)	6,585,312	166,615	15,878,312	not attempted

Which engine to use#

Whole-graph PageRank: use GFQL on GPU (engine="polars-gpu", cuGraph). The solver is a small share of the GFQL time; the rest is the conversion of the edge frame and the join of scores back onto the nodes, which is where the next gains are.
Filter and 1-hop: use GFQL on CPU (engine="polars"). It is faster than GraphFrames on both graphs, and the GPU streaming executor does not help at these result sizes.
2-hop from hub seeds: GraphFrames wins on both graphs today. GFQL’s cost is the wavefront seed-rediscovery rule evaluated over the traversed ball; #2024 removed the interpreter loop, and the remaining gap is the rule itself.
PageRank without a GPU: GFQL routes the CPU path through igraph, and loses to GraphFrames on both graphs. The igraph solver itself is 2834.8 ms (diagnostic) of the 52622.4 ms LiveJournal row; the rest is the conversion into igraph and the join-back, tracked in #2032. Use the CPU path for convenience, not for speed.
Larger than one node’s memory: see Friendster (1.8B edges): the ceiling.

The tasks#

filter: keep nodes with degree >= threshold. SNAP graphs have no attributes, so both systems compute degree during load. The load carries that cost, not the query. The shared threshold makes the filter identical across systems.

# GFQL
from graphistry import n
from graphistry.compute.predicates.numeric import ge
g.gfql([n(filter_dict={'degree': ge(42)})], engine="polars")  # or "polars-gpu"

# GraphFrames
gf.degrees.filter("degree >= 42").count()

1-hop and 2-hop: undirected expansion from a fixed set of 50 high-degree seed nodes.

# GFQL
from graphistry import n, e_undirected
g.gfql([n(filter_dict={'id': is_in(seeds)}), e_undirected(hops=1), n()], engine="polars")

GraphFrames has no k-hop primitive. Its bfs finds shortest paths between predicates and find matches a fixed motif. The Spark side therefore expands with one iterated undirected edge join per hop and ends in .count().

PageRank: full graph, damping 0.85. GFQL GPU calls g.compute_cugraph('pagerank') on an eager cuDF copy of the graph; GFQL CPU calls g.compute_igraph('pagerank') on an eager pandas copy. GraphFrames calls gf.pageRank(resetProbability=0.15, maxIter=20). All return the full vertex set. The shaded part of a GFQL PageRank bar is the solver alone (cuGraph or igraph) on a graph object built outside the timer; the light part is the rest of the query, which is the conversion into that graph object and the join of scores back onto the nodes.

Friendster (1.8B edges): the ceiling#

Friendster has 1,806,067,135 edges and 65,608,366 nodes (SNAP). The eager harness that produced the earlier version of this page could not load it on the test node (about 119 GB unified memory): a pandas edge frame plus a second pass for degrees exceeds physical RAM, a direct cuDF read exceeds the unified pool, and a 90 GB Spark driver heap swaps.

The harness binds from pl.scan_parquet and collects through GFQL’s streaming paths (GFQL_POLARS_CPU_STREAMING=1 for the Polars streaming engine, GFQL_POLARS_GPU_EXECUTOR=streaming for the cudf-polars streaming executor), with a peak-memory receipt at every size. On Friendster the CPU streaming run loaded the graph (scan plus degree pass in about 20 seconds, 55.0 GiB resident), answered the degree filter and the 1-hop from 50 hub seeds (table above), and peaked at 103.6 GiB resident after the 1-hop; a second run answered the 2-hop, a 15,878,312-node ball, in 164890.2 ms at 67.9 GiB resident. The streaming collect keeps the load out of memory, but the traversal still materializes the edges it touches, and that is where the GPU path stops: the cudf-polars streaming executor completed the degree filter at 103.8 GiB resident, then the watchdog ended the run during the 1-hop when host free memory fell to 17 GB against its 20 GB floor, so the GPU column has no Friendster cell. Whole-graph PageRank does not fit on either path: the GPU preflight refused it (an estimated 87 GB peak against an 80 GB budget), and the CPU path was not attempted: its Orkut row peaked at 29.9 GiB resident for 117M edges, and Friendster has fifteen times the edges. That is the single-server ceiling this page measured. GraphFrames on local[*] stays at the boundary it hit above.

Method and limits#

Scope: single node, in memory. local[*] is Spark’s single-node mode. A cluster amortizes scheduling and shuffle cost across machines and changes the trade-off at larger scale. Use a Spark cluster when the data already lives there or the graph exceeds one node’s memory.
Timing: median of 5 runs after 2 warmups per cell, each system loaded once and resident across iterations. Load is not timed.
Materialization: Spark is lazy, so every task ends in .count() or .vertices.count(). GFQL materializes with len(_nodes).
Comparability: a task is comparable only when every system reports the same result size; a cell that disagrees is published as a direct time with a disclosure. Cells marked diagnostic are never quoted as GFQL’s number.
PageRank convergence: GraphFrames runs a fixed maxIter=20; cuGraph runs to its default tolerance. Times compare wall-clock to a usable ranking.
Receipts: one run at a time under a host lock, after two clean checks five minutes apart; a load monitor samples the host every second and a classifier invalidates the run if a process outside the benchmark ran during it. Invalidated attempts stay in the package under stale-attempts/; one Orkut GraphFrames run is valid by reclassification after the classifier learned that Spark’s own shutdown cleanup is the benchmark’s process (RECLASSIFIED.txt in that run’s directory).
Harness: the GFQL streaming harness and every receipt live in pyg-bench; the GraphFrames baseline is benchmarks/gfql/bench_graphframes.py --systems graphframes in this repository, run from a host Spark with the GraphFrames assembly jar.

Provenance#

Every figure on this page is printed from docs/source/_data/gfql_benchmarks.json, which pyg-bench publishes. The documentation build and docs/test_bench_numbers.py reject missing, stale, or unpublished values.

Measurement

Measured:

2026-09-04

Host:

dgx-spark (NVIDIA GB10), 20 CPU

Repetitions:

each system loads once and stays resident; 2 warmups + 5 timed runs per task, median; GraphFrames PageRank 20 iterations

Runtime:

2026-09-04: GFQL in graphistry/test-rapids-official:26.02-gfql-polars (RAPIDS 26.02, Python 3.13, polars 1.35.2, cuDF/cuGraph 26.2) under dgx-guard safe_run (RMM 80 GB, 100 GB for the Friendster rungs; host floor 20 GB); the two CPU PageRank rungs (fix2024/lj-polars-pagerank, orkut-polars-pagerank) in graphistry/test-rapids-official:26.02-gfql-polars-igraph, the same image plus python-igraph 1.0.0 (image id and versions in each rung’s IMAGE_RECEIPT.txt, recorded after the run); GraphFrames on host PySpark 3.5.3 + graphframes assembly jar, OpenJDK 1.8.0_492
2026-09-04: GFQL in graphistry/test-rapids-official:26.02-gfql-polars (RAPIDS 26.02, Python 3.13, polars 1.35.2, cuDF/cuGraph 26.2) under dgx-guard safe_run (RMM 80 GB, 100 GB for the Friendster rungs; host floor 20 GB); GraphFrames on host PySpark 3.5.3 + graphframes assembly jar, OpenJDK 1.8.0_492

Dataset:

SNAP com-LiveJournal (3,997,962 nodes / 34,681,189 edges), com-Orkut (3,072,441 nodes / 117,185,083 edges), com-Friendster (65,608,366 nodes / 1,806,067,135 edges) as parquet edge lists; 50 highest-degree seeds; filter thresholds 42 / 162 / 148 (Friendster: the 90th degree percentile)

Result validation:

every task records the materialized result size (node count of the filter / k-hop ball, scored vertex count for PageRank); a cell is comparable only when every system on that dataset and task reports the same size, else it is published as a direct, diagnostic-only time

Caveats:

Single node (dgx-spark, NVIDIA GB10 unified memory, 20 CPU); GraphFrames on Spark local[*] over all cores with a 90g driver, PySpark 3.5.3; GFQL binds the graph from a lazy Polars scan and its filter and hop rows run under the Polars CPU streaming collect or the cudf-polars streaming executor; PageRank rows re-bind an eager cuDF (GPU) or pandas (CPU) copy outside the timer.
GFQL filter, hop and CPU PageRank rows were measured at the head of pull request #2024 (commit e951e9a2c), the fix for issue #2023 (a per-edge Python loop that made undirected multi-hop ~30x slower at 0.59.0); it had not landed on master when measured. GFQL GPU PageRank rows and the GraphFrames rows were measured at 0.59.0 (3fb216dd); the PageRank code is the same at both commits.
Polars CPU streaming versus eager collect (receipts fix2024/lj-polars and fix2024/orkut-polars, same commit and protocol): filter and 2-hop medians agree within 9% (LiveJournal 2-hop 6.5 s streaming vs 7.0 s eager; Orkut 27.3 s vs 27.3 s) and streaming is faster on 1-hop (LiveJournal 220 ms vs 336 ms; Orkut 514 ms vs 1,031 ms); the published rows are the streaming lane.
CPU PageRank runs the igraph solver: on LiveJournal the solver alone is 2.8 s of the 52.6 s row (the GraphFrames row is 16.3 s) and on Orkut 7.8 s of 170.9 s (GraphFrames 34.9 s); the rest is the frame-to-igraph conversion and the join-back of scores (pygraphistry issue #2032).
Friendster (1.8B edges) filter and 1-hop ran on the Polars CPU streaming lane; resident memory after the 1-hop was 106,095 MiB (103.6 GiB) of the 119 GiB host.
Friendster 2-hop ran on the same lane without the host address-space cap: 164.9 s, a 15,878,312-node ball, 69,526 MiB (67.9 GiB) resident. The first attempt under a 100 GB address-space cap (fix2024/friendster-polars-hop2.failed-attempt1) ended at load with a failed 7.2 GB allocation, exit 133.
Friendster on the cudf-polars streaming executor (fix2024/friendster-polars-gpu-streaming.killed-attempt1): the degree filter completed (4.0 s, a diagnostic from a killed lane, not a cell) with the process at 106,308 MiB resident; the watchdog killed the run during the 1-hop when host free memory fell to 17 GB against the 20 GB floor, exit 137. The GPU column has no Friendster cell.
Friendster GPU PageRank was refused by preflight (fix2024/friendster-polars-gpu-pagerank.refused-attempt1: estimated 87 GB peak against the 80 GB budget). CPU PageRank was not attempted: Orkut’s CPU PageRank peaked at 30,586 MiB resident at 117M edges and Friendster has 15x the edges. GraphFrames on local[*] did not run on Friendster (a 90 GB driver heap swapped in the earlier eager attempt). Friendster rows are direct times with no comparison.
GFQL 0.59.0 (3fb216dd) hop rows on LiveJournal, kept as the measured before-state of issue #2023; the published GraphFrames comparison uses the #2024 rows.
Component of graphframes.lj.pagerank.gfql_polars: the PageRank solver alone on a graph object built outside the timer; the parent time adds the conversion and the join of scores back onto the nodes.
Component of graphframes.lj.pagerank.gfql_polars_gpu: the PageRank solver alone on a graph object built outside the timer; the parent time adds the conversion and the join of scores back onto the nodes.
Component of graphframes.orkut.pagerank.gfql_polars: the PageRank solver alone on a graph object built outside the timer; the parent time adds the conversion and the join of scores back onto the nodes.
Component of graphframes.orkut.pagerank.gfql_polars_gpu: the PageRank solver alone on a graph object built outside the timer; the parent time adds the conversion and the join of scores back onto the nodes.

GFQL vs Apache Spark GraphFrames on One Node

Contents