Skip to content

doc: Pregel Tutorial - #809

Open
rjurney wants to merge 158 commits into
graphframes:mainfrom
rjurney:rjurney/pypi-tutorials
Open

doc: Pregel Tutorial#809
rjurney wants to merge 158 commits into
graphframes:mainfrom
rjurney:rjurney/pypi-tutorials

Conversation

@rjurney

@rjurney rjurney commented Mar 13, 2026

Copy link
Copy Markdown
Collaborator
  1. New, big, fancy, super, duper Pregel tutorial
  2. Moved Stack Exchange data content from Network Motif Finding Tutorial into Data Setup tutorial. Refer to from both motif and Pregel tutorials.
  3. Point at new tutorial(s) from list of tutorials.
  4. New network motif and Pregel tutorial Jupyter notebooks
  5. Some other minor changes...

…s.txt and split out requirements-dev.txt. Version bumps.
rjurney added 2 commits July 29, 2026 10:54
…togram

- Replace __file__-based DATA_DIR with graphframes package path (works in REPL)
- Set spark.sql.caseSensitive on the live session instead of the builder,
  which silently no-ops in the pyspark shell
- Add log-scale power-of-two in-degree histogram (log_hist) to the Pregel
  tutorial and pregel.py, for both AggregateMessages and Pregel examples
@rjurney

rjurney commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator Author

Not closing... I did work on the wrong PR #578 and now have to integrate it. Ugh!

rjurney added 20 commits July 29, 2026 11:14
Add a 'Choosing a Termination Condition' subsection to 'How Fast Does It
Converge?' in the Pregel tutorial: a table of the three conditions, the
cost of the two dynamic ones (each adds a Spark action per superstep),
and per-algorithm guidance for when each applies.

Also fix the user guide's vertex-voting bullet, which named
stopIfAllNonActiveVertices (the actual API is
setStopIfAllNonActiveVertices) and had an unclosed paren.
Switch generate_diagrams.py from the PhantomJS-based mmdc converter to
mermaidx, which bundles its own JS engine and needs no browser or Node
install. The new subgraph syntax separates the node id from the title, so
restore the G numbers in the graphlets overview titles ("G1 single directed
edge") that the bare-title syntax had provided implicitly.

Regenerate all motif and Pregel diagrams with the new renderer, and wire the
SVG versions into the motif tutorial in place of the older PNG figures.
Move the SparkSession setup, path resolution, and the repartition/checkpoint/
cache load of Nodes.parquet and Edges.parquet into 03-data-setup.md, which
now ends with nodes_df, edges_df, and g ready to use. The motif and Pregel
tutorials drop their duplicated copies of that boilerplate and state up front
that they continue from those objects, keeping only the imports each one adds.

Correct the prerequisites while here: Java 21 and Spark 4.1.3, matching the
versions the tutorials were actually written against.
Add a stat.corr call to the reputation propagation example and report the
result: authority and ViewCount correlate at only 0.07, so state that number
rather than implying the two usually track each other. Widen the truncate
width so the question titles are readable, and bold the point that joins grow
quadratically per hop while Pregel just needs a larger setMaxIter.
Replace the "the implementation would use..." hand-wave with a working
average-answer-score-per-tag algorithm, in both the tutorial and pregel.py.
It reverses the Tags edges so both hops travel with sendMsgToDst, sends a
struct carrying score and count together, and unpacks it with two
withVertexColumn definitions - which also demonstrates the vertex-vs-triplet
expression contexts.

Add two diagrams: one mapping each design question to its Pregel paper
concept and GraphFrames API call, one tracing scores from Answer to Question
to Tag. Note in the text that the four questions restate the components of
the original Pregel model, and add the vertex-centric literature behind that
framing to Further Reading - the McCune survey, Tian et al. on "think like a
vertex", and Colyer's walkthrough of the Pregel paper.

Switch figcaptions to white for legibility on the dark docs theme.
…rial assets in black and isort. Add mispelling exceptions.
Comment thread python/pyproject.toml
[tool.black]
line-length = 100
target-version = ["py39"]
include = ["graphframes"]

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default behavior is to just check Python files. This configuration cleared that and had it checking non-Python files from the tutorial's data.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 47 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

python/graphframes/tutorials/download.py:21

  • DEFAULT_DATA_DIR defaults to a path inside the installed package (Path(__file__).parent / "data"). In many environments site-packages is read-only, so graphframes stackexchange ... can fail with a permissions error unless users always pass --data-dir.
    .pre-commit-config.yaml:40
  • The codespell hook exclude regex references python/tutorials/, but this repository’s tutorials live under python/graphframes/tutorials/ (there is no python/tutorials/ directory). As a result, codespell will not skip the intended tutorial content.
        exclude: ^(graphx|python/tutorials/)

docs/src/03-tutorials/03-data-setup.md:129

  • This comment says the spark.sql.caseSensitive setting is needed to let Id and id coexist, but the tutorial data generation script renames Id to StackId and drops Id. The comment is misleading for readers following the documented workflow.
# Lets the Id:(Stack Overflow int) and id:(GraphFrames ULID) coexist
spark.conf.set("spark.sql.caseSensitive", True)

Comment on lines 1 to 16
import click

from graphframes.tutorials import download


@click.group()
def cli():
"""GraphFrames CLI: a collection of commands for graphframes."""
pass


cli.add_command(download.stackexchange)


def main():
# Lazy-import tutorials.download to avoid requiring py7zr/requests
# at import time — those are only in the 'tutorials' optional extra.
from graphframes.tutorials import download

cli.add_command(download.stackexchange)
cli()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants