doc: Pregel Tutorial - #809
Open
rjurney wants to merge 158 commits into
Open
Conversation
rjurney
commented
Mar 13, 2026
Collaborator
- New, big, fancy, super, duper Pregel tutorial
- Moved Stack Exchange data content from Network Motif Finding Tutorial into Data Setup tutorial. Refer to from both motif and Pregel tutorials.
- Point at new tutorial(s) from list of tutorials.
- New network motif and Pregel tutorial Jupyter notebooks
- Some other minor changes...
…s.txt and split out requirements-dev.txt. Version bumps.
…ney/build-upgrades
…ney/build-upgrades
…togram - Replace __file__-based DATA_DIR with graphframes package path (works in REPL) - Set spark.sql.caseSensitive on the live session instead of the builder, which silently no-ops in the pyspark shell - Add log-scale power-of-two in-degree histogram (log_hist) to the Pregel tutorial and pregel.py, for both AggregateMessages and Pregel examples
Collaborator
Author
|
Not closing... I did work on the wrong PR #578 and now have to integrate it. Ugh! |
Add a 'Choosing a Termination Condition' subsection to 'How Fast Does It Converge?' in the Pregel tutorial: a table of the three conditions, the cost of the two dynamic ones (each adds a Spark action per superstep), and per-algorithm guidance for when each applies. Also fix the user guide's vertex-voting bullet, which named stopIfAllNonActiveVertices (the actual API is setStopIfAllNonActiveVertices) and had an unclosed paren.
Switch generate_diagrams.py from the PhantomJS-based mmdc converter to
mermaidx, which bundles its own JS engine and needs no browser or Node
install. The new subgraph syntax separates the node id from the title, so
restore the G numbers in the graphlets overview titles ("G1 single directed
edge") that the bare-title syntax had provided implicitly.
Regenerate all motif and Pregel diagrams with the new renderer, and wire the
SVG versions into the motif tutorial in place of the older PNG figures.
Move the SparkSession setup, path resolution, and the repartition/checkpoint/ cache load of Nodes.parquet and Edges.parquet into 03-data-setup.md, which now ends with nodes_df, edges_df, and g ready to use. The motif and Pregel tutorials drop their duplicated copies of that boilerplate and state up front that they continue from those objects, keeping only the imports each one adds. Correct the prerequisites while here: Java 21 and Spark 4.1.3, matching the versions the tutorials were actually written against.
Add a stat.corr call to the reputation propagation example and report the result: authority and ViewCount correlate at only 0.07, so state that number rather than implying the two usually track each other. Widen the truncate width so the question titles are readable, and bold the point that joins grow quadratically per hop while Pregel just needs a larger setMaxIter.
Replace the "the implementation would use..." hand-wave with a working average-answer-score-per-tag algorithm, in both the tutorial and pregel.py. It reverses the Tags edges so both hops travel with sendMsgToDst, sends a struct carrying score and count together, and unpacks it with two withVertexColumn definitions - which also demonstrates the vertex-vs-triplet expression contexts. Add two diagrams: one mapping each design question to its Pregel paper concept and GraphFrames API call, one tracing scores from Answer to Question to Tag. Note in the text that the four questions restate the components of the original Pregel model, and add the vertex-centric literature behind that framing to Further Reading - the McCune survey, Tian et al. on "think like a vertex", and Colyer's walkthrough of the Pregel paper. Switch figcaptions to white for legibility on the dark docs theme.
…rial assets in black and isort. Add mispelling exceptions.
rjurney
commented
Aug 3, 2026
| [tool.black] | ||
| line-length = 100 | ||
| target-version = ["py39"] | ||
| include = ["graphframes"] |
Collaborator
Author
There was a problem hiding this comment.
The default behavior is to just check Python files. This configuration cleared that and had it checking non-Python files from the tutorial's data.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 47 changed files in this pull request and generated 1 comment.
Suppressed comments (3)
python/graphframes/tutorials/download.py:21
DEFAULT_DATA_DIRdefaults to a path inside the installed package (Path(__file__).parent / "data"). In many environments site-packages is read-only, sographframes stackexchange ...can fail with a permissions error unless users always pass--data-dir.
.pre-commit-config.yaml:40- The codespell hook exclude regex references
python/tutorials/, but this repository’s tutorials live underpython/graphframes/tutorials/(there is nopython/tutorials/directory). As a result, codespell will not skip the intended tutorial content.
exclude: ^(graphx|python/tutorials/)
docs/src/03-tutorials/03-data-setup.md:129
- This comment says the
spark.sql.caseSensitivesetting is needed to letIdandidcoexist, but the tutorial data generation script renamesIdtoStackIdand dropsId. The comment is misleading for readers following the documented workflow.
# Lets the Id:(Stack Overflow int) and id:(GraphFrames ULID) coexist
spark.conf.set("spark.sql.caseSensitive", True)
Comment on lines
1
to
16
| import click | ||
|
|
||
| from graphframes.tutorials import download | ||
|
|
||
|
|
||
| @click.group() | ||
| def cli(): | ||
| """GraphFrames CLI: a collection of commands for graphframes.""" | ||
| pass | ||
|
|
||
|
|
||
| cli.add_command(download.stackexchange) | ||
|
|
||
|
|
||
| def main(): | ||
| # Lazy-import tutorials.download to avoid requiring py7zr/requests | ||
| # at import time — those are only in the 'tutorials' optional extra. | ||
| from graphframes.tutorials import download | ||
|
|
||
| cli.add_command(download.stackexchange) | ||
| cli() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.