Introduction

Almost all components of RisingWave are developed in rust, and they are split to several crates:

1. config contains default configurations for servers.
2. prost contains generated protobuf rust code, e.g. grpc definition and message definition.
3. stream contains our stream compute engine, read Stream Engine for more details.
4. batch contains our batch compute engine for queries against materialized views.
5. frontend contains our SQL query planner and scheduler.
6. storage contains our cloud native storage engine, read State Store Overview for more details.
7. meta contains our meta engine, read Meta Service for more details.
8. utils contains several independent util crates which helps to simplify development. We plan to publish them to crates.io in future when they are more mature.
9. cmd contains all binaries, and cmd_all contains the all-in-one binary risingwave.
10. risedevtool is an awesome developer tool for RisingWave, read RiseDev Guide for more details.

Understanding RisingWave Macros

RisingWave uses a lot of macros to simplify development. You may choose the read to macro definition directly, but something complementary / easier is to use cargo expand to expand the macro and read the expanded code.

You can also setup rust-analyzer in the editor to expand macros on the fly. See https://rust-analyzer.github.io/manual.html#expand-macro-recursively. This is the better option if you want to understand the code interactively.

Example 1: declarative macro commit_meta!()

For instance, within meta::manager::catalog::CatalogManager::finish_create_table_procedure there's a call to the macro commit_meta!().

To understand it, first I dump the expanded code to a file:

cargo expand -p risingwave_meta > meta.rs

Then I search for the function call finish_create_table_procedure in my editor, and compare the original code with the expanded code in meta.rs.

From there I can see that it does the following, followed by some instrumentation code which can be ignored:

async {
    tables.apply_to_txn(&mut trx)?;
    self.env.meta_store().txn(trx).await?;
    tables.commit();
    MetaResult::Ok(())
}.instrument(/* ... */)

Example 2: Procedural macro aggregate

The other example would be the #[aggregate] procedural macro.

To understand it, first I dump the expanded code to a file:

cargo expand -p risingwave_expr > expr.rs

Then we identify the #[aggregate] macro call to examine. In this case it's string_agg.

First, we know that cargo expand merges the individual modules into one big file, namespaced by the module name.

In this case string_agg is its own module, so we can search for mod string_agg in the expr.rs we just generated.

We can see that the expanded string_agg now also contains extern fn string_agg_varchar_varchar_varchar.

Reading the code we understand now that it does the following:

1. Register it to a global registry for aggregates.
2. "it" here refers to an AggFuncSig, whose definition we can checkout.
3. Of note is that it will generate an anonymous struct which implements the AggregateFunction trait.