This blog series is my personal comments about (part of) the development of RisingWave.
Please take it as an unofficial and no-promise supplement.
Memory control policy
- feat(memory): introduce memory control policy for computing tasks by xx01cyx #7767
- Tracking: refactor user-configurable memory control policy #8228
- refactor: introduce memory control policy abstraction by xx01cyx #8253
Recently we have implemented memory control mechanism, and now we are introducing the policy. Since RisingWave is a “streaming database”, it is crucial for us to consider both batch and streaming tasks while balancing the usage of memory between them.
Our streaming executors access storage via
StateTable interface, and storage layer exposes
LocalStateStore interface. This PR wants to move part of the former (memtable) to the latter. It might help to implement more features in the storage layer, and also make it easier to implement other storage engines for bench purpose.
- refactor(optimizer): divide logical optimizer into one for batch and one for streaming. by chenzl25 #8192
- perf(agg): reuse existing agg calls while building
LogicalAggby richardchien #8200
RisingWave is a streaming processing system which aims to provide real time low latency for our users. Making the join tree shallower helps to reduce the latency.
See the illustration below:
… to prevent blocking when evaluting UDFs
RFC: suspend MV
Last time, I mentioned that RisingWave currently tolerates compute errors by default and we are in the process of implementing an error reporting mechanism. However, we had previously discussed another mechanism for handling errors which involved suspending MV. This RFC reintroduces this idea.
Injectivity of column index mapping
ColIndexMapping is an important utility in optimizer used when we wants to map an input column (index) to an output column. It’s a partial mapping from
[0, n) to
[0, m) , where
n is the number of columns in the input, and
m is the number of columns in the output.
When I began to work on RisingWave (one year ago 😲), I worked on column pruning, which removes unnecessary columns from the plan nodes. Naturally it involves a lot with
ColIndexMapping . Althought mathematically simple and intuitive, it’s not easy to do such mappings correctly in programs. I had a hard time understanding
ColIndexMapping at that time, and added many comments and did some refactor to make it less confusing and error-prone.
After long, we met this new bug related to
ColIndexMapping again.. Under curtain circumstances, a plan node will have a non-injective mapping and duplicate columns in the output. In this case, the duplicate columns are unexpectedly hidden after mapping. We fixed this bug by considering the injectivity of the mapping. A more systematic solution is also proposed: Proposal (frontend): Prune duplicate columns #8277.
Rusty stuff 🦀️
We ❤️ Rust! This section is about some general Rust related issues.
- refactor: switch
async_stack_traceto the crates.io version of
await-treeby BugenZhao #8254
- await-tree - crates.io: Rust Package Registry
We have a tracing mechanism called async stack trace in our system, which allows us to “capture a snapshot” of where, why, and how long the async tasks are pending in real-time. It helped us locate some stuck issues, e.g., streaming deadlock, which would be very hard to debug in other ways.
Now we have published it to crates.io, and renamed it to
await-tree . I already invited @BugenZhao to write a blog post to introduce it, and can’t wait to see it. 😍
We also integrated it into the RisingWave dashboard to make it more convenient to use. 😋
Large memory usage by backtrace and debuginfo
We noticed unexpected memory consumption on Meta node. And the cause is (again) backtrace…
So happy to see another 3 first-time contributors this week! 😲🥳
(BTW, I’m also quite interested in where they come from. Are they encouraged by my posts? If so, please let me know! 🤔🤣)
- feat(expr): support
array_distinctby snipekill #8315
- fix(parser): disable single-quoted strings as aliases for column or table by Eridanus117 #8338
@erichgess submitted 2 PRs. I’m quite impressed by the thoroughness of their communication in issue and PR discussions regarding the problem’s situation, cause, and solution. This is one of the biggest reasons why I love open source or working in public; transparent over-communication benefits both current collaborators’ reviews and future contributors’ learning. 🤗
expPG compatibility): if a very large or very small operand is used, then
experrors by erichgess #8309
- bug(unit tests): Cache unit tests fail when
P.S. Welcome to join the RisingWave Slack community.
So much for this week. See you next week (hopefully)!