RisingWave is a distributed SQL database for stream processing written in Rust. As a developer of RisingWave, I’m always excited (and also a little bit overwhelmed) about its progress every day.
So why not share the excitement with more people (and also help myself to get a better understanding)? That’s why I decided to write this blog post about what’s happening in the project. Hope you will enjoy it!
This blog series is my personal comments about (part of) the development of RisingWave.
Please take it as an unofficial and no-promise supplement.
Most exciting things 🤩
Huge reduce of CI time
As my last post mentioned, last week I found some stupidly effective ways to optimize Rust compile time. I managed to reduce the CI time from main 40min/PR 25min30s to main 28min/PR 16-19min, and it looks good this week!
It’s quite a DX improvement. I’m very happy and would like to quote matklad’s blog again:
Compilation time is a multiplier for basically everything. Whether you want to ship more features, to make code faster, to adapt to a change of requirements, or to attract new contributors, build time is a factor in that.
It also is a non-linear factor. Just waiting for the compiler is the smaller problem. The big one is losing the state of the flow or (worse) mental context switch to do something else while the code is compiling. One minute of work for the compiler wastes more than one minute of work for the human.
Let’s take some time to prevent “broken windows”. The effort would pay off!
DDL UX improvements
DDL can take very long time if there are already a lot of existing data to consume. Previously you can only sit there and wait. But now, you can:
- Show DDL’s progress: (by chenzl25 #7914)
dev=> select * from rw_catalog.rw_ddl_progress; ddl_id | ddl_statement | progress --------+-------------------------------+---------- 1026 | CREATE INDEX idx ON sbtest1(c) | 69.02% (1 row)
- and cancel streaming jobs by ctrl-c! (by yezizp2012 #7917)
Most intersting things 😄
OpenDAL into RisingWave!
OpenDAL is a unified data access layer which aims to help access data freely, painlessly, and efficiently.
Previously, RisingWave only supports s3 as a storage backend (and also support other s3-compatible storage). Recently we are trying to add more storage backends.
Last week, we used OpenDAL to add support for using HDFS as a storage backend. This week, we tried more things:
- Tried using google cloud storage in RisingWave (by wcy-fdu #7920). In our initial benchmark, it seems OpenDAL can be faster than s3-compatible protocol!
- Changed the implementation for oss from s3-compatible mode to OpenDAL (by wcy-fdu #7969). S3-compatible mode for oss doesn’t support
delete_objects, and also suffers from some unstable issues.
It seems that OpenDAL is quite promising!
Rusty stuff 🦀️
We ❤️ Rust! This section is about some general Rust related issues.
or , The
or_else methods on
Result can help to avoid expensive computation in the
Err case. (i.e., lazy evaluation!) And the
clippy::or_fun_call lint is for detecting that. It used to be
warn by default, but unfortunately it seems to have a high false positive rate, and is now
allow by default.
But we met a case where it’s very wanted. In a function, we used
ok_or::<RwError> 9 times, so 9
RwError is created regardless of whether it’s needed. But constructing
RwError is very expensive because it captures the backtrace…
What’s worse, on M1 Mac, capturing backtrace is VERY SLOW (~300ms #6131) and can even cause SEGFAULT (#6205)! 😇 It’s not completely resolved yet. We mitigated it by reducing unncessary backtrace capturing, but it’s still a problem.
So for this issue, of course we should use
ok_or_else instead (by xxchan #7945).
zip_eq is a safer version of
zip which checks if the two iterators have the same length. However, it’s notoriously slow (see #6856 by wangrunji0408 for a benchmark), because in the naive implementation, every item in the iterator is checked.
ExactSizeIterator trait and naturally it can have an optimized implementation. Such speciallized implementation does not exist because Rust doesn’t support specialization yet. (See tracking issue for specialization (RFC 1210))
So last week I added two new separated traits to replace
zip_eq : (code here)
debug_assertionsis enabled, otherwise use
zip. It’s a good trade-off between safety and performance, and can be a drop-in replacement for
zip_eq_fast: speciallized implementation for
zip_eq_debugcan be good enough, but I still added this after playing with specialization for a while. Just for fun…!😄)
ExactSizeIterator are implemented for more our iterator implementations, so we can use
zip_eq_fast more often… (by BugenZhao #7939)
Other notable things
Recently we are doing a large refactoring for system parameters in order to achive consistency and mutability for the cluster configurations (Tracking Issue #7381). This is a serious issue as RisingWave grows mature.
This week, the meta part for
ALTER SYTEM is implemented by Gun9niR #7954.
Since RisingWave is PostgreSQL compatible, I had thought that supporting other database tools would be easy. But it turns out to be very hard 😭. The largest obstacles is that database tools ususally rely heavily on the system tables in
pg_catalog to get the metadata of the database so that they can provide a better UX. But there are so many features in the system tables!
We have been constantly making efforts to support more and more system tables in order to integrate RisingWave into other tools. This week, we:
pg_catalog.pg_conversionby yezizp2012 #7964. This is for DBeaver support.
- Found it’s not possible to add
pg_catalog.pg_type#7555, which affects sqlalchemy support. But sqlalchemy-risingwave can be used as a substitute for the PG native plugin.
EXPLAIN format is another thing I had thought would be easy. Although current situation is not so bad, it’s still far from perfect.
This week, we tried to use a “global expression id” to make the plan more readable, but it still has some problems:
- feat: try to improve explain result by fuyufjh #7953
- New project explain format makes planner test unstable #8005
- refactor(optimizer): reset expression display id for explain by chenzl25 #8006
By the way, here’s another intersting stuff: rewrite the
EXPLAIN implementation using the (modified) Wadler-style algebraic pretty printer! See the RFC by ice1000
Although OLAP batch queries are not the main focus of RisingWave, we still want to make it better. This week, we have these optimizer improvements to make batch queries faster:
- feat(optimizer): index accelerating
TopNby Eurekaaw #7726
- feat(optimizer): support like expression rewrite by chenzl25 #7982
Last but not least
- fix: fix idle exit in playground mode of compactor by yezizp2012 #8014. During an HTTP server’s graceful shutdown (
serve_with_shutdown), it will wait for all the connections to be closed. We ran into a situation where some connections were not closed and the server waits forever…
- Tracking: Monitoring, logging and tooling improvements #8018. Observability matters!
- Support error line/column number reporting during parsing #7863. This is a lovely feature to increase UX, but it will require hard work :)
So much for this week. See you next week (hopefully)! 🤗