Reilly Wood

I’ve been using Rust full time for the last month and a bit while contributing to Nushell (more on that later). A lot has changed since I first tried Rust in 2019 and this is my first time working on a big Rust project. Here are some thoughts on the language while they’re still fresh in my head.

Compile times and feedback loops

Rust’s compile times are notoriously slow. Rust development was slow enough on my laptop that I finally gave up on mobile computing and bought a desktop with a top-of-the line CPU (12900K). Along the way I switched from Windows to Linux (more on that later) and started using the mold linker, and now… things are OK!

I’m able to do incremental builds of Nushell (a huge project) in a second or 2, and a full debug build takes 25s. For smaller projects, incremental builds are nearly instant. There’s certainly room to improve here, and the development experience is not great on average hardware, but… this works for me.

Another thing to consider is that the typical Rust feedback loop is tighter than you might expect from the slow compile times. The Rust compiler catches a lot of bugs before a full build needs to happen, and that reduces the need to do a full build and try things out.

Complexity + monotony

Rust is not a simple language. In total I’ve spent nearly 3 months working mostly in Rust, and the language still has a lot of corners that I don’t have a solid grasp on. To improve on this I’m going to need to branch out from Nushell and write a lot of little tools for myself.

Despite the complexity, I’ve found that writing Rust is sometimes a bit… braindead? The type system is very expressive and the compiler catches a ton of errors, so I spend 25% of my time thinking real hard and 75% painting by numbers to make the compiler happy. I can’t quite decide how I feel about this style of development, it can be a little tedious but it also makes for a better end product.

I (sometimes) want a higher-level Rust

Rust has a lot of great things going for it; the tooling, community, package ecosystem, compiler, and syntax are all fantastic. But the focus on systems programming does mean that Rust isn’t quite as ergonomic as it could be for many use cases.

Sometimes I just want a garbage collector! Sometimes I’d be perfectly happy for Rust to implicitly allocate memory if it makes my code work (for example: converting from a &str to a String)! I don’t know if that will ever be possible in standard Rust, but… maybe there’s room for a Rust variant intended for higher-level use cases.

On the other hand, the ability to go as low as you want is great. It’s nice to work in a language with a very “high ceiling”; no matter where your Rust project goes, you won’t have to switch to C or C++.

Big shout-outs to @reillywood, who managed cut the CI times for @nu_shell in half!

If you do @rustlang and see your CI build times creeping up, you should give his fixes a look. Smarter test ordering and build caching makes a huge difference.

— JT (@jntrnr) April 30, 2022

I recently spent a few days tuning Nushell’s GitHub Actions CI pipelines and it paid off: CI used to take about 30 minutes, and now it’s closer to 10. This is not pleasant or glamorous work, but it has a big payoff; every Nu change going forward will spend a lot less time waiting for essential feedback. Here’s how you can do the same.

Use rust-cache

Seriously, it’s really good! GitHub build runners are slow. But GitHub gives every repo 10GB of cache space, and rust-cache takes advantage of that. It caches temporary files for your build dependencies across CI runs, so if you have a lot of dependencies you’ll likely see a big performance boost.

One gotcha to be aware of: GitHub Actions has slightly unintuitive behavior across PRs. PR X is unable to see cache data from PR Y, but they can both see cache data from the base branch (usually main or master). This makes sense from an isolation perspective, but it’s not especially well-documented; I ended up adding an extra CI trigger on main just to fill caches properly.

Split your build and test jobs

Previously we were running cargo build then cargo test in a single job. This was suboptimal for a few reasons:

1. cargo test often needed to recompile crates that had just been built for cargo build. #[cfg(test)] is the most likely culprit here; it makes sense that build output might be different in “test mode”. This has implications for caching too!
2. It’s faster to run build and test in parallel; GitHub gives us 20 build runners for free, and we might as well use them.

Run Clippy after cargo build

Previously we were running Clippy before cargo build. Just switching their order shaved about 5 minutes off every test run! It seems like Clippy can reuse build artifacts from cargo build, but not vice versa.

Use cargo nextest

cargo nextest is “a next-generation test runner for Rust projects.” It’s dead simple to install in CI, and it’s often faster than cargo test. We didn’t see a huge benefit from this (maybe 30-40s faster?), but that’s because our CI time is dominated by compilation; YMMV depending on your code base and test suite.

Conclusion

If you’d like to see the actual changes, they’re all here. Like anything GitHub Actions, this took a lot of tries to get right; those 5 PRs are just the tip of the iceberg, there were a lot more experimental changes in my private fork. I’m hopeful that someday we’ll be able to stop programming in YAML files, but we’re not there yet!

Nushell

Nushell v0.60 is out, and it’s fantastic. This is the first Nu release where I made meaningful contributions (mostly to the website+documentation) and it feels like a good use of my sabbatical time. It’s been interesting figuring out how to sell+explain Nu succinctly; writing good public-facing documentation is hard!

If you haven’t tried Nu, this is a great time to do so; Nu’s not stable yet, but I think you’ll be very pleasantly surprised by the level of polish. I’ve finally made it my default shell on both Windows+Linux.

Most of the work I’m doing for Nushell has a selfish motivation: I want to live in a world where POSIX shells are a thing of the past, and Nushell seems like the most promising way to get there.

Learning

I’ve started working through Crafting Interpreters by Bob Nystrom. My first exposure to Nystrom’s work was Game Programming Patterns, one of the best programming books I’ve ever read. The title’s a little unfortunate because it covers design patterns that are useful in any field of programming; I genuinely think GPP is much more useful to today’s programmer than the book that inspired it.

Crafting Interpreters walks you through building a scripting language from the ground up. The book walks you through an interpreter implementation in Java then C; I’m doing the Java version in C# (personal preference and experience).

Other

I’ve started rekindling some old friendships with people I haven’t seen in person in 2 years, and that’s been really great.

Spring is finally arriving here in Vancouver, so I’ve been finding lots of excuses to be outdoors. My patio’s never been cleaner and I’m looking forward to a lot of spring gardening. I’d like to get some more trellises set up this year; I have a fairly small urban patio so it’s important to make good use of vertical space. “Green to the eye, not green on the ground."

“Work”

I’ve been spending a lot of time writing Nushell documentation. The Nushell core team has been banging out new features rapidly and the documentation hasn’t always kept up; improving the documentation feels like a high-impact way to help out.

Nushell’s got a big release coming up later this month and it has the potential to attract many new users. I want the documentation to be in a better place by then, I’m going to keep spending time on this.

I’ve tried to make a few commits to Nushell itself, but I keep bouncing off Rust. I’m reasonably familiar with Rust, but this is my first time working on a big Rust project. The compile times are painful and they inhibit rapid iteration; I haven’t found a development approach that I like yet.

Reading

Anthony Bourdain’s Typhoid Mary was pretty disappointing (3 stars?); you can tell that Bourdain was short on source material so most of the book is him imagining what Mary might have done.

I started rereading the Wheel of Time series in December and I’m on book 9 now. My interest started to wane around book 8; I think I’ll press onward but I’m really looking forward to the later books (written by another author).

Other

I think I’m a Dance Dance Revolution guy now. I ordered a fancy mat from Poland and have been playing Club Fantastic. I’m still pretty bad but it’s a lot of fun.

I’m officially unemployed again 😎. In the interest of self-accountability, I’m going to try to document what I’m up to on my break; expect more frequent updates to this blog.

HYTRADBOI

I bought a ticket to Have You Tried Rubbing A Database On It?, which could loosely be described as a hipster database conference; lots of people using databases in unusual ways, not much in the way of enterprise RDBMSs. The speaker list is like a Who’s Who for offbeat database work, and I’m really looking forward to it.

Nushell

I’ve been using Nushell as my shell on both Windows and Linux, about half the time. Nushell is a fascinating project; it’s a shell that operates on structured data like PowerShell, but without PowerShell’s (many) pain points.

Nushell has recently seen some massive upgrades (the parsing and evaluation engine was completely rewritten) and it’s a very good time to give it a try. It’s still early days, but I’m hopeful Nushell will be able to displace POSIX shells; it’s liberating to work with much richer data types than plain text:

Nu is a way of saying “what if we didn’t need tools like awk so often?” Since you’re working with structured data, as we add more support for file types, it’s less often you need to reach for “awk”, “jq”, “grep”, and the array of other tools to open and work with common file types. In a way, it’s taking the original spirit of Unix — where you use pipelines to combine a set of tools — and imagining how that original spirit would work today, with what we know about programming languages and tools.

Building data-centric apps with a reactive relational database

This essay touches on a lot of my favourite things: SQLite! The intersection of native apps and web UI! iTunes clones! In a nutshell, it’s a very cool approach to building GUI applications in which all of the application’s state lives in a local database.

It’s more of a provocation than a fully finished system, but I think it shows promise. I’d like to see a bit more investigation of “escape hatches”; how hard would it be mix in a little imperative code when SQL/SQLite aren’t the right fit for a task? Also, this was a bit depressing:

One challenge has been inter-process communication. When the reactive graph is running in the UI thread and the SQLite database is on a web worker or native process, each query results in an asynchronous call that has to serialize and deserialize data.

This would have been a non-issue in traditional GUI code (just query SQLite on a background thread in the same process); one more thing we lose as web UI takes over, I guess 😞.