profile picture

Contributing to Rust in 2017 and 2023

June 02, 2024 - images rust

This is a comparison of the contribution experience to the Rust standard library in 2017 and 2023. I have the possibly unique perspective of having contributed to the Rust standard library twice separated by many years.

2017

In 2017, I was a PhD student was working on the Noria project, which was a research project to build a new kind of database. While working on Noria, I ran into a limitation of the standard library's BufReader type. The BufReader type is a type that wraps a reader and provides a buffer to read from. The problem was that there was no way to check if the buffer was empty without reading from it.

After discussing it with a labmate, I started by creating an issue:

@fintelia commented on Oct 16, 2017...

There is currently no way to tell whether there is any data buffered inside a BufReader. This is unfortunate because it means that an application has no way to know whether calls to read() and fill_buffer() will return instantly or trigger a potentially expensive call to the underlying Reader. I propose adding an is_empty() method to BufReader to fill this gap.

And then two days later followed up with a PR:

#[unstable(feature = "bufreader_is_empty", issue = "45323", reason = "recently added")]
pub fn is_empty(&self) -> bool {
    self.pos == self.cap
}

After a brief bit of discussion, the PR was merged on Nov 6, 2017 and was included in the next nightly build. Then about a week later we started using the new method in Noria.

From here the process started to draw out. The functionality I needed was already implemented, so I wasn't in a hurry. Over the next few months discussion continued on the tracking issue. There was a FCP (final comment period) to stabilize the method, but it was called off in favor of a better approach: a BufReader::buffer method.

In July 2018 I checked back on the tracked issue to ask it the new approach was ready to stabilize. A project member told me that it was still blocked on someone implementing BufWriter::buffer which had been suggested as a follow-up. Three months later someone else asked in the tracking issue if there'd been any progress. Noticing that the other method still hadn't been added, I went ahead and made a PR:

#[unstable(feature = "bufreader_buffer", issue = "45323")]
pub fn buffer(&self) -> &[u8] {
    &self.buf
}

This one proved slightly harder to get reviewed. I had to ping my rustbot assigned reviewer twice and a triage team member also had to ping the issue. But the whole process concluded in about a month.

Then in January 2019 I again bumped the main tracking issue to ask if the new methods were ready to stabilize. After two more months with no reply someone else commented on the issue asking if there was any reason not to move forward, so I tried tagging the project members who'd been involved previously. After another month without reply I tried pinging the other project members tagged by the earlier FCP. This time it landed back on the libs team's radar and they decided to stabilize the methods. The stabilization PR merged on May 29th and Rust 1.37.0 including the new methods was released on Aug 15, 2019.

All told, the process took just under a year and ten months from the initial PR to stabilization.

2023

By 2023 I'd been maintaining the image crate for a few years. The crate depends on a number of format-specific crates to encode and decode images in various formats. One problem I'd encountered was that Rust standard library's io traits had some annoying limitations that made it hard to write efficient image decoders. In particular, I learned that the BufReader type didn't have an efficient implementation of the Seek trait due to an API limitation.

Other than a documentation PR, I hadn't contributed to Rust since 2019. Thus I went looking for updated guidelines on contributing. I didn't find clear guidance but it seemed that the right thing to do was to create a Pre-RFC on internals.rust-lang.org:

@fintelia - Sep 10, 2023

BufReader::seek_relative was added to address #31100. As explained in that issue, BufReader's stable API required that the internal buffer was dumped on any call to seek, so the new method was added to expose a way to seek without doing dumping the buffer.

Unfortunately, there isn't really a good way to take advantage of the method in generic code. If a method takes a R: Read + Seek, then only the regular (inefficient) seek implementation is exposed. The method could itself create a BufReader<R> and then call seek_relative, but then depending on what R is, it might end up making a BufReader<BufReader<File>> or BufReader<Cursor<Vec<u8>>> or some other silly type.

If the Seek trait had a seek_relative method (with the straightforward implementation as a default impl) then this could be solved.

I didn't get much of a response on the Pre-RFC, so after a month I went ahead and made a PR:

/// trait Seek
#[unstable(feature = "seek_seek_relative", issue = "none")]
fn seek_relative(&mut self, offset: i64) -> Result<()> {
    self.seek(SeekFrom::Current(offset))?;
    Ok(())
}

// struct BufReader
fn seek_relative(&mut self, offset: i64) -> io::Result<()> {
    self.seek_relative(offset)
}

A project member quickly responded to the PR and explained that API change proposal was required to add new methods to the standard library. I created an issue on the rust-lang/libs-team repository as requested, and it was accepted two days later. Around this time non-Rust factors interfered with my ability to work on the feature, though over the next few weeks I did create a tracking issue for it and respond to some minor feedback. By mid-November the implementation PR was merged.

Next started the first waiting period. Given that the library I was maintaining targeted stable Rust, the availability of the feature on nightly was of no use to me. However, the project expects all features to stay in nightly for an indeterminate period of time before stabilization.

I wasn't sure how long to wait, but four months later tried I bumped the tracking issue asking whether the feature was ready for a FCP. It got no response so two weeks later I posted in the rust-lang zulip chat asking about stabilization. After another week, a project member initiated the final comment period, which concluded without any objections.

Someone else then went ahead and made a stabilization PR. Sadly, it didn't get a response so after two weeks I tried pinging the assigned reviewer. After two more weeks without a response, I asked in the zulip chat what the right thing to do was. Another commenter suggested the proper rustbot command to reasign the reviewer. Shortly after, the PR was merged. Unfortunately, the delay reviewing the stabilization PR caused it to miss the merge window so it'll have to wait over ten weeks for Rust 1.80 to reach stable on July 25, 2024.

Of course, the process won't end there. The image crate -- like much of the Rust ecosystem -- doesn't force upgrading to new rustc versions immediately after they're released. It instead supports the last several Rust versions, which will mean that the new method won't be usable until several months after it's stabilized.

All told it'll be about 15 months from the initial Pre-RFC to when I'll actually be able to use the new method in the image crate. Which is a pretty significant ordeal for a feature I've never actually used yet.

Takeaways

When I got started writing this article I was expecting to find that the process had gotten slower and more circuitous. It certainly felt that way. However, the actual difference was more in the non-process factors. The actual time time to stabilization was "only" ten and a half months, almost a year faster than the previous time!

In 2017, I was using nightly Rust and was able to use the new method almost immediately after it was merged. This was typical for the time. Rust 1.0 had only been released a few years earlier and the ecosystem was still in a state of flux. By 2023 stable Rust was the norm, particularly for libraries. And not just latest stable, but the expectation of a minimum supported Rust version that is at least several releases back. The net result is that instead of waiting one month to use a new feature, I'll be waiting about 15 months.