Stream: general

Topic: rustix interface for `recvmmsg`

Colin Marc (Sep 13 2024 at 12:43):

Hi, I'm trying to make a PR to add sendmmsg/recvmmsg (note the extra m) to rustix. libc has these functions in linux_like, which I gather is a hard requirement. However, I have a few questions as I go:

• It looks like the BSDs have sendmmsg, but it's not a syscall. Does that mean it should be only for linux?
• The interface is shaping up to be pretty unwieldy, something like msgs: &[(&[IoSlice<'_>], &mut SendAncillaryBuffer)]. That's not even getting into the return type. Should I go straight to a higher level interface, maybe something like a builder pattern, or just slog through with that?
• The msghdr.rs interfaces are not well-suited to this (see with_noaddr_msghdr). Is that designed specifically to avoid allocation or something? Should I try to adapt those or just do without?
• Should I just give up and cry?

Thanks in advance!

Edit: thinking more, I think it has to be a builder pattern, or you can't mix sockaddr types easily. Something like:

let res: Vec<usize> = Sendmmsg::builder(fd)
    .send_v4(sa1, &[IoSlice::new("foo")], Default::default())
    .send_v6(sa2, &[IoSlice::new("bar")], Default::default())
    .done()?;

Dan Gohman (Sep 13 2024 at 16:26):

It looks like the BSDs have sendmmsg, but it's not a syscall. Does that mean it should be only for linux?

It can go either way. rustix aims for "syscall-like APIs", and sendmmsg is pretty syscall-like. That said, if it's more complex, it's ok to defer for now too.

The interface is shaping up to be pretty unwieldy, something like msgs: &[(&[IoSlice<'_>], &mut SendAncillaryBuffer)]. That's not even getting into the return type. Should I go straight to a higher level interface, maybe something like a builder pattern, or just slog through with that?

My understanding is that sendmmsg is just an optimization, compared to making multiple sendmsg calls. As such, my guess is that it's better to go for the lower-level interface, since a builder would likely impose a dynamic allocation which users looking to optimize would want to avoid. That's just a guess though, and not a hard rule.

The msghdr.rs interfaces are not well-suited to this (see with_noaddr_msghdr). Is that designed specifically to avoid allocation or something? Should I try to adapt those or just do without?
Should I just give up and cry?

My guess is that that existing infrastructure won't work well for you as-is, if you need to support multiple addresses in a single sendmmsg call.

As a possible alternative, I wonder if this would work: have a public type MMsgHdr, that contains a single private mmsghdr and is repr(transparent). Give it constructors for constructing it with different address types and data and control messages. It probably needs lifetime parameters. And a method for reading out the msg_len value. And then let users do their own builder-like thing with a Vec<MMsgHdr>``, or their own fixed-sized [MMsgHdr; N], or whatever, as they see fit, and pass it into sendmmsg as a &mut [MMsgHdr]. Since it's repr(transparent), rustix could pass the raw slice into the sendmmsg syscall. Do you think something like that might work?

Colin Marc (Sep 13 2024 at 16:42):

My understanding is that sendmmsg is just an optimization, compared to making multiple sendmsg calls. As such, my guess is that it's better to go for the lower-level interface, since a builder would likely impose a dynamic allocation which users looking to optimize would want to avoid. That's just a guess though, and not a hard rule.

Even the lower-level interface would require allocating msghdr structs, so it wouldn't save any allocations. Also, the performance benefit is usually from the reduced number of syscalls, not from the allocations, but that's a fair point.

As a possible alternative, I wonder if this would work: have a public type MMsgHdr, that contains a single private mmsghdr and is repr(transparent). Give it constructors for constructing it with different address types and data and control messages. It probably needs lifetime parameters. And a method for reading out the msg_len value. And then let users do their own builder-like thing with a Vec<MMsgHdr>``, or their own fixed-sized [MMsgHdr; N], or whatever, as they see fit, and pass it into sendmmsg as a &mut [MMsgHdr]. Since it's repr(transparent), rustix could pass the raw slice into the sendmmsg syscall. Do you think something like that might work?

Yes, that seems doable! FWIW, the MMsgHdr is just a msghdr and a len, so it would be potentially usable for single-M sendmsg as well.

Also, there's the question of the return value. The syscall returns the number of messages succesfully sent, and the amount of each message sent is saved in in the mmsghdr's length field. Hence Vec<usize>. One benefit of the builder pattern thing is that you could offer a remaining() method to filter out everything that's already been sent, but maybe that's too high-level.

Colin Marc (Sep 18 2024 at 19:42):

Hm, so, I spent some time poking at this, and unfortunately I don't think repr(transparent) works very well.

The problem is that msghdr is just a bunch of pointers:

struct msghdr {
    void         *msg_name;       /* Optional address */
    socklen_t     msg_namelen;    /* Size of address */
    struct iovec *msg_iov;        /* Scatter/gather array */
    size_t        msg_iovlen;     /* # elements in msg_iov */
    void         *msg_control;    /* Ancillary data, see below */
    size_t        msg_controllen; /* Ancillary data buffer len */
    int           msg_flags;      /* Flags on received message */
};

If I have RecvMMsgHdr with a constructor like new(iov: &mut [IoSliceMut], control: RecvAncillaryBuffer), then I can ensure with lifetimes that the iov pointers and control buffer pointer are safe to store on the msghdr. But that's not true of the sockaddr. I could require a &mut SockAddrStorage, but I don't have a way to communicate whether the memory is valid after the call (it depends on whether the kernel set msg_namelen >= 0, I think).

It would be reasonable to allocate the sockaddrstorage on behalf of the user, but I don't have anywhere to store the pointer, since it's repr(transparent).

Any ideas?

Jacob Lifshay (Sep 19 2024 at 03:20):

maybe use 1-ZST typestate to track if it's before/after the call?

Colin Marc (Sep 19 2024 at 07:30):

Since this issue is unique to recvmmsg, I pushed a PR with just sendmmsg: https://github.com/bytecodealliance/rustix/pull/1171

maybe use 1-ZST typestate to track if it's before/after the call?

I'm not a type wizard, would you be able to sketch out an example of what you mean?

Another idea I had would be to make a RecvMMsgBuffer utility with something like fn grow(&mut self, len: usize, mtu: usize). It would maintain a slab of memory for contents/cmgs/sockaddrs and a Vec<mmsghdr>, and let you iterate over filled messages. The goal would be for it to be reusable, obviously.

Last updated: Nov 22 2024 at 16:03 UTC