Stream: git-wasmtime

Topic: wasmtime / issue #4145 Cannot capture stdout using WasiCtx

view this post on Zulip Wasmtime GitHub notifications bot (May 12 2022 at 14:54):

jlkiri opened issue #4145:

If I understand correctly, stdout method on WasiCtxBuilder can be used to set what the instance will treat as stdout. This something needs to implement WasiFile trait and I discovered that File from cap-std create implements it. It also allows creating cap-std-files from usual std files. I guessed that I can then created a usual file, and then through some conversions turn it into a WasiCtxBuilder-usable stdout target. Since all these methods own the file, I obtain the raw fd which I then use to open the file again and read its contents. However it appears to be empty, although I do print text in the Rust program compiled to wasm. Note that the execution itself is successful - I just don't see any side effects. Here's the code that I compile to wasm:

fn main() {
  println!("Hello");
}

and here's what I use to execute it in wasmtime:

fn wasm_execute(
    engine: &wasmtime::Engine,
    module_path: impl AsRef<Path>,
) -> anyhow::Result<String> {
    use cap_std::fs::File as CapStdFile;
    use wasmtime::*;
    use wasmtime_wasi::{add_to_linker, sync::file::File, WasiCtxBuilder};

    let module = Module::from_file(&engine, module_path)?;

    let tmpfile = tempfile::tempfile()?;
    let tmpfile_fd = tmpfile.as_raw_fd();

    let cap_std_file = CapStdFile::from_std(tmpfile);
    let stdout_file = File::from_cap_std(cap_std_file);

    let wasi = WasiCtxBuilder::new().stdout(Box::new(stdout_file)).build();

    let store_limits = StoreLimits::default();
    let mut store = Store::new(&engine, Data { wasi, store_limits });
    store.limiter(|data| &mut data.store_limits);

    let mut linker: Linker<Data> = Linker::new(&engine);
    add_to_linker(&mut linker, |state| &mut state.wasi)?;

    linker.module(&mut store, "", &module)?;
    linker.instantiate(&mut store, &module)?;
    linker
        .get_default(&mut store, "")?
        .typed::<(), (), _>(&store)?
        .call(&mut store, ())?;

    let mut output = String::new();

    let mut stdout_file_handle = unsafe { std::fs::File::from_raw_fd(tmpfile_fd) };
    let bytes_read = stdout_file_handle.read_to_string(&mut output)?;

    tracing::info!(bytes_read); // 0 bytes read

    Ok(output)
}

Could it be that raw_fd is no longer valid at that point?

view this post on Zulip Wasmtime GitHub notifications bot (May 12 2022 at 14:55):

jlkiri edited issue #4145:

If I understand correctly, stdout method on WasiCtxBuilder can be used to set what the instance will treat as stdout. This something needs to implement WasiFile trait and I discovered that File from cap-std create implements it. It also allows creating cap-std-files from usual std files. I guessed that I can then created a usual file, and then through some conversions turn it into a WasiCtxBuilder-usable stdout target. Since all these methods own the file, I obtain the raw fd which I then use to open the file again and read its contents. However it appears to be empty, although I do print text in the Rust program compiled to wasm. Note that the execution itself is successful - I just don't see any side effects. Here's the code that I compile to wasm:

fn main() {
  println!("Hello");
}

and here's what I use to execute it in wasmtime:

fn wasm_execute(
    engine: &wasmtime::Engine,
    module_path: impl AsRef<Path>,
) -> anyhow::Result<String> {
    use cap_std::fs::File as CapStdFile;
    use wasmtime::*;
    use wasmtime_wasi::{add_to_linker, sync::file::File, WasiCtxBuilder};

    let module = Module::from_file(&engine, module_path)?;

    let tmpfile = tempfile::tempfile()?;
    let tmpfile_fd = tmpfile.as_raw_fd();

    let cap_std_file = CapStdFile::from_std(tmpfile);
    let stdout_file = File::from_cap_std(cap_std_file);

    let wasi = WasiCtxBuilder::new().stdout(Box::new(stdout_file)).build();

    let store_limits = StoreLimits::default();
    let mut store = Store::new(&engine, Data { wasi, store_limits });
    store.limiter(|data| &mut data.store_limits);

    let mut linker: Linker<Data> = Linker::new(&engine);
    add_to_linker(&mut linker, |state| &mut state.wasi)?;

    linker.module(&mut store, "", &module)?;
    linker.instantiate(&mut store, &module)?;
    linker
        .get_default(&mut store, "")?
        .typed::<(), (), _>(&store)?
        .call(&mut store, ())?;

    let mut output = String::new();

    let mut stdout_file_handle = unsafe { std::fs::File::from_raw_fd(tmpfile_fd) };
    let bytes_read = stdout_file_handle.read_to_string(&mut output)?;

    tracing::info!(bytes_read); // 0 bytes read

    Ok(output)
}

Could it be that raw_fd is no longer valid at that point?

view this post on Zulip Wasmtime GitHub notifications bot (May 12 2022 at 14:55):

jlkiri edited issue #4145:

If I understand correctly, stdout method on WasiCtxBuilder can be used to set what the instance will treat as stdout. This something needs to implement WasiFile trait and I discovered that File from cap-std create implements it. It also allows creating cap-std-files from usual std files. I guessed that I can then created a usual file, and then through some conversions turn it into a WasiCtxBuilder-usable stdout target. Since all these methods own the file, I obtain the raw fd which I then use to open the file again and read its contents. However it appears to be empty, although I do print text in the Rust program compiled to wasm. Note that the execution itself is successful - I just don't see any side effects. Here's the code that I compile to wasm:

fn main() {
  println!("Hello");
}

and here's what I use to execute it in wasmtime:

fn wasm_execute(
    engine: &wasmtime::Engine,
    module_path: impl AsRef<Path>,
) -> anyhow::Result<String> {
    use cap_std::fs::File as CapStdFile;
    use wasmtime::*;
    use wasmtime_wasi::{add_to_linker, sync::file::File, WasiCtxBuilder};

    let module = Module::from_file(&engine, module_path)?;

    let tmpfile = tempfile::tempfile()?;
    let tmpfile_fd = tmpfile.as_raw_fd();

    let cap_std_file = CapStdFile::from_std(tmpfile);
    let stdout_file = File::from_cap_std(cap_std_file);

    let wasi = WasiCtxBuilder::new().stdout(Box::new(stdout_file)).build();

    let store_limits = StoreLimits::default();
    let mut store = Store::new(&engine, Data { wasi, store_limits });
    store.limiter(|data| &mut data.store_limits);

    let mut linker: Linker<Data> = Linker::new(&engine);
    add_to_linker(&mut linker, |state| &mut state.wasi)?;

    linker.module(&mut store, "", &module)?;
    linker.instantiate(&mut store, &module)?;
    linker
        .get_default(&mut store, "")?
        .typed::<(), (), _>(&store)?
        .call(&mut store, ())?;

    let mut output = String::new();

    let mut stdout_file_handle = unsafe { std::fs::File::from_raw_fd(tmpfile_fd) };
    let bytes_read = stdout_file_handle.read_to_string(&mut output)?;

    tracing::info!(bytes_read); // 0 bytes read

    Ok(output)
}

Could it be that raw_fd is no longer valid at that point?

view this post on Zulip Wasmtime GitHub notifications bot (May 12 2022 at 14:55):

jlkiri edited issue #4145:

If I understand correctly, stdout method on WasiCtxBuilder can be used to set what the instance will treat as stdout. This something needs to implement WasiFile trait and I discovered that File from cap-std crate implements it. It also allows creating cap-std-files from usual std files. I guessed that I can then created a usual file, and then through some conversions turn it into a WasiCtxBuilder-usable stdout target. Since all these methods own the file, I obtain the raw fd which I then use to open the file again and read its contents. However it appears to be empty, although I do print text in the Rust program compiled to wasm. Note that the execution itself is successful - I just don't see any side effects. Here's the code that I compile to wasm:

fn main() {
  println!("Hello");
}

and here's what I use to execute it in wasmtime:

fn wasm_execute(
    engine: &wasmtime::Engine,
    module_path: impl AsRef<Path>,
) -> anyhow::Result<String> {
    use cap_std::fs::File as CapStdFile;
    use wasmtime::*;
    use wasmtime_wasi::{add_to_linker, sync::file::File, WasiCtxBuilder};

    let module = Module::from_file(&engine, module_path)?;

    let tmpfile = tempfile::tempfile()?;
    let tmpfile_fd = tmpfile.as_raw_fd();

    let cap_std_file = CapStdFile::from_std(tmpfile);
    let stdout_file = File::from_cap_std(cap_std_file);

    let wasi = WasiCtxBuilder::new().stdout(Box::new(stdout_file)).build();

    let store_limits = StoreLimits::default();
    let mut store = Store::new(&engine, Data { wasi, store_limits });
    store.limiter(|data| &mut data.store_limits);

    let mut linker: Linker<Data> = Linker::new(&engine);
    add_to_linker(&mut linker, |state| &mut state.wasi)?;

    linker.module(&mut store, "", &module)?;
    linker.instantiate(&mut store, &module)?;
    linker
        .get_default(&mut store, "")?
        .typed::<(), (), _>(&store)?
        .call(&mut store, ())?;

    let mut output = String::new();

    let mut stdout_file_handle = unsafe { std::fs::File::from_raw_fd(tmpfile_fd) };
    let bytes_read = stdout_file_handle.read_to_string(&mut output)?;

    tracing::info!(bytes_read); // 0 bytes read

    Ok(output)
}

Could it be that raw_fd is no longer valid at that point?

view this post on Zulip Wasmtime GitHub notifications bot (May 12 2022 at 14:56):

jlkiri edited issue #4145:

If I understand correctly, stdout method on WasiCtxBuilder can be used to set what the instance will treat as stdout. This something needs to implement WasiFile trait and I discovered that File from cap-std crate implements it. It also allows creating cap-std-files from usual std files. I guessed that I can then created a usual file, and then through some conversions turn it into a WasiCtxBuilder-usable stdout target. Since all these methods own the file, I obtain the raw fd which I then use to open the file again and read its contents. However it appears to be empty, although I do print text in the Rust program compiled to wasm. Note that the execution itself is successful - I just don't see any side effects. Here's the code that I compile to wasm:

fn main() {
  println!("Hello");
}

and here's what I use to execute it in wasmtime:

fn wasm_execute(
    engine: &wasmtime::Engine,
    module_path: impl AsRef<Path>,
) -> anyhow::Result<String> {
    use cap_std::fs::File as CapStdFile;
    use wasmtime::*;
    use wasmtime_wasi::{add_to_linker, sync::file::File, WasiCtxBuilder};

    let module = Module::from_file(&engine, module_path)?;

    let tmpfile = tempfile::tempfile()?;
    let tmpfile_fd = tmpfile.as_raw_fd();

    let cap_std_file = CapStdFile::from_std(tmpfile);
    let stdout_file = File::from_cap_std(cap_std_file);

    let wasi = WasiCtxBuilder::new().stdout(Box::new(stdout_file)).build();

    let store_limits = StoreLimits::default();
    let mut store = Store::new(&engine, Data { wasi, store_limits });
    store.limiter(|data| &mut data.store_limits);

    let mut linker: Linker<Data> = Linker::new(&engine);
    add_to_linker(&mut linker, |state| &mut state.wasi)?;

    linker.module(&mut store, "", &module)?;
    linker.instantiate(&mut store, &module)?;
    linker
        .get_default(&mut store, "")?
        .typed::<(), (), _>(&store)?
        .call(&mut store, ())?;

    let mut output = String::new();

    let mut stdout_file_handle = unsafe { std::fs::File::from_raw_fd(tmpfile_fd) };
    let bytes_read = stdout_file_handle.read_to_string(&mut output)?;

    tracing::info!(bytes_read); // 0 bytes read

    Ok(output)
}

Could it be that raw_fd is no longer valid at that point?

view this post on Zulip Wasmtime GitHub notifications bot (May 12 2022 at 14:56):

jlkiri edited issue #4145:

If I understand correctly, stdout method on WasiCtxBuilder can be used to set what the instance will treat as stdout. This something needs to implement WasiFile trait and I discovered that File from cap-std crate implements it. It also allows creating cap-std-files from usual std files. I guessed that I can then create a usual file, and then through some conversions turn it into a WasiCtxBuilder-usable stdout target. Since all these methods own the file, I obtain the raw fd which I then use to open the file again and read its contents. However it appears to be empty, although I do print text in the Rust program compiled to wasm. Note that the execution itself is successful - I just don't see any side effects. Here's the code that I compile to wasm:

fn main() {
  println!("Hello");
}

and here's what I use to execute it in wasmtime:

fn wasm_execute(
    engine: &wasmtime::Engine,
    module_path: impl AsRef<Path>,
) -> anyhow::Result<String> {
    use cap_std::fs::File as CapStdFile;
    use wasmtime::*;
    use wasmtime_wasi::{add_to_linker, sync::file::File, WasiCtxBuilder};

    let module = Module::from_file(&engine, module_path)?;

    let tmpfile = tempfile::tempfile()?;
    let tmpfile_fd = tmpfile.as_raw_fd();

    let cap_std_file = CapStdFile::from_std(tmpfile);
    let stdout_file = File::from_cap_std(cap_std_file);

    let wasi = WasiCtxBuilder::new().stdout(Box::new(stdout_file)).build();

    let store_limits = StoreLimits::default();
    let mut store = Store::new(&engine, Data { wasi, store_limits });
    store.limiter(|data| &mut data.store_limits);

    let mut linker: Linker<Data> = Linker::new(&engine);
    add_to_linker(&mut linker, |state| &mut state.wasi)?;

    linker.module(&mut store, "", &module)?;
    linker.instantiate(&mut store, &module)?;
    linker
        .get_default(&mut store, "")?
        .typed::<(), (), _>(&store)?
        .call(&mut store, ())?;

    let mut output = String::new();

    let mut stdout_file_handle = unsafe { std::fs::File::from_raw_fd(tmpfile_fd) };
    let bytes_read = stdout_file_handle.read_to_string(&mut output)?;

    tracing::info!(bytes_read); // 0 bytes read

    Ok(output)
}

Could it be that raw_fd is no longer valid at that point?

view this post on Zulip Wasmtime GitHub notifications bot (May 12 2022 at 14:58):

jlkiri edited issue #4145:

If I understand correctly, stdout method on WasiCtxBuilder can be used to set what the instance will treat as stdout. This something needs to implement WasiFile trait and I discovered that File from cap-std crate implements it. It also allows creating cap-std-files from usual std files. I guessed that I can then create a usual file, and then through some conversions turn it into a WasiCtxBuilder-usable stdout target. Since all these methods own the file, I obtain the raw fd which I then use to open the file again and read its contents. However it appears to be empty, although I do print text in the Rust program compiled to wasm. Note that the execution itself is successful - I just don't see any side effects. Also inheriting stdio also works properly. Here's the code that I compile to wasm:

fn main() {
  println!("Hello");
}

and here's what I use to execute it in wasmtime:

fn wasm_execute(
    engine: &wasmtime::Engine,
    module_path: impl AsRef<Path>,
) -> anyhow::Result<String> {
    use cap_std::fs::File as CapStdFile;
    use wasmtime::*;
    use wasmtime_wasi::{add_to_linker, sync::file::File, WasiCtxBuilder};

    let module = Module::from_file(&engine, module_path)?;

    let tmpfile = tempfile::tempfile()?;
    let tmpfile_fd = tmpfile.as_raw_fd();

    let cap_std_file = CapStdFile::from_std(tmpfile);
    let stdout_file = File::from_cap_std(cap_std_file);

    let wasi = WasiCtxBuilder::new().stdout(Box::new(stdout_file)).build();

    let store_limits = StoreLimits::default();
    let mut store = Store::new(&engine, Data { wasi, store_limits });
    store.limiter(|data| &mut data.store_limits);

    let mut linker: Linker<Data> = Linker::new(&engine);
    add_to_linker(&mut linker, |state| &mut state.wasi)?;

    linker.module(&mut store, "", &module)?;
    linker.instantiate(&mut store, &module)?;
    linker
        .get_default(&mut store, "")?
        .typed::<(), (), _>(&store)?
        .call(&mut store, ())?;

    let mut output = String::new();

    let mut stdout_file_handle = unsafe { std::fs::File::from_raw_fd(tmpfile_fd) };
    let bytes_read = stdout_file_handle.read_to_string(&mut output)?;

    tracing::info!(bytes_read); // 0 bytes read

    Ok(output)
}

Could it be that raw_fd is no longer valid at that point?

view this post on Zulip Wasmtime GitHub notifications bot (May 12 2022 at 14:58):

jlkiri edited issue #4145:

If I understand correctly, stdout method on WasiCtxBuilder can be used to set what the instance will treat as stdout. This something needs to implement WasiFile trait and I discovered that File from cap-std crate implements it. It also allows creating cap-std-files from usual std files. I guessed that I can then create a usual file, and then through some conversions turn it into a WasiCtxBuilder-usable stdout target. Since all these methods own the file, I obtain the raw fd which I then use to open the file again and read its contents. However it appears to be empty, although I do print text in the Rust program compiled to wasm. Note that the execution itself is successful - I just don't see any side effects. Inheriting stdio also works properly. Here's the code that I compile to wasm:

fn main() {
  println!("Hello");
}

and here's what I use to execute it in wasmtime:

fn wasm_execute(
    engine: &wasmtime::Engine,
    module_path: impl AsRef<Path>,
) -> anyhow::Result<String> {
    use cap_std::fs::File as CapStdFile;
    use wasmtime::*;
    use wasmtime_wasi::{add_to_linker, sync::file::File, WasiCtxBuilder};

    let module = Module::from_file(&engine, module_path)?;

    let tmpfile = tempfile::tempfile()?;
    let tmpfile_fd = tmpfile.as_raw_fd();

    let cap_std_file = CapStdFile::from_std(tmpfile);
    let stdout_file = File::from_cap_std(cap_std_file);

    let wasi = WasiCtxBuilder::new().stdout(Box::new(stdout_file)).build();

    let store_limits = StoreLimits::default();
    let mut store = Store::new(&engine, Data { wasi, store_limits });
    store.limiter(|data| &mut data.store_limits);

    let mut linker: Linker<Data> = Linker::new(&engine);
    add_to_linker(&mut linker, |state| &mut state.wasi)?;

    linker.module(&mut store, "", &module)?;
    linker.instantiate(&mut store, &module)?;
    linker
        .get_default(&mut store, "")?
        .typed::<(), (), _>(&store)?
        .call(&mut store, ())?;

    let mut output = String::new();

    let mut stdout_file_handle = unsafe { std::fs::File::from_raw_fd(tmpfile_fd) };
    let bytes_read = stdout_file_handle.read_to_string(&mut output)?;

    tracing::info!(bytes_read); // 0 bytes read

    Ok(output)
}

Could it be that raw_fd is no longer valid at that point?

view this post on Zulip Wasmtime GitHub notifications bot (May 12 2022 at 15:02):

alexcrichton commented on issue #4145:

I believe you're reading the same file descriptor that was used for writes, so I think you might need to seek to the beginning to reset the internal cursor?

view this post on Zulip Wasmtime GitHub notifications bot (May 12 2022 at 15:03):

bjorn3 commented on issue #4145:

I think you will need to seek to the start of the file. Also you should probably use cap_std::fs::File directly instead of trying to convert it into an std::fs::File. I'm pretty sure the code you have right now would close the fd twice.

view this post on Zulip Wasmtime GitHub notifications bot (May 12 2022 at 15:14):

jlkiri commented on issue #4145:

Seeking to start of the file solved the problem!

Also you should probably use cap_std::fs::File directly instead of trying to convert it into an std::fs::File
Are you talking about this line?

File::from_cap_std(cap_std_file);

This File is wasmtime::File.

view this post on Zulip Wasmtime GitHub notifications bot (May 12 2022 at 15:14):

jlkiri edited a comment on issue #4145:

Seeking to start of the file solved the problem!

Also you should probably use cap_std::fs::File directly instead of trying to convert it into an std::fs::File

Are you talking about this line?

File::from_cap_std(cap_std_file);

This File is wasmtime::File.

view this post on Zulip Wasmtime GitHub notifications bot (May 12 2022 at 15:14):

jlkiri edited a comment on issue #4145:

Seeking to start of the file solved the problem! Thanks!

Also you should probably use cap_std::fs::File directly instead of trying to convert it into an std::fs::File

Are you talking about this line?

File::from_cap_std(cap_std_file);

This File is wasmtime::File.

view this post on Zulip Wasmtime GitHub notifications bot (May 12 2022 at 15:14):

jlkiri edited a comment on issue #4145:

Seeking to the start of the file solved the problem! Thanks!

Also you should probably use cap_std::fs::File directly instead of trying to convert it into an std::fs::File

Are you talking about this line?

File::from_cap_std(cap_std_file);

This File is wasmtime::File.

view this post on Zulip Wasmtime GitHub notifications bot (May 12 2022 at 16:06):

bjorn3 commented on issue #4145:

Are you talking about this line?

No, I'm talking about std::fs::File::from_raw_fd(tmpfile_fd). I think you should use cap_std_file there instead. You may need to change File::from_cap_std(cap_std_file) to File::from_cap_std(cap_std_file.clone()) for that to be possible.

view this post on Zulip Wasmtime GitHub notifications bot (May 13 2022 at 07:34):

jlkiri commented on issue #4145:

Oh I see! Thanks, I thought that the file cannot have multiple owners by cloning.

view this post on Zulip Wasmtime GitHub notifications bot (May 13 2022 at 07:34):

jlkiri closed issue #4145:

If I understand correctly, stdout method on WasiCtxBuilder can be used to set what the instance will treat as stdout. This something needs to implement WasiFile trait and I discovered that File from cap-std crate implements it. It also allows creating cap-std-files from usual std files. I guessed that I can then create a usual file, and then through some conversions turn it into a WasiCtxBuilder-usable stdout target. Since all these methods own the file, I obtain the raw fd which I then use to open the file again and read its contents. However it appears to be empty, although I do print text in the Rust program compiled to wasm. Note that the execution itself is successful - I just don't see any side effects. Inheriting stdio also works properly. Here's the code that I compile to wasm:

fn main() {
  println!("Hello");
}

and here's what I use to execute it in wasmtime:

fn wasm_execute(
    engine: &wasmtime::Engine,
    module_path: impl AsRef<Path>,
) -> anyhow::Result<String> {
    use cap_std::fs::File as CapStdFile;
    use wasmtime::*;
    use wasmtime_wasi::{add_to_linker, sync::file::File, WasiCtxBuilder};

    let module = Module::from_file(&engine, module_path)?;

    let tmpfile = tempfile::tempfile()?;
    let tmpfile_fd = tmpfile.as_raw_fd();

    let cap_std_file = CapStdFile::from_std(tmpfile);
    let stdout_file = File::from_cap_std(cap_std_file);

    let wasi = WasiCtxBuilder::new().stdout(Box::new(stdout_file)).build();

    let store_limits = StoreLimits::default();
    let mut store = Store::new(&engine, Data { wasi, store_limits });
    store.limiter(|data| &mut data.store_limits);

    let mut linker: Linker<Data> = Linker::new(&engine);
    add_to_linker(&mut linker, |state| &mut state.wasi)?;

    linker.module(&mut store, "", &module)?;
    linker.instantiate(&mut store, &module)?;
    linker
        .get_default(&mut store, "")?
        .typed::<(), (), _>(&store)?
        .call(&mut store, ())?;

    let mut output = String::new();

    let mut stdout_file_handle = unsafe { std::fs::File::from_raw_fd(tmpfile_fd) };
    let bytes_read = stdout_file_handle.read_to_string(&mut output)?;

    tracing::info!(bytes_read); // 0 bytes read

    Ok(output)
}

Could it be that raw_fd is no longer valid at that point?

Last updated: Oct 23 2024 at 20:03 UTC