Victor Adossi (Jun 29 2023 at 07:10): Victor Adossi (Jun 29 2023 at 07:10):Hey all, thanks for the awesome work!
I have what I think is a quick beginner level question here --
Is there a way to access memory from a non-JS, purely Rust context?
I see wasm_bindgen::memory, which I can get to give me the buffer, but after the module has been written out, it's failing to find import (ex. unknown import: __wbindgen_placeholder__::__wbindgen_describe has not been defined)
I'm in a wasm32-unknown-unknown context, but wasm32-wasi would be great to know as well -- how do I just read data into a Vec<u8> from the "default" memory module in pure Rust?
If I drop down and try to do it with WAT (let's say exporting a function that returns the data), it seems like I'd be limited to byte-by-byte reads
Lann Martin (Jun 29 2023 at 13:06):If you are referring to Rust in a guest module then "yes"; Rust pointers are just offsets into "normal" memory. If you can give more detail about what you're looking to do there might be an easier approach.
fitzgen (he/him) (Jun 29 2023 at 15:15):wasm-bindgen is only for targeting Wasm on the Web/node/JS hosts, so trying to use any wasm-bindgen stuff on wasm32-wasi and Wasmtime isn't going to work out
Instead, you should either
wit-bindgen, which makes it so that you can't access memory directly, because components are shared-nothing. instead you only communicate via WIT's interfaces and its typeswasmtime::Caller::get_export) Victor Adossi (Jun 29 2023 at 23:29):Appreciate the responses ya'll!
Hey @Lann Martin thanks -- so what I'm trying to do is read memory in a guest module, from that guest module. Reading the guest module's own attached memory.
I don't have a pointer yet, per-say -- I have Rust code that is doing wasm things.
Your suggestion is making me think that actually I just need the start & length of the memory to read it... So basically, I write a function to tell me where the memory starts and how long, then use that from the rust code. Something like:
extern "C" {
pub fn get_main_mem_start() -> i32;
pub fn get_main_mem_len() -> i32;
}
Then in the rust code I can do some unsafe reading of the memory from start?
@fitzgen (he/him) thanks for the detail -- at first I'm trying to do this with wasm32-unknown-unknown but I will keep this in mind for the wasm32-wasi attempt.
I think your second point makes it really clear how I'm not explaining well enough! I'm trying to do all of this from inside the guest.
Basically, I want to stuff a value into memory inside the guest module (currently, a wasm module, not a preview1/preview2 component), and read it back out, from the guest.
In practice this starts with "read data from memory", inside the Rust module (just pretending like it's there), which I can't find a reasonable way to do just yet.
Oh and another note, I want to do all this without modifying the WasiCtx at instantiation time. If a guest module has memory in it (which is written into the file itself), it seems like it should be able to read that data without being fed a constructed memory.
Lann Martin (Jun 30 2023 at 00:00):Ah, so you are looking to read static data that is preinitialized in the module? The details of that will depend on how you are inserting that external data. You could, for example, initialize data from a file at compile time with e.g. https://doc.rust-lang.org/std/macro.include_bytes.html, and then there would be nothing wasm-specific to do.
Victor Adossi (Jun 30 2023 at 00:01):As far as the byte-by-byte read solution which seemed more straight forward
Some more code that seems like it would work, but doesn't:
// Retrieve the default memory
let mem_id = module
.memories
.iter()
.next()
.ok_or_else(|| anyhow!("list of memories was unexpectedly empty"))?
.id();
let mut load_fn = FunctionBuilder::new(&mut module.types, &[ValType::I32], &[ValType::I32]);
load_fn
.name("load_data".into())
.func_body()
.local_get(read_offset)
.load(
mem_id,
LoadKind::I32_8 {
kind: ExtendedLoad::ZeroExtend,
},
MemArg {
align: 1,
offset: 0,
},
);
load_fn.finish(vec![read_offset], &mut module.funcs);
From the guest, I try to do the following:
extern "C" {
pub fn load_data(offset: i32) -> i32;
}
When I call that though, wasmtime complains that the import has not been defined, but in the WAT I can see the function:
....
(import "env" "load_data" (func $load_data (;3;) (type 3)))
....
(func $#func1498<load_data> (@name "load_data") (;1498;) (type 3) (param i32) (result i32)
local.get 0
i32.load8_u
)
Another thing that's occurring to me is that maybe walrus actually doesn't work on wasm32-unknown-unknown any more? the function naming I see being used (@name, etc) is quite unique compared to the other locally defined functions)
Victor Adossi (Jun 30 2023 at 00:03):Lann Martin said:
Ah, so you are looking to read static data that is preinitialized in the module? The details of that will depend on how you are inserting that external data. You could, for example, initialize data from a file at compile time with e.g. https://doc.rust-lang.org/std/macro.include_bytes.html, and then there would be nothing wasm-specific to do.
That's the interesting bit! I'm trying to inject this data into the module itself, with walrus.
I thought the best way to do do this would be to insert a memory section with the data, but multiple memories are not supported just yet, so I'm trying to use the main memory module to do it (IIRC it's empty/waiting to be filled, normally).
If it was just packing it in on the rust side it would have been trivial, thanks to include_bytes :)
What I'm finding is that I can't even call an injected function correctly either (since technically I could convert any bunch of bytes into a horrible hacky function which returns each byte as a lookup)...
Lann Martin (Jun 30 2023 at 00:11):It should be possible to do that with wasm bulk memory operations, but doing that from Rust is beyond my experience.
Lann Martin (Jun 30 2023 at 00:12):@Kyle Brown does something similar in his template generator, but by generating wasm instructions directly.
Lann Martin (Jun 30 2023 at 00:15): Victor Adossi (Jun 30 2023 at 00:31):Hey @Lann Martin thanks for noodling on it a bit! I think I have a path forward and there are so many avenues to try! I just really expected to be able to at least inject a function into the guest module (without instantiation), at the very least -- I'm sure there's something obvious I'm missing here.
Victor Adossi (Jul 12 2023 at 17:42):For anyone who might be trying to do this -- @Guy Bedford has an excellent example here:
https://github.com/bytecodealliance/WASI-Virt
(the original commit in the PR was a bit more instructive as it's a bit more raw)
While I ended up hacking my thing together by reserving some space & modifying WAT files, the more right way is basically:
static on the rust side Rust)unsafe -- see get_env_ptr in the raw code)walrus)unsafe fn)thanks @Bailey Hayes for forwarding this to me as an example!
Victor Adossi (Jul 12 2023 at 17:43):The wasi-virt usecase is slightly different, but the early version of the PR was exactly what I was trying to do.
The wasi-virt ended up looking quite different (using the "env" import directly now, it looks like), but still really impressive/great to look at for understanding
Victor Adossi (Jul 12 2023 at 17:44):Gonna mark this resolved since there is high quality code that demonstrates this now
Victor Adossi (Jul 12 2023 at 17:44):JK, can't mark it resolved, if any admins come by this, feel free to mark it resolved!
Last updated: Oct 23 2024 at 20:03 UTC