Wenwen Wang (Jul 16 2023 at 13:12): Wenwen Wang (Jul 16 2023 at 13:12):Hi everyone! I am developing a testing framework to find performance problems in WebAssembly runtimes. This framework is part of a research proposal. I'm wondering whether I can request a collaboration letter from you, showing that once the proposal is funded I can use the developed framework to test Wasmtime to find its potential performance problems. Or, could you please tell me who should I contact regarding this issue? Thank you very much!
Chris Fallin (Jul 17 2023 at 05:17):Can you say more what this entails? From whom are you hoping to have a letter (we are a loose association of various contributors), what would it say, etc? Are you hoping for a formal promise that we would collaborate or…? Without knowing more about what you would be working on and without some history of discussion with us I’m not sure we could offer such a formal thing, though of course we’re always happy to actually collaborate on relevant projects.
Wenwen Wang (Jul 17 2023 at 12:50):Chris Fallin said:
Can you say more what this entails? From whom are you hoping to have a letter (we are a loose association of various contributors), what would it say, etc? Are you hoping for a formal promise that we would collaborate or…? Without knowing more about what you would be working on and without some history of discussion with us I’m not sure we could offer such a formal thing, though of course we’re always happy to actually collaborate on relevant projects.
Thank you so much for your prompt response, Chris! Sure, of course, I can talk about more details of the project and the collaboration letter. Is it okay to send you DM?
Chris Fallin (Jul 17 2023 at 14:50):No, I'd prefer not to take this to DMs -- mostly I'm trying to get the discussion in the public channel going in the right direction, so the right folks can see it (that's likely not me, I would not be the person to write such a letter)
Wenwen Wang (Jul 17 2023 at 17:23):No problem. I will talk about them here.
The techniques I will develop are mainly performance testing for Wasm runtimes. Specifically, we will generate various Wasm binaries to stress a Wasm runtime to find its performance problems, e.g., performance bugs, missed optimizations, and improperly implemented optimizations. We will also generate WASI test programs to test the performance of WASI implementations in Wasm runtimes. Finally, we will generate well-focused Wasm binaries to detect code commits that can cause potential performance regression issues during the development of a Wasm runtime.
Regarding the letter, I would like to have the letter from the project manager of Wasmtime if possible. It would be great if it can indicate the support for the project and a collaboration intention if it is get funded, e.g., testing Wasmtime to find its performance problems or using/evaluating the developed testing framework in daily development.
Chris Fallin (Jul 17 2023 at 17:31):So, we currently have Sightglass which seems to cover a lot of the same needs -- how does this work differ?
Wenwen Wang (Jul 17 2023 at 17:45):Thanks for your question, Chris! I think the key difference is that we plan to generate a large number of Wasm binaries to test the performance, instead of using existing benchmarks, which may not achieve good coverage in terms of performance testing. Also, we use several carefully designed test oracles to determine the existence of performance problems. For example, we generate performance-similar Wasm binaries to check whether a Wasm runtime exhibit similar performances when running the Wasm binaries. I think the techniques developed in our project can complement Sightglass to achieve better testing effectiveness and efficiency.
Alex Crichton (Jul 17 2023 at 17:46):I'll echo what Chris already mentioned about we're sort of a loose collaboration, there's no single project manager for Wasmtime for example. It's developed by a number of different companies so there's not a single manager at a single company who's "in charge" for example.
Personally I think it's interesting to collect data points in lots of various ways about performance, but I think doing so in a robust fashion can be quite challening. Additionally benchmarks have a way of not always aging very well, too. I'd echo here again what Chris mentioned with Sightglass where it's a system that Wasmtime already, so it'd be the most useful to us to integrate with what we already have (and in theory that can cover other runtimes as well too).
That being said though as an open source project you're more than welcome to take Wasmtime and analyze it and benchmark it however you'd like, you don't really need explicit consent from us to do that. Could you perhaps clarify what our role as a project in this would be? Are we basically saying we'd find such analysis interesting or are you thinking we would have a much more active role in shaping the benchmark suite and such?
Wenwen Wang (Jul 17 2023 at 17:57):Many thanks for your response, Alex! Your suggestions are very helpful to me! Perhaps, I should clarify that the letter is used to indicate that Wasmtime developers are interested in the performance testing techniques developed in the project and would like to provide comments and feedback from an industry perspective about the developed techniques if a usable research prototype is available, and potentially integrate the techniques into existing testing framework Sightglass if possible. What do you think about this?
Chris Fallin (Jul 17 2023 at 18:00):My perspective at least is that we're always interested in discussions, but a letter is a bit heavyweight and feels somehow like it's making a commitment out of turn: the amount to which we can collaborate really depends on how the project turns out, how much time we have at any given point, etc. Maybe we can collaborate a bit more informally first?
Karel Hrkal (kajacx) (Jul 17 2023 at 19:01):Hello, I am not a contributor to wasmtime, but I am working on a project that might be interesting to have a look at if you are looking to measure performance of wasm modules and their runtimes.
I am working on wasm-bridge, a Rust crate that would allow you tu load and run wasm modules in Rust on the web with the same code that you use on the desktop.
It might be interesting to compare the performance of the same waam module in these situations:
I have not optimized the code for performance yet (in fact, I have not even measured the difference), so I would be interested to see the results.
As a side note, wasmer is doing basically the same thing with their "js" feature. The wasm module is executed by the js runtime, you can interact with it from Rust the same way.
Wenwen Wang (Jul 17 2023 at 19:45):Chris Fallin said:
My perspective at least is that we're always interested in discussions, but a letter is a bit heavyweight and feels somehow like it's making a commitment out of turn: the amount to which we can collaborate really depends on how the project turns out, how much time we have at any given point, etc. Maybe we can collaborate a bit more informally first?
Thanks, Chris! I understand your concerns. To be clear, I don't think the letter requires any commitment from your side. It is actually a kind of support letter to show Wasmtime developers' interest in the proposed research project. We can definitely collaborate first, but I need to submit the proposal (deadline: July 26) to secure a grant to support our development.
Chris Fallin (Jul 17 2023 at 19:48):OK. At least from my side, I am not able to supply such a letter. Best of luck.
Last updated: Dec 23 2024 at 13:07 UTC