Luke Street (Feb 19 2025 at 22:40): Luke Street (Feb 19 2025 at 22:40):Hi! While using jco transpile, I've had a few issues with the generated types. Here's an example WIT and the current jco transpile output: https://gist.github.com/encounter/41cf4793ac1ff6e5e62f3077830f5cbe
import { diff, types } from 'objdiff-wasm';
export function runDiff(...) {
// ERROR: DiffResult is not exported on namespace ObjdiffCoreTypes
const result: types.DiffResult = diff.runDiff(...);
}
Initially, I tried working around this by adding export type { DiffResult }; in the generated TypeScript namespace. However, it still does not work: export const types: typeof ObjdiffCoreTypes; A variable can't have associated types, so the only things that get exported are the actual functions.
I tried using guest-types, but it leaves the root .d.ts broken. It tries to import the namespaces, despite them not existing anymore.
I settled on a new approach that builds on top of guest-types and turns it into a more fully-fledged solution. (Full output here: https://gist.github.com/encounter/d6d2c946e305b1949f0e3a6fe634e209)
import * as ObjdiffCoreDiff from './interfaces/objdiff-core-diff.js';
export { ObjdiffCoreDiff as diff };
Instead of using namespaces, we can import * the interface, and then export it from the root. This preserves everything: all functions and types are accessible.
declare module 'objdiff:core/diff' {
export { runDiff };
export type { ObjectData };
}
export function runDiff(left: ObjectData | undefined, right: ObjectData | undefined, config: DiffConfig): DiffResult;
import type { ObjectData } from './objdiff-core-types.js';
export type { ObjectData };
In the interface, we have the declare module from guest-types, but we've extended it even further. Functions are exported in both the interface itself and the declared module, so that the import * from the root contains them.
Additionally, we export type any types in the module. Before, only functions and classes were exported.
This means that if you reference it via objdiff:core/diff or by import { diff } from '...', you get all of the functions and associated types.
I'd like to gather feedback. Right now I have it implemented with an --es-modules flag, and internally it replaces opts.guest with opts.es_modules. However, I believe this approach fully fixes the ergonomics of using the generated types, and I'd like to consider making it the default. I'm also happy to submit a PR with what I have so far for further discussion.
Guy Bedford (Feb 19 2025 at 22:42):Thanks for looking into this - a PR against guest-types would be amazing as that flag hasn't been fully developed yet. Are there any limitations to this approach for guest types?
Luke Street (Feb 19 2025 at 22:47):No limitations as far as I'm aware. You could consider this "fully developed" guest-types. But I developed it to fix my issues with the transpiled/non-guest types, so I think it could in theory replace both.
Guy Bedford (Feb 19 2025 at 22:55):guest types are under tested right now, but if it passes transpiled types that is great too
of course taking into account the difference between guest and host types (and specifically that guest types shouldn't type imports, but still does)
Luke Street (Feb 19 2025 at 23:05):Yeah, I guess I should clarify: I'm not sure the scope of what all guest-types needs to do. Currently, it only changes the TS namespaces to ES modules, and that's also what I wanted to do for the host types in this scenario, so it internally replaces the "guest" option with a more generic "es_modules" option.
I assume guest types will need other changes, like you describe, and therefore the guest option would have to be re-implemented separately from this.
So this is specifically targeting host types, but it fully overlaps with how guest types are currently implemented, making it necessary to refactor that. A bit confusing but hopefully that helps.
I'll submit a draft PR soon!
Guy Bedford (Feb 19 2025 at 23:05):can we not replace the TS namespaces with modules across both of host and guest without otherwise changing flags?
Luke Street (Feb 19 2025 at 23:08):Yes, and I personally think that's the best approach, since I'm not aware of any value in keeping the TS namespaces.
That would make the types and guest-types output exactly the same for now, and I assume guest-types would be further developed later.
Milan (Feb 19 2025 at 23:16):wasmcloud's getting started had a manual patch to generated types to switch to module declarations. Better out of the box support would be great! FYI @Victor Adossi
Milan (Feb 19 2025 at 23:27):edit: sorry for the noise, was confusing host and guest types, but getting rid of namespaces all around sounds great :grinning:
Guy Bedford (Feb 19 2025 at 23:28):great to hear - would be fantastic to see a PR along these lines
Luke Street (Feb 20 2025 at 00:35):Submitted: https://github.com/bytecodealliance/jco/pull/571
Guy Bedford (Feb 20 2025 at 00:39):thank you! will review soon
Victor Adossi (Feb 20 2025 at 07:45):Hey @Luke Street would you mind sharing how you're integrating with respect to tsconfig.json and your imports for the types? Curious to see what approach you're taking there given the multiple options.
One thing we've been trying to solve for is ambient TS modules, so that once people have the bindings folder included (via tsconfig.json), they can go straight to code like this:
import { runDiff } from "objdiff:core/diff";
To get this to work, we have to use Ambient Modules which only include the declare module "..." in the file without any other external imports/exports.
This isn't something we have working today, but where we want to get to for code generated for guests.
I bring this up because if I trim the "module augmentations" (which is how TS will view any file with a top level import/export next to a declare module ...), I get code that is less usable than what the guest-types would output (ignoring of course the missing actual definitions.
declare module 'objdiff:core/diff' {
export function runDiff(left: ObjectData | undefined, right: ObjectData | undefined, config: DiffConfig): DiffResult;
}
// import type { ObjectData } from './objdiff-core-types.js';
// export { ObjectData };
// import type { ObjectDiff } from './objdiff-core-types.js';
// export { ObjectDiff };
// import type { DiffConfig } from './objdiff-core-types.js';
// export { DiffConfig };
// import type { DiffResult } from './objdiff-core-types.js';
// export { DiffResult };
NEW
Obviously, there are lots of problems with just commenting out a bunch of relevant type imports/exports, but what I want to point out is that the new re-export methodology might not quite work as well as what we had before in the guest component case -- we get a little further away from the ideal state.
If I trim the declarations in a file like objdiff-core-diff.d.ts in the old guest-types code, w/ a top level
Separately, do you run into the following errors with your generated .d.ts files?
typescript [2666]: Exports and export assignments are not permitted in module augmentations
typescript [2484]: Export declaration conflicts with exported declaration of 'runDiff'.
tsc certainly runs just fine, but it is odd that these pop up.
Victor Adossi (Feb 20 2025 at 07:47):I've made a reproduction repository so it's a bit easier to visualize/hack on these:
https://github.com/vados-cosmonic/jco-typegen-repro
(builds of jco are not included, sorry -- just the output of jco types/jco guest-types)
Victor Adossi (Feb 20 2025 at 07:57):oof ran out of time to edit, so weird formatting remained, but hopefully it's clear.
Luke Street (Feb 20 2025 at 12:50):Hey Victor, thanks for the detailed info. My use case is host types, so I’m importing everything from the root module directly, not using the ambient module imports.
I was under the impression that the ambient modules already worked with the guest-types output, and that my changes would not affect that, since I’m simply adding more exports. Sounds like that assumption was wrong, though :smile:
Thinking about it: maybe this is the distinction we need to make for host and guest types:
declare module part entirely. These are non-ambient types for a JS module that we’re importing, so the declare module part is unnecessary (and confusing).declare module, all exports are inside of there. The goal being to omit real ambient modules. Maybe the root .d.ts /// <references the other interfaces.I just recently ran into an issue where I wanted type imports in an ambient .d.ts, and I was able to solve it using the import() syntax:
declare module 'foo' {
export type Bar = import('./bar').Bar;
}
So maybe this is the solution for generating proper guest types with ambient modules.
I’m happy to experiment with this idea. Let me know if this sounds right to you.
Victor Adossi (Feb 20 2025 at 15:14):I think that distinction makes a lot of sense to me. The key reason to use TS namespaces in the first place IIRC was the ability to split declarations across files, something I'm not sure we ended up doing at all.
One wrinkle we've run into is that people have had problems with even the syntax of declare module that I'm suggestion (and you're suggesting), this came up in another discussion (I can't tell if this is a tooling mismatch somehow or something else -- the allowed syntaxes seem to somewhat differ with what plain tsc will allow)
I definitely support using that syntax to get to ambient imports, if it works well I'd love to adopt it -- it would massively improve ergonomics for jco type generation.
Luke Street (Feb 20 2025 at 23:49):I have fully functional ambient module generation now:
/// <reference path="./objdiff-core-diff-types.d.ts" />
declare module 'objdiff:core/diff' {
export function runDiff(left: Object | undefined, right: Object | undefined, config: DiffConfig): DiffResult;
export type Object = import('objdiff:core/diff-types').Object;
export type DiffConfig = import('objdiff:core/diff-types').DiffConfig;
export type DiffResult = import('objdiff:core/diff-types').DiffResult;
}
It even works with the root .d.ts:
/// <reference path="./interfaces/objdiff-core-diff-types.d.ts" />
/// <reference path="./interfaces/objdiff-core-diff.d.ts" />
/// <reference path="./interfaces/objdiff-core-display-types.d.ts" />
/// <reference path="./interfaces/wasi-logging-logging.d.ts" />
declare module 'objdiff:core/api' {
export type Level = import('wasi:logging/logging@0.1.0-draft').Level;
export * as diffTypes from 'objdiff:core/diff-types';
export * as displayTypes from 'objdiff:core/display-types';
export * as diff from 'objdiff:core/diff';
export function init(level: Level): void;
export function version(): string;
}
Though I realized we're missing another piece, we need to generate ambient modules for imports and concrete modules for exports (and invert that for guest-types), rather than having it all generate one way or the other. More to do :smile:
Victor Adossi (Feb 21 2025 at 04:58):Wow fantastic, thanks for putting in the hard work here!
Yeah I think you're fully read in on the difference between host and guest types now :) it's subtle but I think there's enough there to keep the commands different but I'm not sure the average person can tell the difference w/ the current naming.
Victor Adossi (Feb 21 2025 at 04:59):I like that it's module native all the way down, this should transfer really well to the browser (w/ a competent bundler), you might end up fixing that other issue at the same time!
Victor Adossi (Feb 21 2025 at 05:02):Let me know if/when I should refresh your branch and I'll give it a test (I have your branch pulled locally of course) -- I don't know if my tsc setup is just generating more warnings. Given that the import() syntax is exactly what you have to use in declare module blocks (which I I didn't know before, thanks!) -- I get the feeling we're going to be warning free :)
Luke Street (Feb 21 2025 at 06:31):I thought I had a clever idea: always emit interfaces as ambient modules, and the only difference between host and guest types would be whether the root .d.ts is an ambient module or not. This would be quite elegant, all imports could use the ns:pkg/iface style.
It seems that causes issues when an interface is generated in multiple type files:
test/output/flavorful/interfaces/wasi-io-streams.d.ts(13,16): error TS2300: Duplicate identifier 'OutputStream'.
test/output/many-arguments/interfaces/wasi-io-streams.d.ts(13,16): error TS2300: Duplicate identifier 'OutputStream'.
(Thanks tests!)
I could see this becoming an issue if you import multiple packages that use a common WASI interface, and both have a wasi-io-streams.d.ts with declare module 'wasi:io/streams@0.2.0'. Unless there's a workaround here, I think means we can't use ambient modules in host types at all, given that packages may ship the types and thus conflict with each other.
Would this affect guest types too? My initial thought is no, it'd be reasonable to assume there's only one "set" of guest types that'd exist in a project, given that the purpose of the project would be implementing those types.
I'll definitely update my branch when I have tests passing again.
Victor Adossi (Feb 21 2025 at 06:57):Tests are the real MVPs :)
And yeah -- the multiple packages using common WASI interfaces is as you might expect a very common case.
One thing that I've also thought about in the past is changing the generation methodology to produce folders that have de-duped names as well. Feel free to try that, we're not tied to file-per-interface, IMO, and while there isn't any way to semantically/intelligently compare right now, we can use names to dedup effort I think for now...
Let me know if I'm not understanding the problem you ran into properly!
Victor Adossi (Feb 21 2025 at 07:00):Ahh let me think about that some more... I'm fairly certain it's impossible/unintended to have two completely different modules with the same package, name, interface and version that are both implemented w/ different WIT contents... You should be fine to dedup and go one level up and allow for some cross-component reuse
Luke Street (Feb 21 2025 at 15:42):That’d be nice if we could assume we controlled all the types in the project, but I’m imagining a situation where you import an npm package that has its own wasi types, if it uses a wasm component internally. I’m not aware of any way to dedupe interface types across package boundaries, but if someone has ideas, I’m all ears.
It’s fine, though, the alternative is using the normal relative imports (import … from './interfaces/wasi-io-streams.js') and not using declare module at all in host types. Just means we can’t unify host/guest types like I wanted to.
Claire Smith (Feb 26 2025 at 03:53):hey hey, got this wstd Rust example (sends a HTTP GET request to a URL and logs the result) compiled to WASM and JSified with jco transpile.
It gets about half way through the function before it hits this:
> GET / HTTP/1.1
< HTTP/1.1 200 OK
< content-type: text/html
< etag: "84238dfc8092e5d9c0dac8ef93371a07:1736799080.121134"
< last-modified: Mon, 13 Jan 2025 20:11:20 GMT
< cache-control: max-age=1365
< date: Wed, 26 Feb 2025 03:38:15 GMT
< alt-svc: h3=":443"; ma=93600,h3-29=":443"; ma=93600,quic=":443"; ma=93600; v="43"
< content-length: 1256
< connection: keep-alive
file:///mnt/c/Users/.../node_modules/@bytecodealliance/preview2-shim/lib/io/worker-io.js:323
{ src: src.#id, len }
^
TypeError: Cannot read private member #id from an object whose class did not declare it
at OutputStream.splice (file:///mnt/c/Users/.../node_modules/@bytecodealliance/preview2-shim/lib/io/worker-io.js:323:18)
at trampoline56 (file:///mnt/c/Users/.../comm/comm.js:2917:34)
at wit-component:shim.indirect-wasi:io/streams@0.2.3-[method]output-stream.splice (wasm://wasm/wit-component:shim-b13bbaba:wasm-function[16]:0x283)
at wasm://wasm/001e552a:wasm-function[184]:0x7504
at wasm://wasm/001e552a:wasm-function[281]:0x16dc8
at wasm://wasm/001e552a:wasm-function[279]:0x103e0
at wasm://wasm/001e552a:wasm-function[96]:0x3286
at wasm://wasm/001e552a:wasm-function[329]:0x24c61
at wasm://wasm/001e552a:wasm-function[61]:0x148e
at wit-component:adapter:wasi_snapshot_preview1.wasi:cli/run@0.2.3#run (wasm://wasm/wit-component:adapter:wasi_snapshot_preview1-2fd92fea:wasm-function[29]:0x701)
Node.js v22.5.1
just wondering if this is the right place to ask questions? :smile:
Claire Smith (Feb 26 2025 at 03:56):Claire Smith said:
hey hey, got this wstd Rust example (sends a HTTP GET request to a URL and logs the result) compiled to WASM and JSified with
jco transpile.It gets about half way through the function before it hits this:
> GET / HTTP/1.1 < HTTP/1.1 200 OK < content-type: text/html < etag: "84238dfc8092e5d9c0dac8ef93371a07:1736799080.121134" < last-modified: Mon, 13 Jan 2025 20:11:20 GMT < cache-control: max-age=1365 < date: Wed, 26 Feb 2025 03:38:15 GMT < alt-svc: h3=":443"; ma=93600,h3-29=":443"; ma=93600,quic=":443"; ma=93600; v="43" < content-length: 1256 < connection: keep-alive file:///mnt/c/Users/.../node_modules/@bytecodealliance/preview2-shim/lib/io/worker-io.js:323 { src: src.#id, len } ^ TypeError: Cannot read private member #id from an object whose class did not declare it at OutputStream.splice (file:///mnt/c/Users/.../node_modules/@bytecodealliance/preview2-shim/lib/io/worker-io.js:323:18) at trampoline56 (file:///mnt/c/Users/.../comm/comm.js:2917:34) at wit-component:shim.indirect-wasi:io/streams@0.2.3-[method]output-stream.splice (wasm://wasm/wit-component:shim-b13bbaba:wasm-function[16]:0x283) at wasm://wasm/001e552a:wasm-function[184]:0x7504 at wasm://wasm/001e552a:wasm-function[281]:0x16dc8 at wasm://wasm/001e552a:wasm-function[279]:0x103e0 at wasm://wasm/001e552a:wasm-function[96]:0x3286 at wasm://wasm/001e552a:wasm-function[329]:0x24c61 at wasm://wasm/001e552a:wasm-function[61]:0x148e at wit-component:adapter:wasi_snapshot_preview1.wasi:cli/run@0.2.3#run (wasm://wasm/wit-component:adapter:wasi_snapshot_preview1-2fd92fea:wasm-function[29]:0x701) Node.js v22.5.1just wondering if this is the right place to ask questions? :smile:
What topic should I go in .. I haven't used zulip before, sorry
edit: figuring it out.
Victor Adossi (Feb 26 2025 at 03:58):Hey Claire it absolutely is! This IMO is worth making an issue for @ https://github.com/bytecodealliance/jco :)
Victor Adossi (Feb 26 2025 at 03:59):That said I think the fix is pretty simple, so I've added a PR for it that you could try:
https://github.com/bytecodealliance/jco/pull/576
Claire Smith (Feb 26 2025 at 04:13):Added that change, similar iss-a-ma-ue on line 340 same file
return outputStream.#id;
^
TypeError: Cannot read private member #id from an object whose class did not declare it
image.png
:rolling_on_the_floor_laughing:
Victor Adossi (Feb 26 2025 at 04:41):Ah thanks for the update -- that's odd, let me dig in!
And thanks for filing the issue, I'll work against that :)
Victor Adossi (Feb 26 2025 at 05:00):Hey so good news, I got a fix in for the request, but some bad news, it looks like the wstd wasm is looking for a pollable that doesn't exist (and that it didn't create!)
Victor Adossi (Feb 26 2025 at 05:00):Looking into that, but I've pushed the fix for the member issues
Victor Adossi (Feb 26 2025 at 05:01):When you run with the latest code the HTTP request will happen but you will likely get a different error there
Victor Adossi (Feb 26 2025 at 05:09):Just a little peek at the madness:
creating poll: 2
searching for pollable ready [2], present? true
creating poll: 3
searching for pollable ready [3], present? true
searching for pollable ready [2], present? true
searching for pollable ready [3], present? true
searching for pollable ready [0], present? false
Of course that doesn't say much but the idea is that the http_client example makes some pollables, but then tries to check ready on pollable zero (which it never created)...
Debugging now, this might result in an upstream commit to wstd but otherwise I think the Jco side is working properly now :)
Claire Smith (Feb 26 2025 at 05:24):I do still have issues, but if it's wstd rather than jco causing the issue, why would wasmtime run -S http=y http_client.wasm https://example.com work with the same wasm I used to create the JS-wasm module?
Victor Adossi (Feb 26 2025 at 05:30):That's a good question! I'm not sure, the current theory is a premature drop but still tracing to figure out!
Claire Smith (Feb 26 2025 at 05:31):I gotta run, best of luck, will check back in later so if there's any progress, keen to see :)
Notification Bot (Feb 26 2025 at 09:44):5 messages were moved from this topic to #jco > Incomplete trailers support breaking wstd examples by Victor Adossi.
Last updated: Feb 27 2025 at 22:03 UTC