@Taylor Thomas @Matt Fisher I'm playing with Krustlet in Kind by following https://github.com/deislabs/krustlet/blob/master/docs/howto/krustlet-on-kind.md. I can see the krustlet node with kubectl get node
. I apply this manifest https://github.com/deislabs/krustlet/blob/master/demos/wasi/hello-world-rust/k8s.yaml but i couldn't see the log:
k logs hello-world-wasi-rust Error from server: Get https://172.17.0.1:3000/containerLogs/default/hello-world-wasi-rust/hello-world-wasi-rust: dial tcp 172.17.0.1:3000: connect: connection refused
Do you happen to know what went wrong? I'm on Mac.
That is looking like you may be listening on the wrong IP address. Did you double check that is the bridge address? @Matt Fisher just finished those today, so he probably has more tips
I ran krustlet-wasi --node-ip 172.17.0.1 --pfx-password password
. And I verified the default gateway in the hypervisor is 172.17.0.1:
$ docker run --rm -it --privileged --pid=host walkerlee/nsenter -t 1 -m -u -i -n sh / # ip addr show docker0 5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:20:cf:45:96 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever inet6 fe80::42:20ff:fecf:4596/64 scope link valid_lft forever preferred_lft forever
Perhaps I should use a different node-ip because of Docker for Mac
The error from the krustlet-wasi
process
./krustlet-wasi --pfx-password password --node-ip 172.17.0.1 [2020-04-01T21:06:13Z ERROR kubelet::kubelet] Error handling event: OCI API error: manifest unknown on https://webassembly.azurecr.io/v2/hello-world-wasi-rust/manifests/v1.1.0
Oh so there is another error there too. Did you modify the k8s.yaml to have a different version tag? v1.1.0 doesn't exist
As for the other error, it may be because of docker for mac. @Matt Fisher did you do it on docker for mac?
@Taylor Thomas I got it to work! Thanks for the hints! Two (stupid) mistakes I made: 1) somehow the version tag was changed to v1.1.0, I reverted it back to v0.1.0 2) I don't provide the node-ip and let krustlet-wasi
to figure it out itself: ./krustlet-wasi --pfx-password password
I'm creating a PR to add the notes for Docker for Mac bit for node-ip
Oh so it ended up figuring it out on its own?
Out of curiosity, what IP address did it end up using?
It's the host ip, from krustlet-wasi
's help page:
--hostname <hostname> The hostname for this node, defaults to the hostname of this machine [env: KRUSTLET_HOSTNAME=]
kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kind-control-plane Ready master 3h34m v1.17.0 172.17.0.2 <none> Ubuntu 19.10 4.19.76-linuxkit containerd://1.3.2 oou-ltm.internal.salesforce.com Ready agent 4m47s v1.17.0 10.3.52.6 <none> <unknown> <unknown> mvp
Yeah, I know it will select one of the available interfaces, but I haven't had that work on the Mac very well. Glad it actually worked though
Out of curiosity, why is specifying 172.17.0.1
needed if krustlet can use the hostname of the machine? I guess it's a linux thing?
Because sometimes it can select the wrong interface, especially for minikube (which does use a bridge)
I have found that it works fine on VMs in the cloud
It's also because you have Krustlet running on your host OS (macOS) and Kubernetes running in a Docker container. In order for the kubernetes API to make successful calls for kubectl logs
, it needs to reach Krustlet. A Docker container cannot resolve the host IP address. 172.17.0.1 is the bridge IP where a container can talk to the host
Same thing applies for the minikube guide: you need to use the bridge IP if you want applications in your VirtualBox VM to communicate with applications running on the host OS
In most cases, Krustlet grabs the right IP address (the IP associated with the hostname), but with bridge IPs it's a little different
If you ran Krustlet in the minikube VM or inside the KinD container, this setup wouldn't be necessary
but that's a lot harder to setup for new users
Yeah, you're right. It doesn't seem krustlet
can always find the right hostname. I turned off vpn and it errored out (unsure why krustlet can figure out the hostname on VPN but not outside of VPN):
./krustlet-wasi --pfx-password password thread 'main' panicked at 'unable to get default node IP address: failed to lookup address information: nodename nor servname provided, or not known', crates/kubelet/src/config.rs:92:13 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
I have to ./krustlet-wasi --pfx-password password --node-ip 192.168.1.64
. 192.168.1.64
is my en0 ip
I'm not familiar with bridge IP in Docker for Mac. Is the env0 ip the bridge IP?
Is there any scenario where you wouldn't be interested in getting the logs with kubectl
though?
As this is the main reason for starting the web server, and so it introduces the networking requirement of the control plane accessing the Krustlet node IP..
I'm not familiar with bridge IP in Docker for Mac. Is the env0 ip the bridge IP?
You know, I haven't tested with Docker for Mac. This was on Ubuntu, where Docker creates a docker0
network bridge. Try docker network list
and see if anything comes up
It might be different as Docker for Mac uses xhyve.
docker network list NETWORK ID NAME DRIVER SCOPE c7c426831aa0 bridge bridge local 9eacdd7d7b52 host host local aca588e59558 none null local
bridge:
docker network inspect c7c426831aa0 [ { "Name": "bridge", "Id": "c7c426831aa0646abf9f35f96f35cacca545319ca21d9b223c923383e2b521ab", "Created": "2020-04-01T21:39:08.475587129Z", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "172.17.0.0/16", "Gateway": "172.17.0.1" } ] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": { "171beb96c33077025226a63765f1e8ed2024440760922c8b606b47fa66028f0b": { "Name": "kind-control-plane", "EndpointID": "88c20ca55f3c76b53ec3fec4b49bd26304139ade265e26ad57c43d9286d059ce", "MacAddress": "02:42:ac:11:00:02", "IPv4Address": "172.17.0.2/16", "IPv6Address": "" } }, "Options": { "com.docker.network.bridge.default_bridge": "true", "com.docker.network.bridge.enable_icc": "true", "com.docker.network.bridge.enable_ip_masquerade": "true", "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0", "com.docker.network.bridge.name": "docker0", "com.docker.network.driver.mtu": "1500" }, "Labels": {} } ]
host
docker network inspect 9eacdd7d7b52 [ { "Name": "host", "Id": "9eacdd7d7b52e9585b9f6a124841cb66c754cd619839e5638df3ba4991490d97", "Created": "2019-12-15T02:32:30.276617216Z", "Scope": "local", "Driver": "host", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": {}, "Options": {}, "Labels": {} } ]
ah yeah, looks like your default gateway is 172.17.0.1 based on that output, so that should be your --node-ip
I did try 172.17.0.1. I got https://bytecodealliance.zulipchat.com/#narrow/stream/227959-wasmtime-on.20k8s/topic/Krustlet/near/192593255. If I use 192.168.1.64 (en0), things work :shrug:
weird. That error message looks totally unrelated... That looks like the v0.1.0 vs v1.1.0 issue earlier
it's also possible that xhyve understands the host network, in which case it should be able to address anything in the 192.168.0.0/16 namespace
/shrug
if it works, it works!
You should try a kubectl logs
once you have a WebAssembly module running. I find that you don't know if it'll actually work until you make a call to kubectl logs
.
I just retest 172.17.0.1 with the right v0.1.0. Same error:
k logs hello-world-wasi-rust Error from server: Get https://172.17.0.1:3000/containerLogs/default/hello-world-wasi-rust/hello-world-wasi-rust: dial tcp 172.17.0.1:3000: connect: connection refused
Does kubectl log
talk to the http server on krustlet-wasi
on the host? If it does, en0 makes sense, instead of 172.17.0.1 which is docerk0 in the hypervisor (the VM).
yes
I also tested 0.0.0.0:
k logs hello-world-wasi-rust Error from server: Get https://0.0.0.0:3000/containerLogs/default/hello-world-wasi-rust/hello-world-wasi-rust: dial tcp 0.0.0.0:3000: connect: connection refused
Looks like something special for en0 with Docker for Mac.
Docs for Docker for Mac say the docker0 interface is internal to the VM only
This may be related: https://github.com/moby/moby/issues/22753#issuecomment-234078631
nice! If you want to make an amendment to the document under "step 2: determine your default gateway", that would be much appreciated
I also found this: https://docs.docker.com/docker-for-mac/networking/#use-cases-and-workarounds
PTAL: https://github.com/deislabs/krustlet/pull/148.
Oh you're 1 min ahead of me: https://github.com/deislabs/krustlet/pull/147. Feel free to close mine :slight_smile:
https://twitter.com/_oftaylor/status/1262482967009476610
We just cut the 0.2 release of Krustlet :tada: Thanks to all those in the community who have helped out with this! https://github.com/deislabs/krustlet/releases/tag/v0.2.0 We also published the Kubelet logic as its own separate #rustlang crate so you can build your own provider in Rust! https://crates.io/crates/kubelet
- Taylor Thomas (@_oftaylor)Last updated: Dec 23 2024 at 13:07 UTC