Owen Ou (Apr 01 2020 at 20:58): Owen Ou (Apr 01 2020 at 20:58):@Taylor Thomas @Matt Fisher I'm playing with Krustlet in Kind by following https://github.com/deislabs/krustlet/blob/master/docs/howto/krustlet-on-kind.md. I can see the krustlet node with kubectl get node. I apply this manifest https://github.com/deislabs/krustlet/blob/master/demos/wasi/hello-world-rust/k8s.yaml but i couldn't see the log:
k logs hello-world-wasi-rust Error from server: Get https://172.17.0.1:3000/containerLogs/default/hello-world-wasi-rust/hello-world-wasi-rust: dial tcp 172.17.0.1:3000: connect: connection refused
Do you happen to know what went wrong? I'm on Mac.
Taylor Thomas (Apr 01 2020 at 21:00):That is looking like you may be listening on the wrong IP address. Did you double check that is the bridge address? @Matt Fisher just finished those today, so he probably has more tips
Owen Ou (Apr 01 2020 at 21:04):I ran krustlet-wasi --node-ip 172.17.0.1 --pfx-password password. And I verified the default gateway in the hypervisor is 172.17.0.1:
$ docker run --rm -it --privileged --pid=host walkerlee/nsenter -t 1 -m -u -i -n sh
/ # ip addr show docker0
5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:20:cf:45:96 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:20ff:fecf:4596/64 scope link
valid_lft forever preferred_lft forever
Perhaps I should use a different node-ip because of Docker for Mac
Owen Ou (Apr 01 2020 at 21:09):The error from the krustlet-wasi process
./krustlet-wasi --pfx-password password --node-ip 172.17.0.1 [2020-04-01T21:06:13Z ERROR kubelet::kubelet] Error handling event: OCI API error: manifest unknown on https://webassembly.azurecr.io/v2/hello-world-wasi-rust/manifests/v1.1.0
Taylor Thomas (Apr 01 2020 at 21:18):Oh so there is another error there too. Did you modify the k8s.yaml to have a different version tag? v1.1.0 doesn't exist
Taylor Thomas (Apr 01 2020 at 21:19):As for the other error, it may be because of docker for mac. @Matt Fisher did you do it on docker for mac?
Owen Ou (Apr 01 2020 at 21:24):@Taylor Thomas I got it to work! Thanks for the hints! Two (stupid) mistakes I made: 1) somehow the version tag was changed to v1.1.0, I reverted it back to v0.1.0 2) I don't provide the node-ip and let krustlet-wasi to figure it out itself: ./krustlet-wasi --pfx-password password
Owen Ou (Apr 01 2020 at 21:25):I'm creating a PR to add the notes for Docker for Mac bit for node-ip
Taylor Thomas (Apr 01 2020 at 21:26):Oh so it ended up figuring it out on its own?
Taylor Thomas (Apr 01 2020 at 21:26):Out of curiosity, what IP address did it end up using?
Owen Ou (Apr 01 2020 at 21:28):It's the host ip, from krustlet-wasi's help page:
--hostname <hostname> The hostname for this node, defaults to the hostname of this machine [env: KRUSTLET_HOSTNAME=]
kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kind-control-plane Ready master 3h34m v1.17.0 172.17.0.2 <none> Ubuntu 19.10 4.19.76-linuxkit containerd://1.3.2 oou-ltm.internal.salesforce.com Ready agent 4m47s v1.17.0 10.3.52.6 <none> <unknown> <unknown> mvp
Taylor Thomas (Apr 01 2020 at 21:28):Yeah, I know it will select one of the available interfaces, but I haven't had that work on the Mac very well. Glad it actually worked though
Owen Ou (Apr 01 2020 at 21:33):Out of curiosity, why is specifying 172.17.0.1 needed if krustlet can use the hostname of the machine? I guess it's a linux thing?
Taylor Thomas (Apr 01 2020 at 21:34):Because sometimes it can select the wrong interface, especially for minikube (which does use a bridge)
Taylor Thomas (Apr 01 2020 at 21:34):I have found that it works fine on VMs in the cloud
Matt Fisher (Apr 01 2020 at 21:47):It's also because you have Krustlet running on your host OS (macOS) and Kubernetes running in a Docker container. In order for the kubernetes API to make successful calls for kubectl logs, it needs to reach Krustlet. A Docker container cannot resolve the host IP address. 172.17.0.1 is the bridge IP where a container can talk to the host
Matt Fisher (Apr 01 2020 at 21:48):Same thing applies for the minikube guide: you need to use the bridge IP if you want applications in your VirtualBox VM to communicate with applications running on the host OS
Matt Fisher (Apr 01 2020 at 21:49):In most cases, Krustlet grabs the right IP address (the IP associated with the hostname), but with bridge IPs it's a little different
Matt Fisher (Apr 01 2020 at 21:49):If you ran Krustlet in the minikube VM or inside the KinD container, this setup wouldn't be necessary
Matt Fisher (Apr 01 2020 at 21:49):but that's a lot harder to setup for new users
Owen Ou (Apr 01 2020 at 21:50):Yeah, you're right. It doesn't seem krustlet can always find the right hostname. I turned off vpn and it errored out (unsure why krustlet can figure out the hostname on VPN but not outside of VPN):
./krustlet-wasi --pfx-password password thread 'main' panicked at 'unable to get default node IP address: failed to lookup address information: nodename nor servname provided, or not known', crates/kubelet/src/config.rs:92:13 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Owen Ou (Apr 01 2020 at 21:51):I have to ./krustlet-wasi --pfx-password password --node-ip 192.168.1.64. 192.168.1.64 is my en0 ip
Owen Ou (Apr 01 2020 at 21:51):I'm not familiar with bridge IP in Docker for Mac. Is the env0 ip the bridge IP?
Radu Matei (Apr 01 2020 at 21:53):Is there any scenario where you wouldn't be interested in getting the logs with kubectl though?
As this is the main reason for starting the web server, and so it introduces the networking requirement of the control plane accessing the Krustlet node IP..
Matt Fisher (Apr 01 2020 at 21:55):I'm not familiar with bridge IP in Docker for Mac. Is the env0 ip the bridge IP?
You know, I haven't tested with Docker for Mac. This was on Ubuntu, where Docker creates a docker0 network bridge. Try docker network list and see if anything comes up
Matt Fisher (Apr 01 2020 at 21:56):It might be different as Docker for Mac uses xhyve.
Owen Ou (Apr 01 2020 at 21:57):docker network list NETWORK ID NAME DRIVER SCOPE c7c426831aa0 bridge bridge local 9eacdd7d7b52 host host local aca588e59558 none null local
bridge:
docker network inspect c7c426831aa0
[
{
"Name": "bridge",
"Id": "c7c426831aa0646abf9f35f96f35cacca545319ca21d9b223c923383e2b521ab",
"Created": "2020-04-01T21:39:08.475587129Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.17.0.0/16",
"Gateway": "172.17.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"171beb96c33077025226a63765f1e8ed2024440760922c8b606b47fa66028f0b": {
"Name": "kind-control-plane",
"EndpointID": "88c20ca55f3c76b53ec3fec4b49bd26304139ade265e26ad57c43d9286d059ce",
"MacAddress": "02:42:ac:11:00:02",
"IPv4Address": "172.17.0.2/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
},
"Labels": {}
}
]
host
docker network inspect 9eacdd7d7b52
[
{
"Name": "host",
"Id": "9eacdd7d7b52e9585b9f6a124841cb66c754cd619839e5638df3ba4991490d97",
"Created": "2019-12-15T02:32:30.276617216Z",
"Scope": "local",
"Driver": "host",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": []
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {},
"Options": {},
"Labels": {}
}
]
Matt Fisher (Apr 01 2020 at 21:58):ah yeah, looks like your default gateway is 172.17.0.1 based on that output, so that should be your --node-ip
Owen Ou (Apr 01 2020 at 21:59):I did try 172.17.0.1. I got https://bytecodealliance.zulipchat.com/#narrow/stream/227959-wasmtime-on.20k8s/topic/Krustlet/near/192593255. If I use 192.168.1.64 (en0), things work :shrug:
Matt Fisher (Apr 01 2020 at 22:00):weird. That error message looks totally unrelated... That looks like the v0.1.0 vs v1.1.0 issue earlier
Matt Fisher (Apr 01 2020 at 22:01):it's also possible that xhyve understands the host network, in which case it should be able to address anything in the 192.168.0.0/16 namespace
Matt Fisher (Apr 01 2020 at 22:01):/shrug
Matt Fisher (Apr 01 2020 at 22:02):if it works, it works!
Matt Fisher (Apr 01 2020 at 22:03):You should try a kubectl logs once you have a WebAssembly module running. I find that you don't know if it'll actually work until you make a call to kubectl logs.
Owen Ou (Apr 01 2020 at 22:03):I just retest 172.17.0.1 with the right v0.1.0. Same error:
k logs hello-world-wasi-rust Error from server: Get https://172.17.0.1:3000/containerLogs/default/hello-world-wasi-rust/hello-world-wasi-rust: dial tcp 172.17.0.1:3000: connect: connection refused
Does kubectl log talk to the http server on krustlet-wasi on the host? If it does, en0 makes sense, instead of 172.17.0.1 which is docerk0 in the hypervisor (the VM).
Matt Fisher (Apr 01 2020 at 22:05):yes
Owen Ou (Apr 01 2020 at 22:07):I also tested 0.0.0.0:
k logs hello-world-wasi-rust Error from server: Get https://0.0.0.0:3000/containerLogs/default/hello-world-wasi-rust/hello-world-wasi-rust: dial tcp 0.0.0.0:3000: connect: connection refused
Looks like something special for en0 with Docker for Mac.
Taylor Thomas (Apr 01 2020 at 22:15):Docs for Docker for Mac say the docker0 interface is internal to the VM only
Owen Ou (Apr 01 2020 at 22:16):This may be related: https://github.com/moby/moby/issues/22753#issuecomment-234078631
Matt Fisher (Apr 01 2020 at 22:18):nice! If you want to make an amendment to the document under "step 2: determine your default gateway", that would be much appreciated
Matt Fisher (Apr 01 2020 at 22:20):I also found this: https://docs.docker.com/docker-for-mac/networking/#use-cases-and-workarounds
Owen Ou (Apr 01 2020 at 22:47):PTAL: https://github.com/deislabs/krustlet/pull/148.
Owen Ou (Apr 01 2020 at 22:59):Oh you're 1 min ahead of me: https://github.com/deislabs/krustlet/pull/147. Feel free to close mine :slight_smile:
Radu Matei (May 18 2020 at 22:28):https://twitter.com/_oftaylor/status/1262482967009476610
We just cut the 0.2 release of Krustlet :tada: Thanks to all those in the community who have helped out with this! https://github.com/deislabs/krustlet/releases/tag/v0.2.0 We also published the Kubelet logic as its own separate #rustlang crate so you can build your own provider in Rust! https://crates.io/crates/kubelet
- Taylor Thomas (@_oftaylor)Last updated: Nov 22 2024 at 16:03 UTC