Link does indeed happily sit in silence with Ilphyl as he weeds. But even if there's no words, there's still a kind of communication happening.

Like when Link finds one particularly interesting-looking weed that's got a funny little patten on it. Upon discovery, it immediately gets brought over for Ilphyl to look at.

See? It's got a face!