After reading this so answer and looking at https://github.com/jstarks/npiperelay/ I at least managed to get one direction done using socat and putty's plink (I somehow had a bit of trouble with npiperelay):
socat UNIX-LISTEN:/tmp/openssh-ssh-agent,umask=066,fork EXEC:"PLINK.EXE -serial //./pipe/openssh-ssh-agent",pipes
I actually build an agent-bridge that works nicely to let keepassxc manage my ssh-keys on cygwin.
The other way around seems to be a bit more tricky without writing a dedicated program but possible, as its done by the keepass plugin . Please add to this answer or add more straight forward ways without running tons of processes or writing custom code.