Learning To Be a Better Listener with KALM