I thought that too. Granted, it's not as if Gainsbourg said, "Well, he never sexually assaulted me so I think he's alright." That's not what she was saying - she made it clear that von Trier is 'difficult' - but it was all a bit awkward.
I don't think they were trying to dismiss Bjork's allegations, it's just that neither of them really wanted to talk about such a serious subject. Adam brought it up, of course, probably because he felt he had to, but, good interviewer though he is, he's not really equipped to deal with this stuff.
As you say, it's not really up to Adam Buxton and Charlotte Gainsbourg to weigh in on this. The latter is qualified to talk about her experience of working with von Trier, but I really don't know what else she could say under these circumstances.
I believe Bjork, of course, and I'm pretty sure Adam and Charlotte do too. However, that section of the podcast was really quite... ugh.
EDIT: Really, Gainsbourg should've said "Bjork's ordeal sounds horrendous and I hope cunts like von Trier are never allowed to work again." But Gainsbourg isn't me, so she didn't say that.