I don't know if it's faster to do this in the CharFilter or not. It just seemed to align better with what you are doing.
Since your punctiation filter is basically splitting on non-alpha-numeric, instead of writing code you very well could have used PatternTokenizerFactory with pattern="[^\p{L}\p{N}]+". I love that in Solr you can do so much without writing code. (though ironically I love writing code).
Re: solr.PunctuationFilterFactory