I wouldn't... it may not do awful things to the worms, but if you want to use the compost for edibles, it may do awful things to you! Probably a different story if you're not planning on using in a veggie garden.
Presumably your using a herbicide to kill weeds, so it probably won't kill animals.It may however retain active constituents which will adversely effect any plants to which you apply the compost. Herbicides containing only glyphosate are about the safest, but still risky. Consider other means of weed control. Mulching will control many weed species. Hand weeding is the best, safest method.