The distinction is at the species level. Do you have problems killing insects or spiders in your house or apartment?
No, and I also eat animal meat, but again, the only reason for that which I'm aware of is due to my own biological bias and my impulse to adhere to status quo norms and comforts/desires/fears as well as my ability to remove myself from things that are dissimilar to me (whether it's morally right to or not). That isn't a compelling moral argument.
With insects, you may be able to argue the degree of consciousness and ability to think and feel, perhaps, but we would need to find some measurement that's actually consistent beyond "look, that's what we've always done and what feels natural to us-- it's never going to change!"
You can say "at a species level", but there needs to be a compelling reason for how that changes the moral equation and why rather than just accepting an arbitrary line that feels comfortable regarding how far removed we are from them (in the same way that murdering a stranger or a member of a different race or sex isn't more morally acceptable than murdering someone closer to you or that resembles you).
It may be fair to say "my comforts/instincts/feelings/desires/fears are more important to me than what's actually moral", but it's not fair to say "It must be moral because I'm comfortable with it." What feels good or bad can be very different from what's morally right or wrong. Otherwise, we wouldn't be able to look back in time and see "Oh wait, it turns out we were wrong about that all along and it was actually worse than we felt."