What is the most difficult pest you deal with in your garden?

Big green fat tomatoes worms. They eat and eat until they're 3" long and your plants look like twig trees. You have to check the plants every morning (best to look for them by lying down under the plant and looking up). I squash them! (this gives me a little too much pleasure) Wonder if they're good as fish bait?