First way I know is being linked to from link collections. If there are large numbers of links to your site from such link lists, Google seems to think that you are attempting to game the system, and your site's position will suffer. It might be that this is just a property of their algorithm and that there are no "bad neighborhood" tags, but effectively your site will be punished if you try to submit to many such lists.
Eventually Google seems to forgive, though. Someone I know was left with the task of trying to restore a site into Google's listings after some excuse for a site optimizer had gotten their site delisted from Google. Of course there was nothing that could be done, as the links could not be removed, but as time passed the site eventually appeared in the listings again.
Second way to get slapped I discovered myself -- typosquatting. I made an experiment where I created a page which contained ten variations of the term "sweepstakes", with links to my own page (on another domain). Google's reaction was fast. Just three days passed, and all the subsites on the entire domain got completely delisted from Google. There were four subsites which had a fine ranking on Google for months, but had now completely disappeared from listings, so it was not just random fluctuation.
The page has been removed, and I'm waiting now if the badness tag I got was permanent or if I will be relisted again on the next indexing round. Moral of the story: don't be evil, and Google will like you.
