PINK LADY: Nicole Kidman stars as a woman playing passive-aggressive games with her family in Margot at the Wedding. Credit: Paramount Vantage

PINK LADY: Nicole Kidman stars as a woman playing passive-aggressive games with her family in Margot at the Wedding. Credit: Paramount Vantage

Hollywood's dream factories are in full swing pumping out feel-good holiday fare, but for every action there is an equal and opposite reaction. Ergo: Margot at the Wedding, a feel-bad comedy about family members doing terrible things to each other in the name of love.

This is the new movie from Noah Baumbach, the gifted writer-director whose The Squid and the Whale took a potential mega-downer (a family shattered by divorce) and transformed it into an uplifting outpouring of wit and eloquent insight. The Squid and the Whale was largely based on the filmmaker's personal experiences, as is his new film, although this time around there's a small but crucial difference. Real life and its attendant demons suddenly seem a little too close for comfort in Margot at the Wedding, and Baumbach has begun to look like an artist with an ax to grind.

This doesn't make Margot at the Wedding a bad film by any stretch, but it does make it a difficult one to sit through without squirming. Baumbach still crafts some of the sharpest dialogue around, but the filmmaker seems more interested in purging personal demons than in entertaining his audience, and Margot often steps over that fine line between pain-based comedy and pure pain.

For every mild chuckle the movie prompts, there is an equal and opposite rush of discomfort at some disastrous wrong turn or knee-jerk nastiness or even a brush with a sharp-toothed child scooped from some Freudian nightmare. And just when you think it can't get any more uncomfortable, Margot at the Wedding fills our eyes with the inglorious spectacle of Jack Black's butt.

Like all of Baumbach's films, Margot is an ensemble piece, but at the center of it all is the title character, played by Nicole Kidman as a more nuanced version of the emissary of evil she assayed in The Golden Compass. Margot is a smart, successful Manhattan writer and ostensibly the normal one in her immediate family, but when she returns home for the wedding of her estranged, ne'er-do-well sister, Pauline (Jennifer Jason Leigh), we quickly get the sense of someone with major issues, a woman about to implode. Kidman's character begins baring her fangs with passive-aggressive gusto at everyone in sight, then climbs up the tallest tree around and, in the movie's loudest-proudest metaphor, finds herself immediately and completely stuck.

Ultimately, though, everybody in Margot at the Wedding is up his or her own tree, with each character burdened by an unmanageable crisis or two, and everybody ruthlessly pushing everybody else's buttons. As with Baumbach's other films, most of the characters here are monumentally self-absorbed East Coast artists and intellectuals — Volvo-driving, white-wine-drinking neurotics who relentlessly poke and prod at one another, realizing their mistakes even as they perpetrate them, but powerless to stop their own aggressive, self-defeating behavior.

Nobody is particularly nice to his or her fellow humans here, but Kidman and Leigh (who really do look like they could share the same gene pool) are locked in a love-hate connection of sisterly intensity, getting under each other's skin with laser precision.

The director positions his angsty characters amid idyllic surroundings, at a rambling country home in the Hamptons, where the wedding unfolds more like a competition than a celebration. Baumbach has always seemed a decidedly Eurocentric filmmaker, what with his tendency toward Rohmer-like chatter and Bergmanesque mind games, but the auteur he calls to mind most here, curiously enough, is an American, albeit an American under European influence: Woody Allen.

Specifically, Baumbach reminds us of Woody during his initial transition from crafter of neurotic comedies to chronicler of contemporary angst, pure and anything but simple. Unfortunately, if The Squid and the Whale was Baumbach's Annie Hall, then Margot at the Wedding is very nearly his Interiors, filled with characters so essentially unlikable that it's hard to see the things that are genuinely interesting about them.

Even the character we could assume will lighten up the proceedings — Jack Black as Pauline's slacker fiancé, Malcolm — doesn't quite do the trick. Black seems like he's trying way too hard to pull off a Bill Murray here, a manic comedian distilling his shtick into deadpan Lost in Translation/Broken Flowers non sequiturs.

Playing against type, Black turns Malcolm into a timid cynic whose inarticulate comments frequently trail off into a mumbled "whatever." The movie wants us to find humor in Malcolm's painful awkwardness and in his complete lack of humor, but it's a joke that never really connects — much like the mustache that Black's character sports for much of the film and that he explains away, to the uncomfortable silence of anyone unfortunate enough to be listening, by noting, "It's meant to be funny."

The movie unfolds as a collection of odd little throwaway moments, though hard information does manifest itself from time to time, thrown at us in furious bursts. (Although, what with all the mind games, we're often challenged to read between the lines to discover what's true and what's not.)

It turns out that Pauline has known Malcolm for less than a year and may only be marrying him because she's pregnant; Margot increasingly emerges as a self-medicating control freak, running from a rocky marriage and sliding into an unhealthy affair with a former lover; then there's the mysterious third sister who may or may not exist ("Poor Becky," Kidman and Leigh murmur in unison) and the children destined to recycle the sins of their parents.

Meanwhile, the weird neighbors wander about in their skivvies injecting pig carcasses with hypodermic needles; someone appears to have kidnapped the family dog; and everyone talks and talks and talks, appearing for all the world to be honest to a fault, even as the secrets and lies compound.

When the characters aren't openly antagonizing each other, they're spying on one another and, by the end of the film, chaos is king and everyone is colliding like human bumper cars. Margot at the Wedding is clearly as ambitious a comedy of horrors as you'll find this holiday season, but its relentless unpleasantness becomes its own dead end.