Rocco was born Alexander Federico Petricone, Jr. in Cambridge, Massachusetts but raised in Somerville, Massachusetts, the son of Mary (née Di Biase) and Alexander Sam Petricone. Known by the nickname "Bobo" as a young man, Rocco had connections to the Winter Hill Gang in the early 1960s (a mostly Irish gang, despite his Italian-American background). Howie Carr describes Alex as a "fringe member" of the Winter Hill Gang and the Patriarca crime family of which the Winter Hill Gang had close business relations with.
What would later be known as the Irish Gang War started on Labor Day weekend of 1961. The Somerville-based Winter Hill Gang and McLaughlin Gang before the incident had coexisted relatively peacefully together were partying together on Salisbury Beach. George McLaughlin, the youngest of the criminal gang leaders made some unwelcome drunken advances toward the girlfriend of Alex. When George McLaughlin refused to take "no" for an answer, he was severely beaten by Alex and others and dumped unconscious in front of the emergency ward entrance of a local hospital. Before the McLaughln-Winter Hill Gang was over, more than forty criminals would be murdered on the streets of Boston.
Petricone was a suspect in the October 1961 murder of Bernie McLaughlin, but was never charged. He then moved to California in 1962 and began using the name Alex Rocco, lost a considerable amount of weight, and got into Hollywood. He first worked as a bartender in Santa Monica, California and took acting lessons from actor Leonard Nimoy, a Boston native himself. Leonard was not impressed with Alex's heavy Boston accent and told him to take speech lessons. Alex followed through with Leonard's instructions and after ridding himself of the accent came back to study under Nimoy and character actor and teacher Jeff Corey.
After moving to Los Angeles, Rocco became a member of the Bahá'í Faith. He first married Sandra Elaine Garrett in 1964 who bore him two sons, actor Marc Rocco on June 7, 1965 and Lucien and one daughter named Jennifer. Sandra died in June 2002 of cancer and he remarried to Shannon Wilcox on October 15, 2005. 
The Winter Hill Gang did not know of Alex's acting career until the spring of 1972 when The Godfather was released. John Martorano, James Martorano, Stephen Flemmi, Vincent Flemmi and James J. Bulger all went to see the film and were said to be amused.
Rocco played the part of Moe Greene, a Las Vegas casino owner, in the film The Godfather. Greene's character represented the top Jewish mobster in the United States.
Rocco also played a gangster in the film The Friends of Eddie Coyle about the Boston Irish Mob, a part very close to his own life. Other notable movies in which Rocco has appeared include The Wedding Planner, as Salvatore and appeared uncredited in Smokin' Aces. He also played a small part in the Disney/Pixar film, A Bug's Life as Thorny. In the film That Thing You Do!, Rocco played the founder of Playtone Records.
He also has a recurring part in the long running animated series The Simpsons as the head of Itchy and Scratchy Studios, Roger Meyers, Jr.. In DVD commentaries, Rocco has expressed true gratitude to The Simpsons' staff for allowing him his first voiceover role. He has also taken a part on an episode of Family Guy (in the episode "Mind Over Murder", he had a memorable role as the masculine woman) and he is known for having played Charlie Polniaczek, Jo's father on The Facts of Life. From 1989 to 1990, Rocco was a regular on the television comedy series The Famous Teddy Z in which he played "Al Floss", a slick cheesy Hollywood talent agent and foil to Jon Cryer in the title role, an ex-mailroom clerk turned superagent. Rocco received an Emmy Award as Best Supporting Actor in a Comedy Series for this role.
In 2008, Alex Rocco starred in Audi's Super Bowl commercial for the R8 supercar. The commercial was inspired by one of the films Rocco was in: The Godfather. He portrays a rich man who finds the front fascia of his luxury car in his bed, a nod to the scene from the original movie in which Jack Woltz, a rich movie producer, finds the head of his prized racehorse in his bed.