The church was full, with the only surviving mother of one of the girls, Maxine McNair, sitting in the front row.
Holder called the girls' deaths "a seminal and tragic moment" in U.S. history and recalled gains that followed their killings like the Civil Rights Act and the Voting Rights Act.
Alluding to the Supreme Court decision this year that struck down a key part of the voting law, Holder said the struggle continues decades later.
"This a fight that we will continue," Holder said.
The dynamite bomb went off outside the church Sept. 15, 1963. Of the Klansmen convicted years later, one remains imprisoned. Two others died in prison.
Two young men, both black, were shot to death in Birmingham in the chaos that followed the bombing.
Birmingham was strictly segregated at the time of the bombing, which occurred as city schools were being racially integrated for the first time. The all-black 16th Street Baptist was a gathering spot for civil rights demonstrations for months before the blast.
The bombing became a powerful symbol of the depth of racial hatred in the South and helped build momentum for later laws, including the 1964 Civil Rights Act and the Voting Rights Act of 1965.
During the morning commemoration, an honor guard composed of black and whites officers and firefighters watched over ceremonies with mixed-race crowd, something that would have been unthinkable in Birmingham in 1963. That same year, white police officers and firefighters used dogs and water hoses on black demonstrators marching for equal rights.
President Barack Obama issued a statement noting that earlier this year the four girls were posthumously awarded the Congressional Gold Medal, one of the country's highest civilian honors.
"That horrific day in Birmingham, Alabama quickly became a defining moment for the Civil Rights Movement. It galvanized Americans all across the country to stand up for equality and broadened support for a movement that would eventually lead to the passage of the Civil Rights Act of 1964," Obama said.
Rev. Bernice King, a daughter of the late Rev. Martin Luther King Jr., noted the changed city in a prayer.
"We thank you father for the tremendous progress we have made in 50 years, that we can sit in the safe confines of this sanctuary being protected by the city of Birmingham when 50 years ago the city turned its eye and its ears away from us," she said.