Every year, he would be on the causeway in Barrington, cheering on the racers.
He felt so passionately about the races that he built a boat out of plywood and got a driver so that he could be a part of the action, too.
"I like to call myself a land captain," jokes Penney.
"I never missed a race."
Each year, thousands of people would crowd the beaches to watch the excitement. Young and old alike would get hooked by racing fever.
Association president Becky Nickerson remembers a few years back when a three-year-old knew all the racers names and all the boats by sight as they whipped around the corner.
Penney can understand the feeling - he remembers going to his first race in the 1980s.
"I was young," he said. "I loved it."
Sadly, for the second year in a row, there are no boat races scheduled, much to the disappointment of fans like Penney.
"This is the home of the boat racing, right here in the causeway," said Penney.
Many reasons behind it
Nickerson said there are a lot of fans from all over the province who would come to the races and generations of followers in the community.
"Out of everyone, Clifford is the saddest," said Nickerson.
She said there were many reasons for the races not to continue. Expenses were high while volunteer power was low. The insurance costs for a season ran upwards of $15,000 and the boat costs were high for racers.
A group of racers would travel to races in other communities, including Yarmouth,
Spectators began to dwindle in numbers after bringing in various safety rules to keep racers and spectators safe, said Nickerson.
Despite the complications, the association was still ready to bring back the races for the 2017 season and held three meetings to register racers for the year.
Only three people signed up.
"If we can't get racers to bring in their boats and if you can't get people to come, then what is the point?" said Nickerson.
She, too, is sad not see the races continue. The racers, she said, became family as they spent each racing weekend together.
"That's the worst thing," she said. "We built a family and we miss each other."
The association isn't giving up hope for a future of boat racing, she said.
"The association is there for when racers are ready to come back," said Nickerson.