So where is the problem?
Although there was a list of minor defects to be fixed at handover, they did not maintain constant contact with the builder. The builder did not fix all the defects immediately as they most likely had other jobs to attend to as well, but that does not absolve them of the responsibility. However, like most things in life, "if you don't ask, you don't get", so by not maintaining at least weekly correspondence of the status of the defects and using that same opportunity to raise any new defects may appear, getting defects fixed in a timely manner is unlikely.
What happens now?
I advised them that they should meet with the builder on site immediately and go through all the defects from oldest to newest, and where it is agreed that it was a true defect to the work (and not caused by occupation). They then should both agree to a program of rectification whereby after the works are completed, then the retention sum be released.
All this stress could have been avoided if there was constant communication with the builder.
So, surely you're not leaving your defects notice to your builder until last minute, right?