Police allege the offences were committed when the man stayed at the childrens’ grandparents home.