Any clerical assistants who continue on the House pay rolls under the provisions of section 5324 of this title shall, while so continued, perform their duties under the direction of the Clerk of the House, and he is authorized and directed to remove from such pay rolls any such clerks who are not attending to the duties for which their services are continued.