Is there a length limit for custom instructions?

Yes. Instructions cannot be empty and cannot be longer than 3000 characters. The counter under the text box shows how much of the limit you have used; if you go over it you will see the error "Instructions cannot be longer than 3000 characters" and the instructions will not be saved.