Can I choose which language a public form is displayed in?

Yes. Each form has a "Language" setting chosen from the platform languages; it is required ("Language is required"). The public form's labels are shown in that language when customers open it.