October 7, 2025Learning to Prompt in Unknown Environments: A POMDP Framework with Compositional Actions for Large Language Models