When in doubt I usually use the wizards and see what they do to the code in the source pane. In this case action is "openDocument" or "editDocument". If you select "Create document" as the default action, xp:this.action is not included.
However, if you add xp:this.action, then go to the All Properties panel and look at the options in the drop-down for data for action, the option is "newDocument", so that looks to be the answer for creating a document.
if (x==y) {return "newDocument" } else {return "openDocument"}
As I'm sure you're aware, for openDocument/editDocument you'll also need to include documentId parameter. For creating a document I expect that will be a blank string.