I think that 2. is wrong. If an unchecked exception is thrown, it is considered a System Exception, which result in transaction rollback. The only unchecked exceptions that do not cause rollback are the ones annotated as @ApplicationException(rollback=false).In this case, since you are asked only to fail the checkout and not rollback the transaction, you can do one of the following -
1. Do not call setRollbackOnly() and throw a checked application exception. For example, throw new InvalidCartItemException(), where InvalidCartItemException is a normal checked exception.
2. Do not call setRollbackOnly() and throw an unchecked application exception that is not marked with rollback=true.
So, I think it should be:
2. Do not call setRollbackOnly() and throw an unchecked application exception which is annotated as @ApplicationException with rollback=false.