1. Reasons for choosing RocketMQ

ActiveMQ, RabbitMQ, ZeroMQ, Kafka, RocketMQ Selection

2. Integration Thought

RocketMQ provides a transaction message review to view the official Demo

@SpringBootApplication
public class ProducerApplication implements CommandLineRunner {
    private static final String TX_PGROUP_NAME = "myTxProducerGroup";
    @Resource
    private RocketMQTemplate rocketMQTemplate;
    @Value("${demo.rocketmq.transTopic}")
    private String springTransTopic;
    
    public static void main(String[] args) {
        SpringApplication.run(ProducerApplication.class, args);
    }

    @Override
    public void run(String... args) throws Exception {
        // Send transactional messages
        testTransaction();
    }


    private void testTransaction() throws MessagingException {
        String[] tags = new String[]{"TagA", "TagB", "TagC", "TagD", "TagE"};
        for (int i = 0; i < 10; i++) {
            try {

                Message msg = MessageBuilder
                                        .withPayload("Hello RocketMQ " + i)
                                        .setHeader(RocketMQHeaders.TRANSACTION_ID, "KEY_" + i)
                                        .build();
                SendResult sendResult = rocketMQTemplate.sendMessageInTransaction(TX_PGROUP_NAME,
                                                                        springTransTopic + ":" + tags[i % tags.length],
                                                                        msg,
                                                                        null);
                System.out.printf("------ send Transactional msg body = %s , sendResult=%s %n",
                                    msg.getPayload(),
                                    sendResult.getSendStatus());

                Thread.sleep(10);
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }

    @RocketMQTransactionListener(txProducerGroup = TX_PGROUP_NAME)
    class TransactionListenerImpl implements RocketMQLocalTransactionListener {
        private AtomicInteger transactionIndex = new AtomicInteger(0);

        private ConcurrentHashMap<String, Integer> localTrans = new ConcurrentHashMap<>();

        @Override
        public RocketMQLocalTransactionState executeLocalTransaction(Message msg, Object arg) {
            String transId = (String)msg.getHeaders().get(RocketMQHeaders.TRANSACTION_ID);
            System.out.printf("#### executeLocalTransaction is executed, msgTransactionId=%s %n", transId);
            int value = transactionIndex.getAndIncrement();
            int status = value % 3;
            localTrans.put(transId, status);
            if (status == 0) {
                // Return local transaction with success(commit), in this case,
                // this message will not be checked in checkLocalTransaction()
                System.out.printf("    # COMMIT # Simulating msg %s related local transaction exec succeeded! ### %n", msg.getPayload());
                return RocketMQLocalTransactionState.COMMIT;
            }

            if (status == 1) {
                // Return local transaction with failure(rollback) , in this case,
                // this message will not be checked in checkLocalTransaction()
                System.out.printf("    # ROLLBACK # Simulating %s related local transaction exec failed! %n", msg.getPayload());
                return RocketMQLocalTransactionState.ROLLBACK;
            }

            System.out.printf("    # UNKNOW # Simulating %s related local transaction exec UNKNOWN! \n");
            return RocketMQLocalTransactionState.UNKNOWN;
        }

        @Override
        public RocketMQLocalTransactionState checkLocalTransaction(Message msg) {
            String transId = (String)msg.getHeaders().get(RocketMQHeaders.TRANSACTION_ID);
            RocketMQLocalTransactionState retState = RocketMQLocalTransactionState.COMMIT;
            Integer status = localTrans.get(transId);
            if (null != status) {
                switch (status) {
                    case 0:
                        retState = RocketMQLocalTransactionState.UNKNOWN;
                        break;
                    case 1:
                        retState = RocketMQLocalTransactionState.COMMIT;
                        break;
                    case 2:
                        retState = RocketMQLocalTransactionState.ROLLBACK;
                        break;
                }
            }
            System.out.printf("------ !!! checkLocalTransaction is executed once," +
                    " msgTransactionId=%s, TransactionState=%s status=%s %n",
                transId, retState, status);
            return retState;
        }
    }

}

You need to send a message in testTransaction(), then implement the executeLocalTransaction() method in the TransactionListenerImpl class to execute the entire local transaction, and then implement a transaction message review in checkLocalTransaction().

Looking at the source code, you can see that the testTransaction() method and executeLocalTransaction() are in the same thread, just wrapping the RocketMQTemplate.

3. Problems and Solutions

3.1 Several issues with transactional messages:

  1. Transactional messages sent by messages do not have a strict sequence of inquiries and local transactions. How to ensure that the transaction operation must have been completed at the time of the review.
  2. Transaction message callbacks use transaction_id queries to find where transaction_id is stored and to ensure that the business operations associated with transaction_id are executed successfully.
  3. How to isolate a transaction from a business by returning it to an inquiry to ensure that it does not intrude into the code.
  4. How do downstream consumers guarantee interface idempotency.
  5. How downstream consumers can improve idempotency query performance.
  6. How to isolate idempotent operations from the business and ensure that they do not intrude into the code.

3.2 Solutions

  1. Since there may be delays in the database or other business operations, it is not guaranteed that the business operation is completed at the time of the lookup. You can do multiple lookups, set the maximum number of lookups, and not discard MQ message persistence for manual recovery.
  2. Local message tables can be used to send messages that fall to the ground, while facets, inheritance, and so on can be used to isolate the messages from the business code, to ensure that the local message repository is not intruded, note that the local message repository and the local business repository must be guaranteed within the same transaction!
  3. Transaction message lookup can use the local message table at point 2 to judge the execution result of a local transaction based on the transaction_id query. As with point 2, there are some ways to isolate the transaction message lookup code from the business code so as not to intrude.
  4. Methods of idempotency:
    • Database Unique Constraints
    • State Machine CAS One-way Flow
    • Message Reduplication Table
  5. Before executing local business, first judge if the business id exists, then directly return to the success of consumption. After executing local business, consumer information can fall asynchronously into redis.Note: Local business and message idempotency operations need to be guaranteed to be in the same transaction, while redis fall outside of the transaction.
  6. A better solution would be database unique constraints + message de-resampling tables, set unique constraints on business IDS in message de-resampling tables, and isolate message-to-ground operations from local businesses to ensure no intrusion.
  7. Timely cleans up the local message table of history (message de-tables).