MTR(mini-transaction)设计与实现

简介

首先来看MTR的定义:

An internal phase of InnoDB processing, when making changes at thephysicallevel to internal data structures duringDMLoperations. A mini-transaction (mtr) has no notion ofrollback; multiple mini-transactions can occur within a singletransaction. Mini-transactions write information to theredo logthat is used duringcrash recovery. A mini-transaction can also happen outside the context of a regular transaction, for example duringpurgeprocessing by background threads.

MTR主要的目的是为了保证数据的一致性(比如多个事务或者发生数据库异常时). 因此MTR一般来说都伴随着写Redo log,因为redolog就是为了在recover的时候能够正常恢复数据.

一般来说在一个MTR中会做两个事情.

  • 写redolog
  • 挂载脏页到flush list.

  • 源码分析

MTR的接口

一般使用逻辑如下:

Mtr start
Process something
Mtr commit

来看个例子，比如在btree中打印目录以及btree info.

1
2
3
4
5
6
7
8
9
10
mtr_start(&mtr);

root = btr_root_block_get(index, RW_SX_LATCH, &mtr);

btr_print_recursive(index, root, width, &heap, &offsets, &mtr);
if (heap) {
  mem_heap_free(heap);
}

mtr_commit(&mtr);

可以看到使用比较简单，和我们上面的描述一致。

因此我们来看对应的这两个接口. 这里要注意mtr_start不仅有正常的接口，还有一个syn和async的接口，mtr_start就是mtr_start_sync.也就是默认的mtr是同步的.而async的mtr只能做只读操作.

1
2
3
4
5
6
7
8
9
10
11
/** Start a mini-transaction. */
#define mtr_start(*m*) (m)->start()

*/** Start a synchronous mini-transaction */*
#define mtr_start_sync(*m*) (m)->start(true)

*/** Start an asynchronous read-only mini-transaction */*
#define mtr_start_ro(*m*) (m)->start(true, true)

*/** Commit a mini-transaction. */*
#define mtr_commit(*m*) (m)->commit()

这里的m就是 struct mtr_t ,这个结构就是mini-transaction在InnoDB中的抽象.

数据结构

接下来来分析mtr_t这个结构, 这个结构有一个内部的数据结构用来保存MTR的一些状态.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
struct Impl {
    */** memo stack for locks etc. */*
    mtr_buf_t m_memo;

    */** mini-transaction log */*
    mtr_buf_t m_log;

    */** true if mtr has made at least one buffer pool page dirty */*
    bool m_made_dirty;

    */** true if inside ibuf changes */*
    bool m_inside_ibuf;

    */** true if the mini-transaction modified buffer pool pages */*
    bool m_modifications;

    */** Count of how many page initial log records have been*
*written to the mtr log */*
    ib_uint32_t m_n_log_recs;

    */** specifies which operations should be logged; default*
*value MTR_LOG_ALL */*
    mtr_log_t m_log_mode;

    */** State of the transaction */*
    mtr_state_t m_state;

    */** Flush Observer */*
    FlushObserver *m_flush_observer;

#ifdef UNIV_DEBUG
    */** For checking corruption. */*
    ulint m_magic_n;
#endif */* UNIV_DEBUG */*

    */** Owning mini-transaction */*
    mtr_t *m_mtr;
  };

::注释写的比较简略，这里我们来详细看几个比较重要的字段::

首先是m_state,这个表示当前MTR的状态，主要有3个状态，分别是激活，提交中以及提交完毕.

1
2
3
4
5
6
enum mtr_state_t {
  MTR_STATE_INIT = 0,
  MTR_STATE_ACTIVE = 12231,
  MTR_STATE_COMMITTING = 56456,
  MTR_STATE_COMMITTED = 34676
};

然后是 m_log_mode,它主要是表示当前所需要记录的操作类型，可以看到一共分为4种操作类型.

  1. MTR_LOG_ALL 表示LOG所有的操作（包括写redolog以及加脏页到flush list)
  2. MTR_LOG_NONE 不记录任何操作.
  3. MTR_LOG_NO_REDO 不生成REDO log,可是会加脏页到flush list
  4. MTR_LOG_SHORT_INSERTS 这个也是不记录任何操作，纯粹只是使用了MTR的一些功能(只在copy page的使用).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
*/** Logging modes for a mini-transaction */*
enum mtr_log_t {
  */** Default mode: log all operations modifying disk-based data */*
  MTR_LOG_ALL = 21,

  */** Log no operations and dirty pages are not added to the flush list */*
  MTR_LOG_NONE = 22,

  */** Don't generate REDO log but add dirty pages to flush list */*
  MTR_LOG_NO_REDO = 23,

  */** Inserts are logged in a shorter form */*
  MTR_LOG_SHORT_INSERTS = 24
};

m_log 则是当前的MTR提交的log内容,后面我们回来分析m_log的格式.

然后我们来看mtr_t这个结构里面仅有的几个字段. 可以看到m_impl就是上面我们介绍的Impl,而m_commit_lsn表示在commit的时候(commit)的lsn, 这里还有一个比较关键的数据结构，那就是Command,这个数据结构主要是抽象了MTR的具体操作。也就是说对于Redo log的修改其实是在Command这个结构中执行的.

1
2
3
4
5
6
7
8
9
10
11
12
private:
 Impl m_impl;

 */** LSN at commit time */*
 lsn_t m_commit_lsn;

 */** true if it is synchronous mini-transaction */*
 bool m_sync;

 class Command;

 friend class Command;

mtr_t::start

接下来我们来看MTR的启动函数mtr_t::start.这个函数包含两个参数，第一个sync表示是否当前的mtr是同步，第二个是read_only,这个表示当前mtr 是否只读.默认情况下sync=true, read_only=false.

这里的初始化可以看到state会被初始化为MTR_STATE_ACTIVE,其他的参数都是初始化为默认值.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
void mtr_t::start(bool sync, bool read_only) {
  UNIV_MEM_INVALID(this, sizeof(*this));

  UNIV_MEM_INVALID(&m_impl, sizeof(m_impl));

  m_sync = sync;

  m_commit_lsn = 0;

  new (&m_impl.m_log) mtr_buf_t();
  new (&m_impl.m_memo) mtr_buf_t();

  m_impl.m_mtr = this;
  m_impl.m_log_mode = MTR_LOG_ALL;
  m_impl.m_inside_ibuf = false;
  m_impl.m_modifications = false;
  m_impl.m_made_dirty = false;
  m_impl.m_n_log_recs = 0;
  m_impl.m_state = MTR_STATE_ACTIVE;
  m_impl.m_flush_observer = NULL;

  ut_d(m_impl.m_magic_n = MTR_MAGIC_N);
}

Start完毕之后，就是提交修改了(commit).

mtr_t::commit

  1. 首先会设置m_state为COMMITTING状态
  2. 然后进行判断是否需要执行所做的修改.

这里可以看到只要满足下面两个条件之一就会去执行MTR.

  1. m_n_log_recs 大于0 也就是将要写入到mar log的页的个数.
  2. 当前mtr修改buffer pool pages并且不生成redolog操作.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
*/** Commit a mini-transaction. */*
void mtr_t::commit() {
  ut_ad(is_active());
  ut_ad(!is_inside_ibuf());
  ut_ad(m_impl.m_magic_n == MTR_MAGIC_N);
  m_impl.m_state = MTR_STATE_COMMITTING;

  Command cmd(this);

  if (m_impl.m_n_log_recs > 0 ||
      (m_impl.m_modifications && m_impl.m_log_mode == MTR_LOG_NO_REDO)) {
    ut_ad(!srv_read_only_mode || m_impl.m_log_mode == MTR_LOG_NO_REDO);

    cmd.execute();
  } else {
    cmd.release_all();
    cmd.release_resources();
  }
}

Command::execute

因此我们来看最终的执行方法 execute 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
void mtr_t::Command::execute() {
  ut_ad(m_impl->m_log_mode != MTR_LOG_NONE);

  ulint len;

#ifndef UNIV_HOTBACKUP
  len = prepare_write();

  if (len > 0) {
    mtr_write_log_t write_log;

    write_log.m_left_to_write = len;

    auto handle = log_buffer_reserve(*log_sys, len);

    write_log.m_handle = handle;
    write_log.m_lsn = handle.start_lsn;
    write_log.m_rec_group_start_lsn = handle.start_lsn;

    m_impl->m_log.for_each_block(write_log);

    ut_ad(write_log.m_left_to_write == 0);
    ut_ad(write_log.m_lsn == handle.end_lsn);

    log_wait_for_space_in_log_recent_closed(*log_sys, handle.start_lsn);

    DEBUG_SYNC_C("mtr_redo_before_add_dirty_blocks");

    add_dirty_blocks_to_flush_list(handle.start_lsn, handle.end_lsn);

    log_buffer_close(*log_sys, handle);

    m_impl->m_mtr->m_commit_lsn = handle.end_lsn;

  } else {
    DEBUG_SYNC_C("mtr_noredo_before_add_dirty_blocks");

    add_dirty_blocks_to_flush_list(0, 0);
  }
#endif */* !UNIV_HOTBACKUP */*

  release_all();
  release_resources();
}

函数不长，我们分段来看。

首先在excecute中会先进行写之前的操作，主要是进行一些校验以及最终返回将要写入的redolog长度,这个函数就是(prepare_write).

prepare_write

先来看几个变量.

  1. m_log.size() 这个返回当前m_log buffer的字节长度.
  2. m_n_log_recs 这个表示当前mtr将要写入的页的个数.

因此prepare_write这个函数就是根据m_n_log_recs来判断是否是多个record，从而来设置不同的标记.

  • 如果是单个record(n_recs == 1), 则设置到m_log的最高位为1.
  • 如果是多个record(n_recs > 1),则多写一个字节到record的最末.

最终只有多个redord才会更改len,不然默认就是m_log.size().

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
ulint mtr_t::Command::prepare_write() {
...............................................
  */* An ibuf merge could happen when loading page to apply log*
*records during recovery. During the ibuf merge mtr is used. */*

  ut_a(!recv_recovery_is_on() || !recv_no_ibuf_operations);

  ulint len = m_impl->m_log.size();
  ut_ad(len > 0);

  ulint n_recs = m_impl->m_n_log_recs;
  ut_ad(n_recs > 0);

  ut_ad(log_sys != nullptr);

  ut_ad(m_impl->m_n_log_recs == n_recs);

  */* This was not the first time of dirtying a*
*tablespace since the latest checkpoint. */*

  ut_ad(n_recs == m_impl->m_n_log_recs);

  if (n_recs <= 1) {
    ut_ad(n_recs == 1);

    */* Flag the single log record as the*
*only record in this mini-transaction. */*

    *m_impl->m_log.front()->begin() |= MLOG_SINGLE_REC_FLAG;

  } else {
    */* Because this mini-transaction comprises*
*multiple log records, append MLOG_MULTI_REC_END*
*at the end. */*

    mlog_catenate_ulint(&m_impl->m_log, MLOG_MULTI_REC_END, MLOG_1BYTE);
    ++len;
  }

  ut_ad(m_impl->m_log_mode == MTR_LOG_ALL);
  ut_ad(m_impl->m_log.size() == len);
  ut_ad(len > 0);

  return (len);
}

计算完毕之后，我们就进入真正的执行阶段了(Command. execute)，这里流程是这样子的:

  1. 首先需要构造一个mtr_write_log_t 结构.
    mtr_write_log_t write_log;

    1. 这个结构主要是将mtr中的内容写入到redo log 中.这里先来看他的字段.

    2. m_handle 这个主要是用来保存从redolog得到的一些字段

      1. lock_no 表示shared锁
      2. start_lsn表示当前mtr的起始lsn
      3. end_lsn表示当前mtr的结束lsn.
        1
        2
        3
        4
        5
        6
        7
        8
        9
        typedef size_t log_lock_no_t;

        struct Log_handle {
          log_lock_no_t lock_no;

          lsn_t start_lsn;

          lsn_t end_lsn;
        };

    3. m_lsn 起始lsn

    4. m_rec_group_start_lsn ?

    5. m_left_to_write 表示还需要写入到redolog的内容的长度，因此这个值默认就是len.

1
2
3
4
5
6
7
struct mtr_write_log_t {
..................................
  Log_handle m_handle;
  lsn_t m_lsn;
  lsn_t m_rec_group_start_lsn;
  ulint m_left_to_write;
};
  1. 分配log buf以及初始化write_log.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
void mtr_t::Command::execute() {
..........................
  if (len > 0) {
    mtr_write_log_t write_log;

    write_log.m_left_to_write = len;

    auto handle = log_buffer_reserve(*log_sys, len);

    write_log.m_handle = handle;
    write_log.m_lsn = handle.start_lsn;
    write_log.m_rec_group_start_lsn = handle.start_lsn;
.........................................
}

log_buffer_reserve

在执行mtr的时候会首先调用log_buffer_reserve在redolog中分配对应的buf长度. 这个函数主要是计算sn以及lsn ,而sn和lsn的区别是在于数据写入到redolog的时候，redolog是按照block来写的，而每一个block都会有header和footer，因此这里sn是写入者看到的lsn，而lsn则是在磁盘上的真正的lsn.
下面就是代码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
Log_handle log_buffer_reserve(log_t &log, size_t len) {
  Log_handle handle;

  handle.lock_no = log_buffer_s_lock_enter(log);
......................................
  srv_stats.log_write_requests.inc();

  ut_a(srv_shutdown_state <= SRV_SHUTDOWN_FLUSH_PHASE);
  ut_a(len > 0);

  */* Reserve space in sequence of data bytes: */*
  const sn_t start_sn = log.sn.fetch_add(len);

  */* Ensure that redo log has been initialized properly. */*
  ut_a(start_sn > 0);
......................................

  */* Headers in redo blocks are not calculated to sn values: */*
  const sn_t end_sn = start_sn + len;
...........................................
  */* Translate sn to lsn (which includes also headers in redo blocks): */*
  handle.start_lsn = log_translate_sn_to_lsn(start_sn);
  handle.end_lsn = log_translate_sn_to_lsn(end_sn);
...................................................

  return (handle);
}
  1. 然后就是从mtr的buffer中写入内容到redolog的buffer.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
void mtr_t::Command::execute() {
.....................
m_impl->m_log.for_each_block(write_log);
.......................
}

  template <typename Functor>
  bool for_each_block(Functor &functor) const {
    for (const block_t *block = UT_LIST_GET_FIRST(m_list); block != NULL;
         block = UT_LIST_GET_NEXT(m_node, block)) {
      if (!functor(block)) {
        return (false);
      }
    }

    return (true);
  }

从上面的代码我们可以看到写入最红柿会遍历m_log的buf，然后再调用write_log类的方法来真正写入block.因此我们需要再次回到mtr_write_log_t这个结构.

这里是通过重载()来实现函数调用的，参数block表示将要写入redolog的内容.这里看到一个循环勒啊些股

  1. 首先是调用 log_buffer_write来写入block到redolog.
  2. 更新m_left_to_write.
  3. 如果内容写完则 log_buffer_set_first_record_group ？
  4. 调用 log_buffer_write_completed完成buffer的写入
  5. 更新m_lsn,以便于下次使用.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
struct mtr_write_log_t {
  */** Append a block to the redo log buffer.*
*@return whether the appending should continue */*
  bool operator()(const mtr_buf_t::block_t *block) {
    lsn_t start_lsn;
    lsn_t end_lsn;

    ut_ad(block != nullptr);

    if (block->used() == 0) {
      return (true);
    }

    start_lsn = m_lsn;

    end_lsn = log_buffer_write(*log_sys, m_handle, block->begin(),
                               block->used(), start_lsn);

    ut_a(end_lsn % OS_FILE_LOG_BLOCK_SIZE <
         OS_FILE_LOG_BLOCK_SIZE - LOG_BLOCK_TRL_SIZE);

    m_left_to_write -= block->used();

    if (m_left_to_write == 0
        && m_rec_group_start_lsn / OS_FILE_LOG_BLOCK_SIZE !=
               end_lsn / OS_FILE_LOG_BLOCK_SIZE) {
      log_buffer_set_first_record_group(*log_sys, m_handle, end_lsn);
    }

    log_buffer_write_completed(*log_sys, m_handle, start_lsn, end_lsn);

    m_lsn = end_lsn;

    return (true);
  }
.........................
};

通过上面的代码，可以看到核心就是两个调用，一个是log_buffer_write, 一个是log_buffer_write_completed.

log_buffer_write

先来看log_buffer_write,这个函数主要是写入到redo log buffer(log->buf).

  1. 这里首先根据start_lsn(也就是前一次写入之后的lsn),来计算当前的redolog block的偏移(也就是上一次写入之后的可写位置).
  2. 然后得到当前的block剩余的大小
  3. 如果当前block可以写入在直接copy到当前的block
  4. 否则只copy部分(left)内容到当前block,然后再次进入循环再写入一个新的block.
  5. 最后则是返回最终写入完毕后的lsn.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
lsn_t log_buffer_write(log_t &log, const Log_handle &handle, const byte *str,
                       size_t str_len, lsn_t start_lsn) {
............................

  const lsn_t end_sn = log_translate_lsn_to_sn(start_lsn) + str_len;

  byte *buf_end = log.buf + log.buf_size;

  byte *ptr = log.buf + (start_lsn % log.buf_size);

  lsn_t lsn = start_lsn;

  while (true) {
    const auto offset = lsn % OS_FILE_LOG_BLOCK_SIZE;

    const auto left = OS_FILE_LOG_BLOCK_SIZE - LOG_BLOCK_TRL_SIZE - offset;

    ut_a(left > 0);
    ut_a(left < OS_FILE_LOG_BLOCK_SIZE);

    size_t len, lsn_diff;

    if (left > str_len) {

      len = str_len;

      lsn_diff = str_len;

    } else {

      len = left;

      lsn_diff = left + LOG_BLOCK_TRL_SIZE + LOG_BLOCK_HDR_SIZE;
    }
..............................
    std::memcpy(ptr, str, len);

    str_len -= len;
    str += len;
    lsn += lsn_diff;
    ptr += lsn_diff;

    if (ptr >= buf_end) {

      ptr -= log.buf_size;
    }

    if (lsn_diff > left) {
......................................
      log_block_set_first_rec_group(
          reinterpret_cast<byte *>(uintptr_t(ptr) &
                                   ~uintptr_t(LOG_BLOCK_HDR_SIZE)),
          0);

      if (str_len == 0) {

        break;
      }

    } else {
      break;
    }
  }

...............................
  return (lsn);
}

log_buffer_write_completed

然后就是log_buffer_write_completed函数，这个函数主要是用来更新recent_written字段，这个字段主要是用来track已经写入到log buffer的lsn(后续分析redolog的时候会详细分析).

逻辑很简单，就是判断是否recent_written是否还有空间，如果没有则等待，否则加入到recent_written.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
void log_buffer_write_completed(log_t &*log*, const Log_handle &*handle*,
                                lsn_t *start_lsn*, lsn_t *end_lsn*) {

  uint64_t wait_loops = 0;

  while (!log.recent_written.has_space(start_lsn)) {
    ++wait_loops;
    os_thread_sleep(20);
  }

  if (unlikely(wait_loops != 0)) {
    MONITOR_INC_VALUE(MONITOR_LOG_ON_RECENT_WRITTEN_WAIT_LOOPS, wait_loops);
  }
  std::atomic_thread_fence(std::memory_order_release);

  log.recent_written.add_link(start_lsn, end_lsn);
}

log_wait_for_space_in_log_recent_closed

然后我们来看log_wait_for_space_in_log_recent_closed，到达这里的话，则说明我们已经写完log buffer,然后等待加脏页到flush list,而在InnoDB中log.recent_closed用来track在flush list中的脏页，因此这里在加脏页之前需要判断是否link buf已满。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
void log_wait_for_space_in_log_recent_closed(log_t &*log*, lsn_t *lsn*) {
  ut_a(log_lsn_validate(lsn));

  ut_ad(lsn >= log_buffer_dirty_pages_added_up_to_lsn(log));

  uint64_t wait_loops = 0;

  while (!log.recent_closed.has_space(lsn)) {
    ++wait_loops;
    os_thread_sleep(20);
  }

  if (unlikely(wait_loops != 0)) {
    MONITOR_INC_VALUE(MONITOR_LOG_ON_RECENT_CLOSED_WAIT_LOOPS, wait_loops);
  }
}

add_dirty_blocks_to_flush_list

然后就是加脏页到flush list中.

1
2
3
4
5
6
7
8
9
void mtr_t::Command::add_dirty_blocks_to_flush_list(lsn_t *start_lsn*,
                                                    lsn_t *end_lsn*) {
  Add_dirty_blocks_to_flush_list add_to_flush(start_lsn, end_lsn,
                                              m_impl->m_flush_observer);

  Iterate<Add_dirty_blocks_to_flush_list> iterator(add_to_flush);

  m_impl->m_memo.for_each_block_in_reverse(iterator);
}

这里有几个点要注意的.

  • m_impl->m_memo
    • 这个里面保存了需要加入到flush list的block
    • 也就是说在使用mtr的时候，需要自己挂载block到这个数据结构
  • Add_dirty_blocks_to_flush_list
    • 核心是这个数据结构，所有的操作都在这个结构里面

m_memo.for_each_block_in_reverse比较简单，就是从尾部开始遍历，然后调用iterator的()，因此这里我们来看Add_dirty_blocks_to_flush_list的().

这里可以看到最终就是调用add_dirty_page_to_flush_list来吧对应的block加入到flush list,这里不详细分析这块，以后我们分析buffer pool相关代码的时候会再来看这块.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
bool operator()(mtr_memo_slot_t **slot*) const {
  if (slot->object != NULL) {
    if (slot->type == MTR_MEMO_PAGE_X_FIX ||
        slot->type == MTR_MEMO_PAGE_SX_FIX) {
      add_dirty_page_to_flush_list(slot);

    } else if (slot->type == MTR_MEMO_BUF_FIX) {
      buf_block_t *block;
      block = reinterpret_cast<buf_block_t *>(slot->object);
      if (block->made_dirty_with_no_latch) {
        add_dirty_page_to_flush_list(slot);
        block->made_dirty_with_no_latch = false;
      }
    }
  }

#MySQL/InnoDB/MTR