From: Alexei Starovoitov When a prog is rejected the log ends with the basic block that fails and the message. How the verifier got to that block can be told only then, and a reader wants it in front of the block: the way to the block printed last the block, the message printed first The block is not going to be verified again to print it again. The kernel has no copy of the text either: the log is in the buffer of the user. bpf_vlog_stash() reads back what was written since a position and moves the log back to it. bpf_vlog_unstash() appends that text to what was written in between. A buffer that is too small holds a part of the text: a rolling one has lost the start, a fixed one the end. There may be no buffer at all. What is not there is counted. So log_true_size, -ENOSPC and the bytes in the buffer are those of a log that was written in the final order, whatever the size of the buffer. The stash is never larger than the buffer, which may have 1G. When memory is short the allocation has to fail and not to call the OOM killer: __GFP_RETRY_MAYFAIL. The log is left as it is then. The text is not looked at. If another thread has changed it the caller gets these bytes back, cut at a zero byte. Signed-off-by: Alexei Starovoitov --- include/linux/bpf_verifier.h | 9 +++++++ kernel/bpf/log.c | 47 ++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 656ed55b96ec..15619edc8f03 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -1417,6 +1417,15 @@ __printf(2, 3) void bpf_log(struct bpf_verifier_log *log, int bpf_vlog_init(struct bpf_verifier_log *log, u32 log_level, char __user *log_buf, u32 log_size); void bpf_vlog_reset(struct bpf_verifier_log *log, u64 new_pos); + +/* The end of the log, taken out of it by bpf_vlog_stash() */ +struct bpf_vlog_stash { + char *buf; + u64 len, lost_start, lost_end; +}; + +int bpf_vlog_stash(struct bpf_verifier_log *log, u64 pos, struct bpf_vlog_stash *stash); +void bpf_vlog_unstash(struct bpf_verifier_log *log, struct bpf_vlog_stash *stash); int bpf_vlog_finalize(struct bpf_verifier_log *log, u32 *log_size_actual); __printf(3, 4) void verbose_linfo(struct bpf_verifier_env *env, diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c index 1253118dfd05..84659a757b43 100644 --- a/kernel/bpf/log.c +++ b/kernel/bpf/log.c @@ -177,6 +177,53 @@ void bpf_vlog_reset(struct bpf_verifier_log *log, u64 new_pos) log->ubuf = NULL; } +/* + * Take what was written since @pos out of the log. bpf_vlog_unstash() appends it to what is + * written in between. The text is in the buffer of the user and nowhere else. What the buffer + * doesn't hold, the start of the text if it's rolling and the end if it's fixed, is counted. + */ +int bpf_vlog_stash(struct bpf_verifier_log *log, u64 pos, struct bpf_vlog_stash *stash) +{ + u64 from = max(pos, log->start_pos), to = log->end_pos, i; + u32 off, n; + + if (!log->ubuf) + from = to; + else if (log->level & BPF_LOG_FIXED) + to = clamp(log->len_total - 1ULL, from, to); + stash->lost_start = from - pos; + stash->lost_end = log->end_pos - to; + stash->len = to - from; + /* as large as the buffer at worst: fail before the OOM killer is called */ + stash->buf = kvzalloc(stash->len, GFP_KERNEL_ACCOUNT | __GFP_NOWARN | __GFP_RETRY_MAYFAIL); + if (!stash->buf) + return -ENOMEM; + /* two pieces if it wraps around */ + for (i = from; i < to && log->ubuf; i += n) { + div_u64_rem(i, log->len_total, &off); + n = min(to - i, log->len_total - off); + if (copy_from_user(stash->buf + (i - from), log->ubuf + off, n)) + log->ubuf = NULL; + } + bpf_vlog_reset(log, pos); + return 0; +} + +void bpf_vlog_unstash(struct bpf_verifier_log *log, struct bpf_vlog_stash *stash) +{ + u64 i, n; + + log->end_pos += stash->lost_start; + for (i = 0; i < stash->len; i += n) { + n = min(stash->len - i, sizeof(log->kbuf) - 1); + bpf_log(log, "%.*s", (int)n, stash->buf + i); + cond_resched(); + } + log->end_pos += stash->lost_end; + bpf_vlog_update_len_max(log, 1); + kvfree(stash->buf); +} + static void bpf_vlog_reverse_kbuf(char *buf, int len) { int i, j; -- 2.55.0