From ecab0ee65997e8823d47e5a3e7b7fe6e21ba299f Mon Sep 17 00:00:00 2001 From: yanggeorge Date: Mon, 22 Jul 2024 16:39:16 +0800 Subject: [PATCH] Site updated: 2024-07-22 16:39:15 --- en/about/index.html | 18 +- en/sitemap.xml | 8 +- search.xml | 1098 ++++++++++++++++++++--------------------- sitemap.xml | 72 +-- tags/CLion/index.html | 2 - 5 files changed, 598 insertions(+), 600 deletions(-) diff --git a/en/about/index.html b/en/about/index.html index 14d3bf79..7268a0ce 100644 --- a/en/about/index.html +++ b/en/about/index.html @@ -16,15 +16,15 @@ - + - + - - + + @@ -282,12 +282,12 @@
-

这是杨明的技术博客，以下是一些个人项目。

+

This is Ming’s technical blog. Here are some of the personal projects:

    -
  • bt-sniffer 是一个基于netty的收集BT种子元数据的嗅探器。项目相关链接如下：
    • -
  • mini-lisp 是一个精简的Lisp语言解释器，实现了闭包，宏等特性。
    • -
  • simple-socks5 是一个基于netty的socks5代理。
    • -
  • scraper 是一个网页数据采集的工具。
    • +
  • bt-sniffer is a BT seed metadata sniffer based on Netty.
    • +
  • mini-lisp is a minimal Lisp language interpreter with features such as closures and macros.
    • +
  • simple-socks5 is a Socks5 proxy based on Netty.
    • +
  • scraper is a web data collection tool.
diff --git a/en/sitemap.xml b/en/sitemap.xml index bbb29dd0..a34db96f 100644 --- a/en/sitemap.xml +++ b/en/sitemap.xml @@ -2,7 +2,7 @@ - https://threelambda.com/2024/07/18/2024-7-18-slang-ast-hier-tree/ + https://threelambda.com/about/index.html 2024-07-22 @@ -11,7 +11,7 @@ - https://threelambda.com/tags/index.html + https://threelambda.com/2024/07/18/2024-7-18-slang-ast-hier-tree/ 2024-07-22 @@ -20,7 +20,7 @@ - https://threelambda.com/about/2019-4-2-bt-7.html + https://threelambda.com/tags/index.html 2024-07-22 @@ -29,7 +29,7 @@ - https://threelambda.com/about/index.html + https://threelambda.com/about/2019-4-2-bt-7.html 2024-07-22 diff --git a/search.xml b/search.xml index 157e393c..17f4de6e 100644 --- a/search.xml +++ b/search.xml @@ -135,64 +135,6 @@ not unary) +, -, or * between the digits so they evaluate to the target value. python - - R和Python里得到传入参数的变量名 - /2016/12/21/get-val-name-and-value/ - 问题

同事写R程序的时候，问我能不能获取一个变量的name， -我说这个好办啊，在R里可以这样写，用quote()

-
> a <- 1
-> quote(a)
-a
-
-

不过我把问题想简单了，他实际需要的是要获得传入 -参数的name。例如定义一个函数foo(c), 给它传入参数a，我能够在函数 -内部知道传入参数的名字a。

- - -

我说可以增加一个参数嘛。把参数的名字直接传入。

-
> foo(a,"a")
-
-

但是我立刻意识到，这里存在重复，而写代码是提倡 -do not repeat yourself的。而且对R这种可以随意获取环境变量的高级语言。 -肯定有办法做到。

-

python可以做到

但是我还是先看看Python吧，毕竟最近用Python -多一些。Google关键字python print variable name and value,立刻可以获得 -答案。这段代码的思想也很直白，就是对输入的obj在环境变量（名称,对象）键值对 -中查找，返回名称。

-
def namestr(obj, namespace=globals()):
-    return [name for name in namespace if namespace[name] is obj]
-
-

这样可以这样使用

-
>>> a = 1
->>> namestr(a)
-['a']
-
-

R也可以

那么R肯定也有办法做到。Google关键字 -r print variable name and value

-
myfunc <- function(v1) {
-  deparse(substitute(v1))
-}
-
-myfunc(foo)
-[1] "foo"
-
-

真是很不错啊。把结果告诉了同事，他也很高兴。 -但是substitute()和deparse()究竟是啥意思呢？在R里输入?substitute，可以 -看到解释。

-
-

substitute returns the parse tree for -the (unevaluated) expression expr, substituting any variables bound in env.

-
-

够晦涩难懂的吧。不过仔细思考一下，也大概明白了。 -在R里，从语法角度来说，所有都是expr，（这跟其他语言都是statement不同） -在R的函数里，可以对传入的未进行求值的expr进行操作。这点非常有趣。 -而deparse()，比较好理解了至少在我看来，与as.character()的作用是一样的。

-]]> - - python - R - - 使用pandas.read_csv()读取csv文件 /2016/12/22/pandas-read-csv/ @@ -318,6 +260,98 @@ Return the array [2, 1, 1, 0].

总结

以上的解法是首先排序，再进行巧妙处理的一种方法。排序的 时间复杂度是O(N*logN)，接下来的处理也是O(N*logN)。

+]]> + + leetcode + 算法 + python + + + + R和Python里得到传入参数的变量名 + /2016/12/21/get-val-name-and-value/ + 问题

同事写R程序的时候，问我能不能获取一个变量的name， +我说这个好办啊，在R里可以这样写，用quote()

+
> a <- 1
+> quote(a)
+a
+
+

不过我把问题想简单了，他实际需要的是要获得传入 +参数的name。例如定义一个函数foo(c), 给它传入参数a，我能够在函数 +内部知道传入参数的名字a。

+ + +

我说可以增加一个参数嘛。把参数的名字直接传入。

+
> foo(a,"a")
+
+

但是我立刻意识到，这里存在重复，而写代码是提倡 +do not repeat yourself的。而且对R这种可以随意获取环境变量的高级语言。 +肯定有办法做到。

+

python可以做到

但是我还是先看看Python吧，毕竟最近用Python +多一些。Google关键字python print variable name and value,立刻可以获得 +答案。这段代码的思想也很直白，就是对输入的obj在环境变量（名称,对象）键值对 +中查找，返回名称。

+
def namestr(obj, namespace=globals()):
+    return [name for name in namespace if namespace[name] is obj]
+
+

这样可以这样使用

+
>>> a = 1
+>>> namestr(a)
+['a']
+
+

R也可以

那么R肯定也有办法做到。Google关键字 +r print variable name and value

+
myfunc <- function(v1) {
+  deparse(substitute(v1))
+}
+
+myfunc(foo)
+[1] "foo"
+
+

真是很不错啊。把结果告诉了同事，他也很高兴。 +但是substitute()和deparse()究竟是啥意思呢？在R里输入?substitute，可以 +看到解释。

+
+

substitute returns the parse tree for +the (unevaluated) expression expr, substituting any variables bound in env.

+
+

够晦涩难懂的吧。不过仔细思考一下，也大概明白了。 +在R里，从语法角度来说，所有都是expr，（这跟其他语言都是statement不同） +在R的函数里，可以对传入的未进行求值的expr进行操作。这点非常有趣。 +而deparse()，比较好理解了至少在我看来，与as.character()的作用是一样的。

+]]> + + python + R + + + + [leetcode 200]Number of Islands 原创解法 + /2016/09/27/2016-9-27-leetcode-200/ + 题目概述

原题链接

+
Given a 2d grid map of '1's (land) and '0's (water), count the number of islands. An island is surrounded by water and is formed by connecting adjacent lands horizontally or vertically. You may assume all four edges of the grid are all surrounded by water.
+
+Example 1:
+
+11110
+11010
+11000
+00000
+Answer: 1
+
+Example 2:
+
+11000
+11000
+00100
+00011
+Answer: 3
+
+ + +

最终解法

解法的代码先贴出来了，解题思路后补啊。总体来说，就是一行一行的扫描。 +是个不错的在线算法，也就是说如果数据量非常大也没关系，呵呵。

+
class Solution(object):
    def numIslands(self, grid):
        """
        基本思路，一行一行的扫描。
        :type grid: List[List[str]]
        :rtype: int
        """
        print("====")
        n = len(grid)
        if n == 0: return 0

        length = len(grid[0])
        h_pre = {}   # key : 是 “i,j” 字符串。value =》 [accumulator] ，一个list包含了岛的索引。
        h_curr = {}  # 当前行
        a = []  # 保存的是一个个的list，list的长度是1，值对应岛的索引。
        debug = []
        accumulator = 1 # 表示岛的自增索引，每当发现一个新的岛，自增。
        for i in range(0, length):
            if i == 0 and grid[0][i] == '1':
                h_curr['0,' + str(i)] = [accumulator]
                a.append(h_curr['0,' + str(i)])
                debug.append('0,' + str(i))
            elif i > 0 and grid[0][i] == '1':
                if grid[0][i - 1] == '1':
                    h_curr[str(0) + ',' + str(i)] = h_curr[str(0) + ',' + str(i - 1)]
                else:
                    accumulator += 1
                    h_curr['0,' + str(i)] = [accumulator]
                    a.append(h_curr['0,' + str(i)])
                    debug.append('0,' + str(i))

        for i in range(1, n):
            h_pre = h_curr
            h_curr = {}
            for j in range(0, length):
                if j == 0 and grid[i][j] == '1':

                    if grid[i - 1][j] == '1':
                        h_curr[str(i) + ',' + str(j)] = h_pre[str(i - 1) + ',' + str(j)]
                    else:
                        accumulator += 1
                        h_curr[str(i) + ',' + str(j)] = [accumulator]
                        a.append(h_curr[str(i) + ',' + str(j)])
                        debug.append(str(i) + ',' + str(j))

                elif j > 0 and grid[i][j] == '1':

                    if grid[i - 1][j] == '1' and grid[i][j - 1] == '1':
                        pre = h_curr[str(i) + ',' + str(j - 1)]
                        above = h_pre[str(i - 1) + ',' + str(j)]

                        if pre[0] == above[0]:
                            # 对pre的值进行更新
                            h_curr[str(i) + ',' + str(j)] = pre
                        else :
                            h_curr[str(i) + ',' + str(j)] = above
                            v1 = pre[0]
                            v2 = above[0]  # z这里一定要换成静态的值，否则的话当a[k]的值进行更新时，会影响到pre的值。
                            for k in range(0, len(a)):
                                if a[k][0] == v1:
                                    a[k][0] = v2


                    elif grid[i][j - 1] == '1' and grid[i - 1][j] != '1':
                        pre = h_curr[str(i) + ',' + str(j - 1)]
                        h_curr[str(i) + ',' + str(j)] = pre
                    elif grid[i][j - 1] != '1' and grid[i - 1][j] == '1':
                        above = h_pre[str(i - 1) + ',' + str(j)]
                        h_curr[str(i) + ',' + str(j)] = above
                    else:
                        accumulator += 1
                        h_curr[str(i) + ',' + str(j)] = [accumulator]
                        a.append(h_curr[str(i) + ',' + str(j)])
                        debug.append(str(i) + ',' + str(j))
        s = set()
        for item in a:
            s.add(item[0])
        return len(s)


if __name__ == "__main__":
    grid = ["11000",
            "11000",
            "00100",
            "00011"]  # 3
    print(Solution().numIslands(grid))
    grid = ["10111",
            "10101",
            "11101"]  # 1
    print(Solution().numIslands(grid))
    grid = ["1111111",
            "0000001",
            "1111101",
            "1000101",
            "1010101",
            "1011101",
            "1111111"]  # 1
    print(Solution().numIslands(grid))


    grid = ["10011101100000000000",
            "10011001000101010010",
            "00011110101100001010",
            "00011001000111001001",
            "00000001110000000000",
            "10000101011000000101",
            "00010001010101010101",
            "00010100110101101110",
            "00001001100001000101",
            "00100100000100100010",
            "10010000000100101010",
            "01000101011011101100",
            "11010000100000010001",
            "01001110001111101000",
            "00111000110001010000",
            "10010100001000101011",
            "10100000010001010000",
            "01100011101010111100",
            "01000011001010010011",
            "00000011110100011000"]  # 58
    print(Solution().numIslands(grid))
]]> leetcode @@ -385,40 +419,6 @@ The three ranges are : [0, 0], [2, 2], [0, 2] and their respective sums are: -2, 我们的代码速度会更快。

l.pop(pos) # <-- 这里是个接近O(N)的操作
-]]> - - leetcode - 算法 - python - - - - [leetcode 200]Number of Islands 原创解法 - /2016/09/27/2016-9-27-leetcode-200/ - 题目概述

原题链接

-
Given a 2d grid map of '1's (land) and '0's (water), count the number of islands. An island is surrounded by water and is formed by connecting adjacent lands horizontally or vertically. You may assume all four edges of the grid are all surrounded by water.
-
-Example 1:
-
-11110
-11010
-11000
-00000
-Answer: 1
-
-Example 2:
-
-11000
-11000
-00100
-00011
-Answer: 3
-
- - -

最终解法

解法的代码先贴出来了，解题思路后补啊。总体来说，就是一行一行的扫描。 -是个不错的在线算法，也就是说如果数据量非常大也没关系，呵呵。

-
class Solution(object):
    def numIslands(self, grid):
        """
        基本思路，一行一行的扫描。
        :type grid: List[List[str]]
        :rtype: int
        """
        print("====")
        n = len(grid)
        if n == 0: return 0

        length = len(grid[0])
        h_pre = {}   # key : 是 “i,j” 字符串。value =》 [accumulator] ，一个list包含了岛的索引。
        h_curr = {}  # 当前行
        a = []  # 保存的是一个个的list，list的长度是1，值对应岛的索引。
        debug = []
        accumulator = 1 # 表示岛的自增索引，每当发现一个新的岛，自增。
        for i in range(0, length):
            if i == 0 and grid[0][i] == '1':
                h_curr['0,' + str(i)] = [accumulator]
                a.append(h_curr['0,' + str(i)])
                debug.append('0,' + str(i))
            elif i > 0 and grid[0][i] == '1':
                if grid[0][i - 1] == '1':
                    h_curr[str(0) + ',' + str(i)] = h_curr[str(0) + ',' + str(i - 1)]
                else:
                    accumulator += 1
                    h_curr['0,' + str(i)] = [accumulator]
                    a.append(h_curr['0,' + str(i)])
                    debug.append('0,' + str(i))

        for i in range(1, n):
            h_pre = h_curr
            h_curr = {}
            for j in range(0, length):
                if j == 0 and grid[i][j] == '1':

                    if grid[i - 1][j] == '1':
                        h_curr[str(i) + ',' + str(j)] = h_pre[str(i - 1) + ',' + str(j)]
                    else:
                        accumulator += 1
                        h_curr[str(i) + ',' + str(j)] = [accumulator]
                        a.append(h_curr[str(i) + ',' + str(j)])
                        debug.append(str(i) + ',' + str(j))

                elif j > 0 and grid[i][j] == '1':

                    if grid[i - 1][j] == '1' and grid[i][j - 1] == '1':
                        pre = h_curr[str(i) + ',' + str(j - 1)]
                        above = h_pre[str(i - 1) + ',' + str(j)]

                        if pre[0] == above[0]:
                            # 对pre的值进行更新
                            h_curr[str(i) + ',' + str(j)] = pre
                        else :
                            h_curr[str(i) + ',' + str(j)] = above
                            v1 = pre[0]
                            v2 = above[0]  # z这里一定要换成静态的值，否则的话当a[k]的值进行更新时，会影响到pre的值。
                            for k in range(0, len(a)):
                                if a[k][0] == v1:
                                    a[k][0] = v2


                    elif grid[i][j - 1] == '1' and grid[i - 1][j] != '1':
                        pre = h_curr[str(i) + ',' + str(j - 1)]
                        h_curr[str(i) + ',' + str(j)] = pre
                    elif grid[i][j - 1] != '1' and grid[i - 1][j] == '1':
                        above = h_pre[str(i - 1) + ',' + str(j)]
                        h_curr[str(i) + ',' + str(j)] = above
                    else:
                        accumulator += 1
                        h_curr[str(i) + ',' + str(j)] = [accumulator]
                        a.append(h_curr[str(i) + ',' + str(j)])
                        debug.append(str(i) + ',' + str(j))
        s = set()
        for item in a:
            s.add(item[0])
        return len(s)


if __name__ == "__main__":
    grid = ["11000",
            "11000",
            "00100",
            "00011"]  # 3
    print(Solution().numIslands(grid))
    grid = ["10111",
            "10101",
            "11101"]  # 1
    print(Solution().numIslands(grid))
    grid = ["1111111",
            "0000001",
            "1111101",
            "1000101",
            "1010101",
            "1011101",
            "1111111"]  # 1
    print(Solution().numIslands(grid))


    grid = ["10011101100000000000",
            "10011001000101010010",
            "00011110101100001010",
            "00011001000111001001",
            "00000001110000000000",
            "10000101011000000101",
            "00010001010101010101",
            "00010100110101101110",
            "00001001100001000101",
            "00100100000100100010",
            "10010000000100101010",
            "01000101011011101100",
            "11010000100000010001",
            "01001110001111101000",
            "00111000110001010000",
            "10010100001000101011",
            "10100000010001010000",
            "01100011101010111100",
            "01000011001010010011",
            "00000011110100011000"]  # 58
    print(Solution().numIslands(grid))
]]> leetcode @@ -685,6 +685,52 @@ coins = 3*1*5 + 3*5*8 + 1*3*8 + 1*8*1 = 167 的min改成max。如果这两个问题是等价的，那么我们的解法一定可以通过测试。并且似乎也不用去证明了。（其实这是最难的部分。）

以下是通过的memoization解法，算法的复杂度是O(n^2)，与矩阵链问题的复杂度一样：

class Solution2(object):
    def maxCoins(self, nums):
        """
        :type nums: List[int]
        :rtype: int
        """
        n = len(nums)
        list = self.get(nums) # 转换为n+1个矩阵的表示。每个pair tuple对应矩阵的长和宽。
        print(list)
        n += 1
        m = [[None] * n for i in range(n)] # 声明一个子问题空间是n^2的记忆体
        print(m)
        for i in range(n):
            m[i][i] = 0

        return self.recursive(list, m, 0, n - 1)

    def get(self, nums):
        n = len(nums)
        list = [(1, nums[0])]
        for i in range(n):
            if (i + 1 < n):
                list.append((nums[i], nums[i + 1]))
            else:
                list.append((nums[i], 1))
        return list

    def recursive(self, list, m, i, j):
        if m[i][j] is not None:
            return m[i][j]
        else:
            max = -1
            for k in range(i, j):
                a = self.recursive(list, m, i, k)
                b = self.recursive(list, m, k + 1, j)
                v = a + b + list[i][0] * list[k][1] * list[j][1]
                if v > max :
                    max = v
            m[i][j] = max
            return m[i][j]
+]]> + + leetcode + 算法 + python + + + + [leetcode 329]Longest Increasing Path in a Matrix 原创解法 + /2017/03/12/2017-3-12-leetcode-329/ + 题目概述

原题链接

+
Given an integer matrix, find the length of the longest increasing path 
+From each cell, you can either move to four directions: left, right, up or down. You may NOT move diagonally or move outside of the boundary (i.e. wrap-around is not allowed).
+
+example:
+nums = [
+  [9,9,4],
+  [6,6,8],
+  [2,1,1]
+]
+return 4
+
+ + +

回溯法

首先想到的就是回溯法(backtracking)。思路也很简单，就是遍历每个位置， +并以此位置的数字作为开始，向四个方向进行回溯遍历。最终找到最长的序列。 +以下代码的36行和38行进行回溯。不过毫无疑问超时了。在这个基础上还可以 +进一步优化，比如只对in-degree为0的位置的数字进行考察。但是还是不行。

+
class Solution(object):
    def longestIncreasingPath(self, matrix):
        """
        回溯法解法。
        :type matrix: List[List[int]]
        :rtype: int
        """

        if matrix:
            hight = len(matrix)
            width = len(matrix[0])

            s = set((u, v) for u in range(hight) for v in range(width))

            result = [0]
            for item in s:
                p = [item]
                self.find(matrix, item, p, result, s)

            return result[0]

        else:
            return 0

    def find(self, matrix, item, p, result, s):
        if self.has_no_choice(matrix, item, p, s):
            if (len(p) > result[0]):
                result[0] = len(p)
        else:
            u, v = item
            four = [(u + 1, v), (u - 1, v), (u, v + 1), (u, v - 1)]
            for one in four:
                x, y = one
                if (x, y) in s and (x, y) not in p and matrix[x][y] < matrix[u][v]:
                    p.append((x, y))
                    self.find(matrix, (x, y), p, result, s)
                    p.pop()

    def has_no_choice(self, matrix, item, p, s):
        u, v = item
        four = [(u + 1, v), (u - 1, v), (u, v + 1), (u, v - 1)]

        for one in four:
            x, y = one
            if one in s and one not in p and matrix[x][y] < matrix[u][v]:
                return False

        return True
+ +

拓扑排序和记忆体

看看leetcode会给我们什么提示呢？点开show tags可以看到这个问题的标签， +Topological Sort和Memoization。既然使用topsort那么肯定要把这个问题 +转化为DAG问题了。如此一来，令人想到，可以先把矩阵数字使用topsort排成 +一排，然后再利用曾经遇到的求解最长递增子序列的动态规划算法进行求解。 +以下是实现的代码，拓扑排序加动态规划。

+
class Solution(object):
    def longestIncreasingPath(self, matrix):
        """
        :type matrix: List[List[int]]
        :rtype: int
        """

        if matrix:
            hight = len(matrix)
            width = len(matrix[0])

            s = set((u, v) for u in range(hight) for v in range(width))
            G = self.create_graph(matrix, s)
            topsort_list = self.topsort(G)
            result = self.find_longest_path(G, topsort_list)
            return result

        else:
            return 0

    def create_graph(self, matrix, s):
        G = {}
        for item in s:
            u, v = item
            G[item] = set()
            center = matrix[u][v]
            four = [(u + 1, v), (u - 1, v), (u, v + 1), (u, v - 1)]
            for one in four:
                if one in s:
                    x, y = one
                    adjacent = matrix[x][y]
                    if center > adjacent:
                        G[item].add(one)
        return G

    def tr(self, G):
        H = {}
        for u in G:
            H[u] = 0

        for u in G:
            for v in G[u]:
                H[v] += 1
        return H

    def topsort(self, G):

        H = self.tr(G)
        q = deque()
        sorted_list = []
        for u in H :
            if H[u] == 0 :
                q.append(u)
        while q:
            u = q.popleft()
            sorted_list.append(u)
            for v in G[u]:
                H[v] -= 1
                if H[v] == 0 :
                    q.append(v)

        return sorted_list

    def find_longest_path(self, G, topsort_list):
        n = len(topsort_list)

        memo = [1] * n
        for i in range(1, n):
            node = topsort_list[i]
            value = 0
            for pre in range(0, i):
                pre_node = topsort_list[pre]
                if node in G[pre_node]:
                    value = max(value, memo[pre] + 1)
                else:
                    value = max(value, memo[pre])
            memo[i] = value

        result = max(memo)
        #print(memo)
        return result
+ +

进一步优化

上边的解法还是超时了，时间不会消耗在topsort上，因为这个方法是O(|V|+|E|)的。 +主要的时间花在了动态规划上，因为这是一个O(N^2)的算法。 +我们仔细看看第72到76行这个内循环，发现遍历所有当前节点之前的节点是无效的。 +因为可以进入当前节点的节点不会超过４个。所以我们可以修改一下这个内循环。

+
def tr2(self, G):
    H = {}
    for u in G:
        H[u] = set()

    for u in G:
        for v in G[u]:
            H[v].add(u)
    return H

def find_longest_path(self, G, topsort_list):
    n = len(topsort_list)

    dic = { node : 1 for node in topsort_list }
    H = self.tr2(G)

    s = {topsort_list[0]}
    for i in range(1, n):
        node = topsort_list[i]
        s.add(node)
        value = 1
        for pre_node in H[node] :
            if pre_node in s :
                value = max(value, dic[pre_node] + 1)
        dic[node] = value

    result = max(dic.values())
    return result
+ +

这里我们在１６行使用一个dict来代替之前的memo， +这样就可以把一个O(N^2)改成一个O(N)的了。 +测试一下，幸运的通过了。不过只打败了1.7%。

]]> leetcode @@ -740,59 +786,13 @@ coins = 3*1*5 + 3*5*8 + 1*3*8 + 1*8*1 = 167 - [leetcode 329]Longest Increasing Path in a Matrix 原创解法 - /2017/03/12/2017-3-12-leetcode-329/ - 题目概述

原题链接

-
Given an integer matrix, find the length of the longest increasing path 
-From each cell, you can either move to four directions: left, right, up or down. You may NOT move diagonally or move outside of the boundary (i.e. wrap-around is not allowed).
-
-example:
-nums = [
-  [9,9,4],
-  [6,6,8],
-  [2,1,1]
-]
-return 4
-
- - -

回溯法

首先想到的就是回溯法(backtracking)。思路也很简单，就是遍历每个位置， -并以此位置的数字作为开始，向四个方向进行回溯遍历。最终找到最长的序列。 -以下代码的36行和38行进行回溯。不过毫无疑问超时了。在这个基础上还可以 -进一步优化，比如只对in-degree为0的位置的数字进行考察。但是还是不行。

-
class Solution(object):
    def longestIncreasingPath(self, matrix):
        """
        回溯法解法。
        :type matrix: List[List[int]]
        :rtype: int
        """

        if matrix:
            hight = len(matrix)
            width = len(matrix[0])

            s = set((u, v) for u in range(hight) for v in range(width))

            result = [0]
            for item in s:
                p = [item]
                self.find(matrix, item, p, result, s)

            return result[0]

        else:
            return 0

    def find(self, matrix, item, p, result, s):
        if self.has_no_choice(matrix, item, p, s):
            if (len(p) > result[0]):
                result[0] = len(p)
        else:
            u, v = item
            four = [(u + 1, v), (u - 1, v), (u, v + 1), (u, v - 1)]
            for one in four:
                x, y = one
                if (x, y) in s and (x, y) not in p and matrix[x][y] < matrix[u][v]:
                    p.append((x, y))
                    self.find(matrix, (x, y), p, result, s)
                    p.pop()

    def has_no_choice(self, matrix, item, p, s):
        u, v = item
        four = [(u + 1, v), (u - 1, v), (u, v + 1), (u, v - 1)]

        for one in four:
            x, y = one
            if one in s and one not in p and matrix[x][y] < matrix[u][v]:
                return False

        return True
- -

拓扑排序和记忆体

看看leetcode会给我们什么提示呢？点开show tags可以看到这个问题的标签， -Topological Sort和Memoization。既然使用topsort那么肯定要把这个问题 -转化为DAG问题了。如此一来，令人想到，可以先把矩阵数字使用topsort排成 -一排，然后再利用曾经遇到的求解最长递增子序列的动态规划算法进行求解。 -以下是实现的代码，拓扑排序加动态规划。

-
class Solution(object):
    def longestIncreasingPath(self, matrix):
        """
        :type matrix: List[List[int]]
        :rtype: int
        """

        if matrix:
            hight = len(matrix)
            width = len(matrix[0])

            s = set((u, v) for u in range(hight) for v in range(width))
            G = self.create_graph(matrix, s)
            topsort_list = self.topsort(G)
            result = self.find_longest_path(G, topsort_list)
            return result

        else:
            return 0

    def create_graph(self, matrix, s):
        G = {}
        for item in s:
            u, v = item
            G[item] = set()
            center = matrix[u][v]
            four = [(u + 1, v), (u - 1, v), (u, v + 1), (u, v - 1)]
            for one in four:
                if one in s:
                    x, y = one
                    adjacent = matrix[x][y]
                    if center > adjacent:
                        G[item].add(one)
        return G

    def tr(self, G):
        H = {}
        for u in G:
            H[u] = 0

        for u in G:
            for v in G[u]:
                H[v] += 1
        return H

    def topsort(self, G):

        H = self.tr(G)
        q = deque()
        sorted_list = []
        for u in H :
            if H[u] == 0 :
                q.append(u)
        while q:
            u = q.popleft()
            sorted_list.append(u)
            for v in G[u]:
                H[v] -= 1
                if H[v] == 0 :
                    q.append(v)

        return sorted_list

    def find_longest_path(self, G, topsort_list):
        n = len(topsort_list)

        memo = [1] * n
        for i in range(1, n):
            node = topsort_list[i]
            value = 0
            for pre in range(0, i):
                pre_node = topsort_list[pre]
                if node in G[pre_node]:
                    value = max(value, memo[pre] + 1)
                else:
                    value = max(value, memo[pre])
            memo[i] = value

        result = max(memo)
        #print(memo)
        return result
- -

进一步优化

上边的解法还是超时了，时间不会消耗在topsort上，因为这个方法是O(|V|+|E|)的。 -主要的时间花在了动态规划上，因为这是一个O(N^2)的算法。 -我们仔细看看第72到76行这个内循环，发现遍历所有当前节点之前的节点是无效的。 -因为可以进入当前节点的节点不会超过４个。所以我们可以修改一下这个内循环。

-
def tr2(self, G):
    H = {}
    for u in G:
        H[u] = set()

    for u in G:
        for v in G[u]:
            H[v].add(u)
    return H

def find_longest_path(self, G, topsort_list):
    n = len(topsort_list)

    dic = { node : 1 for node in topsort_list }
    H = self.tr2(G)

    s = {topsort_list[0]}
    for i in range(1, n):
        node = topsort_list[i]
        s.add(node)
        value = 1
        for pre_node in H[node] :
            if pre_node in s :
                value = max(value, dic[pre_node] + 1)
        dic[node] = value

    result = max(dic.values())
    return result
- -

这里我们在１６行使用一个dict来代替之前的memo， -这样就可以把一个O(N^2)改成一个O(N)的了。 -测试一下，幸运的通过了。不过只打败了1.7%。

-]]> - - leetcode - 算法 - python - - - - 编译和运行Clojure分支20081217源码 - /2017/03/23/2017-3-23-compile-and-run-clojure/ - Clojure branch 20081217

为啥要做这件事情呢？

-
    -
  1. Clojure是一门jvm语言，使用了asm。
    2. -
  2. branch 20081217是最初的版本便于研究。
    3. -
+ 编译和运行Clojure分支20081217源码 + /2017/03/23/2017-3-23-compile-and-run-clojure/ + Clojure branch 20081217

为啥要做这件事情呢？

+
    +
  1. Clojure是一门jvm语言，使用了asm。
    2. +
  2. branch 20081217是最初的版本便于研究。
    3. +

下载branch 20081217

本来直接这样一条语句很简单的事情，但是由于网络太慢。我只好 @@ -1415,82 +1415,6 @@ socket.timeout: timed out Process finished with exit code 0

可以看到，确实是110个1。

-]]> - - bitTorrent - protocol - p2p - - - - 使用delve调试K3s - /2019/11/19/debug-k3s/ - k3s是什么

K3s是什么？k8s的精简版。编译之后执行程序大小不到50M。 -可以用在物联网的边缘计算侧。如果想深入了解k8s，那么k3s是个很好的起点。 -那么如果能够断点调试k3s，就更好了。下面我们来看看怎么做。

- - -

步骤

    -
  • 建议使用linux 64位操作系统。这样可以native构建。
    • -
  • 建议性能好一些的机器，虚拟机编译会很慢
    • -
  • 建议安装好docker
    • -
  • 配置好go的环境，设置GOPATH，同时把$GOPATH/bin加入到PATH
    • -
  • 安装Goland这个集成开发环境
    • -
  • 从github克隆k3s的代码，加上depth参数则不下载历史，速度会快很多
    $ git clone --depth 1 https://github.com/rancher/k3s.git $GOPATH/src/github.com/rancher/k3s
    • -
  • 安装delve的debug工具。完成之后会生成可执行文件$GOPATH/bin/dlv
    $ git clone --depth 1 https://github.com/go-delve/delve.git $GOPATH/src/github.com/go-delve/delve
    $ cd $GOPATH/src/github.com/go-delve/delve
    $ make install
    • -
  • 构建含有调试信息的可执行文件k3s,所在路径是$GOPATH/src/github.com/rancher/k3s/
    $ cd $GOPATH/src/github.com/rancher/k3s
    $ go build -gcflags "all=-N -l" -o k3s
    • -
  • 用delve执行。这时候线程会监听2345的远程调试接入。
    $ cd $GOPATH/src/github.com/rancher/k3s
    $ dlv --listen=:2345 --headless=true --api-version=2 exec -- ./k3s server --docker --disable-agent
    • -
  • Goland中添加k3s项目。项目根路径为$GOPATH，然后配置增加一个remote调试。在运行之前，在main.go上打一个断点。
    • -
-

运行远程调试之后，成功。 -remote-debug

-]]> - - k3s - delve - golang - - - - BitTorrent协议（一）之解析种子文件 - /2019/01/09/2019-1-9-bt-1/ - bt种子文件

bt通过种子文件分享已经是一个过去时了，2009年btChina就已经关闭了。现在一般都是 -使用磁力链接来分享文件。那么为什么种子文件分享不再流行了呢？为什么要用磁力链接呢？ -磁力链接怎么实现的呢？

-

嗯这是这个系列要研究的问题。但是要研究磁力链接的实现原理，最好先从种子文件开始。

- - -

种子文件(metainfo files)的定义

官网文档 BEP3 中的metainfo files章节 -讲的很清楚。

-

简单的说就是把tracker列表和分享的文件信息编码为一个二进制的文件。

-
    -
  • tracker列表是类似如下的列表
    • -
-
http://tracker.trackerfix.com:80/announce

udp://9.rarbg.me:2710/announce

udp://9.rarbg.to:2710/announce
- -

如果想要找到下载源，就要通过tracker找到peer节点。

-
    -
  • 分享的文件信息（info）
    • -
-

包含了文件的大小，分块个数，分块的sha1散列值。

-

编码方式(bencoding)

bt文件的编码逻辑取名为bencoding。

-
-

Strings are length-prefixed base ten followed by a colon and the string. For example 4:spam corresponds to ‘spam’.

-

Integers are represented by an ‘i’ followed by the number in base 10 followed by an ‘e’. For example i3e corresponds to 3 and i-3e corresponds to -3. Integers have no size limitation. i-0e is invalid. All encodings with a leading zero, such as i03e, are invalid, other than i0e, which of course corresponds to 0.

-

Lists are encoded as an ‘l’ followed by their elements (also bencoded) followed by an ‘e’. For example l4:spam4:eggse corresponds to [‘spam’, ‘eggs’].

-

Dictionaries are encoded as a ‘d’ followed by a list of alternating keys and their corresponding values followed by an ‘e’. For example, d3:cow3:moo4:spam4:eggse corresponds to {‘cow’: ‘moo’, ‘spam’: ‘eggs’} and d4:spaml1:a1:bee corresponds to {‘spam’: [‘a’, ‘b’]}. Keys must be strings and appear in sorted order (sorted as raw strings, not alphanumerics).

-
-

翻译为eBNF语法呢，就是如下

-
string : num ':' {CHAR}*

num : 0 
    | [1-9][0-9]+ 
    | '-' [1-9][0-9]+

integer : 'i' num 'e'

list : 'l' {element}* 'e'

dic : 'd' {pair}* 'e'

pair : string element

element : string
        | integer
        | list
        | dic
- -

解码

根据eBNF实现的解码代码如下， 把get_content()方法中path替换为种子文件的路径，运行就可以看到。 -返回的解析结果中会有info_hash，该值是根据info的bencoding的二进制串计算的sha1值。这个值很重要 -因为之后很多协议都会用到。

-
# -*- coding: utf-8 -*-

__author__ = 'ym'
"""
    Date        : '2019/1/7'
    Description :  解析torrent文件
    
"""

from datetime import datetime


class BDecode(object):
    def __init__(self, arr):
        self.arr = arr
        self.n = len(arr)
        self.i = 0

    def parse(self):
        return self.dic()

    def peek(self):
        next = None
        if self.i < self.n:
            next = self.arr[self.i]
        return chr(next)

    def next(self):
        next = None
        if self.i < self.n:
            next = self.arr[self.i]
            self.i += 1
        return chr(next)

    def num(self):
        num_str = ""
        peek = self.peek()

        if peek is None:
            raise Exception("malformed num.")

        if peek == '-' and self.peek(1) in '123456789':
            self.next()  # ignore '-'
            while self.peek() in "0123456789":
                num_str += self.next()
            if num_str[0] in '0' and len(num_str) > 1:
                raise Exception("error : 0 starts with num")
            return -int(num_str)
        elif peek in '0123456789':
            while self.peek() in "0123456789":
                num_str += self.next()
            if num_str[0] in '0' and len(num_str) > 1:
                raise Exception("error : 0 starts with num")
            return int(num_str)
        else:
            raise Exception("malformed num.")

    def string(self, pieces=False):
        length = self.num()
        if self.next() != ':':
            raise Exception("String must contain colon")
        s = self.arr[self.i:(self.i + length)]
        self.i += length

        if not pieces:
            return s.decode("utf8")
        else:
            # pieces maps to a string whose length is a multiple of 20.
            # It is to be subdivided into strings of length 20,
            # each of which is the SHA1 hash of the piece at the corresponding index.
            result = []
            for j in range(0, length, 20):
                hash = s[j:j + 20]
                result.append(hash.hex().lower())
            return result

    def integer(self, timestamp=False):
        if self.next() != "i":
            raise Exception("Integer must begin with i")
        val = self.num()

        if timestamp:
            val = datetime.fromtimestamp(val).__str__()

        if self.next() != "e":
            raise Exception("Integer must end with e")
        return val

    def element(self, pieces=False, timestamp=False):
        peek = self.peek()
        if peek == 'i':
            return self.integer(timestamp)
        elif peek == "l":
            return self.list()
        elif peek == 'd':
            return self.dic()
        elif peek in "0123456789":
            return self.string(pieces)
        else:
            raise Exception("not recognize.")

    def list(self):
        if self.next() != "l":
            raise Exception("list must begin with l")
        result = []
        while self.peek() != 'e':
            result.append(self.element())
        self.next()
        return result

    def dic(self):
        if self.next() != 'd':
            raise Exception("dic must begin with d")
        result = dict()

        while self.peek() != "e":
            key = self.string()
            val = None
            if key == "pieces":
                val = self.element(pieces=True)
            elif key == 'creation date':
                val = self.element(timestamp=True)
            else:
                info_start = None
                info_end = None
                if key == 'info':
                    info_start = self.i
                val = self.element()
                if key == 'info':
                    info_end = self.i
                    result['info_hash'] = self.sha1(self.arr[info_start:info_end])
            result[key] = val

        self.next()
        return result

    def sha1(self, info):
        import hashlib
        p = hashlib.sha1()
        p.update(info)
        return p.hexdigest()


def get_content():
    path = "/Users/ym/tmp/venom.torrent"
    with open(path, "rb") as f:
        return f.read()


def main():
    content = get_content()
    result = BDecode(content).parse()
    import pprint
    pprint.pprint(result)

if __name__ == '__main__':
    main()
- -

运行结果

用最近的毒液电影的种子进行解析，打印如下， -其中announce-list就是tracker列表。

-
/usr/local/bin/python3.7 /Users/ym/charm/pytest/ym/bt/parse_torrent.py
{'announce': 'http://tracker.trackerfix.com:80/announce',
 'announce-list': [['http://tracker.trackerfix.com:80/announce'],
                   ['udp://9.rarbg.me:2710/announce'],
                   ['udp://9.rarbg.to:2710/announce']],
 'comment': 'Torrent downloaded from https://rarbg.to',
 'created by': 'mktorrent 1.0',
 'creation date': '2018-11-28 16:04:22',
 'info': {'files': [{'length': 100074, 'path': ['English.srt']},
                    {'length': 31, 'path': ['RARBG.txt']},
                    {'length': 4431023676,
                     'path': ['Venom.2018.720p.WEBRip.x264.AAC2.0-SHITBOX.mp4']}],
          'name': 'Venom.2018.720p.WEBRip.x264.AAC2.0-SHITBOX',
          'piece length': 1048576,
          'pieces': ['a958677e48a77aff63574c885d7fd70915159034',
                     '0c713356b63454a914452cdcc76a8470fb4bc419',
                     ...
                     ...
                     ...
                     'b926b048253bc506cb3f4e52acab9df6b93cf614',
                     '610f8485ab8c56f53f594e09730a34e8529e13b4']},
 'info_hash': '33297ac9c46f071506711f12814a3dd8ed8b73ed'}

Process finished with exit code 0

]]> bitTorrent @@ -2088,150 +2012,79 @@ Process finished with exit code 0 - docker容器内访问mac主机的kafka - /2020/03/09/connect-kafka/ - 从容器内访问主机的kafka

我最近遇到这样一个需求，需要从容器内的ClickHouse访问安装在mac主机的kafka。这个问题似乎很简单， -因为在windows上，虚拟机可以和host组成一个局域网，因此kafka只要绑定此网段的ip地址即可。 -但是在我的mac主机下，这个方案行不通。

+ BitTorrent协议（一）之解析种子文件 + /2019/01/09/2019-1-9-bt-1/ + bt种子文件

bt通过种子文件分享已经是一个过去时了，2009年btChina就已经关闭了。现在一般都是 +使用磁力链接来分享文件。那么为什么种子文件分享不再流行了呢？为什么要用磁力链接呢？ +磁力链接怎么实现的呢？

+

嗯这是这个系列要研究的问题。但是要研究磁力链接的实现原理，最好先从种子文件开始。

-

有几个原因，

-
    -
  1. ClickHouse无法直接安装在mac上，需要编译（当然8G内存也可以编译，但是ninja需要限制job数量，要花很长时间）。
    2. -
  2. 我的mac内存只有8G，使用vbox根本不可能，只能使用docker。
    3. -
  3. 有现成的镜像，直接可以启动。并且host上可以直接连接ClickHouse。
    4. -
-

难点就在于容器和host根本不在一个网段。

-

解决步骤

    -
  1. 修改主机上的kafka的相关配置如下
    2. -
-
listeners=PLAINTEXT://0.0.0.0:9092

advertised.listeners=PLAINTEXT://host.docker.internal:9092
+

种子文件(metainfo files)的定义

官网文档 BEP3 中的metainfo files章节 +讲的很清楚。

+

简单的说就是把tracker列表和分享的文件信息编码为一个二进制的文件。

+
    +
  • tracker列表是类似如下的列表
    • +
+
http://tracker.trackerfix.com:80/announce

udp://9.rarbg.me:2710/announce

udp://9.rarbg.to:2710/announce
-
    -
  1. /etc/hosts文件添加
    2. -
-
127.0.0.1       host.docker.internal
+

如果想要找到下载源，就要通过tracker找到peer节点。

+
    +
  • 分享的文件信息（info）
    • +
+

包含了文件的大小，分块个数，分块的sha1散列值。

+

编码方式(bencoding)

bt文件的编码逻辑取名为bencoding。

+
+

Strings are length-prefixed base ten followed by a colon and the string. For example 4:spam corresponds to ‘spam’.

+

Integers are represented by an ‘i’ followed by the number in base 10 followed by an ‘e’. For example i3e corresponds to 3 and i-3e corresponds to -3. Integers have no size limitation. i-0e is invalid. All encodings with a leading zero, such as i03e, are invalid, other than i0e, which of course corresponds to 0.

+

Lists are encoded as an ‘l’ followed by their elements (also bencoded) followed by an ‘e’. For example l4:spam4:eggse corresponds to [‘spam’, ‘eggs’].

+

Dictionaries are encoded as a ‘d’ followed by a list of alternating keys and their corresponding values followed by an ‘e’. For example, d3:cow3:moo4:spam4:eggse corresponds to {‘cow’: ‘moo’, ‘spam’: ‘eggs’} and d4:spaml1:a1:bee corresponds to {‘spam’: [‘a’, ‘b’]}. Keys must be strings and appear in sorted order (sorted as raw strings, not alphanumerics).

+
+

翻译为eBNF语法呢，就是如下

+
string : num ':' {CHAR}*

num : 0 
    | [1-9][0-9]+ 
    | '-' [1-9][0-9]+

integer : 'i' num 'e'

list : 'l' {element}* 'e'

dic : 'd' {pair}* 'e'

pair : string element

element : string
        | integer
        | list
        | dic
-
    -
  1. 在容器内，访问host.docker.internal:9092
    2. -
-

原理

docker desktop for mac会默认提供一个域名host.docker.internal给容器内 -的应用访问主机的服务。所以，如果主机上启动一个rest服务localhost:8080，则容器内可以通过 -host.docker.internal:8080直接访问。

-

但是如果kafka只是配置为

-
listeners=PLAINTEXT://localhost:9092
-

想直接访问kafka的broker则是不行的。我们会发现，在容器内，telnet host.docker.internal 9092是通的。这是为什么呢？

-

想要连接kafka的broker会进行基本的两步交互。

-
    -
  1. client访问broker的地址”A”，然后broker会发送回去一个advertised地址”B”
    2. -
  2. client接收到地址”B”，然后尝试访问”B”
    3. -
-

如果不设置advertised.listeners，那么默认等于listeners的值。

-

所以对于这样的设置 -B=A=PLAINTEXT://localhost:9092。容器内尝试访问”host.docker.internal:9092”，则会这样子，

-
    -
  1. 容器内访问host.docker.internal:9092，接收到地址B=PLAINTEXT://localhost:9092
    2. -
  2. 然后访问地址localhost:9092，失败。
    3. -
-

而如果在主机上访问则显然是成功的。

-

而我们的解决方法配置是这样的 -A=PLAINTEXT://localhost:9092, B=PLAINTEXT://host.docker.internal:9092

-

所以如果容器内访问的过程就是，

-
    -
  1. 容器内访问host.docker.internal:9092，接收到地址B=PLAINTEXT://host.docker.internal:9092
    2. -
  2. 然后访问host.docker.internal:9092，成功。
    3. -
-

而主机上的过程是，

-
    -
  1. 主机上访问localhost:9092,接收到地址B=PLAINTEXT://host.docker.internal:9092
    2. -
  2. 然后访问host.docker.internal:9092，根据/etc/hosts的域名解析为127.0.0.1:9092，成功。
    3. -
-

测试

我这里用kafkacat测试，实际上容器内的应用可以使用其他kafka客户端。

-

连接broker,并显示metadata

-
root@bb6b85562200:/# kafkacat -b host.docker.internal:9092 -L
Metadata for all topics (from broker 0: host.docker.internal:9092/0):
 1 brokers:
  broker 0 at host.docker.internal:9092
 8 topics:
  topic "test-topic" with 1 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0
  topic "flink-test" with 1 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0
  topic "__consumer_offsets" with 50 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0
    partition 10, leader 0, replicas: 0, isrs: 0
...
...
+

解码

根据eBNF实现的解码代码如下， 把get_content()方法中path替换为种子文件的路径，运行就可以看到。 +返回的解析结果中会有info_hash，该值是根据info的bencoding的二进制串计算的sha1值。这个值很重要 +因为之后很多协议都会用到。

+
# -*- coding: utf-8 -*-

__author__ = 'ym'
"""
    Date        : '2019/1/7'
    Description :  解析torrent文件
    
"""

from datetime import datetime


class BDecode(object):
    def __init__(self, arr):
        self.arr = arr
        self.n = len(arr)
        self.i = 0

    def parse(self):
        return self.dic()

    def peek(self):
        next = None
        if self.i < self.n:
            next = self.arr[self.i]
        return chr(next)

    def next(self):
        next = None
        if self.i < self.n:
            next = self.arr[self.i]
            self.i += 1
        return chr(next)

    def num(self):
        num_str = ""
        peek = self.peek()

        if peek is None:
            raise Exception("malformed num.")

        if peek == '-' and self.peek(1) in '123456789':
            self.next()  # ignore '-'
            while self.peek() in "0123456789":
                num_str += self.next()
            if num_str[0] in '0' and len(num_str) > 1:
                raise Exception("error : 0 starts with num")
            return -int(num_str)
        elif peek in '0123456789':
            while self.peek() in "0123456789":
                num_str += self.next()
            if num_str[0] in '0' and len(num_str) > 1:
                raise Exception("error : 0 starts with num")
            return int(num_str)
        else:
            raise Exception("malformed num.")

    def string(self, pieces=False):
        length = self.num()
        if self.next() != ':':
            raise Exception("String must contain colon")
        s = self.arr[self.i:(self.i + length)]
        self.i += length

        if not pieces:
            return s.decode("utf8")
        else:
            # pieces maps to a string whose length is a multiple of 20.
            # It is to be subdivided into strings of length 20,
            # each of which is the SHA1 hash of the piece at the corresponding index.
            result = []
            for j in range(0, length, 20):
                hash = s[j:j + 20]
                result.append(hash.hex().lower())
            return result

    def integer(self, timestamp=False):
        if self.next() != "i":
            raise Exception("Integer must begin with i")
        val = self.num()

        if timestamp:
            val = datetime.fromtimestamp(val).__str__()

        if self.next() != "e":
            raise Exception("Integer must end with e")
        return val

    def element(self, pieces=False, timestamp=False):
        peek = self.peek()
        if peek == 'i':
            return self.integer(timestamp)
        elif peek == "l":
            return self.list()
        elif peek == 'd':
            return self.dic()
        elif peek in "0123456789":
            return self.string(pieces)
        else:
            raise Exception("not recognize.")

    def list(self):
        if self.next() != "l":
            raise Exception("list must begin with l")
        result = []
        while self.peek() != 'e':
            result.append(self.element())
        self.next()
        return result

    def dic(self):
        if self.next() != 'd':
            raise Exception("dic must begin with d")
        result = dict()

        while self.peek() != "e":
            key = self.string()
            val = None
            if key == "pieces":
                val = self.element(pieces=True)
            elif key == 'creation date':
                val = self.element(timestamp=True)
            else:
                info_start = None
                info_end = None
                if key == 'info':
                    info_start = self.i
                val = self.element()
                if key == 'info':
                    info_end = self.i
                    result['info_hash'] = self.sha1(self.arr[info_start:info_end])
            result[key] = val

        self.next()
        return result

    def sha1(self, info):
        import hashlib
        p = hashlib.sha1()
        p.update(info)
        return p.hexdigest()


def get_content():
    path = "/Users/ym/tmp/venom.torrent"
    with open(path, "rb") as f:
        return f.read()


def main():
    content = get_content()
    result = BDecode(content).parse()
    import pprint
    pprint.pprint(result)

if __name__ == '__main__':
    main()
-

显示消息

-
^Croot@bb6b85562200:/# kafkacat -b host.docker.internal:9092 -C -t flink-test -o beginning
{seqNo: 1, eventTs: 1583739799986, id: even偶数, value: 4.57}
% Reached end of topic flink-test [0] at offset 1
+

运行结果

用最近的毒液电影的种子进行解析，打印如下， +其中announce-list就是tracker列表。

+
/usr/local/bin/python3.7 /Users/ym/charm/pytest/ym/bt/parse_torrent.py
{'announce': 'http://tracker.trackerfix.com:80/announce',
 'announce-list': [['http://tracker.trackerfix.com:80/announce'],
                   ['udp://9.rarbg.me:2710/announce'],
                   ['udp://9.rarbg.to:2710/announce']],
 'comment': 'Torrent downloaded from https://rarbg.to',
 'created by': 'mktorrent 1.0',
 'creation date': '2018-11-28 16:04:22',
 'info': {'files': [{'length': 100074, 'path': ['English.srt']},
                    {'length': 31, 'path': ['RARBG.txt']},
                    {'length': 4431023676,
                     'path': ['Venom.2018.720p.WEBRip.x264.AAC2.0-SHITBOX.mp4']}],
          'name': 'Venom.2018.720p.WEBRip.x264.AAC2.0-SHITBOX',
          'piece length': 1048576,
          'pieces': ['a958677e48a77aff63574c885d7fd70915159034',
                     '0c713356b63454a914452cdcc76a8470fb4bc419',
                     ...
                     ...
                     ...
                     'b926b048253bc506cb3f4e52acab9df6b93cf614',
                     '610f8485ab8c56f53f594e09730a34e8529e13b4']},
 'info_hash': '33297ac9c46f071506711f12814a3dd8ed8b73ed'}

Process finished with exit code 0

]]> - kafka - container - docker + bitTorrent + protocol + p2p - BitTorrent协议（六）之种子嗅探器 - /2019/02/18/2019-2-18-bt-6/ - Sniffer（嗅探器）

实现一个简单的BT种子嗅探器才算是有点实际价值的吧。这样就可以把种子的metadata信息 -缓存下来，提供按照文件名进行检索。

+ 使用delve调试K3s + /2019/11/19/debug-k3s/ + k3s是什么

K3s是什么？k8s的精简版。编译之后执行程序大小不到50M。 +可以用在物联网的边缘计算侧。如果想深入了解k8s，那么k3s是个很好的起点。 +那么如果能够断点调试k3s，就更好了。下面我们来看看怎么做。

-

不过这里只是实现一个demo，有兴趣的话可以看看github上的dht项目。

-

基本原理

简单的说，就是把嗅探节点加入到其他节点的路由表中，等待其他节点发来的announce_peer请求，然后获取种子的metadata信息。

-

我们根据以下的代码具体说明。

-
    -

  1. 89行代码，向路由节点发送find_node请求。这些路由节点。就是25行代码的3个地址。 -这样做的意义有两个，一方面，路由节点会把我们的节点加入到他们的路由表中。另一方面，路由节点会返回一个好节点列表。

    -
    2. -

  2. 252行代码，启动了6个线程。

    -
    3. -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
线程说明
listener代码93行，监听所有接收到的udp数据，并把这些放入到队列recv_q
t_dispatch代码106行，从队列recv_q中取出数据，并进行bdecoding，根据消息类别分别放入对应的队列。一共有三种：应答，请求，错误。
t_hand_reply代码132行，从reply_q中取出数据，解码节点列表。每一个节点都插入到本地的路由表（这里不太好，最好先进行ping_node，确定是好节点再加入），并向其发送find_node消息
t_hand_query代码165行，从query_q中取出请求，分别对四种请求进行应答处理，分别是ping，find_node, get_peers, announce_peer。
t_hand_error错误消息的处理。
t_handle_metadata代码234行，从metadata_q中取出已经获取的种子的metadata，打印并把种子保存再字典表中。
-
    -
  1. 代码165行，t_hand_query线程执行对请求进行处理。
    2. -
-
    -
  • ping: 返回本地节点id
    • -
  • find_node: 从本地路由表中查出与请求节点id最近的8个节点，并返回。（据说把本地id进行返回，有利于本地节点加入其他路由表）
    • -
  • get_peers: 这里进行了简单处理，仅仅返回空节点list。
    • -
  • announce_peer: 这个请求是有节点告诉我们有新的种子文件发布了，并告诉我们info_hash。然后我们拿着info_hash取进行获取metadata的操作。 -代码207行，开启一个独立的获取metadata的线程。如果获取到metadata，则放入metadata_q中。
    • +

    步骤

      +
    • 建议使用linux 64位操作系统。这样可以native构建。
      • +
    • 建议性能好一些的机器，虚拟机编译会很慢
      • +
    • 建议安装好docker
      • +
    • 配置好go的环境，设置GOPATH，同时把$GOPATH/bin加入到PATH
      • +
    • 安装Goland这个集成开发环境
      • +
    • 从github克隆k3s的代码，加上depth参数则不下载历史，速度会快很多
      $ git clone --depth 1 https://github.com/rancher/k3s.git $GOPATH/src/github.com/rancher/k3s
      • +
    • 安装delve的debug工具。完成之后会生成可执行文件$GOPATH/bin/dlv
      $ git clone --depth 1 https://github.com/go-delve/delve.git $GOPATH/src/github.com/go-delve/delve
      $ cd $GOPATH/src/github.com/go-delve/delve
      $ make install
      • +
    • 构建含有调试信息的可执行文件k3s,所在路径是$GOPATH/src/github.com/rancher/k3s/
      $ cd $GOPATH/src/github.com/rancher/k3s
      $ go build -gcflags "all=-N -l" -o k3s
      • +
    • 用delve执行。这时候线程会监听2345的远程调试接入。
      $ cd $GOPATH/src/github.com/rancher/k3s
      $ dlv --listen=:2345 --headless=true --api-version=2 exec -- ./k3s server --docker --disable-agent
      • +
    • Goland中添加k3s项目。项目根路径为$GOPATH，然后配置增加一个remote调试。在运行之前，在main.go上打一个断点。
    -
    # -*- coding: utf-8 -*-
    from __future__ import absolute_import
    from __future__ import division
    from __future__ import print_function

    __author__ = 'ym'
    """
        Date        : '2019/2/3'
        Description :
        
    """

    from ym.bt.routing_table import RoutingTable, BTNode
    from ym.bt.util import id_generator, ip_me, decode_compact_node, encode_compact_node, format_size
    from ym.bt.bencoding_bin import BEncode
    from ym.bt.bdecoding import BDecode
    from ym.bt.request_metadata import request_metadata_top
    from queue import Queue
    import socket
    import threading
    import logging

    log = logging.getLogger()

    NODES = [("router.bittorrent.com", 6881),
             ("router.utorrent.com", 6881),
             ("dht.transmissionbt.com", 6881)]

    ID = id_generator()
    SOURCE_IP = ip_me()
    SOURCE_PORT = 6881
    TIME_OUT = 1

    INFO_HASH_METADATA_DIC = {}
    RLOCK = threading.RLock()


    def boot_step(sock: socket.socket, local_node: BTNode):
        # send find_node query to NODES
        try:
            dic = {"t": "aa", "y": "q", "q": "find_node",
                   "a": {"id": local_node.id.hex(), "target": local_node.id.hex()}}
            data = BEncode(dic).encode_bin()

            for NODE in NODES:
                dst_ip, dst_port = NODE
                sock.sendto(data, (dst_ip, dst_port))
                print("boot_step|send|dst_ip={}|dst_port={}".format(dst_ip, dst_port))
        except Exception as e:
            log.exception(e)
            raise Exception("boot_step error")


    def request_metadata_thread(peer_id: bytes, info_hash, ip, port, metadata_q: Queue):
        try:
            print("request_metadata_thread|info_hash={}|ip={}|port={}".format(info_hash, ip, port))
            metadata_dic = request_metadata_top(peer_id, info_hash, ip, port)
            if metadata_dic is not None:
                metadata_q.put((metadata_dic, info_hash, ip, port))
        except Exception as e:
            log.exception("request_metadata_thread error")
            raise Exception("request metadata error")


    def main():
        sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        sock.setblocking(True)
        sock.bind((SOURCE_IP, SOURCE_PORT))

        # recv queue
        recv_q = Queue(maxsize=0)
        # reply queue
        reply_q = Queue(maxsize=0)
        # query queue
        query_q = Queue(maxsize=0)
        # error queue
        error_q = Queue(maxsize=0)
        # request metadata
        metadata_q = Queue(maxsize=0)

        # declare routing table
        rt = RoutingTable()

        local_node = BTNode(id=ID, ip=SOURCE_IP, port=SOURCE_PORT)
        print("local_node={}".format(local_node))
        rt.insert(local_node)
        # 1. boot step
        print("boot start.")
        boot_step(sock, local_node)
        print("boot end.")

        # 2. listener
        def recv_listener(sock: socket.socket, recv_q: Queue):
            while True:
                try:
                    data, addr = sock.recvfrom(2048)
                    # print("recv_listener|recv|addr={}".format(addr))
                    recv_q.put((data, addr))
                except Exception as e:
                    print("recv_listener error")

        listener = threading.Thread(target=recv_listener, args=(sock, recv_q,))
        listener.daemon = True

        # 3. handle recv msg
        def dispatch_recv(recv_q: Queue, reply_q: Queue, query_q: Queue, error_q: Queue):
            while True:
                data, addr = recv_q.get()
                try:
                    dic, i, n = BDecode(data).parse()
                    # print("dispatch_recv|dic={}".format(dic))
                    if dic.get("y") is not None:
                        v = dic['y']
                        if v == 'r':
                            # reply msg
                            reply_q.put((dic, addr))
                        elif v == 'q':
                            # query msg
                            query_q.put((dic, addr))
                        elif v == 'e':
                            # error msg
                            error_q.put((dic, addr))
                        else:
                            print("unknown type msg.")
                except Exception as e:
                    log.exception('dispatch_recv error|data={}'.format(data))

        t_dispatch = threading.Thread(target=dispatch_recv, args=(recv_q, reply_q, query_q, error_q,))
        t_dispatch.daemon = True

        # 4. handle reply
        def handle_reply(sock: socket.socket, reply_q: Queue, rt: RoutingTable):
            while True:
                dic, addr = reply_q.get()
                try:
                    r = dic['r']
                    src_id = r['id']
                    src_ip, src_port = addr
                    src_node = BTNode(id=src_id, ip=src_ip, port=src_port)
                    rt.insert(src_node)
                    print("rt.size={}".format(rt.size()))
                    # print("handle_reply|src_ip={}|src_port={}".format(src_ip, src_port))
                    if r.get('nodes') is not None:
                        # ping every node and put good in rt.
                        nodes = decode_compact_node(r['nodes'])
                        dic = {"t": "aa", "y": "q", "q": "find_node",
                               "a": {"id": local_node.id.hex(), "target": local_node.id.hex()}}
                        data = BEncode(dic).encode_bin()
                        for node in nodes:
                            try:
                                dst_ip, dst_port, id = node
                                sock.sendto(data, (dst_ip, dst_port))
                            except Exception as e:
                                log.exception("send node error|node={}".format(node))
                    elif r.get('peers') is not None:
                        # ping every peer and put good in rt.
                        print("handle_reply|peers")
                except Exception as e:
                    log.exception("handle_reply error|dic={}".format(dic), e)

        t_hand_reply = threading.Thread(target=handle_reply, args=(sock, reply_q, rt,))
        t_hand_reply.daemon = True

        # 5. handle query
        def handle_query(sock: socket.socket, query_q: Queue):
            while True:
                dic, addr = query_q.get()
                dst_ip, dst_port = addr
                try:
                    print("handle_query|addr={}|dic={}".format(addr, dic))
                    if dic['q'] == 'ping':
                        # handle ping query
                        reply = {"t": dic['t'], "y": "r", "r": {"id": local_node.id.hex()}}
                        data = BEncode(reply).encode_bin()
                        sock.sendto(data, (dst_ip, dst_port))
                        # print("handle_query|ping|send_reply|dst_ip={}|dst_port={}|data={}".format(dst_ip, dst_port, data))
                    elif dic['q'] == 'find_node':
                        # handle find_node query
                        nodes = rt.find_closer(bytes.fromhex(dic['a']['id']))
                        print("local_node.id={}".format(local_node.id.hex()))
                        print(nodes)
                        compact_node = encode_compact_node(nodes)
                        reply = {"t": dic['t'], "y": "r", "r": {"id": local_node.id.hex(), "nodes": compact_node}}
                        data = BEncode(reply).encode_bin()
                        sock.sendto(data, (dst_ip, dst_port))
                        print("handle_query|find_node|send_reply|dst_ip={}|dst_port={}|data={}".format(dst_ip, dst_port,
                                                                                                       data))
                    elif dic['q'] == 'get_peers':
                        # handle get_peers query
                        reply = {"t": dic['t'], "y": "r",
                                 "r": {"id": local_node.id.hex(), "token": "alenym".encode("utf8"), "nodes": b""}}
                        data = BEncode(reply).encode_bin()
                        sock.sendto(data, (dst_ip, dst_port))
                        print("handle_query|get_peers|send_reply|dst_ip={}|dst_port={}|data={}".format(dst_ip, dst_port,
                                                                                                       data))
                    elif dic['q'] == 'announce_peer':
                        # handle announce_peer
                        print("handle_query|announce_peer|dic={}".format(dic))
                        info_hash = dic['a'].get('info_hash')
                        with RLOCK:
                            if info_hash is None or info_hash in INFO_HASH_METADATA_DIC:
                                continue

                        target_ip = dst_ip
                        target_port = dst_port

                        t_request_meta = threading.Thread(target=request_metadata_thread,
                                                          args=(
                                                              local_node.id, info_hash, target_ip, target_port, metadata_q))
                        t_request_meta.start()


                    else:
                        print("handle_query|other|dic={}".format(dic))
                except Exception as e:
                    log.exception("handle_query error|dic={}".format(dic))

        t_hand_query = threading.Thread(target=handle_query, args=(sock, query_q,))
        t_hand_query.daemon = True

        # 6. handle error
        def handle_error(error_q: Queue):
            while True:
                dic, addr = error_q.get()
                try:
                    print("handle_error|addr={}|dic={}".format(addr, dic))
                except Exception as e:
                    log.exception('handle_error error', e)

        t_hand_error = threading.Thread(target=handle_error, args=(error_q,))
        t_hand_error.daemon = True

        # 7. handle requested metadata
        def handle_metadata(metadata_q: Queue):
            while True:
                try:
                    metadata_dic, info_hash, ip, port = metadata_q.get()
                    format_s = "handle_metadata|info_hash={}|ip={}|port={}|name={}|size={}"
                    file_name = metadata_dic['name']
                    size = format_size(metadata_dic.get('length'))
                    print(format_s.format(info_hash, ip, port, file_name, size))
                    with RLOCK:
                        if info_hash not in INFO_HASH_METADATA_DIC:
                            INFO_HASH_METADATA_DIC[info_hash] = metadata_dic
                except Exception as e:
                    log.exception("handle_metadata error")


        t_handle_metadata = threading.Thread(target=handle_metadata, args=(metadata_q,))
        t_handle_metadata.daemon = True

        listener.start()
        t_dispatch.start()
        t_hand_reply.start()
        t_hand_query.start()
        t_hand_error.start()
        t_handle_metadata.start()

        listener.join()
        t_dispatch.join()
        t_hand_reply.join()
        t_hand_query.join()
        t_hand_error.join()
        t_handle_metadata.join()


    if __name__ == '__main__':
        main()
    -

    运行日志

    我们在一台有公网ip的电脑上运行十几分钟，过滤日志，查看接收到的种子信息如下，

    -
    root@ubuntu:~# cat log.txt | grep "handle_metadata"
    handle_metadata|info_hash=5553330daa12bde6a2f71ab26f7b26688219f276|ip=69.80.12.126|port=11715|name=War for the Planet of the Apes 2017 1080p BluRay x264 DTS 5.1 MSubS-Hon3y|size=None
    handle_metadata|info_hash=5556f3a9c605dd009f92fddb91848f565439e4f4|ip=59.169.228.207|port=62215|name=(C94) [くろすこスイッチ (くろすこ)] 冷泉さんといちゃいちゃする本 (ガールズ&パンツァー).zip|size=7.80M
    handle_metadata|info_hash=555333edb2519c3aa93db4a150f6e021f5a138ff|ip=96.40.42.32|port=32796|name=Princess Go Round|size=None
    handle_metadata|info_hash=572d4df79a7151d9466d371b960cfece91289bf4|ip=1.175.76.59|port=13283|name=(同人ゲーム) [181102][RJ237312][ピンポイント／キングピン] 妻が隠していたビデオ…～元カレ寝取らせ観察記～ DL版 (files).rar|size=1.08G
    handle_metadata|info_hash=572edd8670066945538e4fe823294b45f2c2e3a3|ip=14.199.224.142|port=22979|name=0407raw021|size=None
    handle_metadata|info_hash=574b27358fc1604e65babf32abe839267d36ba4b|ip=14.199.224.142|port=22979|name=o0Akrios0o@www.sexinsex.com@假裝租屋 事實上在陽台勾引住宅區人妻|size=None
    handle_metadata|info_hash=54730eeeb5d74a58f49c6da72eb90be922556f0a|ip=219.98.7.166|port=18586|name=DSAM-29-DVD|size=None
    handle_metadata|info_hash=57039c3ef3343c5b288c5ff219c66f837a88e808|ip=14.199.224.142|port=22979|name=tmem|size=None
    handle_metadata|info_hash=5687578db92c7df4ae3ab7e33c895f04b1e27d37|ip=14.199.224.142|port=22979|name=GNE-148|size=None
    handle_metadata|info_hash=57b832066b2103e29f3a4af256e25175da6dbdc2|ip=14.199.224.142|port=22979|name=[Thz.la]supa-154|size=None
    handle_metadata|info_hash=56936a12cdb3a1a837c8faa206a2dea189dadf66|ip=14.199.224.142|port=22979|name=club-022_1.wmv|size=2.96G
    handle_metadata|info_hash=570e89058161385b1d7dfadfcdc2d9f276ab829b|ip=14.199.224.142|port=22979|name=avidol.us-PTBI-026.wmv|size=2.71G
    handle_metadata|info_hash=57c30a1ddecac71142208425c610837b961f4de6|ip=14.199.224.142|port=22979|name=[HD]sad-039.wmv|size=3.52G
    handle_metadata|info_hash=56794273cf092e0b4f671885a16cf2dbe484f559|ip=14.199.224.142|port=22979|name=KRE-002.wmv|size=1.20G
    handle_metadata|info_hash=5710c35d85a9a0ed8cadc10bdcd09db5bb0c46fd|ip=14.199.224.142|port=22979|name=0111-xv1088|size=None
    handle_metadata|info_hash=561b55ec9a956f208f58b798d781bd5577a47b9e|ip=14.199.224.142|port=22979|name=52.R18-099|size=None
    handle_metadata|info_hash=57c262437e7cf0ca5f24bff947757026f9cef9a8|ip=14.199.224.142|port=22979|name=DF-35976|size=None
    handle_metadata|info_hash=565bedb7fc17d20f017a668e2931cb1d30b53556|ip=14.199.224.142|port=22979|name=judexkwok@片瀬まこ合集06|size=None
    handle_metadata|info_hash=57e93ca1527cd35045f0ca75d0c96579dcf96d2a|ip=14.199.224.142|port=22979|name=SMDV-10-DVD|size=None
    handle_metadata|info_hash=56747025a196cb78db9f94d8ca933677f57e54b0|ip=14.199.224.142|port=22979|name=HUNT-759.mp4|size=2.01G
    handle_metadata|info_hash=56e45f76194dadb2c799c7f2c9b34bc2fff07cee|ip=14.199.224.142|port=22979|name=0510-sama538|size=None
    handle_metadata|info_hash=57fae1a68ee9241a593c1623f64feb7927b40469|ip=14.199.224.142|port=22979|name=[thz.la]chunta-219|size=None
    handle_metadata|info_hash=56b89964da08b923b20722b89ccfc6ae4928aca3|ip=14.199.224.142|port=22979|name=HUNT-665_2.mp4|size=1.60G
    handle_metadata|info_hash=57528d8633ee8b823aea17e47def172082072dd5|ip=14.199.224.142|port=22979|name=HUNT|size=None
    handle_metadata|info_hash=57b14ca4cf260763e170e515b1f235ae5dca487f|ip=14.199.224.142|port=22979|name=SAMA-477|size=None
    handle_metadata|info_hash=57bc38a6f182dbd6be5b42371d645cc43cfa12e1|ip=14.199.224.142|port=22979|name=HUNT-710.mp4|size=1.82G
    handle_metadata|info_hash=57f2b65672462ff81834a928cf8e9863687dc220|ip=14.199.224.142|port=22979|name=3208|size=None
    ...
    - -

    有些种子文件为什么没有文件大小呢？例如size=None,

    -
    handle_metadata|info_hash=54730eeeb5d74a58f49c6da72eb90be922556f0a|ip=219.98.7.166|port=18586|name=DSAM-29-DVD|size=None
    - -

    用info_hash过滤日志，可以看到metadata包含了一个files项，该项包含了多个文件。

    -
    root@ubuntu:~# cat log.txt | grep "54730eeeb5d74a58f49c6da72eb90be922556f0a"| grep "metadata_dic" 
    request_metadata|info_hash=54730eeeb5d74a58f49c6da72eb90be922556f0a|metadata_dic={'files': [{'ed2k': b'\xc4"%lT\xa14s\xd86\xd7a\xb1:\x8bC', 'filehash': '788c531731917e92334c03423df5433b4d3d941d', 'length': 228076, 'path': ['C9~Fang Ping Bi Cheng Xu防屏蔽程序.rar'], 'path.utf-8': ['C9~Fang Ping Bi Cheng Xu防屏蔽程序.rar']}, {'ed2k': b'\xff,5\xb4W\x1b\xf1\xb0\x16\x13=\x0c\xc1\xc5\xb56', 'filehash': 'b760813c2b6690123eb174b5561ce6b4a0b6e000', 'length': 1966849, 'path': ['HOTAVXXX,Free Adult Movie, Fastest & Newest Porn Movie Site.jpg'], 'path.utf-8': ['HOTAVXXX,Free Adult Movie, Fastest & Newest Porn Movie Site.jpg']}, {'ed2k': b'\xfaggnH\xf8e}O\x97\x04\xbb\x86\x9e\x92\xf5', 'filehash': '9a8b43c663ecc46d3a29196cf527d8431fef842b', 'length': 180, 'path': ['HOTAVXXX~最新最快的AV影片每日更新.url'], 'path.utf-8': ['HOTAVXXX~最新最快的AV影片每日更新.url']}, {'ed2k': b'\xff,5\xb4W\x1b\xf1\xb0\x16\x13=\x0c\xc1\xc5\xb56', 'filehash': 'b760813c2b6690123eb174b5561ce6b4a0b6e000', 'length': 1966849, 'path': ['HOTAVXXX、自由な成人映画、最も速く&最新ポルノ映画サイト.jpg'], 'path.utf-8': ['HOTAVXXX、自由な成人映画、最も速く&最新ポルノ映画サイト.jpg']}, {'ed2k': b'$\x13\xfd\\\x8b\xa5\xee=\xc0\xfb\x88\xff~\x03/\x18', 'filehash': 'dddbfe3e917739e1da7afcdd8a78e1a6677b6648', 'length': 235, 'path': ['QQ愛真人視頻交友聊天室.url'], 'path.utf-8': ['QQ愛真人視頻交友聊天室.url']}, {'ed2k': b'/\xe2\xcf\x1b\xc06\xea[\xda\x93\xb9\xd0\\\xc1\x14\x98', 'filehash': '715c4e653db4bb74bb7e40073d55113f512299e9', 'length': 628211, 'path': ['SIS001全面封殺.jpg'], 'path.utf-8': ['SIS001全面封殺.jpg']}, {'ed2k': b'\xf3\x19\x893\x8b\r\xc4\r\x99|W)\xf4\xce\x00J', 'filehash': 'c4eadc74cdbf36df8a66710f62932a60b1a0949a', 'length': 267, 'path': ['[城風 - C9]~成人精品長篇區 最新http一手資源.url'], 'path.utf-8': ['[城風 - C9]~成人精品長篇區 最新http一手資源.url']}, {'ed2k': b'r\xbab\x10i\xc4\x10<f$\x10+p\x84\xd2Z', 'filehash': 'c6a1cb245511e4013c7510220ce3a6d912f8290f', 'length': 217, 'path': ['dioguitar23(第六天魔王)@草榴社區.url'], 'path.utf-8': ['dioguitar23(第六天魔王)@草榴社區.url']}, {'ed2k': b'\xe4z.\xa4\xfa\x88h\xfe\x07\xedy\xc3\x13D\xe8\x7f', 'filehash': 'dddd22e1c4027a8e5c6cec98a4ef3dd29b951682', 'length': 229, 'path': ['dioguitar23@ HD1080.org.url'], 'path.utf-8': ['dioguitar23@ HD1080.org.url']}, {'ed2k': b'\xf9\xee!X\xb3\x9d\xd9\x104\xc52\xc4\x872&a', 'filehash': '7191792635319be376c5ff3edd251e7bda81d069', 'length': 267, 'path': ['dioguitar23@AV 天空.url'], 'path.utf-8': ['dioguitar23@AV 天空.url']}, {'ed2k': b'\xba\x18f&*y&\xf5H\xe8\x1f;8x\x1eA', 'filehash': 'd774c64c18ec44e607ba6825675e841332a0d4f4', 'length': 188, 'path': ['dioguitar23@D.C.資訊交流網.url'], 'path.utf-8': ['dioguitar23@D.C.資訊交流網.url']}, {'ed2k': b'\x91\x97\x15O\rxh\x7fa\xbf^\xbf \x187K', 'filehash': 'f06835576489512e2c4a3ff58a5c4a2fed3131a3', 'length': 174, 'path': ['dioguitar23@KTzone.url'], 'path.utf-8': ['dioguitar23@KTzone.url']}, {'ed2k': b"\xf7\xf1\x9c\xa0+\x15\xc5_\xdb2u'\x81\xcb\xefU", 'filehash': '25a54a9fdab494a60fdbd206089ee75c08913f24', 'length': 235, 'path': ['dioguitar23@SexInSex! Board.url'], 'path.utf-8': ['dioguitar23@SexInSex! Board.url']}, {'ed2k': b'\x10k\x08\xafU\xcb\x05\x8b\x0f\xf9\xaf\xcb\r\xdd\xf3\xc4', 'filehash': 'a9bb602e06bb33483a1960db95bc65b535831c5d', 'length': 1086, 'path': ['dioguitar23@公仔箱論壇.url'], 'path.utf-8': ['dioguitar23@公仔箱論壇.url']}, {'ed2k': b'6-*X\x12\xf3f\xbe\xd8q\x15\xbe\x91[\xee4', 'filehash': 'd19dd08e932c7db940491e5ac259c6ad0ff225c3', 'length': 188, 'path': ['dioguitar23@痴漢俱樂部.url'], 'path.utf-8': ['dioguitar23@痴漢俱樂部.url']}, {'ed2k': b'\x89\xe1CN\xef\xd1\x83\xc2\x8c?,|;\xdb\t\x10', 'filehash': '47255a6ce6a43dc9af1de780fcffd210de793bfd', 'length': 190, 'path': ['dioguitar23@香港廣場.url'], 'path.utf-8': ['dioguitar23@香港廣場.url']}, {'ed2k': b'\xaaJh&\x1f\x88\x93\xda\x9bL\xca\xd9\x9c+V\xa9', 'filehash': '168c3faa1f01a31b31c8d5a1bb7e699333e0bfe4', 'length': 226, 'path': ['dioguitar23_Plus28 討論區.url'], 'path.utf-8': ['dioguitar23_Plus28 討論區.url']}, {'ed2k': b"\x04k`{\x0b'\xee\x80\xb4\x8e\x91=\xa92\xaf\x95", 'filehash': 'c7181e934379905222bf363317d8ae8ddfb12eb4', 'length': 220, 'path': ['dioguitar23_Touch99.url'], 'path.utf-8': ['dioguitar23_Touch99.url']}, {'ed2k': b'\xaf\xbe+\xd5\xde\x96\xdc\xe0\xa8a\x18\x87b\x8d\xf6\x0c', 'filehash': 'a92caa360c647d14ae05655c6e43e9f28d7edcf5', 'length': 208, 'path': ['dioguitar23_WK綜合論壇.url'], 'path.utf-8': ['dioguitar23_WK綜合論壇.url']}, {'ed2k': b'\x05>\x9b.\nS\xb2\x9c&L\xbc5H\x82\xcc\xcb', 'filehash': '6fa079db3cafb196163e9a6b126f60a5d4aa85b1', 'length': 156, 'path': ['dioguitar23_mimip2p.url'], 'path.utf-8': ['dioguitar23_mimip2p.url']}, {'ed2k': b'\xc3|\x1fKA\xb7\xdd\xb9\x87\xd3}\x04\x8f}\xb3m', 'filehash': '1a709fb06ff75ced80995ab8fcdefa69744d7aa0', 'length': 229, 'path': ['dioguitar23_※=色界论坛=※ (开放注册） 色界论坛.url'], 'path.utf-8': ['dioguitar23_※=色界论坛=※ (开放注册） 色界论坛.url']}, {'ed2k': b'>\x13\x98U3q\xf6u\xe6\x06\xc4\xa4\x91\xb24\xc5', 'filehash': '350cdfa69ce763f8c2fad85e1f0f4323c3fb5dfa', 'length': 208, 'path': ['dioguitar23_九九情色帝国.url'], 'path.utf-8': ['dioguitar23_九九情色帝国.url']}, {'ed2k': b'E\x0c\x8a\xed5{\x00\xe6\xdb\xfc\x9b\xb1|U%\xf5', 'filehash': '384d925cb1da6605d49014622ed9f0b455efe343', 'length': 261, 'path': ['dioguitar23_性吧春暖花开,春暖花开性吧有你.url'], 'path.utf-8': ['dioguitar23_性吧春暖花开,春暖花开性吧有你.url']}, {'ed2k': b'\xefE\xbe\xc2\xcf(\xf5\xe3\x18\x04\xdan\x8f\x08j\xcf', 'filehash': '2509953b0831793667a904ee08229da3114835cf', 'length': 214, 'path': ['dioguitar23_找樂子論壇.url'], 'path.utf-8': ['dioguitar23_找樂子論壇.url']}, {'ed2k': b"QuDc'zc]\xcb\xeb\xc0\x86-\xfe\xa5\xf4", 'filehash': '2e92574d6c46add45048b08aa15b4eae132c2cc1', 'length': 235, 'path': ['dioguitar23_無限討論區.url'], 'path.utf-8': ['dioguitar23_無限討論區.url']}, {'ed2k': b'tf\xe0\xda\x8c&\x96A\x9d\x9e~P\xe9\xd6\x83\xdb', 'filehash': '6b2a83a72df99bd7f2a197cb6fd423a41e061b70', 'length': 138, 'path': ['hav.tv-新幹線ONLINE~慶祝開站包月對折優惠~免費影片天看到爽!!.url'], 'path.utf-8': ['hav.tv-新幹線ONLINE~慶祝開站包月對折優惠~免費影片天看到爽!!.url']}, {'ed2k': b'\'\xf0\x08\xf1 {\xa9"\x13//(\xc7o\x9d`', 'filehash': '226f0d140af4689833613f0f21a95608fff324ef', 'length': 4324384768, 'path': ['hotavxxx.com_DSAM-29.ISO'], 'path.utf-8': ['hotavxxx.com_DSAM-29.ISO']}, {'ed2k': b'\x92]\xef\x0b\xa5\xee\xff\xd9\xb9\xd2\xd2\xc2J\x9e\x1a\xdc', 'filehash': 'eb8316fc354049e3bc1fb251bf6928dc55ed157a', 'length': 131523, 'path': ['hotavxxx.com_DSAM-29.jpg'], 'path.utf-8': ['hotavxxx.com_DSAM-29.jpg']}, {'ed2k': b'0\xaf\xc8Q\xc9\xa3\xfb\xacn\xf4\xde\xc1\r\x9c\xda\xbd', 'filehash': 'aa6975d6341d2a079661c066eb5244ee0fd1aaff', 'length': 2254263, 'path': ['hotavxxx.com_DSAM-29A.jpg'], 'path.utf-8': ['hotavxxx.com_DSAM-29A.jpg']}, {'ed2k': b'\xa1\xaf\x1b\xd7\x84\xd2g\x87\x99>\x82\xec\x94j\xb1U', 'filehash': '7254c5e02a0119f38e9c0e7576a1d8ac04b59f67', 'length': 214, 'path': ['❤dioguitar23❤18P2P-2013年04月2日 18p2p開放註冊3天.url'], 'path.utf-8': ['❤dioguitar23❤18P2P-2013年04月2日 18p2p開放註冊3天.url']}, {'ed2k': b'\xe0\x82]\xe7\xe2\xc3\xe0\x9bn\x81\x1d\xa4\x0fr\x9f+', 'filehash': 'cad3021392437e48d35faa57861e8ce0a89a6a0d', 'length': 233, 'path': ['アジア表動画公開板 [城風 - C9].url'], 'path.utf-8': ['アジア表動画公開板 [城風 - C9].url']}, {'ed2k': b'\x9a\xdeh5D\x1e}5`FeM\xf3\xcf\x1dt', 'filehash': '37402d4e9e9fedb26ff5c833fbc399fb5462a4af', 'length': 168, 'path': ['京色屋~即日起大降價.一片只要30元起.藍光片只要150元.url'], 'path.utf-8': ['京色屋~即日起大降價.一片只要30元起.藍光片只要150元.url']}, {'ed2k': b'W\x9a\xf4E\x1d\xb0b(\x05_v\x00\xef\xe5\xe4\xe6', 'filehash': '6d6df82810c4aa00152c3912d28bc13b6589a780', 'length': 223, 'path': ['堂本屋 TW DMM~即日起大降價.一片只要30元起.藍光片只要150元.url'], 'path.utf-8': ['堂本屋 TW DMM~即日起大降價.一片只要30元起.藍光片只要150元.url']}, {'ed2k': b'\x83\x05t=\xda\x85\x84\xbc\x1c\xb3J\xac\x04\x94\xfea', 'filehash': '52dcc77f3f2aad3ffd1cf4af73157deb1aeb8f4c', 'length': 2994, 'path': ['更多精彩.rar'], 'path.utf-8': ['更多精彩.rar']}, {'ed2k': b'<\xf8Y\xfd\x08\x99.\xc5\xec\x0e\x03V\r\xb4%\x06', 'filehash': 'ab91ce91df44509574450c875435f3c2a7bf18cf', 'length': 226, 'path': ['金花娱乐城-澳门网上真人赌博，90后性感荷官24小时在线存提款（5分钟到账）.url'], 'path.utf-8': ['金花娱乐城-澳门网上真人赌博，90后性感荷官24小时在线存提款（5分钟到账）.url']}, {'ed2k': b'\xfc*"\xad\xdbva\xdf\xbf\x92\x0e\x83.\x08/\x84', 'filehash': 'b60f8d54f1756a04753905be9a9dc7982f33ad14', 'length': 235, 'path': ['魔王の家-情慾視界~最新最快的成人資訊平台 魔王の家,http--bbs.hotavxxx.com.url'], 'path.utf-8': ['魔王の家-情慾視界~最新最快的成人資訊平台 魔王の家,http--bbs.hotavxxx.com.url']}], 'name': 'DSAM-29-DVD', 'name.utf-8': 'DSAM-29-DVD', 'piece length': 1048576, 'pieces': ['557c428f62f38fe8ac8d11411bc9796abb28254d', '53effb4976edd7d32c9be7a0315bfd40ff93875e', 'c38ea041059367193bc3e9cc6fac40f67cfb377f', '63d640ba1942f834e4ec0340f84d8c0703a1f992', '8b783f6a2dc8ec649ad78acc8c49f892a8bf7663', '961a8f0b0cdfe010c979061082982c3548173948', '57dd2532983cf4d30bd442281d007ce0189b6eac', 'd858de50e2a5d5dd60945ac687baa159697d88cd', 'f7a262965c87d1c4d3cc10660176c4d063d16807', '43f7e4f982c4a47d94757071e8b280798d5fbfa9', ...
    +

    运行远程调试之后，成功。 +remote-debug

    ]]> - bitTorrent - protocol - p2p + k3s + delve + golang @@ -2271,55 +2124,150 @@ csn为UTF-8。

     - 在mac下跑一个Ingress的例子 - /2020/07/06/run-ingress-example-on-mac/ - Ingress是什么

    在Kubernetes中，Ingress是一个对象，该对象允许从Kubernetes集群外部访问Kubernetes服务。 您可以 -通过创建一组规则来配置访问权限，这些规则定义了哪些入站连接可以访问哪些服务。

    + BitTorrent协议（六）之种子嗅探器 + /2019/02/18/2019-2-18-bt-6/ + Sniffer（嗅探器）

    实现一个简单的BT种子嗅探器才算是有点实际价值的吧。这样就可以把种子的metadata信息 +缓存下来，提供按照文件名进行检索。

    -

    这里有篇文章很好的说明了Ingress，并给出了例子 —— Kubernetes Ingress with Nginx Example

    -

    但是要想跑一下，首先要有一个k8s集群。下面首先在mac上安装一个k8s集群。

    -

    在mac上安装k8s集群

    我的mbp的配置是8G内存。

    +

    不过这里只是实现一个demo，有兴趣的话可以看看github上的dht项目。

    +

    基本原理

    简单的说，就是把嗅探节点加入到其他节点的路由表中，等待其他节点发来的announce_peer请求，然后获取种子的metadata信息。

    +

    我们根据以下的代码具体说明。

      -
    1. 下载Docker.dmg。
      2. -
    2. 从这里下载搭建k8s所需要的镜像——https://github.com/gotok8s/k8s-docker-desktop-for-mac -使用的方法就是如下，我用的docker desktop的kubernates的版本是1.19.3所以，相应的要把文件images中的版本号进行修改，以匹配。
      $ git clone https://github.com/gotok8s/k8s-docker-desktop-for-mac.git
      $ cd k8s-docker-desktop-for-mac
      $ cat images
      k8s.gcr.io/kube-proxy:v1.19.3=gotok8s/kube-proxy:v1.19.3
      k8s.gcr.io/kube-controller-manager:v1.19.3=gotok8s/kube-controller-manager:v1.19.3
      k8s.gcr.io/kube-scheduler:v1.19.3=gotok8s/kube-scheduler:v1.19.3
      k8s.gcr.io/kube-apiserver:v1.19.3=gotok8s/kube-apiserver:v1.19.3
      k8s.gcr.io/coredns:1.7.0=gotok8s/coredns:1.7.0
      k8s.gcr.io/pause:3.2=gotok8s/pause:3.2
      k8s.gcr.io/etcd:3.4.13-0=gotok8s/etcd:3.4.13-0
      $ ./load_images.sh
      3. -
    3. 启动docker desktop，并在dashboard中修改docker使用的资源大小。我分配了5G， -交换内存设置为4G，cpu数量为3
      4. -
    4. 勾选Enable Kubernetes和Show system containers (advanced)，启动，然后耐心等待。
      5. +

    5. 89行代码，向路由节点发送find_node请求。这些路由节点。就是25行代码的3个地址。 +这样做的意义有两个，一方面，路由节点会把我们的节点加入到他们的路由表中。另一方面，路由节点会返回一个好节点列表。

      +
      6. +

    6. 252行代码，启动了6个线程。

      +
    -

    需要等待的时间比较长，因为还会下载多个镜像,如docker/desktop-kubernetes等等。

    -

    安装ingress-nginx-controller

    我们从ingress-nginx的官网可以看到不同的安装方式，对于 Docker for Mac，一行命令搞定

    -
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud/deploy.yaml
    - -

    这个需要下载镜像quay.io/kubernetes-ingress-controller/nginx-ingress-controller，所以也需要时间。直到看到如下pods

    -
    ingress-nginx   ingress-nginx-admission-create-frllm       0/1     Completed   0          56m
    ingress-nginx   ingress-nginx-admission-patch-gbwpd        0/1     Completed   0          56m
    ingress-nginx   ingress-nginx-controller-8f7b9d799-c67xw   1/1     Running     2          56m
    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    线程说明
    listener代码93行，监听所有接收到的udp数据，并把这些放入到队列recv_q
    t_dispatch代码106行，从队列recv_q中取出数据，并进行bdecoding，根据消息类别分别放入对应的队列。一共有三种：应答，请求，错误。
    t_hand_reply代码132行，从reply_q中取出数据，解码节点列表。每一个节点都插入到本地的路由表（这里不太好，最好先进行ping_node，确定是好节点再加入），并向其发送find_node消息
    t_hand_query代码165行，从query_q中取出请求，分别对四种请求进行应答处理，分别是ping，find_node, get_peers, announce_peer。
    t_hand_error错误消息的处理。
    t_handle_metadata代码234行，从metadata_q中取出已经获取的种子的metadata，打印并把种子保存再字典表中。
    +
      +
    1. 代码165行，t_hand_query线程执行对请求进行处理。
      2. +
    +
      +
    • ping: 返回本地节点id
      • +
    • find_node: 从本地路由表中查出与请求节点id最近的8个节点，并返回。（据说把本地id进行返回，有利于本地节点加入其他路由表）
      • +
    • get_peers: 这里进行了简单处理，仅仅返回空节点list。
      • +
    • announce_peer: 这个请求是有节点告诉我们有新的种子文件发布了，并告诉我们info_hash。然后我们拿着info_hash取进行获取metadata的操作。 +代码207行，开启一个独立的获取metadata的线程。如果获取到metadata，则放入metadata_q中。
      • +
    +
    # -*- coding: utf-8 -*-
    from __future__ import absolute_import
    from __future__ import division
    from __future__ import print_function

    __author__ = 'ym'
    """
        Date        : '2019/2/3'
        Description :
        
    """

    from ym.bt.routing_table import RoutingTable, BTNode
    from ym.bt.util import id_generator, ip_me, decode_compact_node, encode_compact_node, format_size
    from ym.bt.bencoding_bin import BEncode
    from ym.bt.bdecoding import BDecode
    from ym.bt.request_metadata import request_metadata_top
    from queue import Queue
    import socket
    import threading
    import logging

    log = logging.getLogger()

    NODES = [("router.bittorrent.com", 6881),
             ("router.utorrent.com", 6881),
             ("dht.transmissionbt.com", 6881)]

    ID = id_generator()
    SOURCE_IP = ip_me()
    SOURCE_PORT = 6881
    TIME_OUT = 1

    INFO_HASH_METADATA_DIC = {}
    RLOCK = threading.RLock()


    def boot_step(sock: socket.socket, local_node: BTNode):
        # send find_node query to NODES
        try:
            dic = {"t": "aa", "y": "q", "q": "find_node",
                   "a": {"id": local_node.id.hex(), "target": local_node.id.hex()}}
            data = BEncode(dic).encode_bin()

            for NODE in NODES:
                dst_ip, dst_port = NODE
                sock.sendto(data, (dst_ip, dst_port))
                print("boot_step|send|dst_ip={}|dst_port={}".format(dst_ip, dst_port))
        except Exception as e:
            log.exception(e)
            raise Exception("boot_step error")


    def request_metadata_thread(peer_id: bytes, info_hash, ip, port, metadata_q: Queue):
        try:
            print("request_metadata_thread|info_hash={}|ip={}|port={}".format(info_hash, ip, port))
            metadata_dic = request_metadata_top(peer_id, info_hash, ip, port)
            if metadata_dic is not None:
                metadata_q.put((metadata_dic, info_hash, ip, port))
        except Exception as e:
            log.exception("request_metadata_thread error")
            raise Exception("request metadata error")


    def main():
        sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        sock.setblocking(True)
        sock.bind((SOURCE_IP, SOURCE_PORT))

        # recv queue
        recv_q = Queue(maxsize=0)
        # reply queue
        reply_q = Queue(maxsize=0)
        # query queue
        query_q = Queue(maxsize=0)
        # error queue
        error_q = Queue(maxsize=0)
        # request metadata
        metadata_q = Queue(maxsize=0)

        # declare routing table
        rt = RoutingTable()

        local_node = BTNode(id=ID, ip=SOURCE_IP, port=SOURCE_PORT)
        print("local_node={}".format(local_node))
        rt.insert(local_node)
        # 1. boot step
        print("boot start.")
        boot_step(sock, local_node)
        print("boot end.")

        # 2. listener
        def recv_listener(sock: socket.socket, recv_q: Queue):
            while True:
                try:
                    data, addr = sock.recvfrom(2048)
                    # print("recv_listener|recv|addr={}".format(addr))
                    recv_q.put((data, addr))
                except Exception as e:
                    print("recv_listener error")

        listener = threading.Thread(target=recv_listener, args=(sock, recv_q,))
        listener.daemon = True

        # 3. handle recv msg
        def dispatch_recv(recv_q: Queue, reply_q: Queue, query_q: Queue, error_q: Queue):
            while True:
                data, addr = recv_q.get()
                try:
                    dic, i, n = BDecode(data).parse()
                    # print("dispatch_recv|dic={}".format(dic))
                    if dic.get("y") is not None:
                        v = dic['y']
                        if v == 'r':
                            # reply msg
                            reply_q.put((dic, addr))
                        elif v == 'q':
                            # query msg
                            query_q.put((dic, addr))
                        elif v == 'e':
                            # error msg
                            error_q.put((dic, addr))
                        else:
                            print("unknown type msg.")
                except Exception as e:
                    log.exception('dispatch_recv error|data={}'.format(data))

        t_dispatch = threading.Thread(target=dispatch_recv, args=(recv_q, reply_q, query_q, error_q,))
        t_dispatch.daemon = True

        # 4. handle reply
        def handle_reply(sock: socket.socket, reply_q: Queue, rt: RoutingTable):
            while True:
                dic, addr = reply_q.get()
                try:
                    r = dic['r']
                    src_id = r['id']
                    src_ip, src_port = addr
                    src_node = BTNode(id=src_id, ip=src_ip, port=src_port)
                    rt.insert(src_node)
                    print("rt.size={}".format(rt.size()))
                    # print("handle_reply|src_ip={}|src_port={}".format(src_ip, src_port))
                    if r.get('nodes') is not None:
                        # ping every node and put good in rt.
                        nodes = decode_compact_node(r['nodes'])
                        dic = {"t": "aa", "y": "q", "q": "find_node",
                               "a": {"id": local_node.id.hex(), "target": local_node.id.hex()}}
                        data = BEncode(dic).encode_bin()
                        for node in nodes:
                            try:
                                dst_ip, dst_port, id = node
                                sock.sendto(data, (dst_ip, dst_port))
                            except Exception as e:
                                log.exception("send node error|node={}".format(node))
                    elif r.get('peers') is not None:
                        # ping every peer and put good in rt.
                        print("handle_reply|peers")
                except Exception as e:
                    log.exception("handle_reply error|dic={}".format(dic), e)

        t_hand_reply = threading.Thread(target=handle_reply, args=(sock, reply_q, rt,))
        t_hand_reply.daemon = True

        # 5. handle query
        def handle_query(sock: socket.socket, query_q: Queue):
            while True:
                dic, addr = query_q.get()
                dst_ip, dst_port = addr
                try:
                    print("handle_query|addr={}|dic={}".format(addr, dic))
                    if dic['q'] == 'ping':
                        # handle ping query
                        reply = {"t": dic['t'], "y": "r", "r": {"id": local_node.id.hex()}}
                        data = BEncode(reply).encode_bin()
                        sock.sendto(data, (dst_ip, dst_port))
                        # print("handle_query|ping|send_reply|dst_ip={}|dst_port={}|data={}".format(dst_ip, dst_port, data))
                    elif dic['q'] == 'find_node':
                        # handle find_node query
                        nodes = rt.find_closer(bytes.fromhex(dic['a']['id']))
                        print("local_node.id={}".format(local_node.id.hex()))
                        print(nodes)
                        compact_node = encode_compact_node(nodes)
                        reply = {"t": dic['t'], "y": "r", "r": {"id": local_node.id.hex(), "nodes": compact_node}}
                        data = BEncode(reply).encode_bin()
                        sock.sendto(data, (dst_ip, dst_port))
                        print("handle_query|find_node|send_reply|dst_ip={}|dst_port={}|data={}".format(dst_ip, dst_port,
                                                                                                       data))
                    elif dic['q'] == 'get_peers':
                        # handle get_peers query
                        reply = {"t": dic['t'], "y": "r",
                                 "r": {"id": local_node.id.hex(), "token": "alenym".encode("utf8"), "nodes": b""}}
                        data = BEncode(reply).encode_bin()
                        sock.sendto(data, (dst_ip, dst_port))
                        print("handle_query|get_peers|send_reply|dst_ip={}|dst_port={}|data={}".format(dst_ip, dst_port,
                                                                                                       data))
                    elif dic['q'] == 'announce_peer':
                        # handle announce_peer
                        print("handle_query|announce_peer|dic={}".format(dic))
                        info_hash = dic['a'].get('info_hash')
                        with RLOCK:
                            if info_hash is None or info_hash in INFO_HASH_METADATA_DIC:
                                continue

                        target_ip = dst_ip
                        target_port = dst_port

                        t_request_meta = threading.Thread(target=request_metadata_thread,
                                                          args=(
                                                              local_node.id, info_hash, target_ip, target_port, metadata_q))
                        t_request_meta.start()


                    else:
                        print("handle_query|other|dic={}".format(dic))
                except Exception as e:
                    log.exception("handle_query error|dic={}".format(dic))

        t_hand_query = threading.Thread(target=handle_query, args=(sock, query_q,))
        t_hand_query.daemon = True

        # 6. handle error
        def handle_error(error_q: Queue):
            while True:
                dic, addr = error_q.get()
                try:
                    print("handle_error|addr={}|dic={}".format(addr, dic))
                except Exception as e:
                    log.exception('handle_error error', e)

        t_hand_error = threading.Thread(target=handle_error, args=(error_q,))
        t_hand_error.daemon = True

        # 7. handle requested metadata
        def handle_metadata(metadata_q: Queue):
            while True:
                try:
                    metadata_dic, info_hash, ip, port = metadata_q.get()
                    format_s = "handle_metadata|info_hash={}|ip={}|port={}|name={}|size={}"
                    file_name = metadata_dic['name']
                    size = format_size(metadata_dic.get('length'))
                    print(format_s.format(info_hash, ip, port, file_name, size))
                    with RLOCK:
                        if info_hash not in INFO_HASH_METADATA_DIC:
                            INFO_HASH_METADATA_DIC[info_hash] = metadata_dic
                except Exception as e:
                    log.exception("handle_metadata error")


        t_handle_metadata = threading.Thread(target=handle_metadata, args=(metadata_q,))
        t_handle_metadata.daemon = True

        listener.start()
        t_dispatch.start()
        t_hand_reply.start()
        t_hand_query.start()
        t_hand_error.start()
        t_handle_metadata.start()

        listener.join()
        t_dispatch.join()
        t_hand_reply.join()
        t_hand_query.join()
        t_hand_error.join()
        t_handle_metadata.join()


    if __name__ == '__main__':
        main()
    +

    运行日志

    我们在一台有公网ip的电脑上运行十几分钟，过滤日志，查看接收到的种子信息如下，

    +
    root@ubuntu:~# cat log.txt | grep "handle_metadata"
    handle_metadata|info_hash=5553330daa12bde6a2f71ab26f7b26688219f276|ip=69.80.12.126|port=11715|name=War for the Planet of the Apes 2017 1080p BluRay x264 DTS 5.1 MSubS-Hon3y|size=None
    handle_metadata|info_hash=5556f3a9c605dd009f92fddb91848f565439e4f4|ip=59.169.228.207|port=62215|name=(C94) [くろすこスイッチ (くろすこ)] 冷泉さんといちゃいちゃする本 (ガールズ&パンツァー).zip|size=7.80M
    handle_metadata|info_hash=555333edb2519c3aa93db4a150f6e021f5a138ff|ip=96.40.42.32|port=32796|name=Princess Go Round|size=None
    handle_metadata|info_hash=572d4df79a7151d9466d371b960cfece91289bf4|ip=1.175.76.59|port=13283|name=(同人ゲーム) [181102][RJ237312][ピンポイント／キングピン] 妻が隠していたビデオ…～元カレ寝取らせ観察記～ DL版 (files).rar|size=1.08G
    handle_metadata|info_hash=572edd8670066945538e4fe823294b45f2c2e3a3|ip=14.199.224.142|port=22979|name=0407raw021|size=None
    handle_metadata|info_hash=574b27358fc1604e65babf32abe839267d36ba4b|ip=14.199.224.142|port=22979|name=o0Akrios0o@www.sexinsex.com@假裝租屋 事實上在陽台勾引住宅區人妻|size=None
    handle_metadata|info_hash=54730eeeb5d74a58f49c6da72eb90be922556f0a|ip=219.98.7.166|port=18586|name=DSAM-29-DVD|size=None
    handle_metadata|info_hash=57039c3ef3343c5b288c5ff219c66f837a88e808|ip=14.199.224.142|port=22979|name=tmem|size=None
    handle_metadata|info_hash=5687578db92c7df4ae3ab7e33c895f04b1e27d37|ip=14.199.224.142|port=22979|name=GNE-148|size=None
    handle_metadata|info_hash=57b832066b2103e29f3a4af256e25175da6dbdc2|ip=14.199.224.142|port=22979|name=[Thz.la]supa-154|size=None
    handle_metadata|info_hash=56936a12cdb3a1a837c8faa206a2dea189dadf66|ip=14.199.224.142|port=22979|name=club-022_1.wmv|size=2.96G
    handle_metadata|info_hash=570e89058161385b1d7dfadfcdc2d9f276ab829b|ip=14.199.224.142|port=22979|name=avidol.us-PTBI-026.wmv|size=2.71G
    handle_metadata|info_hash=57c30a1ddecac71142208425c610837b961f4de6|ip=14.199.224.142|port=22979|name=[HD]sad-039.wmv|size=3.52G
    handle_metadata|info_hash=56794273cf092e0b4f671885a16cf2dbe484f559|ip=14.199.224.142|port=22979|name=KRE-002.wmv|size=1.20G
    handle_metadata|info_hash=5710c35d85a9a0ed8cadc10bdcd09db5bb0c46fd|ip=14.199.224.142|port=22979|name=0111-xv1088|size=None
    handle_metadata|info_hash=561b55ec9a956f208f58b798d781bd5577a47b9e|ip=14.199.224.142|port=22979|name=52.R18-099|size=None
    handle_metadata|info_hash=57c262437e7cf0ca5f24bff947757026f9cef9a8|ip=14.199.224.142|port=22979|name=DF-35976|size=None
    handle_metadata|info_hash=565bedb7fc17d20f017a668e2931cb1d30b53556|ip=14.199.224.142|port=22979|name=judexkwok@片瀬まこ合集06|size=None
    handle_metadata|info_hash=57e93ca1527cd35045f0ca75d0c96579dcf96d2a|ip=14.199.224.142|port=22979|name=SMDV-10-DVD|size=None
    handle_metadata|info_hash=56747025a196cb78db9f94d8ca933677f57e54b0|ip=14.199.224.142|port=22979|name=HUNT-759.mp4|size=2.01G
    handle_metadata|info_hash=56e45f76194dadb2c799c7f2c9b34bc2fff07cee|ip=14.199.224.142|port=22979|name=0510-sama538|size=None
    handle_metadata|info_hash=57fae1a68ee9241a593c1623f64feb7927b40469|ip=14.199.224.142|port=22979|name=[thz.la]chunta-219|size=None
    handle_metadata|info_hash=56b89964da08b923b20722b89ccfc6ae4928aca3|ip=14.199.224.142|port=22979|name=HUNT-665_2.mp4|size=1.60G
    handle_metadata|info_hash=57528d8633ee8b823aea17e47def172082072dd5|ip=14.199.224.142|port=22979|name=HUNT|size=None
    handle_metadata|info_hash=57b14ca4cf260763e170e515b1f235ae5dca487f|ip=14.199.224.142|port=22979|name=SAMA-477|size=None
    handle_metadata|info_hash=57bc38a6f182dbd6be5b42371d645cc43cfa12e1|ip=14.199.224.142|port=22979|name=HUNT-710.mp4|size=1.82G
    handle_metadata|info_hash=57f2b65672462ff81834a928cf8e9863687dc220|ip=14.199.224.142|port=22979|name=3208|size=None
    ...
    -

    跑一个Ingress的例子

    Kubernetes Ingress with Nginx Example中的yaml文件需要外网才能看到。

    -

    我在此列出三个yaml文件

    -

    apple.yaml:

    -
    kind: Pod
    apiVersion: v1
    metadata:
      name: apple-app
      labels:
        app: apple
    spec:
      containers:
        - name: apple-app
          image: hashicorp/http-echo
          args:
            - "-text=apple"

    ---

    kind: Service
    apiVersion: v1
    metadata:
      name: apple-service
    spec:
      selector:
        app: apple
      ports:
        - port: 5678 # Default port for image
    +

    有些种子文件为什么没有文件大小呢？例如size=None,

    +
    handle_metadata|info_hash=54730eeeb5d74a58f49c6da72eb90be922556f0a|ip=219.98.7.166|port=18586|name=DSAM-29-DVD|size=None
    -

    banana.yaml :

    -
    kind: Pod
    apiVersion: v1
    metadata:
      name: banana-app
      labels:
        app: banana
    spec:
      containers:
        - name: banana-app
          image: hashicorp/http-echo
          args:
            - "-text=banana"

    ---

    kind: Service
    apiVersion: v1
    metadata:
      name: banana-service
    spec:
      selector:
        app: banana
      ports:
        - port: 5678 # Default port for image
    +

    用info_hash过滤日志，可以看到metadata包含了一个files项，该项包含了多个文件。

    +
    root@ubuntu:~# cat log.txt | grep "54730eeeb5d74a58f49c6da72eb90be922556f0a"| grep "metadata_dic" 
    request_metadata|info_hash=54730eeeb5d74a58f49c6da72eb90be922556f0a|metadata_dic={'files': [{'ed2k': b'\xc4"%lT\xa14s\xd86\xd7a\xb1:\x8bC', 'filehash': '788c531731917e92334c03423df5433b4d3d941d', 'length': 228076, 'path': ['C9~Fang Ping Bi Cheng Xu防屏蔽程序.rar'], 'path.utf-8': ['C9~Fang Ping Bi Cheng Xu防屏蔽程序.rar']}, {'ed2k': b'\xff,5\xb4W\x1b\xf1\xb0\x16\x13=\x0c\xc1\xc5\xb56', 'filehash': 'b760813c2b6690123eb174b5561ce6b4a0b6e000', 'length': 1966849, 'path': ['HOTAVXXX,Free Adult Movie, Fastest & Newest Porn Movie Site.jpg'], 'path.utf-8': ['HOTAVXXX,Free Adult Movie, Fastest & Newest Porn Movie Site.jpg']}, {'ed2k': b'\xfaggnH\xf8e}O\x97\x04\xbb\x86\x9e\x92\xf5', 'filehash': '9a8b43c663ecc46d3a29196cf527d8431fef842b', 'length': 180, 'path': ['HOTAVXXX~最新最快的AV影片每日更新.url'], 'path.utf-8': ['HOTAVXXX~最新最快的AV影片每日更新.url']}, {'ed2k': b'\xff,5\xb4W\x1b\xf1\xb0\x16\x13=\x0c\xc1\xc5\xb56', 'filehash': 'b760813c2b6690123eb174b5561ce6b4a0b6e000', 'length': 1966849, 'path': ['HOTAVXXX、自由な成人映画、最も速く&最新ポルノ映画サイト.jpg'], 'path.utf-8': ['HOTAVXXX、自由な成人映画、最も速く&最新ポルノ映画サイト.jpg']}, {'ed2k': b'$\x13\xfd\\\x8b\xa5\xee=\xc0\xfb\x88\xff~\x03/\x18', 'filehash': 'dddbfe3e917739e1da7afcdd8a78e1a6677b6648', 'length': 235, 'path': ['QQ愛真人視頻交友聊天室.url'], 'path.utf-8': ['QQ愛真人視頻交友聊天室.url']}, {'ed2k': b'/\xe2\xcf\x1b\xc06\xea[\xda\x93\xb9\xd0\\\xc1\x14\x98', 'filehash': '715c4e653db4bb74bb7e40073d55113f512299e9', 'length': 628211, 'path': ['SIS001全面封殺.jpg'], 'path.utf-8': ['SIS001全面封殺.jpg']}, {'ed2k': b'\xf3\x19\x893\x8b\r\xc4\r\x99|W)\xf4\xce\x00J', 'filehash': 'c4eadc74cdbf36df8a66710f62932a60b1a0949a', 'length': 267, 'path': ['[城風 - C9]~成人精品長篇區 最新http一手資源.url'], 'path.utf-8': ['[城風 - C9]~成人精品長篇區 最新http一手資源.url']}, {'ed2k': b'r\xbab\x10i\xc4\x10<f$\x10+p\x84\xd2Z', 'filehash': 'c6a1cb245511e4013c7510220ce3a6d912f8290f', 'length': 217, 'path': ['dioguitar23(第六天魔王)@草榴社區.url'], 'path.utf-8': ['dioguitar23(第六天魔王)@草榴社區.url']}, {'ed2k': b'\xe4z.\xa4\xfa\x88h\xfe\x07\xedy\xc3\x13D\xe8\x7f', 'filehash': 'dddd22e1c4027a8e5c6cec98a4ef3dd29b951682', 'length': 229, 'path': ['dioguitar23@ HD1080.org.url'], 'path.utf-8': ['dioguitar23@ HD1080.org.url']}, {'ed2k': b'\xf9\xee!X\xb3\x9d\xd9\x104\xc52\xc4\x872&a', 'filehash': '7191792635319be376c5ff3edd251e7bda81d069', 'length': 267, 'path': ['dioguitar23@AV 天空.url'], 'path.utf-8': ['dioguitar23@AV 天空.url']}, {'ed2k': b'\xba\x18f&*y&\xf5H\xe8\x1f;8x\x1eA', 'filehash': 'd774c64c18ec44e607ba6825675e841332a0d4f4', 'length': 188, 'path': ['dioguitar23@D.C.資訊交流網.url'], 'path.utf-8': ['dioguitar23@D.C.資訊交流網.url']}, {'ed2k': b'\x91\x97\x15O\rxh\x7fa\xbf^\xbf \x187K', 'filehash': 'f06835576489512e2c4a3ff58a5c4a2fed3131a3', 'length': 174, 'path': ['dioguitar23@KTzone.url'], 'path.utf-8': ['dioguitar23@KTzone.url']}, {'ed2k': b"\xf7\xf1\x9c\xa0+\x15\xc5_\xdb2u'\x81\xcb\xefU", 'filehash': '25a54a9fdab494a60fdbd206089ee75c08913f24', 'length': 235, 'path': ['dioguitar23@SexInSex! Board.url'], 'path.utf-8': ['dioguitar23@SexInSex! Board.url']}, {'ed2k': b'\x10k\x08\xafU\xcb\x05\x8b\x0f\xf9\xaf\xcb\r\xdd\xf3\xc4', 'filehash': 'a9bb602e06bb33483a1960db95bc65b535831c5d', 'length': 1086, 'path': ['dioguitar23@公仔箱論壇.url'], 'path.utf-8': ['dioguitar23@公仔箱論壇.url']}, {'ed2k': b'6-*X\x12\xf3f\xbe\xd8q\x15\xbe\x91[\xee4', 'filehash': 'd19dd08e932c7db940491e5ac259c6ad0ff225c3', 'length': 188, 'path': ['dioguitar23@痴漢俱樂部.url'], 'path.utf-8': ['dioguitar23@痴漢俱樂部.url']}, {'ed2k': b'\x89\xe1CN\xef\xd1\x83\xc2\x8c?,|;\xdb\t\x10', 'filehash': '47255a6ce6a43dc9af1de780fcffd210de793bfd', 'length': 190, 'path': ['dioguitar23@香港廣場.url'], 'path.utf-8': ['dioguitar23@香港廣場.url']}, {'ed2k': b'\xaaJh&\x1f\x88\x93\xda\x9bL\xca\xd9\x9c+V\xa9', 'filehash': '168c3faa1f01a31b31c8d5a1bb7e699333e0bfe4', 'length': 226, 'path': ['dioguitar23_Plus28 討論區.url'], 'path.utf-8': ['dioguitar23_Plus28 討論區.url']}, {'ed2k': b"\x04k`{\x0b'\xee\x80\xb4\x8e\x91=\xa92\xaf\x95", 'filehash': 'c7181e934379905222bf363317d8ae8ddfb12eb4', 'length': 220, 'path': ['dioguitar23_Touch99.url'], 'path.utf-8': ['dioguitar23_Touch99.url']}, {'ed2k': b'\xaf\xbe+\xd5\xde\x96\xdc\xe0\xa8a\x18\x87b\x8d\xf6\x0c', 'filehash': 'a92caa360c647d14ae05655c6e43e9f28d7edcf5', 'length': 208, 'path': ['dioguitar23_WK綜合論壇.url'], 'path.utf-8': ['dioguitar23_WK綜合論壇.url']}, {'ed2k': b'\x05>\x9b.\nS\xb2\x9c&L\xbc5H\x82\xcc\xcb', 'filehash': '6fa079db3cafb196163e9a6b126f60a5d4aa85b1', 'length': 156, 'path': ['dioguitar23_mimip2p.url'], 'path.utf-8': ['dioguitar23_mimip2p.url']}, {'ed2k': b'\xc3|\x1fKA\xb7\xdd\xb9\x87\xd3}\x04\x8f}\xb3m', 'filehash': '1a709fb06ff75ced80995ab8fcdefa69744d7aa0', 'length': 229, 'path': ['dioguitar23_※=色界论坛=※ (开放注册） 色界论坛.url'], 'path.utf-8': ['dioguitar23_※=色界论坛=※ (开放注册） 色界论坛.url']}, {'ed2k': b'>\x13\x98U3q\xf6u\xe6\x06\xc4\xa4\x91\xb24\xc5', 'filehash': '350cdfa69ce763f8c2fad85e1f0f4323c3fb5dfa', 'length': 208, 'path': ['dioguitar23_九九情色帝国.url'], 'path.utf-8': ['dioguitar23_九九情色帝国.url']}, {'ed2k': b'E\x0c\x8a\xed5{\x00\xe6\xdb\xfc\x9b\xb1|U%\xf5', 'filehash': '384d925cb1da6605d49014622ed9f0b455efe343', 'length': 261, 'path': ['dioguitar23_性吧春暖花开,春暖花开性吧有你.url'], 'path.utf-8': ['dioguitar23_性吧春暖花开,春暖花开性吧有你.url']}, {'ed2k': b'\xefE\xbe\xc2\xcf(\xf5\xe3\x18\x04\xdan\x8f\x08j\xcf', 'filehash': '2509953b0831793667a904ee08229da3114835cf', 'length': 214, 'path': ['dioguitar23_找樂子論壇.url'], 'path.utf-8': ['dioguitar23_找樂子論壇.url']}, {'ed2k': b"QuDc'zc]\xcb\xeb\xc0\x86-\xfe\xa5\xf4", 'filehash': '2e92574d6c46add45048b08aa15b4eae132c2cc1', 'length': 235, 'path': ['dioguitar23_無限討論區.url'], 'path.utf-8': ['dioguitar23_無限討論區.url']}, {'ed2k': b'tf\xe0\xda\x8c&\x96A\x9d\x9e~P\xe9\xd6\x83\xdb', 'filehash': '6b2a83a72df99bd7f2a197cb6fd423a41e061b70', 'length': 138, 'path': ['hav.tv-新幹線ONLINE~慶祝開站包月對折優惠~免費影片天看到爽!!.url'], 'path.utf-8': ['hav.tv-新幹線ONLINE~慶祝開站包月對折優惠~免費影片天看到爽!!.url']}, {'ed2k': b'\'\xf0\x08\xf1 {\xa9"\x13//(\xc7o\x9d`', 'filehash': '226f0d140af4689833613f0f21a95608fff324ef', 'length': 4324384768, 'path': ['hotavxxx.com_DSAM-29.ISO'], 'path.utf-8': ['hotavxxx.com_DSAM-29.ISO']}, {'ed2k': b'\x92]\xef\x0b\xa5\xee\xff\xd9\xb9\xd2\xd2\xc2J\x9e\x1a\xdc', 'filehash': 'eb8316fc354049e3bc1fb251bf6928dc55ed157a', 'length': 131523, 'path': ['hotavxxx.com_DSAM-29.jpg'], 'path.utf-8': ['hotavxxx.com_DSAM-29.jpg']}, {'ed2k': b'0\xaf\xc8Q\xc9\xa3\xfb\xacn\xf4\xde\xc1\r\x9c\xda\xbd', 'filehash': 'aa6975d6341d2a079661c066eb5244ee0fd1aaff', 'length': 2254263, 'path': ['hotavxxx.com_DSAM-29A.jpg'], 'path.utf-8': ['hotavxxx.com_DSAM-29A.jpg']}, {'ed2k': b'\xa1\xaf\x1b\xd7\x84\xd2g\x87\x99>\x82\xec\x94j\xb1U', 'filehash': '7254c5e02a0119f38e9c0e7576a1d8ac04b59f67', 'length': 214, 'path': ['❤dioguitar23❤18P2P-2013年04月2日 18p2p開放註冊3天.url'], 'path.utf-8': ['❤dioguitar23❤18P2P-2013年04月2日 18p2p開放註冊3天.url']}, {'ed2k': b'\xe0\x82]\xe7\xe2\xc3\xe0\x9bn\x81\x1d\xa4\x0fr\x9f+', 'filehash': 'cad3021392437e48d35faa57861e8ce0a89a6a0d', 'length': 233, 'path': ['アジア表動画公開板 [城風 - C9].url'], 'path.utf-8': ['アジア表動画公開板 [城風 - C9].url']}, {'ed2k': b'\x9a\xdeh5D\x1e}5`FeM\xf3\xcf\x1dt', 'filehash': '37402d4e9e9fedb26ff5c833fbc399fb5462a4af', 'length': 168, 'path': ['京色屋~即日起大降價.一片只要30元起.藍光片只要150元.url'], 'path.utf-8': ['京色屋~即日起大降價.一片只要30元起.藍光片只要150元.url']}, {'ed2k': b'W\x9a\xf4E\x1d\xb0b(\x05_v\x00\xef\xe5\xe4\xe6', 'filehash': '6d6df82810c4aa00152c3912d28bc13b6589a780', 'length': 223, 'path': ['堂本屋 TW DMM~即日起大降價.一片只要30元起.藍光片只要150元.url'], 'path.utf-8': ['堂本屋 TW DMM~即日起大降價.一片只要30元起.藍光片只要150元.url']}, {'ed2k': b'\x83\x05t=\xda\x85\x84\xbc\x1c\xb3J\xac\x04\x94\xfea', 'filehash': '52dcc77f3f2aad3ffd1cf4af73157deb1aeb8f4c', 'length': 2994, 'path': ['更多精彩.rar'], 'path.utf-8': ['更多精彩.rar']}, {'ed2k': b'<\xf8Y\xfd\x08\x99.\xc5\xec\x0e\x03V\r\xb4%\x06', 'filehash': 'ab91ce91df44509574450c875435f3c2a7bf18cf', 'length': 226, 'path': ['金花娱乐城-澳门网上真人赌博，90后性感荷官24小时在线存提款（5分钟到账）.url'], 'path.utf-8': ['金花娱乐城-澳门网上真人赌博，90后性感荷官24小时在线存提款（5分钟到账）.url']}, {'ed2k': b'\xfc*"\xad\xdbva\xdf\xbf\x92\x0e\x83.\x08/\x84', 'filehash': 'b60f8d54f1756a04753905be9a9dc7982f33ad14', 'length': 235, 'path': ['魔王の家-情慾視界~最新最快的成人資訊平台 魔王の家,http--bbs.hotavxxx.com.url'], 'path.utf-8': ['魔王の家-情慾視界~最新最快的成人資訊平台 魔王の家,http--bbs.hotavxxx.com.url']}], 'name': 'DSAM-29-DVD', 'name.utf-8': 'DSAM-29-DVD', 'piece length': 1048576, 'pieces': ['557c428f62f38fe8ac8d11411bc9796abb28254d', '53effb4976edd7d32c9be7a0315bfd40ff93875e', 'c38ea041059367193bc3e9cc6fac40f67cfb377f', '63d640ba1942f834e4ec0340f84d8c0703a1f992', '8b783f6a2dc8ec649ad78acc8c49f892a8bf7663', '961a8f0b0cdfe010c979061082982c3548173948', '57dd2532983cf4d30bd442281d007ce0189b6eac', 'd858de50e2a5d5dd60945ac687baa159697d88cd', 'f7a262965c87d1c4d3cc10660176c4d063d16807', '43f7e4f982c4a47d94757071e8b280798d5fbfa9', ...
    +]]>     + + bitTorrent + protocol + p2p + +     + + docker容器内访问mac主机的kafka + /2020/03/09/connect-kafka/ + 从容器内访问主机的kafka

    我最近遇到这样一个需求，需要从容器内的ClickHouse访问安装在mac主机的kafka。这个问题似乎很简单， +因为在windows上，虚拟机可以和host组成一个局域网，因此kafka只要绑定此网段的ip地址即可。 +但是在我的mac主机下，这个方案行不通。

    + -

    ingress.yaml:

    -
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: example-ingress
      annotations:
        ingress.kubernetes.io/rewrite-target: /
    spec:
      rules:
      - http:
          paths:
            - path: /apple
              backend:
                serviceName: apple-service
                servicePort: 5678
            - path: /banana
              backend:
                serviceName: banana-service
                servicePort: 5678
    +

    有几个原因，

    +
      +
    1. ClickHouse无法直接安装在mac上，需要编译（当然8G内存也可以编译，但是ninja需要限制job数量，要花很长时间）。
      2. +
    2. 我的mac内存只有8G，使用vbox根本不可能，只能使用docker。
      3. +
    3. 有现成的镜像，直接可以启动。并且host上可以直接连接ClickHouse。
      4. +
    +

    难点就在于容器和host根本不在一个网段。

    +

    解决步骤

      +
    1. 修改主机上的kafka的相关配置如下
      2. +
    +
    listeners=PLAINTEXT://0.0.0.0:9092

    advertised.listeners=PLAINTEXT://host.docker.internal:9092
    -

    执行命令，等待一下。因为要下载一个镜像“jettech/kube-webhook-certgen”

    -
    $ kubectl apply -f apple.yaml
    $ kubectl apply -f banana.yaml
    $ kubectl create -f ingress.yaml
    +
      +
    1. /etc/hosts文件添加
      2. +
    +
    127.0.0.1       host.docker.internal
    -

    查看ingress

    -
    $ kubectl describe ingress -n default example-ingress
    Name:             example-ingress
    Namespace:        default
    Address:          localhost
    Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
    Rules:
      Host        Path  Backends
      ----        ----  --------
      *           
                  /apple    apple-service:5678 (10.1.0.30:5678)
                  /banana   banana-service:5678 (10.1.0.28:5678)
    Annotations:  ingress.kubernetes.io/rewrite-target: /
    Events:
      Type    Reason  Age   From                      Message
      ----    ------  ----  ----                      -------
      Normal  CREATE  45m   nginx-ingress-controller  Ingress default/example-ingress
      Normal  UPDATE  44m   nginx-ingress-controller  Ingress default/example-ingress
      Normal  CREATE  14m   nginx-ingress-controller  Ingress default/example-ingress
    +
      +
    1. 在容器内，访问host.docker.internal:9092
      2. +
    +

    原理

    docker desktop for mac会默认提供一个域名host.docker.internal给容器内 +的应用访问主机的服务。所以，如果主机上启动一个rest服务localhost:8080，则容器内可以通过 +host.docker.internal:8080直接访问。

    +

    但是如果kafka只是配置为

    +
    listeners=PLAINTEXT://localhost:9092
    +

    想直接访问kafka的broker则是不行的。我们会发现，在容器内，telnet host.docker.internal 9092是通的。这是为什么呢？

    +

    想要连接kafka的broker会进行基本的两步交互。

    +
      +
    1. client访问broker的地址”A”，然后broker会发送回去一个advertised地址”B”
      2. +
    2. client接收到地址”B”，然后尝试访问”B”
      3. +
    +

    如果不设置advertised.listeners，那么默认等于listeners的值。

    +

    所以对于这样的设置 +B=A=PLAINTEXT://localhost:9092。容器内尝试访问”host.docker.internal:9092”，则会这样子，

    +
      +
    1. 容器内访问host.docker.internal:9092，接收到地址B=PLAINTEXT://localhost:9092
      2. +
    2. 然后访问地址localhost:9092，失败。
      3. +
    +

    而如果在主机上访问则显然是成功的。

    +

    而我们的解决方法配置是这样的 +A=PLAINTEXT://localhost:9092, B=PLAINTEXT://host.docker.internal:9092

    +

    所以如果容器内访问的过程就是，

    +
      +
    1. 容器内访问host.docker.internal:9092，接收到地址B=PLAINTEXT://host.docker.internal:9092
      2. +
    2. 然后访问host.docker.internal:9092，成功。
      3. +
    +

    而主机上的过程是，

    +
      +
    1. 主机上访问localhost:9092,接收到地址B=PLAINTEXT://host.docker.internal:9092
      2. +
    2. 然后访问host.docker.internal:9092，根据/etc/hosts的域名解析为127.0.0.1:9092，成功。
      3. +
    +

    测试

    我这里用kafkacat测试，实际上容器内的应用可以使用其他kafka客户端。

    +

    连接broker,并显示metadata

    +
    root@bb6b85562200:/# kafkacat -b host.docker.internal:9092 -L
    Metadata for all topics (from broker 0: host.docker.internal:9092/0):
     1 brokers:
      broker 0 at host.docker.internal:9092
     8 topics:
      topic "test-topic" with 1 partitions:
        partition 0, leader 0, replicas: 0, isrs: 0
      topic "flink-test" with 1 partitions:
        partition 0, leader 0, replicas: 0, isrs: 0
      topic "__consumer_offsets" with 50 partitions:
        partition 0, leader 0, replicas: 0, isrs: 0
        partition 10, leader 0, replicas: 0, isrs: 0
    ...
    ...
    -

    最后测试

    -

    $ curl -kL http://localhost/apple
    apple

    $ curl -kL http://localhost/banana
    banana

    $ curl -kL http://localhost/notfound
    default backend - 404

    +

    显示消息

    +
    ^Croot@bb6b85562200:/# kafkacat -b host.docker.internal:9092 -C -t flink-test -o beginning
    {seqNo: 1, eventTs: 1583739799986, id: even偶数, value: 4.57}
    % Reached end of topic flink-test [0] at offset 1
    ]]>     + kafka + container docker - k8s - ingress - mac     @@ -2350,6 +2298,54 @@ csn为UTF-8。

    file     + + 基于CLion和gdbserver实现远程调试c程序 + /2020/09/08/remote-debug-with-clion/ + 远程调试c程序

    最近基于tsar（阿里开源的一个基于c语言的监控程序）做二次开发， +因为以前从来没有在工作中写过c，所以这个简单的工作花了两周时间，期间用gdb进行调试，用valgrind检查内存泄漏。 +但是最让我不舒服的就是gdb调试了，虽然gdb很给力，但是毕竟由奢入俭难，之前写Java，Python，Go都是可以用IDE进行 +debug的。有图形化界面还是效率高很多，而对于新手，能够方便的debug源码就可以快速的理解项目。

    +

    那么怎么办呢？

    + + + +

    CLion

    我一直以为CLion可以很好的理解cmake项目，但是对于大量的基于makefile编译的项目，则不能很好的解析，所以也无法利用CLion +进行代码调试。

    +

    但是并不是这样，虽然CLion无法理解代码中各种符号之间的依赖关系，但是调试只要有debug info和源码就可以进行图形化调试。

    +

    参考CLion远程开发的Remote GDB Server，很简单就实现了。

    +

    远程调试tsar

      +

    1. mac下安装了CLion，linux下编译tsar。 +在mac下，tsar项目源码路径——/Users/ym/work/operation +在linux下为,tsar项目源码路径——/home/keyvalue/ym/operation

      +
      2. +

    2. 在CLion下创建一个GDB remote debug配置。 +参数配置：

      +
      3. +
    +
      +

    • target remote args
      10.4.104.153:1234 # 这是gdbserver的ip:port

      +
      • +

    • path mappings +remote path为/home/keyvalue/ym/operation，local path为/Users/ym/work/operation

      +
      • +
    +

    gdb remote debug config

    +
      +

    1. 在linux下运行命令

      +
      $ sudo gdbserver :1234 src/tsar --cron
      Process src/tsar created; pid = 10216
      Listening on port 1234
      +
      2. +

    2. CLion下启动之前的GDB remote debug配置，就可以愉快的断点调试了。

      +
      3. +
    +

    tsar remote debug

    +]]>     + + debug + CLion + gdb + remote + +     自建根证书，中间证书和Server端X.509证书并搭建nginx验证Server端证书有效性 /2020/10/09/x509-ca/ @@ -2405,51 +2401,55 @@ csn为UTF-8。

     - 基于CLion和gdbserver实现远程调试c程序 - /2020/09/08/remote-debug-with-clion/ - 远程调试c程序

    最近基于tsar（阿里开源的一个基于c语言的监控程序）做二次开发， -因为以前从来没有在工作中写过c，所以这个简单的工作花了两周时间，期间用gdb进行调试，用valgrind检查内存泄漏。 -但是最让我不舒服的就是gdb调试了，虽然gdb很给力，但是毕竟由奢入俭难，之前写Java，Python，Go都是可以用IDE进行 -debug的。有图形化界面还是效率高很多，而对于新手，能够方便的debug源码就可以快速的理解项目。

    -

    那么怎么办呢？

    + 在mac下跑一个Ingress的例子 + /2020/07/06/run-ingress-example-on-mac/ + Ingress是什么

    在Kubernetes中，Ingress是一个对象，该对象允许从Kubernetes集群外部访问Kubernetes服务。 您可以 +通过创建一组规则来配置访问权限，这些规则定义了哪些入站连接可以访问哪些服务。

    - -

    CLion

    我一直以为CLion可以很好的理解cmake项目，但是对于大量的基于makefile编译的项目，则不能很好的解析，所以也无法利用CLion -进行代码调试。

    -

    但是并不是这样，虽然CLion无法理解代码中各种符号之间的依赖关系，但是调试只要有debug info和源码就可以进行图形化调试。

    -

    参考CLion远程开发的Remote GDB Server，很简单就实现了。

    -

    远程调试tsar

      -

    1. mac下安装了CLion，linux下编译tsar。 -在mac下，tsar项目源码路径——/Users/ym/work/operation -在linux下为,tsar项目源码路径——/home/keyvalue/ym/operation

      -
      2. -

    2. 在CLion下创建一个GDB remote debug配置。 -参数配置：

      -
      3. -
    -
      -

    • target remote args
      10.4.104.153:1234 # 这是gdbserver的ip:port

      -
      • -

    • path mappings -remote path为/home/keyvalue/ym/operation，local path为/Users/ym/work/operation

      -
      • -
    -

    gdb remote debug config

    -
      -

    1. 在linux下运行命令

      -
      $ sudo gdbserver :1234 src/tsar --cron
      Process src/tsar created; pid = 10216
      Listening on port 1234
      -
      2. -

    2. CLion下启动之前的GDB remote debug配置，就可以愉快的断点调试了。

      -
      3. +

      这里有篇文章很好的说明了Ingress，并给出了例子 —— Kubernetes Ingress with Nginx Example

      +

      但是要想跑一下，首先要有一个k8s集群。下面首先在mac上安装一个k8s集群。

      +

      在mac上安装k8s集群

      我的mbp的配置是8G内存。

      +
        +
      1. 下载Docker.dmg。
        2. +
      2. 从这里下载搭建k8s所需要的镜像——https://github.com/gotok8s/k8s-docker-desktop-for-mac +使用的方法就是如下，我用的docker desktop的kubernates的版本是1.19.3所以，相应的要把文件images中的版本号进行修改，以匹配。
        $ git clone https://github.com/gotok8s/k8s-docker-desktop-for-mac.git
        $ cd k8s-docker-desktop-for-mac
        $ cat images
        k8s.gcr.io/kube-proxy:v1.19.3=gotok8s/kube-proxy:v1.19.3
        k8s.gcr.io/kube-controller-manager:v1.19.3=gotok8s/kube-controller-manager:v1.19.3
        k8s.gcr.io/kube-scheduler:v1.19.3=gotok8s/kube-scheduler:v1.19.3
        k8s.gcr.io/kube-apiserver:v1.19.3=gotok8s/kube-apiserver:v1.19.3
        k8s.gcr.io/coredns:1.7.0=gotok8s/coredns:1.7.0
        k8s.gcr.io/pause:3.2=gotok8s/pause:3.2
        k8s.gcr.io/etcd:3.4.13-0=gotok8s/etcd:3.4.13-0
        $ ./load_images.sh
        3. +
      3. 启动docker desktop，并在dashboard中修改docker使用的资源大小。我分配了5G， +交换内存设置为4G，cpu数量为3
        4. +
      4. 勾选Enable Kubernetes和Show system containers (advanced)，启动，然后耐心等待。
      -

      tsar remote debug

      +

      需要等待的时间比较长，因为还会下载多个镜像,如docker/desktop-kubernetes等等。

      +

      安装ingress-nginx-controller

      我们从ingress-nginx的官网可以看到不同的安装方式，对于 Docker for Mac，一行命令搞定

      +
      kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud/deploy.yaml
      + +

      这个需要下载镜像quay.io/kubernetes-ingress-controller/nginx-ingress-controller，所以也需要时间。直到看到如下pods

      +
      ingress-nginx   ingress-nginx-admission-create-frllm       0/1     Completed   0          56m
      ingress-nginx   ingress-nginx-admission-patch-gbwpd        0/1     Completed   0          56m
      ingress-nginx   ingress-nginx-controller-8f7b9d799-c67xw   1/1     Running     2          56m
      + +

      跑一个Ingress的例子

      Kubernetes Ingress with Nginx Example中的yaml文件需要外网才能看到。

      +

      我在此列出三个yaml文件

      +

      apple.yaml:

      +
      kind: Pod
      apiVersion: v1
      metadata:
        name: apple-app
        labels:
          app: apple
      spec:
        containers:
          - name: apple-app
            image: hashicorp/http-echo
            args:
              - "-text=apple"

      ---

      kind: Service
      apiVersion: v1
      metadata:
        name: apple-service
      spec:
        selector:
          app: apple
        ports:
          - port: 5678 # Default port for image
      + +

      banana.yaml :

      +
      kind: Pod
      apiVersion: v1
      metadata:
        name: banana-app
        labels:
          app: banana
      spec:
        containers:
          - name: banana-app
            image: hashicorp/http-echo
            args:
              - "-text=banana"

      ---

      kind: Service
      apiVersion: v1
      metadata:
        name: banana-service
      spec:
        selector:
          app: banana
        ports:
          - port: 5678 # Default port for image
      + +

      ingress.yaml:

      +
      apiVersion: extensions/v1beta1
      kind: Ingress
      metadata:
        name: example-ingress
        annotations:
          ingress.kubernetes.io/rewrite-target: /
      spec:
        rules:
        - http:
            paths:
              - path: /apple
                backend:
                  serviceName: apple-service
                  servicePort: 5678
              - path: /banana
                backend:
                  serviceName: banana-service
                  servicePort: 5678
      + +

      执行命令，等待一下。因为要下载一个镜像“jettech/kube-webhook-certgen”

      +
      $ kubectl apply -f apple.yaml
      $ kubectl apply -f banana.yaml
      $ kubectl create -f ingress.yaml
      + +

      查看ingress

      +
      $ kubectl describe ingress -n default example-ingress
      Name:             example-ingress
      Namespace:        default
      Address:          localhost
      Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
      Rules:
        Host        Path  Backends
        ----        ----  --------
        *           
                    /apple    apple-service:5678 (10.1.0.30:5678)
                    /banana   banana-service:5678 (10.1.0.28:5678)
      Annotations:  ingress.kubernetes.io/rewrite-target: /
      Events:
        Type    Reason  Age   From                      Message
        ----    ------  ----  ----                      -------
        Normal  CREATE  45m   nginx-ingress-controller  Ingress default/example-ingress
        Normal  UPDATE  44m   nginx-ingress-controller  Ingress default/example-ingress
        Normal  CREATE  14m   nginx-ingress-controller  Ingress default/example-ingress
      + +

      最后测试

      +

      $ curl -kL http://localhost/apple
      apple

      $ curl -kL http://localhost/banana
      banana

      $ curl -kL http://localhost/notfound
      default backend - 404

      ]]> - debug - CLion - gdb - remote + docker + k8s + ingress + mac @@ -2591,129 +2591,6 @@ search addom.xinaogroup.com coredns - - [leetcode 390]Elimination Game原创解法 - /2022/02/06/2022-2-6-leetcode-390/ - 题目概述

      leetcode现在支持Go了，这次用Go写一个算法题——消除游戏。

      -

      原题链接

      -
      You have a list arr of all integers in the range [1, n] sorted in a strictly increasing order. Apply the following algorithm on arr:
-
-Starting from left to right, remove the first number and every other number afterward until you reach the end of the list.
-Repeat the previous step again, but this time from right to left, remove the rightmost number and every other number from the remaining numbers.
-Keep repeating the steps again, alternating left to right and right to left, until a single number remains.
-Given the integer n, return the last number that remains in arr.
-
      -

      Example 1:

      -

      Input: n = 9

      -

      Output: 6

      -

      Explanation:

      -

      arr = [1, 2, 3, 4, 5, 6, 7, 8, 9]

      -

      arr = [2, 4, 6, 8]

      -

      arr = [2, 6]

      -

      arr = [6]

      -

      简单说就是从左至右消除，再从右到左消除，如果不唯一继续这个过程。对于这个例子，输入是9，那么开始就有1-9， -9个数。第一次，消除{1,3,5,7,9}, 第二次消除{8,4},第三次消除{2},最后留下数字6。

      -

      老办法，先用naive、暴力的方法求解，再优化。

      - - -

      暴力求解方法

      基本思路就是

      -
        -
      1. 生成一个数组，含有1-n，n个整数。
        2. -
      2. 从左到右消除，就是把偶数位置的数字依序移动到数组的左侧，并截取保留。例如从左至右消除数组[1,2,3,4,5,6,7,8,9], -移动偶数位置到左侧后，[2,4,6,8,5,6,7,8,9], 截取左侧后为[2,4,6,8]
        3. -
      3. 从右至左消除，就是把奇数位置的数字依序移动到数组的右侧，并截取保留。例如从右至左消除数组[2,4,6,8],移动奇数 -位置的数到右侧后，[2,4,2,6], 截取右侧后为[2,6]
        4. -
      4. 如果不为1，继续步骤2
        5. -
      -

      代码如下

      -
      //LastRemaining1 has time complexity O(n) and space complexity O(n)
      func LastRemaining1(n int) int {
      	arr := gen(n)

      	leftToRight := true
      	for len(arr) > 1 {
      		if leftToRight {
      			for i := 1; i < len(arr); i += 2 {
      				arr[i/2] = arr[i]
      			}
      			length := len(arr) / 2
      			arr = arr[0:length]
      		} else {
      			count := 0
      			for i := 1; len(arr)-1-i >= 0; i += 2 {
      				arr[len(arr)-1-i/2] = arr[len(arr)-1-i]
      				count += 1
      			}
      			arr = arr[len(arr)-count:]
      		}

      		leftToRight = !leftToRight
      	}

      	return arr[0]
      }

      func gen(n int) []int {
      	arr := make([]int, n)
      	for i := 0; i < n; i++ {
      		arr[i] = i + 1
      	}
      	return arr
      }
      - -

      这个时间和空间复杂度都是 O(n)\mathcal{O}(n) ，空间复杂度是 O(n)\mathcal{O}(n) -好理解。那么时间复杂度是怎么计算的呢？ 对于输入n，第一次消除遍历 n2\dfrac{n}{2} ，第二次消除遍历 n22\dfrac{n}{2^2} ， -第三次消除遍历n23\dfrac{n}{2^3} …因此总时间就是

      -n2+n22+n23+…≈n\dfrac{n}{2} + \dfrac{n}{2^2} + \dfrac{n}{2^3} + \ldots \approx n - -

      分析和优化后的算法

      暴力解法直观，但是空间占用太多了，当n很大的时候，直接栈溢出了。那么该怎么优化呢。只能是找规律，通过发现递推关系来求解。

      -

      也就是说，假如我们要求解的函数为y=f(n)y=\mathcal{f}(n)，其中n就是输入的自然数。y为剩余的最后的数字。 -根据以往的经验我们考虑分治法，找到递推关系 f(n+1)=anf(n)+an−1f(n−1)+…\mathcal{f}(n+1) = a_n\mathcal{f}(n) + a_{n-1}\mathcal{f}(n-1) + \ldots

      -

      很显然，我们知道 f(1)=1\mathcal{f}(1) = 1 , f(2)=2\mathcal{f}(2) = 2 , f(3)=2\mathcal{f}(3) = 2

      -

      通过观察，我们可以发现这样一个递归过程，经过一轮从左至右和从右至左的消除后，余下一个长度为n4\dfrac{n}{4} 的新的数列。 -如果长度n4≤3\dfrac{n}{4} \leq 3 ，那么可以直接知道f(n4)\mathcal{f}(\dfrac{n}{4}) 的值，该值就是新数列的编号。 -取出新数列的该编号的值并返回。 -如果数列的长度大于3，那么返回新数列中编号为 f(n4)\mathcal{f}(\dfrac{n}{4}) 的数字。

      -

      也就是说我们发现了递推关系 f(n)=operator∗f(n4)\mathcal{f}(n) = operator*\mathcal{f}(\dfrac{n}{4})

      -

      举例说明我们的递归过程，当n=9时，经过一轮从左至右和从右至左的消除后，余下一个长度为2的数列[2,6], 因为f(2)=2\mathcal{f}(2) = 2 ， -所以返回数列[2,6]编号为2的数，也就是数字6。

      -

      代码实现如下

      -
      //LastRemaining2 has time complexity O(log(n)) and space complexity O(log(n))
      //Runtime: 3 ms, faster than 85.71% of Go online submissions for Elimination Game.
      //Memory Usage: 2.6 MB, less than 97.62% of Go online submissions for Elimination Game.
      func LastRemaining2(n int) int {
      	return f(n)
      }

      func f(n int) int {
      	if n == 1 {
      		return 1
      	}
      	if n == 2 || n == 3 {
      		return 2
      	}

      	//经过一轮从左至右和从右至左消除后
      	last := last(n)                    //数列的最后一个数字
      	length := length(n)                //数列的长度
      	return last - 4*(length-f(length)) //返回数列中，编号为f(length)的数字
      }

      func last(n int) int {
      	return n - (n % 2) - 2
      }

      func length(n int) int {
      	n /= 2
      	n /= 2
      	return n
      }
      -

      时间复杂度和空间复杂度都是 O(log⁡n)\mathcal{O}(\log{}n) ，因为递归调用的次数为 log⁡4n=log⁡n2\log_{4}n = \dfrac{\log{}n}{2} 。

      -]]>       - - leetcode - 算法 - go - -       - - CLion2021调试Makefile项目 - /2021/07/15/2021-7-15-clion-makefile-debug/ - CLion介绍

      CLion是一款针对C/C++项目的跨平台的集成IDE（A cross-platform IDE for C and C++）。2020版本以前，只支持cmake项目， -但是2021版本对Makefile项目的支持度增加了。我们看看如何对Makefile项目进行断点调试。

      - - -

      我们用一个实际的项目作为例子。

      -

      Go1.4源码

      Go语言项目从源码编译有几种方式，其中一种方式是基于Bootstrap toolchain from C source code, -也就是说，首先编译Go 1.4版本，然后用编译出来的Go，编译最新的Go版本。(UPDATE: Mac12.1 Monterey系统上不支持)

      -

      那么我下载这个Go1.4版本，go1.4-bootstrap-20171003.tar.gz， -解压到某个路径下。

      -
      ~/work/go1.4/ ls          
      AUTHORS      LICENSE      README       api          doc          include      misc         robots.txt   test
      CONTRIBUTORS PATENTS      VERSION      bin          favicon.ico  lib          pkg          src
      - -

      进入src路径下，运行make.bash文件，则开始编译。（当然需要build相关的工具）。为了看清楚bash脚本执行的内容，我们修改make.bash的 -第一行改为 set -ex, 这样会打印详细的执行内容如下，

      -
      g1.4/src/ $  ./make.bash 
      + '[' '!' -f run.bash ']'
      + case "$(uname)" in
      ++ uname
      + ld --version
      + grep 'gold.* 2\.20'
      + for se_mount in /selinux /sys/fs/selinux
      + '[' -d /selinux -a -f /selinux/booleans/allow_execstack -a -x /usr/sbin/selinuxenabled ']'
      + for se_mount in /selinux /sys/fs/selinux
      + '[' -d /sys/fs/selinux -a -f /sys/fs/selinux/booleans/allow_execstack -a -x /usr/sbin/selinuxenabled ']'
      ++ uname -s
      + '[' Darwin == GNU/kFreeBSD ']'
      + rm -f ./runtime/runtime_defs.go
      + echo '# Building C bootstrap tool.'
      # Building C bootstrap tool.
      + echo cmd/dist
      cmd/dist
      ++ cd ..
      ++ pwd
      + export GOROOT=/Users/ym/work/go1.4
      + GOROOT=/Users/ym/work/go1.4
      + GOROOT_FINAL=/Users/ym/work/go1.4
      + DEFGOROOT='-DGOROOT_FINAL="/Users/ym/work/go1.4"'
      + mflag=
      + case "$GOHOSTARCH" in
      ++ uname
      + '[' Darwin == Darwin ']'
      + mflag=' -mmacosx-version-min=10.6'
      ++ type -t gcc
      ++ type -t clang
      + '[' -z '' -a -z file -a -n file ']'
      + gcc -mmacosx-version-min=10.6 -O2 -Wall -Werror -o cmd/dist/dist -Icmd/dist '-DGOROOT_FINAL="/Users/ym/work/go1.4"' cmd/dist/arm.c cmd/dist/buf.c cmd/dist/build.c cmd/dist/buildgc.c cmd/dist/buildgo.c cmd/dist/buildruntime.c cmd/dist/main.c cmd/dist/plan9.c cmd/dist/unix.c cmd/dist/windows.c
      ++ ./cmd/dist/dist env -p
      + eval 'CC="clang"' 'CC_FOR_TARGET="clang"' 'GOROOT="/Users/ym/work/go1.4"' 'GOBIN="/Users/ym/me/go/test/bin"' 'GOARCH="amd64"' 'GOOS="darwin"' 'GOHOSTARCH="amd64"' 'GOHOSTOS="darwin"' 'GOTOOLDIR="/Users/ym/work/go1.4/pkg/tool/darwin_amd64"' 'GOCHAR="6"' 'PATH="/Users/ym/me/go/test/bin:/usr/local/opt/openssl@1.1/bin:/usr/local/opt/postgresql@12/bin:/Users/ym/work/go1.4/bin:/usr/local/Homebrew/bin:/usr/local/sbin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Apple/usr/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Users/ym/mybin/apache-maven-3.6.0/bin:/Users/ym/me/go/test/bin:/Users/ym/.rvm/bin:/Users/ym/mybin/spark-2.4.3-bin-hadoop2.7/bin/"'
      ++ CC=clang
      ++ CC_FOR_TARGET=clang
      ++ GOROOT=/Users/ym/work/go1.4
      ++ GOBIN=/Users/ym/me/go/test/bin
      ++ GOARCH=amd64
      ++ GOOS=darwin
      ++ GOHOSTARCH=amd64
      ++ GOHOSTOS=darwin
      ++ GOTOOLDIR=/Users/ym/work/go1.4/pkg/tool/darwin_amd64
      ++ GOCHAR=6
      ++ PATH='/Users/ym/me/go/test/bin:/usr/local/opt/openssl@1.1/bin:/usr/local/opt/postgresql@12/bin:/Users/ym/work/go1.4/bin:/usr/local/Homebrew/bin:/usr/local/sbin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Apple/usr/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Users/ym/mybin/apache-maven-3.6.0/bin:/Users/ym/me/go/test/bin:/Users/ym/.rvm/bin:/Users/ym/mybin/spark-2.4.3-bin-hadoop2.7/bin/'
      + '[' '' = true ']'
      + echo

      + '[' '' = --dist-tool ']'
      + echo '# Building compilers and Go bootstrap tool for host, darwin/amd64.'
      # Building compilers and Go bootstrap tool for host, darwin/amd64.
      + buildall=-a
      + '[' '' = --no-clean ']'
      + ./cmd/dist/dist bootstrap -a -v
      lib9
      libbio
      liblink
      cmd/cc
      cmd/gc
      cmd/6l
      /Users/ym/work/go1.4/src/cmd/6l/../ld/dwarf.c:1479:15: warning: implicit conversion from 'int' to 'char' changes value from 156 to -100 [-Wconstant-conversion]
      /Users/ym/work/go1.4/src/cmd/6l/../ld/dwarf.c:1763:21: warning: implicit conversion from 'int' to 'char' changes value from 144 to -112 [-Wconstant-conversion]
      /Users/ym/work/go1.4/src/cmd/6l/../ld/lib.h:168:13: note: expanded from macro 'cput'
      cmd/6a
      cmd/6c
      /Users/ym/work/go1.4/src/cmd/6c/txt.c:995:28: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
      /Users/ym/work/go1.4/src/cmd/6c/txt.c:1045:28: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
      cmd/6g
      /Users/ym/work/go1.4/src/cmd/6g/peep.c:771:13: warning: converting the enum constant to a boolean [-Wint-in-bool-context]
      runtime
      errors
      sync/atomic
      sync
      io
      unicode
      unicode/utf8
      unicode/utf16
      bytes
      math
      strings
      strconv
      bufio
      sort
      container/heap
      encoding/base64
      syscall
      time
      os
      reflect
      fmt
      encoding
      encoding/json
      flag
      path/filepath
      path
      io/ioutil
      log
      regexp/syntax
      regexp
      go/token
      go/scanner
      go/ast
      go/parser
      os/exec
      os/signal
      net/url
      text/template/parse
      text/template
      go/doc
      go/build
      cmd/go
      + cp cmd/dist/dist /Users/ym/work/go1.4/pkg/tool/darwin_amd64/dist
      + /Users/ym/work/go1.4/pkg/tool/darwin_amd64/go_bootstrap clean -i std
      + echo

      + '[' amd64 '!=' amd64 -o darwin '!=' darwin ']'
      + echo '# Building packages and commands for darwin/amd64.'
      # Building packages and commands for darwin/amd64.
      + CC=clang
      + /Users/ym/work/go1.4/pkg/tool/darwin_amd64/go_bootstrap install -ccflags '' -gcflags '' -ldflags '' -v std
      runtime
      errors
      sync/atomic
      unicode
      unicode/utf8
      math
      sort
      encoding
      unicode/utf16
      container/list
      sync
      crypto/subtle
      container/ring
      image/color
      runtime/race
      container/heap
      io
      syscall
      image/color/palette
      hash
      crypto/cipher
      hash/crc32
      crypto/hmac
      hash/adler32
      hash/crc64
      hash/fnv
      bytes
      strings
      bufio
      text/tabwriter
      path
      html
      compress/bzip2
      time
      strconv
      math/rand
      math/cmplx
      os
      reflect
      regexp/syntax
      crypto
      encoding/base64
      net/url
      crypto/aes
      crypto/rc4
      crypto/md5
      crypto/sha1
      crypto/sha256
      crypto/sha512
      encoding/pem
      encoding/ascii85
      encoding/base32
      image
      path/filepath
      net
      os/signal
      io/ioutil
      os/exec
      regexp
      image/draw
      image/jpeg
      fmt
      encoding/binary
      cmd/pprof/internal/svg
      crypto/des
      index/suffixarray
      cmd/internal/goobj
      cmd/internal/rsc.io/arm/armasm
      cmd/internal/rsc.io/x86/x86asm
      debug/dwarf
      debug/gosym
      debug/plan9obj
      flag
      log
      go/token
      encoding/json
      encoding/xml
      text/template/parse
      go/scanner
      debug/elf
      debug/macho
      debug/pe
      go/ast
      compress/flate
      text/template
      math/big
      encoding/hex
      mime
      net/textproto
      cmd/internal/objfile
      net/http/internal
      compress/gzip
      runtime/pprof
      cmd/pack
      cmd/pprof/internal/profile
      cmd/pprof/internal/tempfile
      archive/tar
      archive/zip
      cmd/addr2line
      cmd/nm
      crypto/elliptic
      encoding/asn1
      crypto/rand
      go/parser
      go/printer
      go/doc
      crypto/ecdsa
      crypto/rsa
      crypto/dsa
      crypto/x509/pkix
      mime/multipart
      cmd/objdump
      cmd/pprof/internal/plugin
      crypto/x509
      html/template
      go/build
      cmd/cgo
      go/format
      cmd/fix
      cmd/gofmt
      crypto/tls
      cmd/pprof/internal/symbolizer
      cmd/pprof/internal/symbolz
      cmd/pprof/internal/report
      cmd/yacc
      compress/lzw
      compress/zlib
      database/sql/driver
      database/sql
      encoding/csv
      encoding/gob
      cmd/pprof/internal/commands
      image/gif
      cmd/pprof/internal/driver
      image/png
      log/syslog
      net/mail
      os/user
      runtime/debug
      testing
      testing/iotest
      net/http
      net/smtp
      testing/quick
      text/scanner
      cmd/go
      cmd/pprof/internal/fetch
      expvar
      net/http/cgi
      net/http/cookiejar
      net/http/httptest
      net/http/httputil
      net/http/pprof
      cmd/pprof
      net/rpc
      net/http/fcgi
      net/rpc/jsonrpc
      + echo

      + rm -f /Users/ym/work/go1.4/pkg/tool/darwin_amd64/go_bootstrap
      + '[' '' '!=' --no-banner ']'
      + /Users/ym/work/go1.4/pkg/tool/darwin_amd64/dist banner

      ---
      Installed Go for darwin/amd64 in /Users/ym/work/go1.4
      Installed commands in /Users/ym/me/go/test/bin
      - -

      我们看到15-33行, 首先编译了cmd/dist文件夹下的c代码，生成了一个dist可执行文件，并调用了 /cmd/dist/dist env -p

      -
      -# Building C bootstrap tool.
-+ echo cmd/dist
-cmd/dist
-++ cd ..
-++ pwd
-+ export GOROOT=/Users/ym/work/go1.4
-+ GOROOT=/Users/ym/work/go1.4
-+ GOROOT_FINAL=/Users/ym/work/go1.4
-+ DEFGOROOT='-DGOROOT_FINAL="/Users/ym/work/go1.4"'
-+ mflag=
-+ case "$GOHOSTARCH" in
-++ uname
-+ '[' Darwin == Darwin ']'
-+ mflag=' -mmacosx-version-min=10.6'
-++ type -t gcc
-++ type -t clang
-+ '[' -z '' -a -z file -a -n file ']'
-+ gcc -mmacosx-version-min=10.6 -O2 -Wall -Werror -o cmd/dist/dist -Icmd/dist '-DGOROOT_FINAL="/Users/ym/work/go1.4"' cmd/dist/arm.c cmd/dist/buf.c cmd/dist/build.c cmd/dist/buildgc.c cmd/dist/buildgo.c cmd/dist/buildruntime.c cmd/dist/main.c cmd/dist/plan9.c cmd/dist/unix.c cmd/dist/windows.c 
-++ ./cmd/dist/dist env -p
-
      - -

      那么我们来看看怎么在CLion中调试/cmd/dist/dist env -p

      -

      CLion调试cmd/dist

      如果要用CLion直接打开cmd/dist文件夹，会提示创建cmake项目，但是我们想直接使用调试Makefile项目。 -那么我们在cmd/dist目录下创建一个Makefile文件，内容如下

      -
      CFLAGS = -mmacosx-version-min=10.6 -g -Wall -Werror '-DGOROOT_FINAL="/Users/ym/work/go1.4"'
      CC = gcc

      SRC=arm.c buf.c build.c buildgc.c buildgo.c buildruntime.c main.c plan9.c unix.c windows.c
      INCLUDE_DIR = ./


      TARGET = dist

      all: $(TARGET)

      $(TARGET):
      	$(CC) $(CFLAGS) -I$(INCLUDE_DIR) $(SRC) -o $@


      clean:
      	rm $(TARGET)

      .PHONY: all clean
      - -

      其实就是根据make.bash打印的gcc编译命令改写的。需要注意的是要增加-g选项，这样才能调试。

      -

      然后参考Run/Debug Configuration: Makefile Application

      -

      配置几个参数，核心要指定的就是Target和Executable文件 -clion-debug-makefile-config

      -

      最后先clean，然后断点调试unix.c文件中的main入口函数。

      -

      clion-makefile-debug-show

      -

      成功了。

      -]]>       - - debug - clion - makefile - -       ASN.1语言规范介绍 /2020/11/08/asn-1/ @@ -2832,6 +2709,129 @@ byte & 0x7F 表示length的编码长度。 ASN.1 + + CLion2021调试Makefile项目 + /2021/07/15/2021-7-15-clion-makefile-debug/ + CLion介绍

      CLion是一款针对C/C++项目的跨平台的集成IDE（A cross-platform IDE for C and C++）。2020版本以前，只支持cmake项目， +但是2021版本对Makefile项目的支持度增加了。我们看看如何对Makefile项目进行断点调试。

      + + +

      我们用一个实际的项目作为例子。

      +

      Go1.4源码

      Go语言项目从源码编译有几种方式，其中一种方式是基于Bootstrap toolchain from C source code, +也就是说，首先编译Go 1.4版本，然后用编译出来的Go，编译最新的Go版本。(UPDATE: Mac12.1 Monterey系统上不支持)

      +

      那么我下载这个Go1.4版本，go1.4-bootstrap-20171003.tar.gz， +解压到某个路径下。

      +
      ~/work/go1.4/ ls          
      AUTHORS      LICENSE      README       api          doc          include      misc         robots.txt   test
      CONTRIBUTORS PATENTS      VERSION      bin          favicon.ico  lib          pkg          src
      + +

      进入src路径下，运行make.bash文件，则开始编译。（当然需要build相关的工具）。为了看清楚bash脚本执行的内容，我们修改make.bash的 +第一行改为 set -ex, 这样会打印详细的执行内容如下，

      +
      g1.4/src/ $  ./make.bash 
      + '[' '!' -f run.bash ']'
      + case "$(uname)" in
      ++ uname
      + ld --version
      + grep 'gold.* 2\.20'
      + for se_mount in /selinux /sys/fs/selinux
      + '[' -d /selinux -a -f /selinux/booleans/allow_execstack -a -x /usr/sbin/selinuxenabled ']'
      + for se_mount in /selinux /sys/fs/selinux
      + '[' -d /sys/fs/selinux -a -f /sys/fs/selinux/booleans/allow_execstack -a -x /usr/sbin/selinuxenabled ']'
      ++ uname -s
      + '[' Darwin == GNU/kFreeBSD ']'
      + rm -f ./runtime/runtime_defs.go
      + echo '# Building C bootstrap tool.'
      # Building C bootstrap tool.
      + echo cmd/dist
      cmd/dist
      ++ cd ..
      ++ pwd
      + export GOROOT=/Users/ym/work/go1.4
      + GOROOT=/Users/ym/work/go1.4
      + GOROOT_FINAL=/Users/ym/work/go1.4
      + DEFGOROOT='-DGOROOT_FINAL="/Users/ym/work/go1.4"'
      + mflag=
      + case "$GOHOSTARCH" in
      ++ uname
      + '[' Darwin == Darwin ']'
      + mflag=' -mmacosx-version-min=10.6'
      ++ type -t gcc
      ++ type -t clang
      + '[' -z '' -a -z file -a -n file ']'
      + gcc -mmacosx-version-min=10.6 -O2 -Wall -Werror -o cmd/dist/dist -Icmd/dist '-DGOROOT_FINAL="/Users/ym/work/go1.4"' cmd/dist/arm.c cmd/dist/buf.c cmd/dist/build.c cmd/dist/buildgc.c cmd/dist/buildgo.c cmd/dist/buildruntime.c cmd/dist/main.c cmd/dist/plan9.c cmd/dist/unix.c cmd/dist/windows.c
      ++ ./cmd/dist/dist env -p
      + eval 'CC="clang"' 'CC_FOR_TARGET="clang"' 'GOROOT="/Users/ym/work/go1.4"' 'GOBIN="/Users/ym/me/go/test/bin"' 'GOARCH="amd64"' 'GOOS="darwin"' 'GOHOSTARCH="amd64"' 'GOHOSTOS="darwin"' 'GOTOOLDIR="/Users/ym/work/go1.4/pkg/tool/darwin_amd64"' 'GOCHAR="6"' 'PATH="/Users/ym/me/go/test/bin:/usr/local/opt/openssl@1.1/bin:/usr/local/opt/postgresql@12/bin:/Users/ym/work/go1.4/bin:/usr/local/Homebrew/bin:/usr/local/sbin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Apple/usr/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Users/ym/mybin/apache-maven-3.6.0/bin:/Users/ym/me/go/test/bin:/Users/ym/.rvm/bin:/Users/ym/mybin/spark-2.4.3-bin-hadoop2.7/bin/"'
      ++ CC=clang
      ++ CC_FOR_TARGET=clang
      ++ GOROOT=/Users/ym/work/go1.4
      ++ GOBIN=/Users/ym/me/go/test/bin
      ++ GOARCH=amd64
      ++ GOOS=darwin
      ++ GOHOSTARCH=amd64
      ++ GOHOSTOS=darwin
      ++ GOTOOLDIR=/Users/ym/work/go1.4/pkg/tool/darwin_amd64
      ++ GOCHAR=6
      ++ PATH='/Users/ym/me/go/test/bin:/usr/local/opt/openssl@1.1/bin:/usr/local/opt/postgresql@12/bin:/Users/ym/work/go1.4/bin:/usr/local/Homebrew/bin:/usr/local/sbin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Apple/usr/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Users/ym/mybin/apache-maven-3.6.0/bin:/Users/ym/me/go/test/bin:/Users/ym/.rvm/bin:/Users/ym/mybin/spark-2.4.3-bin-hadoop2.7/bin/'
      + '[' '' = true ']'
      + echo

      + '[' '' = --dist-tool ']'
      + echo '# Building compilers and Go bootstrap tool for host, darwin/amd64.'
      # Building compilers and Go bootstrap tool for host, darwin/amd64.
      + buildall=-a
      + '[' '' = --no-clean ']'
      + ./cmd/dist/dist bootstrap -a -v
      lib9
      libbio
      liblink
      cmd/cc
      cmd/gc
      cmd/6l
      /Users/ym/work/go1.4/src/cmd/6l/../ld/dwarf.c:1479:15: warning: implicit conversion from 'int' to 'char' changes value from 156 to -100 [-Wconstant-conversion]
      /Users/ym/work/go1.4/src/cmd/6l/../ld/dwarf.c:1763:21: warning: implicit conversion from 'int' to 'char' changes value from 144 to -112 [-Wconstant-conversion]
      /Users/ym/work/go1.4/src/cmd/6l/../ld/lib.h:168:13: note: expanded from macro 'cput'
      cmd/6a
      cmd/6c
      /Users/ym/work/go1.4/src/cmd/6c/txt.c:995:28: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
      /Users/ym/work/go1.4/src/cmd/6c/txt.c:1045:28: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
      cmd/6g
      /Users/ym/work/go1.4/src/cmd/6g/peep.c:771:13: warning: converting the enum constant to a boolean [-Wint-in-bool-context]
      runtime
      errors
      sync/atomic
      sync
      io
      unicode
      unicode/utf8
      unicode/utf16
      bytes
      math
      strings
      strconv
      bufio
      sort
      container/heap
      encoding/base64
      syscall
      time
      os
      reflect
      fmt
      encoding
      encoding/json
      flag
      path/filepath
      path
      io/ioutil
      log
      regexp/syntax
      regexp
      go/token
      go/scanner
      go/ast
      go/parser
      os/exec
      os/signal
      net/url
      text/template/parse
      text/template
      go/doc
      go/build
      cmd/go
      + cp cmd/dist/dist /Users/ym/work/go1.4/pkg/tool/darwin_amd64/dist
      + /Users/ym/work/go1.4/pkg/tool/darwin_amd64/go_bootstrap clean -i std
      + echo

      + '[' amd64 '!=' amd64 -o darwin '!=' darwin ']'
      + echo '# Building packages and commands for darwin/amd64.'
      # Building packages and commands for darwin/amd64.
      + CC=clang
      + /Users/ym/work/go1.4/pkg/tool/darwin_amd64/go_bootstrap install -ccflags '' -gcflags '' -ldflags '' -v std
      runtime
      errors
      sync/atomic
      unicode
      unicode/utf8
      math
      sort
      encoding
      unicode/utf16
      container/list
      sync
      crypto/subtle
      container/ring
      image/color
      runtime/race
      container/heap
      io
      syscall
      image/color/palette
      hash
      crypto/cipher
      hash/crc32
      crypto/hmac
      hash/adler32
      hash/crc64
      hash/fnv
      bytes
      strings
      bufio
      text/tabwriter
      path
      html
      compress/bzip2
      time
      strconv
      math/rand
      math/cmplx
      os
      reflect
      regexp/syntax
      crypto
      encoding/base64
      net/url
      crypto/aes
      crypto/rc4
      crypto/md5
      crypto/sha1
      crypto/sha256
      crypto/sha512
      encoding/pem
      encoding/ascii85
      encoding/base32
      image
      path/filepath
      net
      os/signal
      io/ioutil
      os/exec
      regexp
      image/draw
      image/jpeg
      fmt
      encoding/binary
      cmd/pprof/internal/svg
      crypto/des
      index/suffixarray
      cmd/internal/goobj
      cmd/internal/rsc.io/arm/armasm
      cmd/internal/rsc.io/x86/x86asm
      debug/dwarf
      debug/gosym
      debug/plan9obj
      flag
      log
      go/token
      encoding/json
      encoding/xml
      text/template/parse
      go/scanner
      debug/elf
      debug/macho
      debug/pe
      go/ast
      compress/flate
      text/template
      math/big
      encoding/hex
      mime
      net/textproto
      cmd/internal/objfile
      net/http/internal
      compress/gzip
      runtime/pprof
      cmd/pack
      cmd/pprof/internal/profile
      cmd/pprof/internal/tempfile
      archive/tar
      archive/zip
      cmd/addr2line
      cmd/nm
      crypto/elliptic
      encoding/asn1
      crypto/rand
      go/parser
      go/printer
      go/doc
      crypto/ecdsa
      crypto/rsa
      crypto/dsa
      crypto/x509/pkix
      mime/multipart
      cmd/objdump
      cmd/pprof/internal/plugin
      crypto/x509
      html/template
      go/build
      cmd/cgo
      go/format
      cmd/fix
      cmd/gofmt
      crypto/tls
      cmd/pprof/internal/symbolizer
      cmd/pprof/internal/symbolz
      cmd/pprof/internal/report
      cmd/yacc
      compress/lzw
      compress/zlib
      database/sql/driver
      database/sql
      encoding/csv
      encoding/gob
      cmd/pprof/internal/commands
      image/gif
      cmd/pprof/internal/driver
      image/png
      log/syslog
      net/mail
      os/user
      runtime/debug
      testing
      testing/iotest
      net/http
      net/smtp
      testing/quick
      text/scanner
      cmd/go
      cmd/pprof/internal/fetch
      expvar
      net/http/cgi
      net/http/cookiejar
      net/http/httptest
      net/http/httputil
      net/http/pprof
      cmd/pprof
      net/rpc
      net/http/fcgi
      net/rpc/jsonrpc
      + echo

      + rm -f /Users/ym/work/go1.4/pkg/tool/darwin_amd64/go_bootstrap
      + '[' '' '!=' --no-banner ']'
      + /Users/ym/work/go1.4/pkg/tool/darwin_amd64/dist banner

      ---
      Installed Go for darwin/amd64 in /Users/ym/work/go1.4
      Installed commands in /Users/ym/me/go/test/bin
      + +

      我们看到15-33行, 首先编译了cmd/dist文件夹下的c代码，生成了一个dist可执行文件，并调用了 /cmd/dist/dist env -p

      +
      +# Building C bootstrap tool.
++ echo cmd/dist
+cmd/dist
+++ cd ..
+++ pwd
++ export GOROOT=/Users/ym/work/go1.4
++ GOROOT=/Users/ym/work/go1.4
++ GOROOT_FINAL=/Users/ym/work/go1.4
++ DEFGOROOT='-DGOROOT_FINAL="/Users/ym/work/go1.4"'
++ mflag=
++ case "$GOHOSTARCH" in
+++ uname
++ '[' Darwin == Darwin ']'
++ mflag=' -mmacosx-version-min=10.6'
+++ type -t gcc
+++ type -t clang
++ '[' -z '' -a -z file -a -n file ']'
++ gcc -mmacosx-version-min=10.6 -O2 -Wall -Werror -o cmd/dist/dist -Icmd/dist '-DGOROOT_FINAL="/Users/ym/work/go1.4"' cmd/dist/arm.c cmd/dist/buf.c cmd/dist/build.c cmd/dist/buildgc.c cmd/dist/buildgo.c cmd/dist/buildruntime.c cmd/dist/main.c cmd/dist/plan9.c cmd/dist/unix.c cmd/dist/windows.c 
+++ ./cmd/dist/dist env -p
+
      + +

      那么我们来看看怎么在CLion中调试/cmd/dist/dist env -p

      +

      CLion调试cmd/dist

      如果要用CLion直接打开cmd/dist文件夹，会提示创建cmake项目，但是我们想直接使用调试Makefile项目。 +那么我们在cmd/dist目录下创建一个Makefile文件，内容如下

      +
      CFLAGS = -mmacosx-version-min=10.6 -g -Wall -Werror '-DGOROOT_FINAL="/Users/ym/work/go1.4"'
      CC = gcc

      SRC=arm.c buf.c build.c buildgc.c buildgo.c buildruntime.c main.c plan9.c unix.c windows.c
      INCLUDE_DIR = ./


      TARGET = dist

      all: $(TARGET)

      $(TARGET):
      	$(CC) $(CFLAGS) -I$(INCLUDE_DIR) $(SRC) -o $@


      clean:
      	rm $(TARGET)

      .PHONY: all clean
      + +

      其实就是根据make.bash打印的gcc编译命令改写的。需要注意的是要增加-g选项，这样才能调试。

      +

      然后参考Run/Debug Configuration: Makefile Application

      +

      配置几个参数，核心要指定的就是Target和Executable文件 +clion-debug-makefile-config

      +

      最后先clean，然后断点调试unix.c文件中的main入口函数。

      +

      clion-makefile-debug-show

      +

      成功了。

      +]]>       + + debug + clion + makefile + +       + + [leetcode 390]Elimination Game原创解法 + /2022/02/06/2022-2-6-leetcode-390/ + 题目概述

      leetcode现在支持Go了，这次用Go写一个算法题——消除游戏。

      +

      原题链接

      +
      You have a list arr of all integers in the range [1, n] sorted in a strictly increasing order. Apply the following algorithm on arr:
+
+Starting from left to right, remove the first number and every other number afterward until you reach the end of the list.
+Repeat the previous step again, but this time from right to left, remove the rightmost number and every other number from the remaining numbers.
+Keep repeating the steps again, alternating left to right and right to left, until a single number remains.
+Given the integer n, return the last number that remains in arr.
+
      +

      Example 1:

      +

      Input: n = 9

      +

      Output: 6

      +

      Explanation:

      +

      arr = [1, 2, 3, 4, 5, 6, 7, 8, 9]

      +

      arr = [2, 4, 6, 8]

      +

      arr = [2, 6]

      +

      arr = [6]

      +

      简单说就是从左至右消除，再从右到左消除，如果不唯一继续这个过程。对于这个例子，输入是9，那么开始就有1-9， +9个数。第一次，消除{1,3,5,7,9}, 第二次消除{8,4},第三次消除{2},最后留下数字6。

      +

      老办法，先用naive、暴力的方法求解，再优化。

      + + +

      暴力求解方法

      基本思路就是

      +
        +
      1. 生成一个数组，含有1-n，n个整数。
        2. +
      2. 从左到右消除，就是把偶数位置的数字依序移动到数组的左侧，并截取保留。例如从左至右消除数组[1,2,3,4,5,6,7,8,9], +移动偶数位置到左侧后，[2,4,6,8,5,6,7,8,9], 截取左侧后为[2,4,6,8]
        3. +
      3. 从右至左消除，就是把奇数位置的数字依序移动到数组的右侧，并截取保留。例如从右至左消除数组[2,4,6,8],移动奇数 +位置的数到右侧后，[2,4,2,6], 截取右侧后为[2,6]
        4. +
      4. 如果不为1，继续步骤2
        5. +
      +

      代码如下

      +
      //LastRemaining1 has time complexity O(n) and space complexity O(n)
      func LastRemaining1(n int) int {
      	arr := gen(n)

      	leftToRight := true
      	for len(arr) > 1 {
      		if leftToRight {
      			for i := 1; i < len(arr); i += 2 {
      				arr[i/2] = arr[i]
      			}
      			length := len(arr) / 2
      			arr = arr[0:length]
      		} else {
      			count := 0
      			for i := 1; len(arr)-1-i >= 0; i += 2 {
      				arr[len(arr)-1-i/2] = arr[len(arr)-1-i]
      				count += 1
      			}
      			arr = arr[len(arr)-count:]
      		}

      		leftToRight = !leftToRight
      	}

      	return arr[0]
      }

      func gen(n int) []int {
      	arr := make([]int, n)
      	for i := 0; i < n; i++ {
      		arr[i] = i + 1
      	}
      	return arr
      }
      + +

      这个时间和空间复杂度都是 O(n)\mathcal{O}(n) ，空间复杂度是 O(n)\mathcal{O}(n) +好理解。那么时间复杂度是怎么计算的呢？ 对于输入n，第一次消除遍历 n2\dfrac{n}{2} ，第二次消除遍历 n22\dfrac{n}{2^2} ， +第三次消除遍历n23\dfrac{n}{2^3} …因此总时间就是

      +n2+n22+n23+…≈n\dfrac{n}{2} + \dfrac{n}{2^2} + \dfrac{n}{2^3} + \ldots \approx n + +

      分析和优化后的算法

      暴力解法直观，但是空间占用太多了，当n很大的时候，直接栈溢出了。那么该怎么优化呢。只能是找规律，通过发现递推关系来求解。

      +

      也就是说，假如我们要求解的函数为y=f(n)y=\mathcal{f}(n)，其中n就是输入的自然数。y为剩余的最后的数字。 +根据以往的经验我们考虑分治法，找到递推关系 f(n+1)=anf(n)+an−1f(n−1)+…\mathcal{f}(n+1) = a_n\mathcal{f}(n) + a_{n-1}\mathcal{f}(n-1) + \ldots

      +

      很显然，我们知道 f(1)=1\mathcal{f}(1) = 1 , f(2)=2\mathcal{f}(2) = 2 , f(3)=2\mathcal{f}(3) = 2

      +

      通过观察，我们可以发现这样一个递归过程，经过一轮从左至右和从右至左的消除后，余下一个长度为n4\dfrac{n}{4} 的新的数列。 +如果长度n4≤3\dfrac{n}{4} \leq 3 ，那么可以直接知道f(n4)\mathcal{f}(\dfrac{n}{4}) 的值，该值就是新数列的编号。 +取出新数列的该编号的值并返回。 +如果数列的长度大于3，那么返回新数列中编号为 f(n4)\mathcal{f}(\dfrac{n}{4}) 的数字。

      +

      也就是说我们发现了递推关系 f(n)=operator∗f(n4)\mathcal{f}(n) = operator*\mathcal{f}(\dfrac{n}{4})

      +

      举例说明我们的递归过程，当n=9时，经过一轮从左至右和从右至左的消除后，余下一个长度为2的数列[2,6], 因为f(2)=2\mathcal{f}(2) = 2 ， +所以返回数列[2,6]编号为2的数，也就是数字6。

      +

      代码实现如下

      +
      //LastRemaining2 has time complexity O(log(n)) and space complexity O(log(n))
      //Runtime: 3 ms, faster than 85.71% of Go online submissions for Elimination Game.
      //Memory Usage: 2.6 MB, less than 97.62% of Go online submissions for Elimination Game.
      func LastRemaining2(n int) int {
      	return f(n)
      }

      func f(n int) int {
      	if n == 1 {
      		return 1
      	}
      	if n == 2 || n == 3 {
      		return 2
      	}

      	//经过一轮从左至右和从右至左消除后
      	last := last(n)                    //数列的最后一个数字
      	length := length(n)                //数列的长度
      	return last - 4*(length-f(length)) //返回数列中，编号为f(length)的数字
      }

      func last(n int) int {
      	return n - (n % 2) - 2
      }

      func length(n int) int {
      	n /= 2
      	n /= 2
      	return n
      }
      +

      时间复杂度和空间复杂度都是 O(log⁡n)\mathcal{O}(\log{}n) ，因为递归调用的次数为 log⁡4n=log⁡n2\log_{4}n = \dfrac{\log{}n}{2} 。

      +]]>       + + leetcode + 算法 + go + +       通过ChatGPT实现Rust语言的八皇后算法 /2023/01/12/2023-1-12-8-queens-chatgpt/ diff --git a/sitemap.xml b/sitemap.xml index 95660f45..73720616 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -83,7 +83,7 @@ - https://threelambda.com/2022/02/06/2022-2-6-leetcode-390/ + https://threelambda.com/2021/07/15/2021-7-15-clion-makefile-debug/ 2024-07-22 @@ -92,7 +92,7 @@ - https://threelambda.com/2021/07/15/2021-7-15-clion-makefile-debug/ + https://threelambda.com/2022/02/06/2022-2-6-leetcode-390/ 2024-07-22 @@ -119,7 +119,7 @@ - https://threelambda.com/2020/10/09/x509-ca/ + https://threelambda.com/2020/09/08/remote-debug-with-clion/ 2024-07-22 @@ -128,7 +128,7 @@ - https://threelambda.com/2020/09/08/remote-debug-with-clion/ + https://threelambda.com/2020/10/09/x509-ca/ 2024-07-22 @@ -146,7 +146,7 @@ - https://threelambda.com/2020/07/06/run-ingress-example-on-mac/ + https://threelambda.com/2020/08/14/write-file-with-spring-integration/ 2024-07-22 @@ -155,7 +155,7 @@ - https://threelambda.com/2020/08/14/write-file-with-spring-integration/ + https://threelambda.com/2020/07/06/run-ingress-example-on-mac/ 2024-07-22 @@ -164,7 +164,7 @@ - https://threelambda.com/2019/02/18/2019-2-18-bt-6/ + https://threelambda.com/2019/04/28/2019-4-28-byte-to-string-and-back/ 2024-07-22 @@ -173,7 +173,7 @@ - https://threelambda.com/2019/04/28/2019-4-28-byte-to-string-and-back/ + https://threelambda.com/2019/02/18/2019-2-18-bt-6/ 2024-07-22 @@ -182,7 +182,7 @@ - https://threelambda.com/2019/11/19/debug-k3s/ + https://threelambda.com/2019/01/25/2019-1-25-bt-5/ 2024-07-22 @@ -200,7 +200,7 @@ - https://threelambda.com/2019/01/25/2019-1-25-bt-5/ + https://threelambda.com/2019/11/19/debug-k3s/ 2024-07-22 @@ -317,7 +317,7 @@ - https://threelambda.com/2017/11/27/find-top-n/ + https://threelambda.com/2017/03/12/2017-3-12-leetcode-329/ 2024-07-22 @@ -326,7 +326,7 @@ - https://threelambda.com/2017/03/12/2017-3-12-leetcode-329/ + https://threelambda.com/2017/11/27/find-top-n/ 2024-07-22 @@ -389,7 +389,7 @@ - https://threelambda.com/2016/12/31/leetcode-327/ + https://threelambda.com/2016/09/27/2016-9-27-leetcode-200/ 2024-07-22 @@ -398,7 +398,7 @@ - https://threelambda.com/2016/09/27/2016-9-27-leetcode-200/ + https://threelambda.com/2016/12/31/leetcode-327/ 2024-07-22 @@ -407,7 +407,7 @@ - https://threelambda.com/2016/12/21/get-val-name-and-value/ + https://threelambda.com/2016/12/22/pandas-read-csv/ 2024-07-22 @@ -416,7 +416,7 @@ - https://threelambda.com/2016/12/22/pandas-read-csv/ + https://threelambda.com/2016/12/21/get-val-name-and-value/ 2024-07-22 @@ -473,14 +473,14 @@ - https://threelambda.com/tags/R/ + https://threelambda.com/tags/%E6%8A%80%E5%B7%A7/ 2024-07-22 weekly 0.2 - https://threelambda.com/tags/%E6%8A%80%E5%B7%A7/ + https://threelambda.com/tags/R/ 2024-07-22 weekly 0.2 @@ -662,112 +662,112 @@ - https://threelambda.com/tags/k8s/ + https://threelambda.com/tags/spring/ 2024-07-22 weekly 0.2 - https://threelambda.com/tags/ingress/ + https://threelambda.com/tags/integration/ 2024-07-22 weekly 0.2 - https://threelambda.com/tags/mac/ + https://threelambda.com/tags/write/ 2024-07-22 weekly 0.2 - https://threelambda.com/tags/spring/ + https://threelambda.com/tags/file/ 2024-07-22 weekly 0.2 - https://threelambda.com/tags/integration/ + https://threelambda.com/tags/debug/ 2024-07-22 weekly 0.2 - https://threelambda.com/tags/write/ + https://threelambda.com/tags/CLion/ 2024-07-22 weekly 0.2 - https://threelambda.com/tags/file/ + https://threelambda.com/tags/gdb/ 2024-07-22 weekly 0.2 - https://threelambda.com/tags/certificate/ + https://threelambda.com/tags/remote/ 2024-07-22 weekly 0.2 - https://threelambda.com/tags/X-509/ + https://threelambda.com/tags/certificate/ 2024-07-22 weekly 0.2 - https://threelambda.com/tags/nginx/ + https://threelambda.com/tags/X-509/ 2024-07-22 weekly 0.2 - https://threelambda.com/tags/openssl/ + https://threelambda.com/tags/nginx/ 2024-07-22 weekly 0.2 - https://threelambda.com/tags/ssl/ + https://threelambda.com/tags/openssl/ 2024-07-22 weekly 0.2 - https://threelambda.com/tags/debug/ + https://threelambda.com/tags/ssl/ 2024-07-22 weekly 0.2 - https://threelambda.com/tags/CLion/ + https://threelambda.com/tags/k8s/ 2024-07-22 weekly 0.2 - https://threelambda.com/tags/gdb/ + https://threelambda.com/tags/ingress/ 2024-07-22 weekly 0.2 - https://threelambda.com/tags/remote/ + https://threelambda.com/tags/mac/ 2024-07-22 weekly 0.2 @@ -788,7 +788,7 @@ - https://threelambda.com/tags/go/ + https://threelambda.com/tags/ASN-1/ 2024-07-22 weekly 0.2 @@ -809,7 +809,7 @@ - https://threelambda.com/tags/ASN-1/ + https://threelambda.com/tags/go/ 2024-07-22 weekly 0.2 diff --git a/tags/CLion/index.html b/tags/CLion/index.html index c9915b1c..d0996b03 100644 --- a/tags/CLion/index.html +++ b/tags/CLion/index.html @@ -385,5 +385,3 @@

      搜索

      -dy> -