elastic search 编译和调试

背景

熟悉elastic search

构建

./gradlew localDistro

• 第一步： 关闭安全相关检查，我本地是http，不需要https

### config/elasticsearch.yml 的这个选项改成false ， 这样可以关闭https校验
# Enable security features
xpack.security.enabled: false

• 改代码 elasticsearch/distribution/tools/server-cli/src/main/java/org/elasticsearch/server/cli/ServerProcess.java 添加jdb相关参数：

        // also pass through distribution type
        jvmOptions.add("-Des.distribution.type=" + processInfo.sysprops().get("es.distribution.type"));
        jvmOptions.add("-agentlib:jdwp=transport=dt_socket,server=y,address=9999");   // 添加这一行 ， 让jdb可以调试

• 运行elasticsearch , 这时候会卡在启动的时候，等待jdb连接

./elasticsearch

• 使用jdb调试

使用gradle 拉取代码之后,需要获取lucene的代码,并解压到/home/ubuntu/lucene目录

jar -xf /home/ubuntu/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/9.7.0/35359f1763c9d7a0f04188c4933311be3c07b60e/lucene-core-9.7.0-sources.jar

## 这里/home/dai/ 是我放elasticsearch 的地方 , `/home/ubuntu/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/9.7.0/35359f1763c9d7a0f04188c4933311be3c07b60e/` 是我的gradle 的默认下载路径,可以自己用find去找
jdb -attach 9999  -sourcepath  /home/ubuntu/elasticsearch/distribution/tools/cli-launcher/src/main/java/:/home/ubuntu/elasticsearch/server/src/main/java/:/home/ubuntu/lucene/

• 断点

stop in org.elasticsearch.rest.action.search.RestSearchAction.prepareRequest

堆栈

使用jdb调试：

断点是：stop in org.elasticsearch.rest.action.search.RestSearchAction.prepareRequest

Breakpoint hit: 
Breakpoint hit: "thread=elasticsearch[myhost][transport_worker][T#5]", org.elasticsearch.rest.action.search.RestSearchAction.prepareRequest(), line=100 bci=0
100            if (request.hasParam("min_compatible_shard_node")) {

elasticsearch[myhost][transport_worker][T#5][1] where
  [1] org.elasticsearch.rest.action.search.RestSearchAction.prepareRequest (RestSearchAction.java:100)
  [2] org.elasticsearch.rest.BaseRestHandler.handleRequest (BaseRestHandler.java:80)
  [3] org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest (SecurityRestFilter.java:96)
  [4] org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest (SecurityRestFilter.java:76)
  [5] org.elasticsearch.rest.RestController.dispatchRequest (RestController.java:414)
  [6] org.elasticsearch.rest.RestController.tryAllHandlers (RestController.java:543)
  [7] org.elasticsearch.rest.RestController.dispatchRequest (RestController.java:316)
  [8] org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest (AbstractHttpServerTransport.java:453)
  [9] org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest (AbstractHttpServerTransport.java:549)
  [10] org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest (AbstractHttpServerTransport.java:426)
  [11] org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest (Netty4HttpPipeliningHandler.java:128)
  [12] org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead (Netty4HttpPipeliningHandler.java:118)
  [13] io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:442)
  [14] io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:420)
  [15] io.netty.channel.AbstractChannelHandlerContext.fireChannelRead (AbstractChannelHandlerContext.java:412)
  [16] io.netty.handler.codec.MessageToMessageDecoder.channelRead (MessageToMessageDecoder.java:103)
  [17] io.netty.handler.codec.MessageToMessageCodec.channelRead (MessageToMessageCodec.java:111)
  [18] io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:442)
  [19] io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:420)
  [20] io.netty.channel.AbstractChannelHandlerContext.fireChannelRead (AbstractChannelHandlerContext.java:412)
  [21] io.netty.handler.codec.MessageToMessageDecoder.channelRead (MessageToMessageDecoder.java:103)
  [22] io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:444)
  [23] io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:420)
  [24] io.netty.channel.AbstractChannelHandlerContext.fireChannelRead (AbstractChannelHandlerContext.java:412)
  [25] io.netty.handler.codec.MessageToMessageDecoder.channelRead (MessageToMessageDecoder.java:103)
  [26] io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:444)
  [27] io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:420)
  [28] io.netty.channel.AbstractChannelHandlerContext.fireChannelRead (AbstractChannelHandlerContext.java:412)
  [29] io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead (ByteToMessageDecoder.java:346)
  [30] io.netty.handler.codec.ByteToMessageDecoder.channelRead (ByteToMessageDecoder.java:318)
  [31] io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:444)
  [32] io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:420)
  [33] io.netty.channel.AbstractChannelHandlerContext.fireChannelRead (AbstractChannelHandlerContext.java:412)
  [34] io.netty.handler.codec.MessageToMessageDecoder.channelRead (MessageToMessageDecoder.java:103)
  [35] io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:444)
  [36] io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:420)
  [37] io.netty.channel.AbstractChannelHandlerContext.fireChannelRead (AbstractChannelHandlerContext.java:412)
  [38] io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead (DefaultChannelPipeline.java:1,410)
  [39] io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:440)
  [40] io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:420)
  [41] io.netty.channel.DefaultChannelPipeline.fireChannelRead (DefaultChannelPipeline.java:919)
  [42] io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read (AbstractNioByteChannel.java:166)
  [43] io.netty.channel.nio.NioEventLoop.processSelectedKey (NioEventLoop.java:788)
  [44] io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain (NioEventLoop.java:689)
  [45] io.netty.channel.nio.NioEventLoop.processSelectedKeys (NioEventLoop.java:652)
  [46] io.netty.channel.nio.NioEventLoop.run (NioEventLoop.java:562)
  [47] io.netty.util.concurrent.SingleThreadEventExecutor$4.run (SingleThreadEventExecutor.java:997)
  [48] io.netty.util.internal.ThreadExecutorMap$2.run (ThreadExecutorMap.java:74)
  [49] java.lang.Thread.runWith (Thread.java:1,636)
  [50] java.lang.Thread.run (Thread.java:1,623)

query phase

elasticsearch[myhost][search][T#7][1] where
  [1] org.elasticsearch.search.query.QueryPhase.execute (QueryPhase.java:62)
  [2] org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase (SearchService.java:516)
  [3] org.elasticsearch.search.SearchService.executeQueryPhase (SearchService.java:668)
  [4] org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2 (SearchService.java:541)
  [5] org.elasticsearch.search.SearchService$$Lambda$7604/0x00007fceb1297320.get (null)
  [6] org.elasticsearch.action.ActionRunnable$2.accept (ActionRunnable.java:51)
  [7] org.elasticsearch.action.ActionRunnable$2.accept (ActionRunnable.java:48)
  [8] org.elasticsearch.action.ActionRunnable$3.doRun (ActionRunnable.java:73)
  [9] org.elasticsearch.common.util.concurrent.AbstractRunnable.run (AbstractRunnable.java:26)
  [10] org.elasticsearch.common.util.concurrent.TimedRunnable.doRun (TimedRunnable.java:33)
  [11] org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun (ThreadContext.java:983)
  [12] org.elasticsearch.common.util.concurrent.AbstractRunnable.run (AbstractRunnable.java:26)
  [13] java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1,144)
  [14] java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:642)
  [15] java.lang.Thread.runWith (Thread.java:1,636)
  [16] java.lang.Thread.run (Thread.java:1,623)
elasticsearch[myhost][search][T#7][1]

查看reader

Breakpoint hit: "thread=elasticsearch[myhost][search][T#7]", org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(), line=150 bci=0
150            final ContextIndexSearcher searcher = searchContext.searcher();

elasticsearch[myhost][search][T#7][1] next
> 
Step completed: "thread=elasticsearch[myhost][search][T#7]", org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(), line=151 bci=5
151            final IndexReader reader = searcher.getIndexReader();

elasticsearch[myhost][search][T#7][1] next
> 
Step completed: "thread=elasticsearch[myhost][search][T#7]", org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(), line=152 bci=10
152            QuerySearchResult queryResult = searchContext.queryResult();

elasticsearch[myhost][search][T#7][1] print reader
 reader = "ExitableDirectoryReader(FilterLeafReader(FieldUsageTrackingLeafReader(reader=FilterLeafReader(_0(9.7.0):c1:[diagnostics={timestamp=1694357055349, source=flush, lucene.version=9.7.0, os.version=6.2.0-31-generic, os.arch=amd64, os=Linux, java.vendor=Oracle Corporation, java.runtime.version=20.0.2+9-78}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=bv125vla2ovjxnipt5j9ssmby))))"
elasticsearch[myhost][search][T#7][1] 

search

elasticsearch[myhost][search][T#7][1] print query
 query = "age:38"
elasticsearch[myhost][search][T#7][1] step in
> 
Step completed: "thread=elasticsearch[myhost][search][T#7]", org.elasticsearch.search.internal.ContextIndexSearcher.search(), line=340 bci=0
340            final C firstCollector = collectorManager.newCollector();

elasticsearch[myhost][search][T#7][1] where 
  [1] org.elasticsearch.search.internal.ContextIndexSearcher.search (ContextIndexSearcher.java:340)
  [2] org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch (QueryPhase.java:206)
  [3] org.elasticsearch.search.query.QueryPhase.executeQuery (QueryPhase.java:134)
  [4] org.elasticsearch.search.query.QueryPhase.execute (QueryPhase.java:63)
  [5] org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase (SearchService.java:516)
  [6] org.elasticsearch.search.SearchService.executeQueryPhase (SearchService.java:668)
  [7] org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2 (SearchService.java:541)
  [8] org.elasticsearch.search.SearchService$$Lambda$7604/0x00007fceb1297320.get (null)
  [9] org.elasticsearch.action.ActionRunnable$2.accept (ActionRunnable.java:51)
  [10] org.elasticsearch.action.ActionRunnable$2.accept (ActionRunnable.java:48)
  [11] org.elasticsearch.action.ActionRunnable$3.doRun (ActionRunnable.java:73)
  [12] org.elasticsearch.common.util.concurrent.AbstractRunnable.run (AbstractRunnable.java:26)
  [13] org.elasticsearch.common.util.concurrent.TimedRunnable.doRun (TimedRunnable.java:33)
  [14] org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun (ThreadContext.java:983)
  [15] org.elasticsearch.common.util.concurrent.AbstractRunnable.run (AbstractRunnable.java:26)
  [16] java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1,144)
  [17] java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:642)
  [18] java.lang.Thread.runWith (Thread.java:1,636)
  [19] java.lang.Thread.run (Thread.java:1,623)

bulkScorer 和Scorer

核心变成了这两个函数:

• bulkScorer
• scorer

           BulkScorer bulkScorer = weight.bulkScorer(ctx);
            if (bulkScorer != null) {
                if (cancellable.isEnabled()) {
                    bulkScorer = new CancellableBulkScorer(bulkScorer, cancellable::checkCancelled);
                }
                try {
                    bulkScorer.score(leafCollector, liveDocs);
                } catch (CollectionTerminatedException e) {
                  ...
                }

这里的weight:

elasticsearch[ubuntu-Vostro-3690][search_worker][T#2][1] dump weight
 weight = {
    similarity: instance of org.elasticsearch.index.similarity.SimilarityService$PerFieldSimilarity(id=25223)
    simScorer: instance of org.apache.lucene.search.similarities.BM25Similarity$BM25Scorer(id=25224)
    termStates: instance of org.apache.lucene.index.TermStates(id=25225)
    scoreMode: instance of org.apache.lucene.search.ScoreMode(id=25226)
    $assertionsDisabled: true
    this$0: instance of org.apache.lucene.search.TermQuery(id=25227)
    org.apache.lucene.search.Weight.parentQuery: instance of org.apache.lucene.search.TermQuery(id=25227)
}

搜索过程

elasticsearch[myhost][search_worker][T#5][1] list
246          // float. And then monotonicity is preserved through composition via
247          // x -> 1 + x and x -> 1 - 1/x.
248          // Finally we expand weight * (1 - 1 / (1 + freq * 1/norm)) to
249          // weight - weight / (1 + freq * 1/norm), which runs slightly faster.
250 =>       float normInverse = cache[((byte) encodedNorm) & 0xFF];
251          return weight - weight / (1f + freq * normInverse);
252        }
253    
254        @Override
255        public Explanation explain(Explanation freq, long encodedNorm) {
elasticsearch[myhost][search_worker][T#5][1] where
  [1] org.apache.lucene.search.similarities.BM25Similarity$BM25Scorer.score (BM25Similarity.java:250)
  [2] org.apache.lucene.search.LeafSimScorer.score (LeafSimScorer.java:60)
  [3] org.apache.lucene.search.TermScorer.score (TermScorer.java:75)
  [4] org.apache.lucene.search.TopScoreDocCollector$SimpleTopScoreDocCollector$1.collect (TopScoreDocCollector.java:73)
  [5] org.apache.lucene.search.Weight$DefaultBulkScorer.scoreRange (Weight.java:274)
  [6] org.apache.lucene.search.Weight$DefaultBulkScorer.score (Weight.java:254)
  [7] org.elasticsearch.search.internal.CancellableBulkScorer.score (CancellableBulkScorer.java:45)
  [8] org.apache.lucene.search.BulkScorer.score (BulkScorer.java:38)
  [9] org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf (ContextIndexSearcher.java:538)
  [10] org.elasticsearch.search.internal.ContextIndexSearcher.search (ContextIndexSearcher.java:480)
  [11] org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4 (ContextIndexSearcher.java:396)
  [12] org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda$7626/0x00007fceb12a4e58.call (null)
  [13] java.util.concurrent.FutureTask.run (FutureTask.java:317)
  [14] org.elasticsearch.common.util.concurrent.TimedRunnable.doRun (TimedRunnable.java:33)
  [15] org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun (ThreadContext.java:983)
  [16] org.elasticsearch.common.util.concurrent.AbstractRunnable.run (AbstractRunnable.java:26)
  [17] java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1,144)
  [18] java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:642)
  [19] java.lang.Thread.runWith (Thread.java:1,636)
  [20] java.lang.Thread.run (Thread.java:1,623)

es 常用 crud

• 写入

curl -XPOST http://localhost:9200/test/_doc -H "Content-Type: application/json" -d  '{"name":"John Smith","age":"38"}'

• 查询

curl -X GET  -H "Content-Type: application/json"    http://localhost:9200/test/_search -d '{"query":{"match":{"age":"38"}}}'

相关阅读

• https://intellij-support.jetbrains.com/hc/en-us/community/posts/360006994800-Automatically-download-sources-of-dependencies-with-gradle
• https://stackoverflow.com/questions/28404149/how-to-download-javadocs-and-sources-for-jar-using-gradle-2-0