数据库执行一段时间后，有大量假死的查询出现

[xufwind95]

问题描述

• iotdb在线上运行一段时候后，出现很多长时间不终止的查询
• 不做任何处理再次执行和这些查询相关的语句(使用相同的timeseries)，新查询也会失去响应
• 查询其它的未出现在这里面的timeseries，也会出现这种情况
• 查询数据库，能找到很多这种处于假死状态的查询语句
  

配置

• 系统 CentOS Linux release 7.6.1810
• iotdb版本: 0.12.4 单节点

业务数据流程

• 使用etl工具streamsets，从kafka抽取数据，通过mqtt协议写入到iotdb
• 下游为多个客户端执行分析查询，存在多个客户端查询同一个timeseries的情况
• 上游从kafka到iotdb的流程，从监控看一直是正常的

[JackieTien97]

方便用jstack -l pid > jstack.log把iotdb目前的堆栈打出来，放到这里不，我们看一下。

另外，可以把处于假死的查询sql复制一下，粘贴在issue里。

不做任何处理再次执行和这些查询相关的语句(使用相同的timeseries)，新查询也会失去响应
查询其它的未出现在这里面的timeseries，也会出现这种情况

这两句的意思是说，目前iotdb无法响应如何查询吗

[xufwind95]

• KILL QUERY <QUERY_ID> 后，这些假死的查询仍然存在；
• 日志中能看到: WARN o.a.i.d.q.c.QueryTimeManager:71 - Query is time out (2000ms) with queryId 45057 ,但是连接始终无法释放
• 重启后假死查询全部消失

[xufwind95]

刚重启了，看不到之前的stack日志，重启完，在页面狂点了几下，又出现这种情况了，下面是部分stack日志，不够的话我找QQ群把stack日志文件传过来：
"logback-1" #24 daemon prio=5 os_prio=0 tid=0x00007fd08c654000 nid=0xac38 waiting on condition [0x00007fbfb5f28000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007fc09d9697f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers:
- None

"Service Thread" #23 daemon prio=9 os_prio=0 tid=0x00007fd08c152000 nid=0xac36 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C1 CompilerThread17" #22 daemon prio=9 os_prio=0 tid=0x00007fd08c14f000 nid=0xac35 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C1 CompilerThread16" #21 daemon prio=9 os_prio=0 tid=0x00007fd08c14d000 nid=0xac34 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C1 CompilerThread15" #20 daemon prio=9 os_prio=0 tid=0x00007fd08c14a800 nid=0xac33 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00007fd08c148800 nid=0xac32 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C1 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00007fd08c146800 nid=0xac31 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C1 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00007fd08c144000 nid=0xac30 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C2 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007fd08c142000 nid=0xac2f waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C2 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007fd08c140000 nid=0xac2e waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007fd08c13e000 nid=0xac2d waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007fd08c13c000 nid=0xac2c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007fd08c139800 nid=0xac2b waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007fd08c137800 nid=0xac2a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007fd08c135000 nid=0xac29 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007fd08c12b000 nid=0xac28 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007fd08c128800 nid=0xac27 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd08c126800 nid=0xac26 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd08c124800 nid=0xac25 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd08c121800 nid=0xac24 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd08c120000 nid=0xac23 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd08c0ed000 nid=0xac22 in Object.wait() [0x00007fbfb76f5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007fc09d993f90> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x00007fc09d993f90> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

Locked ownable synchronizers:
- None

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd08c0ea800 nid=0xac21 in Object.wait() [0x00007fbfb77f6000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007fc09d99b6b8> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00007fc09d99b6b8> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

Locked ownable synchronizers:
- None

"VM Thread" os_prio=0 tid=0x00007fd08c0e0800 nid=0xac20 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd08c021000 nid=0xabe8 runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd08c022800 nid=0xabe9 runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd08c024800 nid=0xabea runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd08c026800 nid=0xabeb runnable

"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007fd08c028000 nid=0xabec runnable

"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007fd08c02a000 nid=0xabed runnable

"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007fd08c02b800 nid=0xabee runnable

"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007fd08c02d800 nid=0xabef runnable

"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007fd08c02f800 nid=0xabf0 runnable

"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007fd08c031000 nid=0xabf1 runnable

"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007fd08c033000 nid=0xabf2 runnable

"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007fd08c035000 nid=0xabf3 runnable

"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007fd08c036800 nid=0xabf4 runnable

"GC task thread#13 (ParallelGC)" os_prio=0 tid=0x00007fd08c038800 nid=0xabf5 runnable

"GC task thread#14 (ParallelGC)" os_prio=0 tid=0x00007fd08c03a800 nid=0xabf6 runnable

"GC task thread#15 (ParallelGC)" os_prio=0 tid=0x00007fd08c03c000 nid=0xabf7 runnable

"GC task thread#16 (ParallelGC)" os_prio=0 tid=0x00007fd08c03e000 nid=0xabf8 runnable

"GC task thread#17 (ParallelGC)" os_prio=0 tid=0x00007fd08c040000 nid=0xabf9 runnable

"GC task thread#18 (ParallelGC)" os_prio=0 tid=0x00007fd08c041800 nid=0xabfa runnable

"GC task thread#19 (ParallelGC)" os_prio=0 tid=0x00007fd08c043800 nid=0xabfb runnable

"GC task thread#20 (ParallelGC)" os_prio=0 tid=0x00007fd08c045000 nid=0xabfc runnable

"GC task thread#21 (ParallelGC)" os_prio=0 tid=0x00007fd08c047000 nid=0xabfd runnable

"GC task thread#22 (ParallelGC)" os_prio=0 tid=0x00007fd08c049000 nid=0xabfe runnable

"GC task thread#23 (ParallelGC)" os_prio=0 tid=0x00007fd08c04a800 nid=0xabff runnable

"GC task thread#24 (ParallelGC)" os_prio=0 tid=0x00007fd08c04c800 nid=0xac00 runnable

"GC task thread#25 (ParallelGC)" os_prio=0 tid=0x00007fd08c04e800 nid=0xac02 runnable

"GC task thread#26 (ParallelGC)" os_prio=0 tid=0x00007fd08c050000 nid=0xac03 runnable

"GC task thread#27 (ParallelGC)" os_prio=0 tid=0x00007fd08c052000 nid=0xac04 runnable

"GC task thread#28 (ParallelGC)" os_prio=0 tid=0x00007fd08c054000 nid=0xac05 runnable

"GC task thread#29 (ParallelGC)" os_prio=0 tid=0x00007fd08c055800 nid=0xac06 runnable

"GC task thread#30 (ParallelGC)" os_prio=0 tid=0x00007fd08c057800 nid=0xac07 runnable

"GC task thread#31 (ParallelGC)" os_prio=0 tid=0x00007fd08c059800 nid=0xac08 runnable

"GC task thread#32 (ParallelGC)" os_prio=0 tid=0x00007fd08c05b000 nid=0xac09 runnable

"GC task thread#33 (ParallelGC)" os_prio=0 tid=0x00007fd08c05d000 nid=0xac0a runnable

"GC task thread#34 (ParallelGC)" os_prio=0 tid=0x00007fd08c05e800 nid=0xac0b runnable

"GC task thread#35 (ParallelGC)" os_prio=0 tid=0x00007fd08c060800 nid=0xac0d runnable

"GC task thread#36 (ParallelGC)" os_prio=0 tid=0x00007fd08c062800 nid=0xac0e runnable

"GC task thread#37 (ParallelGC)" os_prio=0 tid=0x00007fd08c064000 nid=0xac0f runnable

"GC task thread#38 (ParallelGC)" os_prio=0 tid=0x00007fd08c066000 nid=0xac10 runnable

"GC task thread#39 (ParallelGC)" os_prio=0 tid=0x00007fd08c068000 nid=0xac11 runnable

"GC task thread#40 (ParallelGC)" os_prio=0 tid=0x00007fd08c069800 nid=0xac12 runnable

"GC task thread#41 (ParallelGC)" os_prio=0 tid=0x00007fd08c06b800 nid=0xac13 runnable

"GC task thread#42 (ParallelGC)" os_prio=0 tid=0x00007fd08c06d800 nid=0xac14 runnable

"GC task thread#43 (ParallelGC)" os_prio=0 tid=0x00007fd08c06f000 nid=0xac15 runnable

"GC task thread#44 (ParallelGC)" os_prio=0 tid=0x00007fd08c071000 nid=0xac16 runnable

"GC task thread#45 (ParallelGC)" os_prio=0 tid=0x00007fd08c072800 nid=0xac17 runnable

"GC task thread#46 (ParallelGC)" os_prio=0 tid=0x00007fd08c074800 nid=0xac18 runnable

"GC task thread#47 (ParallelGC)" os_prio=0 tid=0x00007fd08c076800 nid=0xac19 runnable

"GC task thread#48 (ParallelGC)" os_prio=0 tid=0x00007fd08c078000 nid=0xac1a runnable

"GC task thread#49 (ParallelGC)" os_prio=0 tid=0x00007fd08c07a000 nid=0xac1c runnable

"GC task thread#50 (ParallelGC)" os_prio=0 tid=0x00007fd08c07c000 nid=0xac1d runnable

"GC task thread#51 (ParallelGC)" os_prio=0 tid=0x00007fd08c07d800 nid=0xac1e runnable

"GC task thread#52 (ParallelGC)" os_prio=0 tid=0x00007fd08c07f800 nid=0xac1f runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007fd08c157000 nid=0xac37 waiting on condition

JNI global references: 496

[xufwind95]

这次出错执行的sql是这个: select time, XXX_point_name from root.abc.def order by time desc limit 10 offset 1947370;
这个报错后，当前执行跟这个timeseires有关的其它查询还能执行，像 select time, XXX_point_name from root.abc.def order by time desc limit 10;
长时间过后这种语句应该也不能执行了……

[github-actions]

Hi, this is your first issue in IoTDB project. Thanks for your report. Welcome to join the community!

[JackieTien97]

刚重启了，看不到之前的stack日志，重启完，在页面狂点了几下，又出现这种情况了，下面是部分stack日志，不够的话我找QQ群把stack日志文件传过来：

可能需要完整的stack日志

[JackieTien97]

@Alima777 看一下 为啥kill都不能kill掉

[Alima777]

Try to check whether your dataset or statement is closed correctly.

[Alima777]

I try to reproduce the situation you described. If I don't close the dataset, the result will be like as you said.

[xufwind95]

刚重启了，看不到之前的stack日志，重启完，在页面狂点了几下，又出现这种情况了，下面是部分stack日志，不够的话我找QQ群把stack日志文件传过来：

可能需要完整的stack日志

日志通过QQ群发了哈，查询出现假死的是啥情况啊，是我们什么地方没操作对么，升级版本能不能避免呢

[jixuan1989]

刚重启了，看不到之前的stack日志，重启完，在页面狂点了几下，又出现这种情况了，下面是部分stack日志，不够的话我找QQ群把stack日志文件传过来：

可能需要完整的stack日志

日志通过QQ群发了哈，查询出现假死的是啥情况啊，是我们什么地方没操作对么，升级版本能不能避免呢

I did not find any log files on QQ group.
Is the log too large? why not upload the files here directly?

[xufwind95]

这里不能上传文件吧

刚重启了，看不到之前的stack日志，重启完，在页面狂点了几下，又出现这种情况了，下面是部分stack日志，不够的话我找QQ群把stack日志文件传过来：

可能需要完整的stack日志

日志通过QQ群发了哈，查询出现假死的是啥情况啊，是我们什么地方没操作对么，升级版本能不能避免呢

I did not find any log files on QQ group. Is the log too large? why not upload the files here directly?

stack.log

[Alima777]

@jixuan1989

我觉得这个问题应该就是没正常关结果集或者 statement 导致的啊...reopen 的疑问在哪?