[P0] Secure exit of the kernels

[Jackcuii]

We should make sure all the kernels can exit safely, within this model: the array may not run the instructions exactly at the timestep mapped onto the array. The model only respect the order of the instructions on the same core.

A better placement/strong condition of the RETURN instruction or other conditions (such as all the pred are zero) are needed.

[tancheng]

the array may not run the instructions exactly at the timestep mapped onto the array. The model only respect the order of the instructions on the same core.

This is exactly the purpose of the time_step, it only keeps the order, instead of exact cycle. That's why we didn't call it cycle.

return is a problem.

dataflow/test/e2e/relu/relu_kernel.mlir

Line 90 in 79d9d8d

// MAPPING-NEXT: "neura.return"() {dfg_id = 3 : i32, mapping_locs = [{id = 9 : i32, index_per_ii = 4 : i32, invalid_iterations = 1 : i32, resource = "tile", time_step = 9 : i32, x = 1 : i32, y = 2 : i32}]} : () -> ()

@ShangkunLi what's your thought? This return has no operand. It would never be triggered then. So accelerator would never stop, though it won't generate valid results at certain point.

[ShangkunLi]

the array may not run the instructions exactly at the timestep mapped onto the array. The model only respect the order of the instructions on the same core.

This is exactly the purpose of the time_step, it only keeps the order, instead of exact cycle. That's why we didn't call it cycle.

return is a problem.

dataflow/test/e2e/relu/relu_kernel.mlir

Line 90 in 79d9d8d

// MAPPING-NEXT: "neura.return"() {dfg_id = 3 : i32, mapping_locs = [{id = 9 : i32, index_per_ii = 4 : i32, invalid_iterations = 1 : i32, resource = "tile", time_step = 9 : i32, x = 1 : i32, y = 2 : i32}]} : () -> ()
@ShangkunLi what's your thought? This return has no operand. It would never be triggered then. So accelerator would never stop, though it won't generate valid results at certain point.

For return ops that have explicit operands, we can keep the current solution.

For those without operands, we can identify all the execution conditions of the pred-blocks of the return block and add an operand to the return op. The added operand is the or of all the pred-blocks' conditions.

[tancheng]

we can identify all the execution conditions of the pred-blocks of the return block and add an operand to the return op.

What the IR would look like? Trying to understand how the conditions become the return's operand.

[tancheng]

Based on offline discussion, return without operand, could be or of all the pred-blocks' conditions; if a pred-block is a unconditional branch, we can copy the return into that pred_block.

[guosran]

I have addressed this issue in PR #220 :)

[ShangkunLi]

Based on offline discussion, return without operand, could be or of all the pred-blocks' conditions; if a pred-block is a unconditional branch, we can copy the return into that pred_block.

Hi~ @tancheng, after discussion with @guosran. I may implement a code skeleton first.

But I find the method we discussed last week may not be able to guarantee a secure exit. Take the bicg kernel as an example:

module attributes {} {
  func.func @kernel(%arg0: i32 {llvm.noundef}, %arg1: i32 {llvm.noundef}, %arg2: !llvm.ptr {llvm.nocapture, llvm.noundef, llvm.readonly}, %arg3: !llvm.ptr {llvm.nocapture, llvm.noundef}, %arg4: !llvm.ptr {llvm.nocapture, llvm.noundef}, %arg5: !llvm.ptr {llvm.nocapture, llvm.noundef, llvm.readonly}, %arg6: !llvm.ptr {llvm.nocapture, llvm.noundef, llvm.readonly}) -> !llvm.void attributes {} {
    %0 = "neura.constant"() <{value = "%arg0"}> : () -> i32
    %1 = "neura.constant"() <{value = "%arg1"}> : () -> i32
    %2 = "neura.constant"() <{value = "%arg3"}> : () -> !llvm.ptr
    %3 = "neura.constant"() <{value = "%arg4"}> : () -> !llvm.ptr
    %4 = "neura.constant"() <{value = 0 : i8}> : () -> i8
    %5 = "neura.constant"() <{value = 0 : i64}> : () -> i64
    %6 = "neura.icmp"(%0) <{cmpType = "sgt"}> {rhs_value = 0 : i32} : (i32) -> i1
    neura.cond_br %6 : i1 then to ^bb1 else to ^bb2
  ^bb1:  // pred: ^bb0
    %7 = neura.zext %0 : i32 -> i64
    %8 = "neura.shl"(%7) {rhs_value = 3 : i64} : (i64) -> i64
    "neura.memset"(%2, %4, %8) <{is_volatile = false}> : (!llvm.ptr, i8, i64) -> ()
    %9 = "neura.icmp"(%1) <{cmpType = "sgt"}> {rhs_value = 0 : i32} : (i32) -> i1
    neura.cond_br %9 : i1 then to ^bb4 else to ^bb8
  ^bb2:  // pred: ^bb0
    %10 = "neura.icmp"(%1) <{cmpType = "sgt"}> {rhs_value = 0 : i32} : (i32) -> i1
    neura.cond_br %10 : i1 then to ^bb3 else to ^bb8
  ^bb3:  // pred: ^bb2
    %11 = neura.zext %1 : i32 -> i64
    %12 = "neura.shl"(%11) {rhs_value = 3 : i64} : (i64) -> i64
    "neura.memset"(%3, %4, %12) <{is_volatile = false}> : (!llvm.ptr, i8, i64) -> ()
    neura.br to ^bb8
  ^bb4:  // pred: ^bb1
    %13 = neura.zext %1 : i32 -> i64
    %14 = neura.zext %0 : i32 -> i64
    neura.br %5 : i64 to ^bb5
  ^bb5(%15: i64):  // 2 preds: ^bb4, ^bb7
    %16 = "neura.gep"(%15) <{operandSegmentSizes = array<i32: 0, 1>}> {lhs_value = "%arg4"} : (i64) -> !llvm.ptr
    "neura.store"(%16) {lhs_value = 0.000000e+00 : f64} : (!llvm.ptr) -> ()
    %17 = "neura.gep"(%15) <{operandSegmentSizes = array<i32: 0, 1>}> {lhs_value = "%arg6"} : (i64) -> !llvm.ptr
    neura.br %5 : i64 to ^bb6
  ^bb6(%18: i64):  // 2 preds: ^bb5, ^bb6
    %19 = "neura.gep"(%18) <{operandSegmentSizes = array<i32: 0, 1>}> {lhs_value = "%arg3"} : (i64) -> !llvm.ptr
    %20 = "neura.load"(%19) : (!llvm.ptr) -> f64
    %21 = "neura.load"(%17) : (!llvm.ptr) -> f64
    %22 = "neura.gep"(%15, %18) <{operandSegmentSizes = array<i32: 0, 2>}> {lhs_value = "%arg2"} : (i64, i64) -> !llvm.ptr
    %23 = "neura.load"(%22) : (!llvm.ptr) -> f64
    %24 = "neura.fmul_fadd"(%21, %23, %20) : (f64, f64, f64) -> f64
    "neura.store"(%24, %19) : (f64, !llvm.ptr) -> ()
    %25 = "neura.load"(%16) : (!llvm.ptr) -> f64
    %26 = "neura.load"(%22) : (!llvm.ptr) -> f64
    %27 = "neura.gep"(%18) <{operandSegmentSizes = array<i32: 0, 1>}> {lhs_value = "%arg5"} : (i64) -> !llvm.ptr
    %28 = "neura.load"(%27) : (!llvm.ptr) -> f64
    %29 = "neura.fmul_fadd"(%26, %28, %25) : (f64, f64, f64) -> f64
    "neura.store"(%29, %16) : (f64, !llvm.ptr) -> ()
    %30 = "neura.add"(%18) {rhs_value = 1 : i64} : (i64) -> i64
    %31 = "neura.icmp"(%30, %14) <{cmpType = "eq"}> : (i64, i64) -> i1
    neura.cond_br %31 : i1 then to ^bb7 else %30 : i64 to ^bb6
  ^bb7:  // pred: ^bb6
    %32 = "neura.add"(%15) {rhs_value = 1 : i64} : (i64) -> i64
    %33 = "neura.icmp"(%32, %13) <{cmpType = "eq"}> : (i64, i64) -> i1
    neura.cond_br %33 : i1 then to ^bb8 else %32 : i64 to ^bb5
  ^bb8:  // 4 preds: ^bb1, ^bb2, ^bb3, ^bb7
    "neura.return"() : () -> ()
  }
}

I plan to distinguish return op (operand being returned values) and return_void op (operand being trigger values).

The main concern is bb3 here. I have tried some approaches for bb3:

1. Take the last result in bb3 (i.e., %12) as a trigger for return_void. But this cannot guarantee the execution of neura.memset after %12
2. Create a constant(true) in bb3 and use this one as the trigger; we have similar problems in 1.
3. Utilize the execution condition of bb3 (from cond_br in bb2) as the trigger; we cannot guarantee the execution of operations in bb3 then.
4. Since the pred block of bb3 is bb2, we cannot merge bb3 & bb2 to utilize the execution condition of bb2.
   Do you have any suggestions?

[tancheng]

But this cannot guarantee the execution of neura.memset after %12

I don't think this is a problem. Return could be viewed as a triggering, it doesn't mean the entire chip magically halt/shut down. Additional mechanism could be implemented to guarantee pending write. e.g., the simplest one could be waiting for 100 cycles once Return is triggered.

I like return_void. For "Create a constant(true) in BB3 and use this one as the trigger", the new constant in BB3 can directly trigger a return_void, what about copying the return_void into BB3?

[ShangkunLi]

But this cannot guarantee the execution of neura.memset after %12

I don't think this is a problem. Return could be viewed as a triggering, it doesn't mean the entire chip magically halt/shut down. Additional mechanism could be implemented to guarantee pending write. e.g., the simplest one could be waiting for 100 cycles once Return is triggered.

I like return_void. For "Create a constant(true) in BB3 and use this one as the trigger", the new constant in BB3 can directly trigger a return_void, what about copying the return_void into BB3?

Got your points. I will implement a version and see if it can work ~