kv,storage: reduce use of time.Now (1% cpu)

[tbg]

time.Now seems cheap but we call it frequently in hot paths, so it ends up being ~1% of CPU in sysbench.

time.Now
GOROOT/src/time/time.go

  Total:       2.04s      2.22s (flat, cum)  1.01%
   1126            .          .           // we avoid ever reporting a monotonic time of 0. 
   1127            .          .           // (Callers may want to use 0 as "time not set".) 
   1128            .          .           var startNano int64 = runtimeNano() - 1 
   1129            .          .            
   1130            .          .           // Now returns the current local time. 
   1131        120ms      300ms           func Now() Time { 
   1132        1.85s      1.85s           	sec, nsec, mono := now() 
   1133         30ms       30ms           	mono -= startNano 
   1134         10ms       10ms           	sec += unixToInternal - minWall 
   1135         10ms       10ms           	if uint64(sec)>>33 != 0 { 
   1136            .          .           		// Seconds field overflowed the 33 bits available when 
   1137            .          .           		// storing a monotonic time. This will be true after 
   1138            .          .           		// March 16, 2157. 
   1139            .          .           		return Time{uint64(nsec), sec + minWall, Local} 
   1140            .          .           	} 
   1141         20ms       20ms           	return Time{hasMonotonic | uint64(sec)<<nsecShift | uint64(nsec), mono, Local} 
   1142            .          .           } 
   1143            .          .            
   1144            .          .           func unixTime(sec int64, nsec int32) Time { 
   1145            .          .           	return Time{uint64(nsec), sec + unixToInternal, Local} 
   1146            .          .           } 

Jira issue: CRDB-43625

Epic CRDB-42584

[RaduBerinde]

Not sure how much this saves, but for code sections that only require a duration between two points (probably most of them), we could use only Since() which does not get the clock, just the monotonic timer. Instead of:

func someFunc() {
  start := time.Now()
  ..
  duration := time.Since(start)
}

we could do

var processStart = time.Now()
func someFunc() {
  start := time.Since(processStart)
  ...
  duration := time.Since(processStart) - start
}

We could add a timeutil helper that does this under the hood.

[RaduBerinde]

Heh, just found this exact suggestion inside the Go runtime: https://github.com/golang/go/blob/master/src/runtime/time_nofake.go#L19C41-L19C45

[stevendanna]

@RaduBerinde Definitely feels like if we want to use that trick we'll want a timeutil helper. We have linters that yell at you if you use time.Now() instead of timeutil.Now(), but the latter strips the monotonic clock reading as a side effect of setting the locale to UTC. Nevermind, we fixed that!

[RaduBerinde]

I'll work on adding something like that.