Package org.apache.nutch.metrics

Class LatencyTracker

java.lang.Object
org.apache.nutch.metrics.LatencyTracker
public class LatencyTracker extends Object
A utility class for tracking latency metrics using TDigest for percentile calculation.

This class wraps a MergingDigest data structure to collect latency samples and emit Hadoop counters with count, sum, and percentile values (p50, p95, p99). MergingDigest supports merging digests from multiple tasks for job-level percentile computation.

Usage: 

 // In mapper/reducer setup
 latencyTracker = new LatencyTracker(NutchMetrics.GROUP_FETCHER, NutchMetrics.FETCHER_LATENCY);

 // During processing
 long start = System.currentTimeMillis();
 // ... operation ...
 latencyTracker.record(System.currentTimeMillis() - start);

 // In cleanup
 latencyTracker.emitCounters(context);

Emits the following counters:

  • {prefix}_count_total - total number of samples
  • {prefix}_sum_ms - sum of all latencies in milliseconds
  • {prefix}_p50_ms - 50th percentile (median) latency
  • {prefix}_p95_ms - 95th percentile latency
  • {prefix}_p99_ms - 99th percentile latency
Since:
1.22

  • Field Details

    • SUFFIX_COUNT_TOTAL

      public static final String SUFFIX_COUNT_TOTAL
      Counter name suffix for total sample count.
      See Also:

    • SUFFIX_SUM_MS

      public static final String SUFFIX_SUM_MS
      Counter name suffix for sum of latencies in milliseconds.
      See Also:

    • SUFFIX_P50_MS

      public static final String SUFFIX_P50_MS
      Counter name suffix for 50th percentile latency in milliseconds.
      See Also:

    • SUFFIX_P95_MS

      public static final String SUFFIX_P95_MS
      Counter name suffix for 95th percentile latency in milliseconds.
      See Also:

    • SUFFIX_P99_MS

      public static final String SUFFIX_P99_MS
      Counter name suffix for 99th percentile latency in milliseconds.
      See Also:

  • Constructor Details

    • LatencyTracker

      public LatencyTracker(String group, String prefix)
      Creates a new LatencyTracker.
      Parameters:
      group - the Hadoop counter group name
      prefix - the prefix for counter names (e.g., "fetch_latency")

  • Method Details

    • record

      public void record(long latencyMs)
      Records a latency sample.
      Parameters:
      latencyMs - the latency in milliseconds

    • merge

      public void merge(LatencyTracker other)
      Merges another LatencyTracker's digest and aggregates count/sum into this one. Used to combine per-thread or per-task metrics before emitting or serializing.
      Parameters:
      other - the other tracker to merge in (not modified)

    • getCount

      public long getCount()
      Returns the number of recorded samples.
      Returns:
      the count of recorded latency samples

    • getSum

      public long getSum()
      Returns the sum of all recorded latencies.
      Returns:
      the sum of latencies in milliseconds

    • getPercentile

      public long getPercentile(double quantile)
      Returns the percentile value for the given quantile.
      Parameters:
      quantile - the quantile (0.0 to 1.0)
      Returns:
      the percentile value in milliseconds

    • toBytes

      public byte[] toBytes()
      Serializes the digest to bytes for transmission to a reducer or side file. Returns an empty array if no samples have been recorded.
      Returns:
      serialized digest bytes, or empty array if count is 0

    • fromBytes

      public static com.tdunning.math.stats.MergingDigest fromBytes(byte[] bytes)
      Deserializes a MergingDigest from bytes (as produced by toBytes()).
      Parameters:
      bytes - serialized digest bytes
      Returns:
      MergingDigest instance, or null if bytes is null or empty

    • emitCountAndSumOnly

      public void emitCountAndSumOnly(TaskInputOutputContext<?,?,?,?> context)
      Emits only count and sum counters (not percentiles). Use in mappers when a reducer will merge TDigests and set job-level percentile counters.

    • emitCounters

      public void emitCounters(TaskInputOutputContext<?,?,?,?> context)
      Emits all latency counters to the Hadoop context.

      Should be called once during cleanup to emit aggregated metrics.

      Parameters:
      context - the Hadoop task context

    • setJobLevelCounters

      public static void setJobLevelCounters(TaskInputOutputContext<?,?,?,?> context, String group, String prefix, long mergedCount, long mergedSum, com.tdunning.math.stats.MergingDigest mergedDigest)
      Sets job-level percentile counters from a merged digest (e.g. in a reducer that merged TDigests from all tasks). Uses the same counter names as emitCounters(TaskInputOutputContext).
      Parameters:
      context - the Hadoop task context
      mergedCount - total count from merged digest
      mergedSum - total sum from merged digest
      mergedDigest - the merged MergingDigest (may be null if mergedCount is 0)