The Empty Channel Scan

Two channels on the same Cassandra cluster hold the same schema and the same replication factor. Channel A has never had a deletion. Channel B had roughly 400,000 old messages purged last week by a moderation tool.

Fetching the newest 50 messages from channel A is a few milliseconds. The identical query against channel B times out, and the coordinator logs this:

WARN  [ReadStage-11] ReadCommand.java:569 - Read 41 live rows and 100000 tombstone cells for query
  SELECT * FROM messages WHERE channel_id = 3417 AND bucket = 419 LIMIT 50
  (see tombstone_warn_threshold)
ERROR [ReadStage-11] ReadCommand.java:582 - Scanned over 100000 tombstones during query
  SELECT * FROM messages WHERE channel_id = 3417 AND bucket = 419 LIMIT 50; query aborted

Nothing was rebalanced and both channels are on healthy nodes. Why did deleting data make reads slower?

Removing keys raised the false positive rate of the partition's Bloom filter, so the read touches SSTables that cannot contain the data.

A delete is written as a tombstone in a new SSTable, and a range read has to merge every tombstone covering the scanned range until compaction runs past the grace period, so the read does strictly more work than before the purge.

Freeing that much space triggered a rebalance, so the partition now lives on a node the coordinator has to hop to.

The delete rewrote the affected SSTables in place, and the read is queued behind that rewrite.