Query-in-Place Logs: The Partition Layout Is the Index

Query-in-Place Logs: The Partition Layout Is the Index An architecture diagram generated by Archify. Glue Data Catalog · schema + table defs · Architecture component · required, or fields drop Glue Data Catalog schema + table defs required, or fields drop App logs · services, JSON · Architecture component App logs services, JSON CloudWatch Logs · subscription filter · Architecture component CloudWatch Logs subscription filter Data Firehose · buffer 60-900s, Parquet · Architecture component · minutes-fresh floor Data Firehose buffer 60-900s, Parquet minutes-fresh floor S3 data lake · dt/hour/service prefix · Architecture component · the prefix is the index S3 data lake dt/hour/service prefix the prefix is the index Athena · SQL, pay per TB scanned · Architecture component Athena SQL, pay per TB scanned Alarms, Live Tail · stays in CloudWatch · Architecture component · seconds-fresh Alarms, Live Tail stays in CloudWatch seconds-fresh Agents, SDKs · Direct PUT · Architecture component Agents, SDKs Direct PUT Compaction job · small files to large · Architecture component · budget for it Compaction job small files to large budget for it metrics rewrite Legend Backend Database Cloud Security Message bus

Write Time Or Read Time

  • • Indexed search pays at ingest and hot retention
  • • Query-in-place pays per TB scanned
  • • No index, so the partition layout is the index
  • • A query without partition columns reads the bucket

The Four Things That Bite

  • • Firehose drops fields the Glue schema lacks, silently
  • • CloudWatch delivers gzip+base64 logEvents arrays, not lines
  • • Partition projection once partitions reach the tens of thousands
  • • Buffering is the freshness floor

Two Log Systems, Not One

  • • Live data and alarms stay in CloudWatch
  • • History lives in S3, queried by Athena
  • • Two query languages, two places to look
  • • Worth it only when retention economics justify it