D&I S3 Footprint — Rolling vs. Permanent Storage, Cost & Tiering

1. Headline metrics

2.00 PB

Total S3 footprint

across 4 accounts, 238 buckets

238

Active buckets

27 in Data Lake Prod, 211 across the other 3 accounts

~$24K/mo

Storage cost, after 20% enterprise discount

~$290.5K/yr — storage only, no request/transfer

1.43 PB

Archived — tiered, permanent

71.8% of footprint, 2 buckets, already cost-optimized

425.6 TB

Rolling landing-zone buffers

21.3% — bounded size, self-cleaning by design

138.6 TB

Permanent, no lifecycle rule

6.9% — dominated by one bucket, see section 2

2. Rolling landing-zone vs. permanent storage

The distinction that matters for a cost/tiering story isn't just "which tier is it in" — it's whether the bucket is designed to stay a bounded size, or whether it accumulates forever. Every bucket with nonzero data was classified by its lifecycle rule into one of three buckets:

Classification	Size	% of footprint	What it means
Rolling landing zone	425.58 TB	21.3%	Flat expiration (e.g. 30/180/183 days), no prior archival step. Size is self-limiting — steady-state size ≈ ingest rate × retention window, not a growing liability. Deleting from these buckets is by design, not neglect.
Archived, permanent	1.43 PB	71.8%	Transitions to Glacier/Deep Archive/Intelligent-Tiering and never expires — this is real long-term data-team storage, already cost-optimized via tiering. This is the actual "data we keep" bucket of the whole footprint.
No lifecycle rule	138.62 TB	6.9%	No expiration and no tiering — grows forever, sitting 100% in Standard. This is the category worth scrutinizing: either it needs a lifecycle rule (tiering and/or expiry), or someone should confirm it's intentional permanent hot storage.

What's actually in the 138.6 TB "no lifecycle rule" category

Bucket	Size	% of category	What it is
`data-lake-prod-bronze`	128.12 TB	92.4%	Data Lake Prod. Raw bronze-layer landing data, 3.88M objects, 100% Standard. Almost certainly still read by downstream silver builds — needs an access-pattern check before adding any lifecycle rule, not a candidate for the same immediate cleanup as pendo/polytomic.
Long tail — 36 buckets	10.50 TB	7.6%	Mostly S3 access-log buckets (buckets that log requests to other buckets, e.g. `bi-workflows-s3-bucket-access-logs-prod` at 6.28 TB / 207.9M objects), plus CloudTrail logs and small tfstate/CloudFront buckets. Each under 1 TB; grows slowly since nothing expires them, but none is individually significant.

Footprint by classification

Bytes by lifecycle classification (rolling / archived / no-lifecycle), across all 4 accounts combined.
A bucket can be 100% Standard and still be "rolling" (e.g. the Everything Else group below) if it just hasn't needed cold tiers yet given its short retention window.

Storage tier occupancy

Bytes by actual S3 storage class today, ranked — 59.0% still sits in Standard ($0.023/GB-mo), the rest already in a cheaper tier.
A horizontal bar reads better than a donut here since the five tiers span three orders of magnitude (1.18 PB down to 28.5 GB) — a donut's angular encoding makes the smallest slices unreadable and crowds the legend, where a ranked bar keeps every value legible at its own scale.

Worked example — the landing-zone pattern:

dw-prod-bi (209 TB, aws-bi account) is the raw Kinesis Firehose ingest point for Okta syslog2 — per-cell (ok1–ok16, op1–op3, eu1) gzip files landing continuously, ~24MB each, several per minute per cell.
Terraform-tagged module = bi_roles/s3_dw_bi.tf, confirming D&I ownership.
Confirmed team-owned and actively consumed — loaded into Snowflake hourly via an Airflow DAG, same pattern as bi-workflows-prod.
Its only lifecycle rule is a flat 183-day expiration — no versioning, no tiering step.
At steady state its size is simply ingest rate × 183 days — it will never grow past that no matter how long the pipeline runs.
This is a landing buffer feeding a downstream system, not an archive. The durable copy of this same syslog data lives in snowflake-archive-prod (Data Lake Prod, 371 TB, tiers to Glacier/Deep Archive within 1 day and never expires) — that's the bucket that represents actual retained data-team storage for syslog.

3. Data Lake Prod (own category)

1.57 PB

Total size

27 buckets, 18.6M objects

~$16K/mo

After 20% discount

~$195.5K/yr (exact: $16,294/mo)

91.5%

Already archived & tiered

data-lake-prod-silver + snowflake-archive-prod

Bucket	Size	Objects	Classification	Lifecycle
`data-lake-prod-silver`	1.06 PB	7.56M	Archived	5 rules, each scoped to one prefix (specific Okta profile dataset) >1–5 MB objects, transitions those prefixes to Intelligent-Tiering day 0 — not a bucket-wide rule, most of the bucket has no matching rule
`snowflake-archive-prod`	371.25 TB	3.15M	Archived	6 rules, mostly 1-day transition to Deep Archive/Glacier — likely destination for syslog2 data
`data-lake-prod-bronze`	128.12 TB	3.88M	No lifecycle	None — 100% Standard, unbounded growth
`data-lake-prod-unity-catalog-metastore`	3.87 TB	270K	Rolling	7-day expiry (Personal Stage policy)
`data-lake-prod-gold`	814.93 GB	69.7K	No lifecycle	None
14 smaller buckets	<500 GB ea.	—	mixed	—

data-lake-prod-bronze is the single bucket behind most of the 138.6 TB "no lifecycle" category — 128.12 TB / 3.88M objects, 92.4% of that whole category by itself, sitting 100% in Standard with zero lifecycle rule (~$2.9K/mo list price untiered). This bucket is likely actively read by downstream silver builds, so before adding any transition/expiry rule, confirm access pattern — an aggressive move to IA/Glacier could hurt query cost/latency elsewhere if it's still hot. Flagged, not yet actioned.

4. Everything Else: data-eng-prod + data-eng-trex + aws-bi

430.25 TB

Total size

211 buckets, 980.8M objects

~$8K/mo

After 20% discount

~$95K/yr (exact: $7,917/mo)

98.0%

Rolling landing-zone buffers

421.71 TB across 80 buckets, all with flat expiry

100%

Sitting in Standard tier

zero Intelligent-Tiering / Glacier anywhere in this group

Every byte in this group is Standard tier — but 98% of it is also on a flat expiration window, which is why it was never tiered.
Most of this data doesn't live long enough (30–183 days) for a move to IA/Glacier to pay back its own transition cost.
This is the opposite pattern from Data Lake Prod, and it's the correct one for this shape of data.
The "gap" isn't missing tiering — it's confirming these retention windows are still the right length.

Per-account split

Account	Total size	% of combined	Buckets	Objects
aws-bi	423.36 TB	98.4%	100	952.0M
data-eng-prod	4.15 TB	1.0%	54	22.5M
data-eng-trex	2.74 TB	0.6%	57	6.4M

Top buckets driving aws-bi (94.4% of the account in 2 buckets)

Bucket	Size	% of aws-bi	Objects	Classification	Notes
`dw-prod-bi`	209.28 TB	49.4%	19.1M	Rolling	Okta syslog2 Firehose landing zone, 183-day expiry. Confirmed team-owned — loaded into Snowflake hourly via Airflow — see worked example above
`bi-workflows-prod`	190.64 TB	45.0%	267.6M	Rolling	Cassandra tracing/log data, 30-day expiry. Confirmed team-owned — loaded into Snowflake via an hourly `workflows_load` DAG
`dh-bi-sfdc-prod`	9.33 TB	2.2%	20.4K	Rolling	183-day expiry
`bi-workflows-s3-bucket-access-logs-prod`	6.28 TB	1.5%	207.9M	No lifecycle	Access logs for bi-workflows-prod — will grow forever until a rule is added
`bi-wms-prod`	4.49 TB	1.1%	7.0M	Rolling	4 domain-scoped 183-day rules

5. Cost Cleanup Run-Rate — data-eng-prod & data-eng-trex

D&I Platform ran a cost cleanup across data-eng-prod and data-eng-trex (us-west-2) — full compute, storage and networking, not S3-only. This scan independently confirms that the S3 half of that cleanup is still holding, two months later. The number that matters here: this is a ~$416K/yr run-rate reduction, not a one-time cleanup credit — it lowers what these two accounts cost every month, going forward, permanently (until something regresses it).

~$40K/mo

May 2026 baseline run-rate

combined data-eng-prod + data-eng-trex (exact: $40,315/mo)

~$5.7K/mo

August 2026 run-rate

after all cleanup fully reflected (exact: $5,677/mo)

-86%

Run-rate reduction

~$35K/mo lower, every month, going forward

~$416K/yr

Annualized savings

~$35K/mo × 12 — the actual size of this number

Monthly cost trend

Month	data-eng-prod	data-eng-trex	Combined	MoM change	Driver
May 2026	~$32.6K	~$7.7K	~$40.3K	—	Baseline
June 2026	~$29.5K	~$7.2K	~$36.7K	-$3.6K (-9%)	`polytomic-prod` terminated (~Jun 13), `tableau_utility_server` terminated (~Jun 27)
July 2026*	~$15.6K	~$5.1K	~$20.7K	-$16.0K (-44%)	S3 versioning cleanup (Jul 17–18), Tableau Bridge decommission + io1 volume deletions (Jul 25)
August 2026 (proj.)	~$3.9K	~$1.7K	~$5.7K	-$15.0K (-73%)	All cleanup fully reflected for a complete month

* July figures are through Jul 25 only (partial month); full-month estimate is ~$17K combined.

Why the S3 buckets looked empty but weren't

Both bi-tableau-prod-bucket and bi-tableau-trex had S3 versioning enabled, and every current object version had a delete marker applied — which makes a bucket render as empty in the AWS Console and in aws s3 ls.
But the actual data — daily Tableau log ZIP archives, some up to 23 GB each — was still sitting in non-current versions, invisible in the UI but still billed. 112 TB of it, combined across both buckets.
The fix (Jul 17–18) was a lifecycle rule: NoncurrentVersionExpiration at 1 day, plus ExpiredObjectDeleteMarker: true to clean up the now-orphaned delete markers themselves.
This scan confirms the fix is still holding: both buckets show 0 bytes today, and versioning remains Enabled on both — exactly as expected, since the fix was the lifecycle rule, not turning versioning off.

Full picture: this was one cleanup pass across compute, storage and networking

Account	Resource	Issue & action	Daily saving
data-eng-prod	`bi-tableau-prod-bucket`	112 TB of orphaned non-current Tableau log ZIPs (see above). Added 1-day noncurrent-version expiry + delete-marker cleanup.	-$76/day
data-eng-prod	Tableau Bridge-2 (EC2 r5.4xlarge)	Ghost standby: 0.1% avg CPU, ~0 MB/week network out, never assigned connections in Tableau Cloud. Removed via Terraform (`terraform destroy -target`).	-$19/day
data-eng-prod	Bridge-2 EBS volumes (io1)	1TB (0 IOPS used) + 100GB root, both provisioned at 3,000 IOPS. Destroyed with the Bridge-2 instance.	-$20/day
data-eng-prod	Bridge-1 root + 1TB data volume	Root io1 (3,000 IOPS, near-zero usage) converted to gp3 in-place, zero downtime. Idle 1TB volume manually deleted.	-$20/day
data-eng-trex	7× orphaned io1 EBS volumes	2TB each, 3,000 IOPS provisioned, unattached and untagged for ~11 months (~$33K already wasted by the time found). Verified unattached, deleted.	-$100/day
data-eng-trex	`bi-tableau-trex`	Same versioning issue as prod, remnant non-current versions after a prior partial cleanup. Same lifecycle fix applied.	-$0.63/day

Total from these six actions: ~-$236/day (~$7.2K/mo). Two earlier terminations (polytomic-prod, ~Jun 13, -$231/day; tableau_utility_server, ~Jun 27, -$89/day) account for most of the June→July drop shown in the trend table above.

Scope: this cleanup never touched aws-bi — it was scoped to data-eng-prod and data-eng-trex only. aws-bi is the account actually driving 423.36 TB of the 430.25 TB "Everything Else" total above (section 4), and was not part of this pass.

6. Open questions / next steps

data-lake-prod-bronze (128 TB, no lifecycle, 100% Standard) is the next tiering candidate — needs an access-pattern check first since it likely feeds downstream silver builds.

Everything Else group is 100% Standard by design (short retention windows don't justify tiering) — worth stating explicitly in the deep dive so it reads as a deliberate tradeoff, not a gap.

Data: CloudWatch AWS/S3 namespace (BucketSizeBytes per StorageType, NumberOfObjects), s3api get-bucket-lifecycle-configuration, s3api get-bucket-tagging. Profiles scanned in order: data-lake-prod, data-eng-prod, aws-bi, data-eng-trex — 240 buckets, 0 errors. Cost estimates are storage-only (no request/transfer), simplified single blended $/GB-month rate per class, list price with 20% enterprise discount applied where noted — directional, not billing-accurate. Snapshot as of 2026-09-25; re-run before presenting if stale.