AWS 12 Months Free Tier Account EBS Disk Full Prevents EC2 Boot? Zero-Downtime Expansion & Root Partition Repair

If your EC2 instance stopped booting because the EBS root disk filled up, the fastest fix is usually not rebuilding the server. In most cases, you can expand the EBS volume online, grow the partition and filesystem, and bring the instance back without data loss. If the machine is already stuck in boot failure, you move to a rescue workflow: stop, detach, attach to a helper instance, clean space or repair the filesystem, then reattach.

What matters in real life is not the theory, but whether your account is ready to let you do the work: billing verified, payment method valid, IAM permissions present, and no compliance review blocking volume changes. I’ve seen teams lose hours because the root cause was “disk full,” but the real blocker was an expired card, a new account under risk review, or a locked IAM policy that prevented ModifyVolume.

First question: is the instance still reachable?

This determines the whole recovery path.

Situation	Best path	Downtime	Typical risk
Instance is running, SSH/SSM still works	Online EBS expansion + filesystem growth	Usually none	Low, if you use the correct device name
Instance is up but services are failing	Free space first, then expand	Short service interruption	Medium, because you may need emergency cleanup
Instance will not boot	Rescue instance or serial console	Minutes to hours	Medium to high, depending on encryption and filesystem damage

If you can still log in, do not stop the instance just to resize the disk. EBS volume modification is online. Stopping the instance only adds downtime unless you need it for a repair workflow.

Why a full EBS root volume can break boot

A Linux system does not always fail at the exact moment the disk hits 100%. What usually happens is more annoying:

• systemd cannot write logs or temporary files.
• Package updates fail halfway and leave broken metadata behind.
• /var, /tmp, or /run fills up and services refuse to start.
• The filesystem becomes read-only after errors, so boot services cannot continue.
• You run out of inodes, not bytes, so df -h looks fine but writes still fail.

The “disk full” label is often incomplete. In incident handling, I always check both:

df -h
df -i
lsblk

If df -h shows space available but df -i is at 100%, the fix is different: delete small-file floods such as logs, cache directories, or container overlay layers.

Zero-downtime expansion: the clean path when EC2 is still alive

For most production systems, this is the preferred route. You expand the EBS volume first, then extend the partition and filesystem inside the guest OS.

Step 1: Expand the volume in AWS

You can do this from the console or CLI. The important part is that the volume modification is online and does not require stopping the instance.

From the CLI, it typically looks like this:

aws ec2 modify-volume \
  --volume-id vol-xxxxxxxx \
  --size 100

Wait until the volume state shows the new size and the modification is completed or optimizing. The instance can keep running during this period.

Step 2: Find the real device name

On Nitro-based instances, the device may appear as NVMe, not /dev/xvda. Do not guess. Use:

lsblk -f

That output tells you the disk, partition, filesystem type, and mount point. Many mistakes happen because someone runs commands on the wrong device path.

Step 3: Grow the partition

If the root filesystem is on a partition, expand the partition table first. A common tool is growpart:

sudo growpart /dev/nvme0n1 1

Here, 1 means partition 1. If your root disk is on /dev/xvda1, the command changes accordingly.

Step 4: Grow the filesystem

Use the command that matches the filesystem:

• ext4: sudo resize2fs /dev/nvme0n1p1
• XFS: sudo xfs_growfs /

XFS is common on RHEL, Amazon Linux, and many enterprise images. ext4 is common on Ubuntu and Debian. If you use the wrong tool, the resize will fail or do nothing useful.

Step 5: Verify before declaring victory

df -h
df -i

Do not stop here if the root volume looks bigger but the application still fails. I have seen cases where the filesystem grew correctly, but a Docker overlay directory, journal logs, or a database data path under /var still had no free space. The OS comes up, but the application does not.

AWS 12 Months Free Tier Account If the instance is not booting: the rescue workflow

When EC2 is stuck in a boot loop or never reaches SSH, your goal is to access the disk outside the broken OS.

Option A: EC2 Serial Console or SSM, if already enabled

If you prepared access in advance, this is the quickest option. Serial Console is useful on Nitro instances, and Systems Manager Session Manager is even better if the agent is healthy enough to start. In practice, many teams discover too late that they never enabled either.

This is why I recommend production accounts keep:

• SSM agent installed and allowed by IAM
• Serial Console permissions for break-glass users
• A working IAM role that can describe and modify EBS volumes

Option B: Stop, detach root, attach to a helper instance

This is the standard emergency method.

1. Stop the broken instance.
2. Detach the root EBS volume.
3. Attach it as a secondary disk to a healthy helper instance.
4. Mount it read-write or read-only depending on your repair plan.
5. Remove the files that consumed all space, or repair the filesystem if needed.
6. Unmount, detach, reattach as the original root volume, and boot again.

If the volume is encrypted with KMS, make sure the helper instance’s role has permission to use the key. This is a common failure point in enterprise accounts: the disk is detached, but you cannot mount it because the KMS policy is too strict.

AWS 12 Months Free Tier Account What to do on the helper instance

First identify the filesystem and mount point:

lsblk -f
sudo blkid

If it is ext4 and you suspect corruption, run a filesystem check before mounting read-write:

sudo e2fsck -f /dev/nvme1n1p1

Then mount it and clean the biggest offenders:

sudo mount /dev/nvme1n1p1 /mnt
sudo du -xhd1 /mnt | sort -h

Common space hogs:

• /var/log
• /var/lib/docker
• /var/cache
• Large application temp directories
• Old kernels under /boot

If the system uses LVM, the repair steps are slightly different:

sudo pvresize /dev/nvme1n1p2
sudo lvextend -r -l +100%FREE /dev/mapper/vg0-root

The -r flag is useful because it expands the filesystem after the logical volume.

When you need bootloader repair

Sometimes the issue is not only disk space. A failed package upgrade, a broken initramfs, or a grub problem can trap the instance in recovery mode. If the volume was mounted on a helper instance and the OS still refuses to boot, chroot into it and rebuild the boot artifacts.

Typical repair tasks:

• Regenerate initramfs
• Update grub configuration
• Check /etc/fstab UUIDs
• Verify the root partition UUID matches the current volume

AWS 12 Months Free Tier Account Do not blindly reinstall the bootloader unless you know the disk layout. On NVMe-based Nitro instances, the device naming differs from older virtualization types, and copying commands from random forum posts is a good way to create a second problem.

Case from the field: the “disk full” alert was really a payment and access problem

A small SaaS team running Ubuntu on t3.small hit a production failure at 02:00. Their root EBS volume was 30 GB, and a log burst from a failed deployment filled /var. The instance stopped taking SSH, and the app health checks failed.

AWS 12 Months Free Tier Account The technical fix was straightforward: expand to 80 GB, grow the ext4 filesystem, and clean 6 GB of old logs. But the real delay came from account readiness:

• The card on the AWS account had expired.
• The new card triggered a billing verification hold.
• The on-call engineer lacked permission to modify EBS volumes.
• SSM was not enabled, so there was no backup access path.

They lost nearly two hours before the account owner updated billing and granted the missing IAM permission. The lesson: disk incidents are also account incidents. If your billing, verification, or IAM setup is weak, a simple volume resize becomes an outage.

What to know before you need emergency recovery: account, billing, and KYC

This is where many cloud teams get surprised. They think storage recovery is purely an operations problem, but in practice the account can block the fix.

1) New account verification can delay urgent actions

AWS 12 Months Free Tier Account On newly created cloud accounts, card verification, phone verification, and sometimes business identity checks may be required before the account is fully usable. If the account has not cleared risk control, you may be able to log in but still get blocked on:

• Creating or enlarging volumes
• Launching instances in certain regions
• Using specific instance families
• Raising quotas quickly

For production systems, do not wait until the first emergency to finish KYC, billing verification, and IAM setup.

2) Payment method differences matter more than people expect

Payment method	Operational behavior	Common issue
Corporate credit card	Usually activates fastest	May trigger fraud review if billing address or country looks inconsistent
Debit card	Sometimes accepted, but less reliable for cloud billing	Insufficient authorization or issuer decline on recurring charges
Prepaid / virtual card	Often problematic for cloud providers	Higher chance of verification failure or later charge rejection
Invoice / enterprise billing	Best for stable long-term use	Setup takes longer; requires company verification and billing approval

If your workloads are business-critical, a card that passes first-time verification is less important than a billing setup that will still work at renewal time. An account that is “cheap to open” but fragile at renewal is a bad trade for production.

3) Risk control reviews can freeze your recovery window

Common triggers I see in the field:

• AWS 12 Months Free Tier Account Sudden high-value charge after a quiet period
• Login from a new country or VPN exit node
• Mismatch between account name and cardholder name
• Multiple failed billing attempts
• Fast creation of many resources after signup

If the account is under review, you may still be able to see the console, but important actions can fail silently or return generic permission errors. That is why production billing should be boring. Use consistent identity data, a stable payment method, and avoid random logins from unstable locations.

4) Shared or transferred accounts are a bad idea for emergency operations

If an organization “buys” an account from a third party instead of creating and controlling it directly, you inherit hidden problems: unknown KYC status, unclear billing ownership, no root-email control, and a much higher chance of suspension during a review. For incident recovery, you need control of the account root, the billing profile, and the IAM break-glass path. Without that, zero-downtime expansion may not be possible even if the technical fix is simple.

Cost comparison: expand the volume, restore from snapshot, or rebuild?

When teams are under pressure, they often choose the wrong recovery method because they only compare storage price, not downtime cost.

Option	Direct cost	Hidden cost	Best use case
Online EBS expansion	Extra GB-month charge on the same volume	Minimal if done correctly	Most production root-disk incidents
Snapshot and restore to a larger volume	Snapshot storage + new volume	Longer recovery time, more steps	Corrupted volume or migration to a fresh disk
Rebuild instance from AMI	New compute + storage	Application redeploy effort, config drift	Severely broken OS or repeated root failures

In many real incidents, the cheapest option is not the least expensive monthly storage price. It is the method that gets the service back fastest without creating new operational risk.

For EBS type choice, gp3 is often the practical default for root volumes because it gives predictable performance and is usually easier to size than older gp2. If you need heavy IOPS, io2 costs more but may be justified for databases or systems with strict latency needs. For a boot volume that filled because of logs or app data, adding capacity on gp3 is usually the most sensible choice.

Common mistakes that turn a simple resize into a bigger outage

• Resizing the EBS volume but forgetting the partition — the console shows a larger disk, but df -h never changes.
• Using the wrong filesystem command — ext4 and XFS are not resized the same way.
• Ignoring inode exhaustion — space exists, but writes still fail.
• AWS 12 Months Free Tier Account Assuming SSH failure means the disk is the only problem — cloud-init, SSHD config, or bootloader issues may also be involved.
• Not checking IAM permissions — the engineer can see the instance but cannot modify the volume.
• Forgetting encrypted volume permissions — the helper instance cannot mount the disk without the right KMS access.

Practical prevention: what I put in production accounts

If you want to avoid this incident repeating, the fix is not “watch the disk occasionally.” It is to build a small operational safety net.

• Set CloudWatch alarms at 80% and 90% disk usage.
• Alert on inode usage, not just bytes.
• Keep SSM enabled for all production instances.
• Keep one break-glass IAM role that can modify EBS volumes.
• Use a payment method that will not expire without notice.
• For enterprise accounts, complete KYC and billing verification before the incident.
• Budget for an oversized root volume rather than squeezing to the minimum.

I usually recommend a root volume size that leaves comfortable headroom after OS patches, logging bursts, and package cache growth. The extra monthly storage cost is tiny compared with the cost of a production boot failure.

Frequently asked questions

Will expanding EBS require EC2 downtime?

Usually no. EBS volume modification is online. The only downtime appears if you need a rescue workflow because the instance already stopped booting.

Can I fix a full root volume if the instance will not boot at all?

Yes. Use Serial Console, SSM if available, or detach the root volume and repair it from a helper instance. If the filesystem is damaged, run a filesystem check first.

What if the volume grew but the OS still shows the old size?

You likely resized only the EBS volume, not the partition and filesystem. Run growpart and then the correct filesystem growth command.

What if df -h looks fine but the system still cannot write?

Check df -i. You may have run out of inodes. That is common with log storms, container layers, and directories full of tiny files.

My AWS account is new and billing is under review. Can I still expand the volume?

Sometimes yes, sometimes no. If risk control or billing verification is holding the account, urgent actions may fail or become delayed. Finish payment verification, confirm your card is valid, and make sure IAM permissions are in place before you rely on the account in production.

AWS 12 Months Free Tier Account Is a virtual card okay for AWS billing?

It can work in some cases, but I would not trust it for a production account. Virtual or prepaid cards are more likely to trigger verification issues or later charge failures.

Should I rebuild from AMI instead of repairing the disk?

Only if the volume is badly corrupted, the OS is broken beyond quick repair, or your recovery runbook already assumes rebuild. For a plain full-disk incident, repairing and expanding is usually faster and safer.

What I would do in order, under pressure

1. Check whether the instance is still reachable through SSH, SSM, or Serial Console.
2. Confirm whether the problem is bytes or inodes.
3. If reachable, expand EBS online first.
4. AWS 12 Months Free Tier Account Grow the partition and filesystem using the correct tool for ext4, XFS, or LVM.
5. If not reachable, move to the rescue instance workflow.
6. If the account is blocked, fix billing, verification, or IAM before burning more time on the disk issue.
7. After recovery, raise the root volume size and add alarms so the same issue does not happen again.

The real lesson is simple: a “disk full” boot failure is usually recoverable in minutes, but only if the cloud account, billing method, and access controls are already in good shape. Technical recovery and account readiness are part of the same incident.