MSDB transaction log was full due to 'CHECKPOINT' despite disk has space. Had to rebuild from scratch

Today I came across something new in my 20 years of DBA life - MSDB transaction log was full due to 'CHECKPOINT'

Due to the secure environment I'm not able to copy-and-paste out anything but you'll get the idea

Issue

• We had a customer disk running out of space, at 0 bytes, it caused everything to stop
• After some cleanup and emergency actions
• Removed pagefile
• Emptied Recycle Bin and disabled saving space for the drives
• We managed to free up 20~80GB+ space, yet the SQL Agent still won't start up, and the error log/event view shows - MSDB transaction log was full due to 'CHECKPOINT'

Things Tried (and did NOT work)

• Obviously restart SQL Server and Agent
• Put MSDB into Emergency mode - nope, not allowed
• Rebuild Log
• Backup Log
• Adding new Log file to MSDB
• Change MSDB recovery model (from SIMPLE to FULL or anything)
• Reboot the physical server
• ^ all above made sense because the LOG is full and cannot do any action

Resolution

• Had to rebuild MSDB using this method
• https://sites.google.com/site/sqlservermsdn/rebuild-msdb-database
• We lost the Backup plan, all SQL Jobs
• had to re-create Backup plan using Maintenance Wizard
• created the jobs elsewhere and paste and re-create

Things to Do AFTER or Prepare beforehand

• Script out your SQL Jobs if possible
• Backup if possible, for restore AFTER the rebuild
• AFTER - had to re-grant permissions to MSDB on SQL Logins

Restoring SYSTEM databases

I had the opportunity to restore a few system databases due to disk inconsistency/corruption. Actually my first time in a long time and was it fun. I have a few tips to share

All below were done on a SQL Server 2005 instance

Restoring MASTER Database!!
- start SQL Server in Single User mode (add -m to the startup parameter)
- You can Google this with lots of results. Microsoft How to on SQL 2008 R2

Restoring MSDB Database
- This is easy, just restore like a user database

Restoring MODEL Database!
- Restore model in RECOVERY mode, never in NORECOVERY mode as SQL Server will stop (due to tempdb unable to create, without a model database)
- If you did NORECOVERY, the only way to get in is manual work.
For me, I started it and detach-and-attach the model database that I copy from another location, as one cannot issue RESTORE command in Single-User mode

net start MSSQLSERVER /c /m /T3608 (you can go into SQL BINN and run sqlserver -c -m -T3608 as well)

sqlcmd -E (to connect to the SQL Server)

Detach and Re-attach model databasesp_detach_db 'model'
go
sp_attach_db 'model', 'DATA.mdf', 'LOG.ldf'
go
use model
go

Command line parameter	Use
-c	Forces an instance of SQL Server not to start as a service.
-m	Allows updates to the system catalog.
-T3608	Recovers the master database only.