Prompt trying to cache databases for closed sessions

SQL Prompt keeps haning and coming up with the error "unload all databases". Then I noticed that it is tyring to cache databases for sessions that I closed. Only workaround I found was to close SSMS and re-open.