It remains unclear whether immune-related adverse events (irAEs) and glucocorticoid use could impact long-term outcomes in patients treated for solid tumors with immune checkpoint inhibitors (ICI). All patients treated with a single-agent ICI for any advanced cancer were included in this retrospective unicentric study. The objectives were to assess the impact of grade ≥3 irAEs, glucocorticoid use and the interruption of immunotherapy on progression-free survival (PFS) and overall survival (OS). In this 828-patient cohort, the first occurrence of grade ≥3 irAEs had no significant impact on PFS or OS. Glucocorticoid administration for the irAEs was associated with a significantly shorter PFS (adjusted HR 3.0; p = 0.00040) and a trend toward shorter OS. ICI interruption was associated with a significantly shorter PFS (adjusted HR 3.5; p < 0.00043) and shorter OS (HR 4.5; p = 0.0027). Glucocorticoid administration and ICI interruption were correlated. In our population of patients treated with single agent ICI, grade ≥3 irAEs did not impact long-term outcomes. However, the need for glucocorticoids and the interruption of immunotherapy resulted in poorer long-term outcomes. The impact of grade ≥3 irAEs reported in other studies might then be explained by the management of the irAEs.