Chronic sclerosing sialadenitis is commonly regarded as a manifestation of IgG4-related disease. We previously found that a high IgG4 expression or IgG4-related disease could accompany nonspecific sialadenitis, whereas chronic sclerosing sialadenitis was not directly associated with IgG4-related disease. Our previous findings lead us to hypothesize that these inflammatory conditions of the submandibular gland signify a continuous progression of disease rather than different disease entities. We, therefore, aimed to determine the presence of IgG4-positivity and genuine IgG4-related disease in a cohort of 165 submandibular gland specimens from patients who underwent surgery due to chronic nonspecific sialadenitis or sialolithiasis. To do so, we re-evaluated histopathological features and divided samples into three groups: (A) nonspecific sialadenitis without known sialolithiasis, (B) sialadenitis with sialolithiasis, and (C) sialolithiasis without sialadenitis. We performed immunohistochemical staining for IgG4, IgG, and CD31, and assessed the Boston consensus statement criteria for IgG4-related disease in IgG4-positive samples. We also reviewed patient records and supplemented follow-up data with a questionnaire among patients with IgG4-positive samples. IgG4-positive plasma cells (range 1–344) were found in 131 samples. Among these, 19 samples were classified as IgG4-positive (≥70 IgG4-positive plasma cells/high-power field). Two IgG4-positive samples were histologically highly suggestive of IgG4-related disease, but only one had a clinically confirmed diagnosis of IgG4-related disease. Our results indicate that patients with sialadenitis and sialolithiasis often present with IgG4-positive lymphoplasmacytic infiltrates, but exceedingly rarely present with genuine IgG4-related disease. In sialolithiasis without sialadenitis, IgG4-positive plasma cells are often absent or appear in small numbers. These results support our hypothesis of a continuum of disease, and indicate that progressive inflammation of the submandibular gland leads to the development of more specific pathological features over time.