Aims. Many studies of the North-South asymmetry of solar activity and its features have been performed. However, most of these studies do not consider whether or not the asymmetry of the time series under consideration is statistically significant. If the asymmetry is statistically insignificant, any study about its behavior is meaningless. Here, we discuss the difficulties found when trying to assess the statistical significance of the North-South asymmetry (hereafter SSNSA) of the most usually considered time series of solar activity. Methods. We distinguish between solar activity time series composed of integer or non-integer and dimensionless data, or composed of non-integer and dimensional data. For each of these cases, we discuss the most suitable statistical tests which can be applied and highlight the difficulties in obtaining valid information about the statistical significance of solar activity time series. Results. Our results suggest that, apart from the need to apply suitable statistical tests, other effects such as data binning, the considered units and the need, in some tests, to consider groups of data, substantially affect the determination of the statistical significance of the asymmetry. Conclusions. The assessment of the statistical significance of the N-S asymmetry of solar activity is difficult and an absolute answer cannot be given, since many different effects influence the results given by the statistical tests. The quantitative results about the statistical significance of the N-S asymmetry of solar activity provided by different authors, as well as studies of its behaviour, must be considered with care because they depend on the chosen values of different parameters or on the considered units.