One of the main bioactive compounds of interest from the Ulva species is the sulfated polysaccharide ulvan, which has recently attracted attention for its anticancer properties. This study investigated the cytotoxic activity of ulvan polysaccharides obtained from Ulva rigida in the following scenarios: (i) in vitro against healthy and carcinogenic cell lines (1064sk (human fibroblasts), HACAT (immortalized human keratinocytes), U-937 (a human leukemia cell line), G-361 (a human malignant melanoma), and HCT-116 (a colon cancer cell line)) and (ii) in vivo against zebrafish embryos. Ulvan exhibited cytotoxic effects on the three human cancer cell lines tested. However, only HCT-116 demonstrated sufficient sensitivity to this ulvan to make it relevant as a potential anticancer treatment, presenting an LC50 of 0.1 mg mL−1. The in vivo assay on the zebrafish embryos showed a linear relationship between the polysaccharide concentration and growth retardation at 7.8 hpf mL mg−1, with an LC50 of about 5.2 mg mL−1 at 48 hpf. At concentrations near the LC50, toxic effects, such as pericardial edema or chorion lysis, could be found in the experimental larvae. Our in vitro study supports the potential use of polysaccharides extracted from U. rigida as candidates for treating human colon cancer. However, the in vivo assay on zebrafish indicated that the potential use of ulvan as a promising, safe compound should be limited to specific concentrations below 0.001 mg mL−1 since it revealed side effects on the embryonic growth rate and osmolar balance.