Purpose -Many open source software packages are available for organizations and individuals to create digital libraries (DLs). However, a simple to use instrument to evaluate these DL software packages does not exist. The objectives of the present work are to develop a checklist for DL evaluation and use this checklist on four DL software packages. Design/methodology/approach -Features that characterized "good" open source DL software were determined from the literature. First identified were essential categories of features that DL software should possess. These categories were then decomposed into supporting features. From these, a checklist that covered all such features was developed. The checklist was then used to evaluate four popular open source DL software packages (CDSware, EPrints, Fedora, and Greenstone) for the purposes of assessing suitability for use in a DL project to be undertaken by the authors. Findings -A checklist consisting of 12 categories of items was developed. Using this, Greenstone was found to be the best performer, followed by CDSware, Fedora and EPrints. Greenstone was the only software package that consistently fulfilled the majority of the criteria in many of the checklist categories. In contrast, EPrints was the worst performer due to its poor support for certain features deemed important in our checklist, and a total absence of functionality in other categories. Originality/value -The present work attempts to develop a comprehensive checklist for assessing DLs. Its flexibility allows users to tailor it to accommodate new categories, items and weighting schemes to reflect the needs of different DL implementations.