PurposeThis paper aims to develop a quality evaluation model for artificial intelligence (AI)-based products/services that is applicable to startups utilizing AI technology. Although AI-based service has risen dramatically and replaced many service offerings, in reality, startups are rarely to develop and evaluate AI services. The features of AI service are fundamentally different from the properties of existing services and have a great influence on the customer's service selection.Design/methodology/approachThis paper reviews startups' development process, existing quality evaluation models and characteristics of services utilizing AI technology, and develops a quality evaluation model for AI-based services. A detailed analysis of a survey (application of the model) on customer satisfaction for AI speakers is provided.FindingsThis paper provides seven key features and 24 evaluation items for evaluating AI-based services.Originality/valueThis paper contributes to the growing need for methodologies that reflect the new era of AI-based products/services in quality evaluation research.