Asynchronous Evolution of Data Mining Workflow Schemes by Strongly Typed Genetic Programming

Pilát, Martin; Křen, Tomáš; Neruda, Roman

doi:10.1109/ictai.2016.0094

Cited by 6 publications

(3 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The ML Bazaar also specifies ML primitives via JSON [54]. Another type-based system for building ML pipelines is described by [47]. These systems can benefit from JSON subschema checking to avoid running and deploying incompatible ML pipelines (Section 5.2.2).…”

Section: Applications Of Subschema Checksmentioning

confidence: 99%

Finding data compatibility bugs with JSON subschema checking

Habib

Shinnar²,

Hirzel³

et al. 2021

Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

JSON is a data format used pervasively in web APIs, cloud computing, NoSQL databases, and increasingly also machine learning. To ensure that JSON data is compatible with an application, one can define a JSON schema and use a validator to check data against the schema. However, because validation can happen only once concrete data occurs during an execution, it may detect data compatibility bugs too late or not at all. Examples include evolving the schema for a web API, which may unexpectedly break client applications, or accidentally running a machine learning pipeline on incorrect data. This paper presents a novel way of detecting a class of data compatibility bugs via JSON subschema checking. Subschema checks find bugs before concrete JSON data is available and across all possible data specified by a schema. For example, one can check if evolving a schema would break API clients or if two components of a machine learning pipeline have incompatible expectations about data. Deciding whether one JSON schema is a subschema of another is non-trivial because the JSON Schema specification language is rich. Our key insight to address this challenge is to first reduce the richness of schemas by canonicalizing and simplifying them, and to then reason about the subschema question on simpler schema fragments using type-specific checkers. We apply our subschema checker to thousands of real-world schemas from different domains. In all experiments, the approach is correct whenever it gives an answer (100% precision and correctness), which is the case for most schema pairs (93.5% recall), clearly outperforming the state-of-the-art tool. Moreover, the approach reveals 43 previously unknown bugs in popular software, most of which have already been fixed, showing that JSON subschema checking helps finding data compatibility bugs early.

show abstract

Section: Applications Of Subschema Checksmentioning

confidence: 99%

Finding data compatibility bugs with JSON subschema checking

Habib

Shinnar²,

Hirzel³

et al. 2021

Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

show abstract

“…Neuroevolution performs well for traditional benchmark tasks, such as the knapsack problems (Denysiuk et al 2019), but also real-life robotics problems (Zimmer and Doncieux 2017). Evolution-based approaches were also successfully adopted for the task of scientific workflow discovery (Pilat et al 2016), offering symbolic descriptions of data mining workflows, directly applicable in practice. Neuroevolution Stanley et al (2019) approaches have shown promising results in the domain of computer vision, where more efficient neural networks were evolved with minimal performance trade-offs (Zoph et al 2018).…”

Section: Evolutionary Computation and Learningmentioning

confidence: 99%

autoBOT: evolving neuro-symbolic representations for explainable low resource text classification

et al. 2021

View full text Add to dashboard Cite

Learning from texts has been widely adopted throughout industry and science. While state-of-the-art neural language models have shown very promising results for text classification, they are expensive to (pre-)train, require large amounts of data and tuning of hundreds of millions or more parameters. This paper explores how automatically evolved text representations can serve as a basis for explainable, low-resource branch of models with competitive performance that are subject to automated hyperparameter tuning. We present autoBOT (automatic Bags-Of-Tokens), an autoML approach suitable for low resource learning scenarios, where both the hardware and the amount of data required for training are limited. The proposed approach consists of an evolutionary algorithm that jointly optimizes various sparse representations of a given text (including word, subword, POS tag, keyword-based, knowledge graph-based and relational features) and two types of document embeddings (non-sparse representations). The key idea of autoBOT is that, instead of evolving at the learner level, evolution is conducted at the representation level. The proposed method offers competitive classification performance on fourteen real-world classification tasks when compared against a competitive autoML approach that evolves ensemble models, as well as state-of-the-art neural language models such as BERT and RoBERTa. Moreover, the approach is explainable, as the importance of the parts of the input space is part of the final solution yielded by the proposed optimization procedure, offering potential for meta-transfer learning.

show abstract

“…The ML Bazaar also specifies ML primitives via JSON [23]. Another type-based system for building ML pipelines is described by [20]. These systems could benefit from JSON subschema checking to avoid running and deploying incompatible ML pipelines.…”

Section: Applications Of Subschemasmentioning

confidence: 99%

Type Safety with JSON Subschema

Habib,

Shinnar,

Hirzel

et al. 2019

Preprint

View full text Add to dashboard Cite

JSON is a popular data format used pervasively in web APIs, cloud computing, NoSQL databases, and increasingly also machine learning. JSON Schema is a language for declaring the structure of valid JSON data. There are validators that can decide whether a JSON document is valid with respect to a schema. Unfortunately, like all instance-based testing, these validators can only show the presence and never the absence of a bug. This paper presents a complementary technique: JSON subschema checking, which can be used for static type checking with JSON Schema. Deciding whether one schema is a subschema of another is non-trivial because of the richness of the JSON Schema specification language. Given a pair of schemas, our approach first canonicalizes and simplifies both schemas, then decides the subschema question on the canonical forms, dispatching simpler subschema queries to type-specific checkers. We apply an implementation of our subschema checking algorithm to 8,548 pairs of real-world JSON schemas from different domains, demonstrating that it can decide the subschema question for most schema pairs and is always correct for schema pairs that it can decide. We hope that our work will bring more static guarantees to hardto-debug domains, such as cloud computing and artificial intelligence. IntroductionJSON (JavaScript Object Notation) is a data serialization format that is widely adopted to store data on disk or send it over the network. Derived from JavaScript, JSON is both humanand machine-readable, and there are now JSON parsers for many programming languages. JSON supports primitive data types, such as strings, numbers, and Booleans, and two data structures: arrays, which represent lists of values, and objects, which represent maps of key-value pairs. The data types can be nested, e.g., to have an array of two objects that each map a key to some primitive value.JSON is used in numerous applications. It is the most popular data exchange format in web APIs, ahead of XML [21]. Cloud-hosted applications also use JSON pervasively, e.g., in micro-services that communicate via JSON data [17]. On the data storage side, not only do traditional database management systems, such as Oracle, IBM DB2, MySQL, and PostgreSQL, now support JSON, but two of the most widely deployed NoSQL database management systems, MongoDB and Cloudant, are entirely based on JSON. Beyond web, cloud, and database applications, JSON is also gaining adoption in artificial intelligence (AI) [13,23].

show abstract

Asynchronous Evolution of Data Mining Workflow Schemes by Strongly Typed Genetic Programming

Cited by 6 publications

References 11 publications

Finding data compatibility bugs with JSON subschema checking

Finding data compatibility bugs with JSON subschema checking

autoBOT: evolving neuro-symbolic representations for explainable low resource text classification

Type Safety with JSON Subschema

Contact Info

Product

Resources

About