Advances in sensing and networking hardware have made the prospect of ambient intelligence more realistic, but the challenge of creating a software framework suitable for ambient intelligence systems remains. We present ADE, the Agent Development Environment, a distributed agent infrastructure with built-in natural language processing capabilities connected to a sophisticated goal manager that controls access to the world via multiple server interfaces for sensing and actuating devices. Unlike other ambient intelligence infrastructures, ADE includes support for multiple autonomous robots integrated into the system. ADE allows developers of ambient intelligence environments to implement agents of varying complexity to meet the varying requirements of each scenario, and it provides facilities to ensure security and fault tolerance for distributed computing. Natural language processing is conducted incrementally, as utterances are acquired, allowing fast, accurate responses from system agents. We describe ADE and a sample of the many experiments and demonstrations conducted using the infrastructure, then an example architecture for a "smart home" is proposed to demonstrate ADE's utility as an infrastructure for ambient intelligence.