AI Builds Cities Through Time: A New Test for Language Models

KI researcher Ethan Mollick from the Wharton School at the University of Pennsylvania challenged current AI models to build and simulate a harbor city over 6,000 years, from 3000 BCE to 3000 CE. Mollick created the “AI Harbot Town Gallery” website to showcase how the models perform with this complex task, which requires coding, design, decision-making, and knowledge.

He assesses the models on code quality and overall design, noting the number of attempts and errors. GPT-3.5 Turbo and GPT-4 performed poorly, requiring additional instructions and producing rudimentary outputs, a 3D cube and a flat surface with blocks, respectively.

Claude 4.8 Opus Extra achieved the best results, generating a dynamic simulation of a harbor city that evolves over time with features like changing ships and controllable timelines. Mollick hopes this method provides a more insightful evaluation of AI capabilities than traditional benchmarks.