How do developers tackle this problem generally? Creating game AI is certainly not the most formal of processes in software development; it requires many ad-hoc modifications and hours of empirical evaluation. The games industry is somewhat immature as a whole, but game AI is probably the furthest behind.
Until recently, AI code took barely a few hundred lines of code, being hacked together with a couple of months to spare before the deadline. Since the turn of the millennium, AI has suddenly been propelled into the limelight and it is expected to scale up—often becoming a centerpiece of the design.
A few different aspects about the typical approach to creating game AI merit analysis, notably the integration with the engine, the design of the system, and the guidelines used throughout the development.
Integration and Design
Historically, in-game agents were simulated as part of the game logic. The 2D positions of the characters were updated at each iteration, like in Pac-Man. Since then, things have moved on slowly. Thankfully, the AI code now is generally separated from the logic. However, the agent is generally given direct access to the game data, free to extract whatever it needs. The separation is only a programming trick to simplify the codebase.
As for software design, AI subsystems are often created as separate libraries (to handle movement, for example). Naturally, these are created in different ways, depending on the development style. The most common approach in game development is the hands-on approach, where you incrementally build up the interface and required functionality from scratch. This might not sound formal—at least in terms of classical software design—but modern agile approaches acknowledge the benefits of such rapid iterations. This is certainly well suited to AI because a large amount of experimentation is required.
Beyond these modular components, the code can become somewhat unmanageable because the unexpected complexity of the task or the deadline pressures sometimes catch programmers. For example, the AI for Return to Castle Wolfenstein uses C function pointers to simulate a finite-state machine, an approach that quickly becomes almost impossible to understand by anyone but the original developer.
Because of the limited resources available, the only important guideline for AI developers is to cut corners wherever possible yet still achieve efficiency. The responsibility of the AI itself is to control characters in a realistic fashion. How this is achieved under the hood is irrelevant. Many applicable techniques could be borrowed from AI research, but few are used in practice. All the common techniques used in classical game AI (for instance, search and scripting) arguably have their roots in computer science instead.
Often, the combination of these two requirements (efficiency and realism) leads to simple AI solutions, such as scripts. This approach is entirely justifiable because it solves the problem in a way that designers can easily control.
This typical approach to game AI has been finely tuned over the years and seems to have reached satisfactory levels. AI in games is competent enough to not stand out. The standard design and integration has immediate benefits (such as simplicity), but can prove inconvenient in many ways.
First, letting the AI access information in the game directly is both dangerous and unnecessary. Second, manually developing all the behaviors for the AI can be tedious using plain programming and causes an exponential growth in time required.
There is undoubtedly room for improvement over these standard approaches, as shown by recent progress in game AI. There is an underlying trend in these innovations.