Writing shaders is similar in many ways to other software engineering tasks. A good dose of common sense can go a long way. Software engineering principles should be applied just as for any other software engineering task. This is especially true in these early generations of programmable graphics hardware. Shader development is more challenging because early OpenGL Shading Language implementations might be incomplete in some ways, compilers will be immature, performance characteristics may differ widely between vendors, and tools to aid shader development are in their infancy. RenderMonkey is one shader development tool that is available now; hopefully others will rapidly follow.
On the other hand, writing shaders for programmable graphics hardware presents some unique challenges. Good decisions need to be made about how to split the computation between the CPU, the vertex processor, and the fragment processor. It is useful to have a solid foundation in mathematics and computer graphics before attempting to write shaders. Thorough knowledge of how OpenGL works is also a key asset, and having some understanding of the underlying graphics hardware can be helpful. It often pays to collaborate with an artist when developing a shader. This can help you develop a shader that is parameterized in such a way that it can be put to a variety of uses in the final application.