"What if we train our people and they leave? ... What if you don't train them and they stay?"
Rarely do you get so much wisdom from < 140 chars. I have seen the latter scenario play out several times and it ain't pretty. In my swamp of software security, there is no way around training because quite simply software people don't know enough about security and security people don't know enough about software. (hey that might fit in under 140 chars!).
There is actually another major benefit beyond knowledge transfer, very often in software security training, you get a decent percent of the top software people and top security people in the company in a room together for two days. This very rarely happens, there is a lot of peer to peer networking that happens. I work to facilitate as much of this as possible.
One of the major benefits in doing in house training is that we don't need to use generic industry wide Petshop type examples, instead we take the company's gnarliest problems and use it as the case studies for the class. More often than not we come up with new ways of looking at the problems and new ideas on approaching a solution.
So yeah, don't worry about keeping your people non trained and in house, instead bring as much knowledge in house as you can.
We've tried to manage our Motley Fool business the Drucker way for 15 years, too. Lemme ask you a question. I'm putting you in charge of your human resources group, in charge of your corporate culture, and you have two choices of how to spend $100,000 -- your call, here, Drucker or Anti-Drucker -- these are the two ways you can invest:
(a) a program to try to get D- employees up to a C, people who have motivational and/or social problems, toxicly bring down the teams they're on, and clearly aren't fit to be hired even by weak competitors in your field, or
(b) a program to invest in your stars -- invest in more training, outside experiences, and leadership opportunities for your best employees
In my swamp of software security training, I have trained both an organization's average developers as well as their top people. I think there is validity to both, because security is a system problem, but the approaches you use are slightly different. In the case of getting the general developer population up to speed you usually want to aim for a checklist type approach and give the developers something like a cookbook full of recipes they can follow. So you are heavier on the prescription of what to do and less about the why.
When I am training the top architects, developers, and security people, then I usually aim to give more of a security architectureframework, so they can then apply the framework depending on the specific problem context.
Developing and delivering training begins by understanding if its a raise the floor effort or a raise the ceiling effort. I think they are both very valid today, so many developers have not had one single day of software security training, and it shows all the way across the software development industry. I did a training class recently, about 15 people in the class at the beginning I asked how many people had been programming at least 5 years, all the hands went up but one. I asked how many people had ever had a single day of software security training. One hand went up. Of course, they had all had multiple week or months of training on j2EE, Weblogic, WAS, SAP, etc. but not a day on security. Software security is easily as complex as programming Weblogic. Would you turn a developer loose on your production Weblogic with no training?
A funny postscript walking out of the room there was one person who had been programming a short time, like 18 months, another had almost 20 years, by the end of the day they both had the same amount of software security training