Shyam is the Chief Architect in the IT Department of YourBank and, as can be expected, plays a key role in all the bank’s major digital initiatives. In one of the bank’s highly visible mission-critical AI/ML program, there is a need to build a complex ML engine that will sit right at the core of the target system. After deliberating and researching on whether to build the whole thing or buy and integrate, Shyam comes across an open source framework that he feels should fit much of what he was looking for. If you were Shyam, how will you approach this?
Since the ML engine will be at the core of the mission-critical platform, it is extremely important for you to approach this challenge in a systematic way, assessing risk versus return. Here’s how you will approach this:
- Research about the open source framework thoroughly to get a sound understanding of the functionality offered, assumptions on input data, maturity of the framework, experience of others who have used the framework, model accuracy, precision and recall, etc.
- Evaluate how the architecture, design, and technology behind the framework will fit into your overall target system architecture and organizational standards
- Look for trusted information on non-functional aspects of the framework where it has been used by others before
- Check the interoperability of the framework with your target environment, especially in the context of the specific cloud provider environments that are firm-wide standard
- Estimate the end-to-end efforts for customization and integration of the framework, validation of the overall architecture and stress testing to the expected performance profile of the target system, and ensuring compliance to the firm’s security architecture
- Weigh all this against the estimates for building the whole thing instead of buying and integrating the framework
- Best case will be if the open source framework fits so well with all the above that you can treat it as a black box and just focus on the integration aspects. Worst case will be if you really need to get down to understanding the framework’s code line by line to be able to trust it to meet your needs.