Thomas Edison once said, “I have not failed, I’ve just found 10,000 ways that won’t work”. Considering this advice is coming from one of the greatest innovators of all time, it seems wise to follow. While the idea of failure has long been discouraged by society, in data science, failure takes an important role in the development of machine learning and artificial intelligence. The fail fast ideology has been adopted as a system design for top companies, and has proven its importance as part of the scientific method. Needless to say, failure is a necessary cycle of life. The objective though, is to fail better, to fail faster, and to learn from our mistakes so we can improve and innovate. The following are tips to help your team fail fast, so you can learn faster. The faster you can learn, the more machine learning and artificial intelligence can be deployed.
1. Collaborate and listen
Yes, it's a bit taboo, but it is known that data scientists sift through many failing models before they find a winner that transforms the face of artificial intelligence. Most data scientists produce thousands of models before they find a winner, meaning thousands fail. But, how can data science teams learn without understanding each others failures? It's important to build on each other's work, and iterate separately in order to build better AI models. You never know when a team might acquire new data that was latent in the first iteration of a model, and might want to revisit it. Understanding failures can save massive amounts of time, ensuring teams don’t repeat the same mistakes or follow a dead end.
2. No humility with reproducibility
Aside from the idea that failure is a scary word, most often failure is not discussed because there is no means to understanding failures. While data science requires failure as part of the process, it is often the most difficult to do in practice. It is generally not supported in the workflow, and can often be unreasonably time consuming and complicated to support such a structure. First, data scientists use endless amounts of tools and languages to produce data science. Data scientists don’t follow one particular flow, there are many different methods and often data scientists need to come up with creative solutions to reach results. This makes it extremely difficult to replicate without tracking and storing models and meta-data, including every step such as parameters, code version, metrics and more. Some companies are lucky enough to support long term projects that will solve simple issues like reproducibility by organizing and formatting all projects into one searchable system, but many don’t have the luxury to spend time on such projects. Now, instead of spending weeks on a dead end model, invest in a structure that will support 100% reproducibility of models and avoid wasted time and possible humiliation.
3. Use resources wisely
Often, data scientists will spend a lot of time and resources such as computing power for models that never had a chance to begin with. Unfortunately the symptoms for a losing model are not always visible at the forefront. There might be a brilliant algorithm that provides outstanding results, though the computing power is too high, or it takes way too much time to deploy. This is important when selecting the best possible model for production.
By tracking every possible model combination, and setting micro-parameters, data scientists can quickly sift out the bad models and embrace the good.
Recording all the steps taken to reach a particular result is arguably the most important aspect of an efficient data science workflow. It is especially important to record those projects that did not lead anywhere. Otherwise, time will be lost in future exploring avenues that have already proved to be dead ends.