{"id":667,"date":"2015-11-11T14:38:19","date_gmt":"2015-11-11T14:38:19","guid":{"rendered":"http:\/\/payne.org\/blog\/?p=667"},"modified":"2017-09-09T11:57:01","modified_gmt":"2017-09-09T11:57:01","slug":"deep-learning-a-sport-of-kings","status":"publish","type":"post","link":"https:\/\/payne.org\/blog\/deep-learning-a-sport-of-kings\/","title":{"rendered":"Deep Learning:  A Sport of Kings?"},"content":{"rendered":"<p>The big news in the machine learning\/deep learning world this week is Google\u2019s release of <a href=\"http:\/\/www.tensorflow.org\/\">TensorFlow<\/a>, their deep learning toolkit. This has prompted some\u00a0to ask: why would they give away \u201ccrown jewels\u201d for such a strategic technology? The question is best answered with a machine learning joke (paraphrased): \u201c<em>the winners usually have the most data, not the best algorithms<\/em>\u201d.<\/p>\n<p><a href=\"https:\/\/en.wikipedia.org\/wiki\/Artificial_neural_network\">Neural networks<\/a> have been around for a while, but it\u2019s only been within the past 10 yrs that researchers have figured out how to train networks with many, many layers (the \u201cdeep\u201d in \u201cdeep learning\u201d). That research has been greatly accelerated by using GPUs as very high-performance, <a href=\"http:\/\/payne.org\/blog\/gpu-overshadows-cpu\/\">general purpose, vector processors<\/a>. If a researcher can turn around an algorithm experiment in a day (<em>vs<\/em> 3 months), a lot more research gets done.<\/p>\n<p>But as the joke suggests, it\u2019s <a href=\"https:\/\/www.youtube.com\/watch?v=7PCkvCPvDXk\">all about that data<\/a>: you need lots and <strong>lots<\/strong> and <strong>LOTS<\/strong> of data to train a high-performance deep learning network. And Google has more data than anyone else\u200a\u2014so they don\u2019t worry so much about giving away algorithms.<\/p>\n<p>(Also, Google, Baidu, Twitter, Facebook, etc. are investing in GPU compute clusters that can only be described as the new \u201cmainframe supercomputers\u201d. Sure, you can rent GPU instances on Amazon, but there\u2019s nothing like having the <a href=\"http:\/\/www.nvidia.com\/object\/tesla-supercomputing-solutions.html\">latest Nvidia board<\/a>\u00a0with lots of RAM and very high-performance interconnect).<\/p>\n<p>What does this all mean for early stage startups? The situation creates several tough hurdles: first, freely available code and technology from Google (<a href=\"http:\/\/www.infoworld.com\/article\/2871752\/machine-learning\/facebook-open-sources-its-machine-learning-magic.html\">and Facebook<\/a>) enables competitors and devalues whatever\u00a0the startup might develop. Second, few startups have access to a large enough proprietary data source to compete at scale. And third, GPU compute clusters need real capital.<\/p>\n<p>What\u2019s left for startups? I see at least two interesting patterns:<\/p>\n<ul>\n<li><strong>Using deep learning as a key feature to enhance another app.<\/strong> \u00a0Use freely available technology\u00a0to add magic. \u00a0Google Photos is a great example of this, and I think every photo and video app will soon be able to recognize stuff, people, people, items, etc. to enhance the functionality.<\/li>\n<li><strong>&#8220;Man-teaches-machine&#8221;.<\/strong> \u00a0Start out with a lot of humans doing some task and capture their work to train a\u00a0network. \u00a0Over time, have the network handle the common cases, with the exceptions \/ ambiguous cases routed to humans for resolution. \u00a0Build a large, proprietary training set, enjoy compounded interest, and profit.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>The big news in the machine learning\/deep learning world this week is Google\u2019s release of TensorFlow, their deep learning toolkit. This has prompted some\u00a0to ask: why would they give away \u201ccrown jewels\u201d for such a strategic technology? The question is &hellip; <a href=\"https:\/\/payne.org\/blog\/deep-learning-a-sport-of-kings\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[],"class_list":["post-667","post","type-post","status-publish","format-standard","hentry","category-software"],"_links":{"self":[{"href":"https:\/\/payne.org\/blog\/wp-json\/wp\/v2\/posts\/667","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/payne.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/payne.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/payne.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/payne.org\/blog\/wp-json\/wp\/v2\/comments?post=667"}],"version-history":[{"count":2,"href":"https:\/\/payne.org\/blog\/wp-json\/wp\/v2\/posts\/667\/revisions"}],"predecessor-version":[{"id":669,"href":"https:\/\/payne.org\/blog\/wp-json\/wp\/v2\/posts\/667\/revisions\/669"}],"wp:attachment":[{"href":"https:\/\/payne.org\/blog\/wp-json\/wp\/v2\/media?parent=667"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/payne.org\/blog\/wp-json\/wp\/v2\/categories?post=667"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/payne.org\/blog\/wp-json\/wp\/v2\/tags?post=667"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}