The claws are out. Adobe’s Platform Evangelist, Lee Brimelow retaliated today against Apple blocking Flash developers on the iPhone with a post on his Flash Blog.

Brimelow holds little back, lambasting the company for trying to exert a “tyrannical control over developers…more importantly, wanting to use developers as pawns in their crusade against Adobe.” He says any real developer could not support Apple’s moves in “good conscience.”

“Personally I will not be giving Apple another cent of my money until there is a leadership change over there. I’ve already moved most of my book, music, and video purchases to Amazon and I will continue to look elsewhere. Now, I want to be clear that I am not suggesting you do the same and I’m also not trying to organize some kind of boycott.”

Brimelow may not be explicitly calling for a boycott (although he is requesting a change in leadership— is he looking at you Jobs?).  But he is issuing a call to arms to developers. The lines are clear: you either stand with us or against us and if you’re against us (and complicit in Apple’s policies) then you’re not a real developer.

And if his sentiment wasn’t clear throughout the post, he caps it all off with: “Go Screw Yourself Apple.”

I don’t expect Apple to respond directly to Brimelow’s rant— it’s not Adobe’s official statement (however, Adobe has clearly seen the post and filtered it: the second paragraph notes “[Sentence regarding Apple's intentions redacted at request from Adobe]“). Nevertheless, they are fighting words. The company must be furious that Apple is potentially locking it out of its products. The problem is there’s very little that Adobe can do besides stomp its feet. At the end of the day, Apple is not obligated to support Adobe’s developers.

The software developer, for its part, is coming to terms with what this all could mean for their bottom line. In a 10-Q filing (released today) Adobe says:

“To the extent new releases of operating systems or other third-party products, platforms or devices, such as the Apple iPhone or iPad, make it more difficult for our products to perform, and our customers are persuaded to use alternative technologies, our business could be harmed.”

UPDATE: They downplayed that “harm” in Adobe CTO Kevin Lynch’s formal response. Lynch says CS5 will thrive with or without Apple’s support. Loosely translated: guess what Apple, you don’t own innovation, watch your back.

“Yesterday Apple released some proposed changes to their SDK license restricting the technologies that developers can use, including Adobe software and others such as Unity and Titanium.

First of all, the ability to package an application for the iPhone or iPad is one feature in one product in Creative Suite. CS5 consists of 15 industry-leading applications, which contain hundreds of new capabilities and a ton of innovation. We intend to still deliver this capability in CS5 and it is up to Apple whether they choose to allow or disallow applications as their rules shift over time.

Secondly, multiscreen is growing beyond Apple’s devices. This year we will see a wide range of excellent smartphones, tablets, smartbooks, televisions and more coming to market and we are continuing to work with partners across this whole range to enable your content and applications to be viewed, interacted with and purchased.”

Full Text of Brimelow’s blog:

Apple Slaps Developers In The Face

By now you have surely heard about the new iPhone 4.0 SDK language that appears to make creating applications in any non-Apple-approved languages a violation of terms. Obviously Adobe is looking into this wording carefully so I will not comment any further until there is an official conclusion.

[Sentence regarding Apple's intentions redacted at request from Adobe]. This has nothing to do whatsoever with bringing the Flash player to Apple’s devices. That is a separate discussion entirely. What they are saying is that they won’t allow applications onto their marketplace solely because of what language was originally used to create them. This is a frightening move that has no rational defense other than wanting tyrannical control over developers and more importantly, wanting to use developers as pawns in their crusade against Adobe. This does not just affect Adobe but also other technologies like Unity3D.

I am positive that there are a large number of Apple employees that strongly disagree with this latest move. Any real developer would not in good conscience be able to support this. The trouble is that we will never hear their discontent because Apple employees are forbidden from blogging, posting to social networks, or other things that we at companies with an open culture take for granted.

Adobe and Apple has had a long relationship and each has helped the other get where they are today. The fact that Apple would make such a hostile and despicable move like this clearly shows the difference between our two companies. All we want is to provide creative professionals an avenue to deploy their work to as many devices as possible. We are not looking to kill anything or anyone. This would be like us putting something in our SDK to make it impossible for 3rd-party editors like FDT to work with our platform. I can tell you that we wouldn’t even think or consider something like that.

Many of Adobe’s supporters have mentioned that we should discontinue the Creative Suite products on OS X as a form of retaliation. Again, this is something that Adobe would never consider in a million years. We are not looking to abuse our loyal users and make them pawns for the sake of trying to hurt another company. What is clear is that Apple most definitely would do that sort of thing as is evidenced by their recent behavior.

Personally I will not be giving Apple another cent of my money until there is a leadership change over there. I’ve already moved most of my book, music, and video purchases to Amazon and I will continue to look elsewhere. Now, I want to be clear that I am not suggesting you do the same and I’m also not trying to organize some kind of boycott. Me deciding not to give money to Apple is not going to do anything to their bottom line. But this is equivalent to me walking into Macy’s to buy a new wallet and the salesperson spits in my face. Chances are I won’t be buying my wallets at Macy’s anymore, no matter how much I like them.

Now let me put aside my role as an official representative of Adobe for a moment as I would look to make it clear what is going through my mind at the moment. Go screw yourself Apple.

Alfred Lin, the COO/CFO of Zappos, has left the company and will join Sequoia Capital. Lin was the no. 2 executive at Zappos at the time of its acquisition by Amazon, and has had a nearly flawless resume as an entrepreneur over the years. Every company he’s worked for has been acquired, and the smallest deal was $265 million. See our post “Alfred Lin Has The Midas Touch: The Man With $2 Billion In Acquisitions Under His Belt.”

Lin isn’t leaving Zappos until the beginning of 2011.

Here’s the email Lin sent to all employees at Zappos earlier today:

Date: Wed, 24 Mar 2010 15:13:50 -0700
From: Alfred Lin
To: employees@zappos.com
Subject: Life Changing Moments

First, I’m sorry for letting you know all this information in an email. If it was doable, I would have preferred to speak to each of you personally, but that is sadly not really possible. The idea of thinking through whom I should tell first and in what order was also very agonizing. Being true to our core values, it is just a lot easier to be open and honest and let everyone know sooner rather than later. Although this still feels a little weird as I write this, we don’t shy away from weirdness at Zappos, so here it is…

First, Rebecca (my wife and my true boss) and I are getting into the baby business. Rebecca’s now 9 weeks along and so far so good. I’ve learned more about the baby business than I would have even expected, like why, in general, you might not want to tell too many people until after the first 12 weeks. Or why, when a woman reaches a certain age, her pregnancy gets classified as a high-risk pregnancy. (I am probably going to get hit over the head for that.) In any case, keep your fingers crossed and we are really delighted to share this happy news with all of you.

Over the past 2 months, Rebecca and I have talked a lot more about what is important to us and what we should do as a family. It is the general stuff you talk about, but you may only talk about superficially until it hits you that you are actually going to be a parent. As agonizing as it is for us to come to this conclusion, we really believe that the right place for us as a family is to move back to the SF Bay Area and be closer to some of our extended family.

To complicate the life discussions, if you had asked me 5 or 10 years ago what I am passionate about and what my higher purpose would be, I would have said that my calling is to help interpret the visions and dreams of entrepreneurs about how the world should be and help build businesses around those visions and dreams. So to that end, I wanted to be a venture capitalist and join Sequoia Capital. They’ve financed and helped built some really special and enormously successful companies, including Google, Yahoo, Paypal, YouTube, Cisco, Oracle, Apple, and also Zappos. I had explored joining them twice, but things never really worked out before. Recently, I went out to beers and then dinner with a few of their partners to catch up on life and in their inebriated state, they made the silly mistake of extending me a position on their team roster (although the envelope was actually marked “For Rebecca”).

You might be shocked to read that I have decided to leave Zappos. It has been a hard decision, and I am still in a bit of a daze myself, but I am also very excited. I have learned so much from all of you along our journey and I am excited to put that knowledge to work to help finance and help build companies in the special ways we have built Zappos. So the silver lining is that hopefully Zappos will give birth to other very special and enormously successful companies.

Tony, Fred and I have had a few conversations with some folks at Amazon and they are committed to helping us find the right person to fill my sample-size shoes (9D). As always with any Zappos employee, that person will, first and foremost, have to live and breathe the Zappos culture, and help continue the good work to build the Zappos brand, business, and culture, independent of Amazon.

Now that the news is out, I also want to assure you that I am not going anywhere anytime soon. I’m committed to Tony, Fred, Amazon, and each and every one of you to making this a very smooth transition. To that end, I plan to stay at Zappos through January 3, 2011. Of course, this timeline may change, since our business continues to grow fast and each and every one of you do such an amazing job that I am sure the transition can and may be completed sooner.

In any case, while parting is such sweet sorrow, it is not time to say our goodbyes yet. We have some work to do and always will…like taking first place in the Corporate Challenge relay tonight. Hope to see you there.

Update: Zappos CEO Tony Hsieh emails employees.

Back in July 2008, we wrote about a company called Paragon Lake that had raised $5.8 million to put kiosks in jewelery stores to help you design custom jewelry. Earlier this year, the company got a new name (it’s now called Gemvara), its CEO departed (it’s now being led by founder Matt Lauzon), and it has a new business model: it’s now a consumer facing site that allows users to create their own custom jewelery from the comfort of their home. Today, the company is announcing that it has raised another $5.2 million in Series B funding. The new round was led by Highland Capital Partners and Canaan Partners, who both led Gemvara’s Series A round.

The new site allows customers to choose from a variety of jewelry designs submitted by artists and designers. Customers can customize each of these designs using one of 7 metals and 16 gemstones (and the number of options is growing), and the piece is manufactured ‘just-in-time’. The artists who designed each piece receive a commission every time one of their works is sold, which generally ranges between 1-10% (and is usually around 5%).

Lauzon says that the company will still support the nearly fifty jewelers who have Gemvara’s kiosks already in their stores, but that Gemvara has shifted its focus to its consumer-facing site because it sees a bigger opportunity there.  The Boston-based company now has 22 employees. Other online jewelers include Blue Nile, Ice.com, and Amazon.

This guest post was written by Kleiner Perkins Caufield & Byers partners John Doerr, Bing Gordon, Chi-Hua Chien and Ellen Pao. We covered KP’s increase in the size of the iFund last week, and additional insights from KP’s Matt Murphy on the iPad. This post goes into a lot more detail on KP’s investments in the first iFund, and what opportunities they see for the iPad.

It’s hard to imagine that once there was no Internet. Just 15 years ago there was no browser, no web point-and-click. It was 1994, and Steve Jobs had left Apple. Steve was making Toy Story, and object-oriented software for Next.

Then one day Bill Joy showed me a beta version of Mosaic, the FIRST web browser. It was magic. Bill said “John, I have NO idea where this is going. You just better dive in.”

The rest of the 90’s were a ONCE-in-a-lifetime experience. Entrepreneurs created the Web, and great ventures – Netscape, Amazon, Ebay, Google, and others. And they changed our lives. Silicon Valley became the Florence of the New, Networked Economy.

The advent of the iPad feels like deja-vu, like it’s happening all over again. Not once, but TWICE-in-a-lifetime.

Inventing The Future

Newsweek put it best… “Steve has the uncanny ability to cook up gadgets we didn’t know we needed… but suddenly can’t live without.” Steve showed us what computer legend Alan Kay told us… namely, “The best way to predict the future is to invent it.”

At Kleiner Perkins we say “If you can’t INVENT the future, the next best thing… is to FUND it.”

The iFund

So two years ago Apple and Kleiner Perkins announced an unexpected collaboration: the $100 million iFund. The best venture fund for mobile entrepreneurs.

Think back to March 6, 2008. The iFund idea was risky – some thought crazy. There was no 3G iPhone. No apps. No appstore. And no iFund.

Today the iFund is 14 ventures. 3 are stealth. We are proud to have backed great entrepreneurs in communications, gaming, publishing, media, mobile advertising, and mobile commerce. The iFund companies include:

• Booyah – Creating new forms of entertainment to the masses by bringing together elements of the real world and the digital world. Booyah’s MyTown is the largest social check-in platform today, with over 1.6 million users checking in 4 million times per day.
• Cooliris – A 3D interface for browsing vast amounts of rich media content on the iPhone and web.
• GOGII – Leading free texting (textPlus), communication, and group social interaction product for mobile, serving over 5 million users who have exchanged over 2 billion messages on textPlus.
• iControl – Stay connected in real-time to what matters most – family, property, home, and business-from anywhere in the world, anytime, day or night.
• InMobi – The largest independent mobile ad network, reaching mobile users in 37 countries.
• ngmoco – The first iPhone/iPad-only games publisher with 20+ titles and the Plus Network for 3rd party developers. ngmoco’s games are installed on over one-third of all iPhone / iPod users and played over 20 million minutes per day.
• Pelago (Whrrl) – Mobile social network and discovery app that allows you to find people, places, events, and trends through the eyes of your friends and other people on the Whrrl platform.
• Pinger – A publisher of great utility and entertainment apps for iPhone, with 13 hit titles on iPhone including Doodle Buddy, Photo Chop and TextFree.
• Shazam – The world’s leading mobile music discovery and socialization application – enabling consumers to experience and share music with others.
• shopkick – Focused on the intersection of mobile and the physical retail world, shopkick’s goal is to dramatically improve the experience of consumers’ shopping experience.
• Zynga mobile – The world’s largest social gaming network, bringing social games to iPhone and iPod Touch.

Each of these companies has received capital from the iFund and participated in bi-annual iFund Summits with other founders, CEOs and industry leaders. Through this network, we’ve seen phenomenal collaboration between iFund companies and tremendous learning in a space that evolves rapidly and changes daily.

iFund ventures have had well over 100 million downloads. And we expect more than $100 million in revenue from these companies this year. We’ve already committed $100 million, and raised another $330 million from others. That’s almost ½ billion dollars.

Good News – Bad News

That’s the good news. Truly good news. But the bad news – is that at the worst possible time, on eve of the iPad, the iFund is out of money.

The first rule of venture capital is (to quote my friend Coach Campbell) is…“you gotta have the friggin’ money.” It’s hard to be a venture capitalist if you don’t have the money.

Doubling Down

So we’re doubling down on the iFund, increasing its size to $200 million dollars. Here’s the real reason why…

The original PCs in the early 80’s were pretty crummy, that is, until 1984 when Apple introduced the mouse and the Mac… Back then Alan Kay, inventor of the Dynabook tablet, said “The Mac is the first PC worth criticizing.”

Fast forward to 2007. When Steve introduced the iPhone, Alan Kay told him “Steve, make the screen size 5 by 8 inches and you’ll rule the world.”

Welcome, iPad

On Saturday (April 3) the iPad arrived. We believe it will rule the world.

I’ve touched it, held it, and caressed it. It feels gorgeous. It feels like touching the future.

It is not a big iPod. But it IS a very big deal.

The New World

We’re going from the Old World to a brave New World.

• From the Old World of the traditional, tired window interfaces… to the wonderful new world of TOUCH.
• From the Old World of Point and Click to the new SWOOSH of Fluidity.
• Instead of old, artificial, indirect interfaces, the iPad is direct and NATURAL.
• Instead of WYSIWyg – what you see is what you get – it is WYTIWis. What You Touch… IS what IS.
• Instead of holding a MOUSE, you’re holding MAGIC.

The iPad is the beginning of the New World, the post-PC era.

One More Thing… The Future

April’s iPad shipment is just the beginning. It is truly, just a beginning.

Twice in the last 15 years we’ve witnessed 100,000 flowers blooming. Flowers of applications for the Web, and then for the iPhone. Both were paradigm shifts in how we interact.

Here comes the third shift: interacting fluidly on full and fast screens with vast information stored locally. And that will start a third renaissance of software.

Beyond Browsers

It’s time to move beyond spreadsheets and word processors, beyond web sites limited by browsers… to interactive, connected applications with incredible simplicity, speed, and fluidity.

To a future that transforms games and entertainment, education and publishing, healthcare, communications and commerce.

And almost everything else.

The tablet, the iPad is where these new dreams will blossom. And where the revolution is happening.

Much More Performance

Bill Joy says the key to more performance is lower power. Over the next decade he sees 3 times better batteries, and 10 times lower power chips. So we should be able to run, for the same price, 30 times as much application.

And as for storage, there’s no reason that can’t be 30x also. Or, about a terabyte of local, faster, solid state storage. (A terabyte is several hundred movies)

Interpersonal Surfaces

What’s important is the new ways tablet computers will be used. They won’t just be reactive, responding to commands. They’ll also be proactive.

They will be much more than personal computers. They’ll be interpersonal surfaces and services. Working seamlessly, unobtrusively, and comfortably in the spaces between us, between you and me and others.

To iPADdicts and Entrepreneurs…

I like to define entrepreneurs this way: Entrepreneurs do MORE than anyone thinks possible. With LESS than anyone thinks possible.

At Kleiner we’re awed by entrepreneurs. They surprise us with the improbable – they don’t know what’s impossible.

So to Steve, Supreme Commander of the Rebel Forces, and to the entrepreneurs at Apple, congratulations!

To iPADdicts and entrepreneurs everywhere: there’s never been a better time than now to start or join a new venture, and launch a new application or service.

Whether you’re seeking $50K or a few million or a great job… contact us.

John Doerr, jdoerr@kpcb.com
Matt Murphy, mmurphy@kpcb.com
Chi-Hua Chien, cchien@kpcb.com
Bing Gordon, bgordon@kpcb.com

Aileen Lee, alee@kpcb.com
Ellen Pao, epao@kpcb.com
Ted Schlein, tschlein@kpcb.com

+1 650 233-2750

www.kpcb.com/ifund

As we’ve reported over the past few days, a number of online video sites are jumping on the HTML5 bandwagon so that their content can be viewed on the iPad (which doesn’t support Adobe’s Flash). However, some online video sites have not adjusted their video content to be iPad friendly (a.k.a these sites are still using Flash). Clicker, a comprehensive search engine for TV content on the web, is launching a special version of its site for the iPad that will only surface videos from the web that will be able to stream on the device.

Clicker, which made its debut at TechCrunch50 last fall, makes it easy to search through the vast amounts of video content available online. Clicker’s index includes over 600,000 full length TV episodes spanning 10,000 shows. The service also allows users to search through premium content including Netflix’s Instant Streaming movies and Amazon Video on Demand (though you have to pay to watch them). Additionally, the site indexes music videos, and has started teaming up with schools to index their lectures and other original content. Clicker doesn’t actually host any of this content — instead, it provides deep links that point you directly to whichever episode you want to watch.

So when iPad users access Clicker’s site on their device, they will be redirected to a specially formatted version of Clicker that has indexed all of the sites and videos on the web that are able to be viewed on the iPad. Clicker’s iPad-friendly site will index HTML5, Quicktime, and H.264 formatted videos. For now, programs available for download via the iTunes store aren’t in the Clicker catalog but should be added within the next few weeks. And as more content becomes available on the iPad, Clicker’s universal search technology will index it and make it available for the iPad-specific site. At launch, the site will only be able to be accessed by iPad devices.

In the blog post announcing the new site, Clicker says they will be launching a fill-fledged iPad app fairly soon. The site will surely be useful for iPad users to find the TV content on the web that works for the device. Of course, iPad users can also access TV content from a host of iPad apps from TV networks and others that are available for download. Clicker, which officially launched to the public in November, just raised an $11 million Series B funding round led by JAFCO Ventures, with existing investors Benchmark Capital and Redpoint Ventures also participating.

With all the talk about Google and Amazon versus Apple, both Internet companies have been quick to release custom applications for the latter’s freshly launched iPad device.

Amazon.com had already introduced its Kindle app for iPad, and over the weekend added a second one to the stable: an IMDb (Internet Movie Database) app (iTunes link).

The Amazon.com subsidiary, which offers a ton of information on movies, TV and celebrities, has actually not released an entirely new app but simply altered its iPhone app to optimize the experience for the bigger screen. According to the company, the IMDb Movies & TV App for iPad lets customers access more than 1.5 million movie and TV titles with information on 3.2 million celebrities, actors, actresses, directors and crew members.

Users can also use the app to watch movie trailers in HD, find localized movie showtimes in the United States, browse photo galleries and shop for DVD and Blu-ray releases. The app is free of charge – it hasn’t garnered any reviews on iTunes so far.

IMDb’s website welcomes over 57 million unique visitors per month, although it looks and feels incredibly old school to me. No doubt, iPad owners with even the slightest interest in movies, TV or celebrities are going to want this app.

Be sure to check out our complete list of the best iPad apps at launch.

Stephen Colbert gave up a kidney, got his hands on an iPad, and talks about its lack of features (like the fact that you can’t call people with it, “just like the iPhone”) and its hidden features, like the ability to make delicious salsa with it.

Also, Colbert pans Amazon and its Kindle.

Bam!

Editor’s note: Big Data has been around for a long time between credit card transactions, phone call records and financial markets. Companies like AT&T, Visa, Bank of America, Ebay, Google, Amazon and more have massive databases they mine for competitive advantage. But lately, Big Data is finding its way to the smallest startups. The Web and cloud computing brings Big Data everywhere. But what exactly is pushing Big Data forward?

To answer that we brought in an expert, Bradford Cross. Bradford is the Co-Founder and Head of Research at FlightCaster. FlightCaster is backed by Y Combinator, Tandem Entrepreneurs and Sherpalo Ventures. The company analyzes large data sets to predict flight delays. Bradford is chair of the Dealing with Big Data track at Cloud Connect this week.

We are in a Renaissance for computer science, engineering, and learning from data right now. The scale of data and computations is an important issue, but the data age is less about the raw size of your data, and more about the cool stuff you can do with it. Now that there is so much data, it is time to unlock its value. Really neat things are happening already—like the way the people of the world can educate themselves on all manner of issues and topics, or the way data and computing serves as leverage in other scientific and technical endeavors. There will be lots of amazing stuff on the web, but innovation will come in other domains as well.

The recent big data trend is about the democratization of large data more than its growth. In articles like the Economist’s recent piece on the data deluge, we hear about big data everywhere. We hear about what big data and the cloud mean for the enterprise, but they have had big data for a long time. eBay manages petabytes in its Teradata and Greenplum data warehouses. Sophisticated startups extracting value from big data is also nothing new—it has been happening at least since the days of Yahoo! and Google, and they have done it without the data warehousing folks.

Now focused early stage startups can get up and running faster than ever. Less technical analysts at companies like Facebook and Twitter can access massive amounts of data easily. Even individuals can undertake cool projects with big data, such as Pete Skomoroch of Data Wrangling did with trending topics for Wikipedia.

Why Now?

We do not have to build all our own hardware and software infrastructure anymore.

Pioneers such as Amazon have given us the cloud, where we have the capability to run very large server clusters at a low startup cost. Pioneers like Google have paved the way for open source projects like Hadoop and HBase, that are backed by big company contributors like Facebook.

The combination has paved the way for a new class of data driven startup like Aardvark (just acquired by Google) and Factual, it has reduced both cost and time to market for these startups, as we showed with Flightcaster. And, it has allowed startups that were not necessarily data driven to become more analytical as they evolved, such as Facebook, LinkedIn, Twitter, and many others.

So we have big data, the cloud, and open source facilitating new data-driven startups. I like to break this trend down from the technical perspective into three chunks; storing data, processing data, and learning from data. I define “learning from data” to mean data mining, AI, machine learning, statistics, and so on.

Supersize my data. Oh wait, I’ll just have a Medium.

The first time I heard the “Medium Data” idea was from Christophe Bisciglia and Todd Lipcon at Cloudera. I think the concept is great. Companies do not have to be at Google scale to have data issues. Scalability issues occur with less than a terabyte of data. If a company works with relational databases and SQL, they can drown in complex data transformations and calculations that do not fit naturally into sequences of set operations. In that sense, the “big data” mantra is misguided at times. For instance, a GigaOm article about big data in the cloud states:

What is becoming increasingly clear is that Big Data is the future of IT. To that end, tackling Big Data will determine the winners and losers in the next wave of cloud computing innovation.

The big issue is not that everyone will suddenly operate at petabyte scale; a lot of folks do not have that much data.

The more important topics are the specifics of the storage and processing infrastructure and what approaches best suit each problem. How much data do you have and what are you trying to do with it? Do you need to do offline batch processing of huge amounts of data to compute statistics? Do you need all your data available online to back queries from a web application or a service API?

Once your data and its processing are large enough to require distributing the data and the work among machines across network boundaries, things get a lot harder. You have to deal with distributed computing and make tradeoffs like a real computer scientist.

Big Data & The Cloud: Viral Buzzwords 4.0!

The cloud, and hosted services, present very interesting opportunities. One of the greatest is that people can leverage the a la carte economics of elastic computing to do things that were prohibitively expensive due to the requirements of building and maintaining their own hardware infrastructure. The interesting parts about the current cloud are its lack of entrance friction and elastic cost efficiency, the speed with which new entrants can set up, and the elastic capability to run 100 machine clusters for 1 hour if that is what is needed.

We started Flightcaster almost a year ago, and it is a good example of how startups can leverage cloud compute and storage resources, mix some open source like Hadoop with some data mining, and create interesting new technologies with relatively low capital upfront.

The cloud is not cheaper in general. Once people scale to a certain point, they move off the cloud onto dedicated hardware—not the other way around. That may change, and better hosted services may play a role in the transition, but that will take a while. In the meantime, the interesting part of the cloud is the use of elastic resources and the ability to get up and going quickly. The interesting part is the freedom it gives startups to try things they would never otherwise do.

Another notable thing about the cloud is the new architectures emerging as a result of economic and resource tradeoffs.

Storage of large amounts of data in the cloud is much cheaper with blobstores like Amazon S3 than it is to maintain an always-up cluster for a distributed datastore. If you do mostly offline batch processing and you do not need bulk storage to be online, then it is an attractive setup.

Storage and NoSQL

Taking another glimpse from the future of big data in the cloud.

A Big Data stack…will also need to emerge before cloud computing will be broadly embraced by the enterprise. In many ways, this cloud stack has already been implemented, albeit in primitive form, at large-scale Internet data centers, which quickly encountered the scaling limitations of traditional SQL databases as the volume of data exploded. Instead, high-performance, scalable/distributed, object-orientated data stores are being developed internally and implemented at scale…large web properties have been building their own so-called “NoSQL” databases, also known as distributed, non-relational database systems (DNRDBMS).

There are several misguided points here. First, there is not going to be a big data or cloud stack. Distributed systems are about making trade offs and a move toward problem-specific solutions rather than one-size-fits-all stacks. Second, enterprises already have their solution—expensive data warehousing and consulting support. Will open source projects like Hadoop supported by people like Cloudera take a chunk of the business? Sure. But as I mentioned earlier, the most interesting part about big data and the cloud is not cheaper alternatives for the enterprise, it is the opportunities it facilitates for data-driven startups.

There is a lot of talk about the NoSQL movement. The big idea here is that distributed systems are hard, require tradeoffs, and sometimes we are better off with data storage and processing that are specific to what we are doing with the data. Sometimes even with a small amount of data on a single node, there are better alternatives to SQL queries and relational databases—time series data has long been a good example.

Processing and Hadoop: The Elephant In The Room

There is a broad range of needs for processing large amounts of data. These range from simple needs like calculations for log analysis that just need to occur at scale, to middle of the road needs like BI, to complex needs like scalable modern machine learning and retrieval systems.

There are a different approaches one can use to service specific needs. Again, we see the pattern of moving away from one-size-fits-all stacks, and toward building for your needs. That said, there are very generic abstractions like Map-Reduce that work well for a lot of use cases. Distributed systems are hard to get right, so when something like Hadoop gets a lot of momentum, it retains that momentum until alternatives have the time to mature enough to solve the hard problems with fault tolerance, performance, and so forth. Not everyone is Leonardo da Vinci, so people should not attempt to create these systems on their own unless they really know what they are doing. In that sense, the cloud and big data are facilitators of open source.

An important aspect of processing at scale is abstraction. Writing complex or even simple computations in raw Map-Reduce is verbose for programmers and intimidating for others who might want to play with the data. Abstractions over Map-Reduce like Pig and Hive make simple things easy, and abstractions like Cascading make hard things possible. The Map-Reduce paradigm, and Hadoop in particular, have been a big success. That said, Map-Reduce is not the only important piece of compute infrastructure. Message queues serve as the backbone of a lot of compute architectures – implementations of AMQP, such as rabbitmq, are a prime example. You can accomplish a lot with producers, consumers, and a messaging system. Distributed storage and processing systems can also be very tricky to configure and deploy, requiring a pretty deep understanding of the system – hence the business case for folks like Cloudera.

Learning from Big Data

Hal Varian, Google’s Chief Economist, recently said,

The sexy job in the next ten years will be statisticians… The ability to take data—to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it

Unfortunately for those of us working on these problems in real life, it is not so simple. The archetypal data-renaissance man is mathematician, statistician, computer scientist, machine learner, and engineer all rolled into one. There are opportunities where you can lack some of these skills and work with a team that supplements your weak points—a startup is not one of those.

Now that we can store so much data, it is attractive to do previously unimaginable things with it. We are sure to see cool applications in fields from the internet to biotechnology to nanotechnology and fundamental materials science research. Almost all advances in every field of science and technology are now heavily dependent upon data and computing. Machine learning is serving a fantastic role as a bridge between mathematical and statistical models and the worlds of AI, computer science, and software engineering. We are exploring applications in learning from text, social networks, data from scientific experiments, and any other data sources we can get our hands on.

The data renaissance does present some difficult issues. There are not many places one can recieve a good education on working on these problems at large scale. Scaling our modeling and optimization algorithms is hard. We need to figure out how to partition and parallelize, or sometimes trade speed and scale for approximately correct calculations. Another issue is that we are often using simplistic models, albeit with pretty good results in many cases. We would like to move toward a deeper approximation of real intelligence.

But the data renaissance is here. Be a part of it.

During his keynote at SXSW this afternoon (live blog here), Twitter CEO Evan Williams just announced a new “At Anywhere” platform, which allows websites to more deeply integrate the service into their sites. The idea is to offer a more seamless experience to Twitter users navigating third party sites like the Huffington Post and the New York Times, giving them Twitter content without forcing them to jump off the page they’re currently viewing. The details on the new platform are still scant, but this is Twitter’s answer to Facebook Connect, which we reported on back in January.

Among the features:

• When you browse a site that uses @anywhere, people and brands that have Twitter accounts will be highlighted with a hyperlink. Mousing over that hyperlink will show a small box (a “hovercard”) containing their Twitter information, including their most recent tweet (in effect it means you don’t have to click over to Twitter’s homepage to see their Twitter profile)
• Publishers will be able to more deeply integrate their own Twitter profiles, making them easier for their readers to ‘follow’ them
• Sites will be able to implement @anywhere with a few lines of Javascript.
• The new platform is launching with a number of major sites and services, including the New York Times, Huffington Post, Meebo, Amazon, Yahoo, Bing, and eBay.

It looks like the platform may eventually be hosted at Twitter.com/anywhere, which currently features a placeholder Twitter account that tweeted “Stay Tuned”. Update This may actually be a Twitter account related to the platform — it just tweeted “If you’re a javascript guru and want to help us build @anywhere and work with publishers @jointheflock”.

From the Twitter blog:

We’ve developed a new set of frameworks for adding this Twitter experience anywhere on the web. Soon, sites many of us visit every day will be able to recreate these open, engaging interactions providing a new layer of value for visitors without sending them to Twitter.com. Our open technology platform is well known and Twitter APIs are already widely implemented but this is a different approach because we’ve created something incredibly simple. Rather than implementing APIs, site owners need only drop in a few lines of javascript. This new set of frameworks is called @anywhere.

When we’re ready to launch, initial participating sites will include Amazon, AdAge, Bing, Citysearch, Digg, eBay, The Huffington Post, Meebo, MSNBC.com, The New York Times, Salesforce.com, Yahoo!, and YouTube. Imagine being able to follow a New York Times journalist directly from her byline, tweet about a video without leaving YouTube, and discover new Twitter accounts while visiting the Yahoo! home page—and that’s just the beginning. Twitter has proven to be compelling in a variety of ways. With @anywhere, web site owners and operators will be able to offer visitors more value with less heavy lifting.

We’re here at the SXSW festival in Austin, Texas where Twitter co-founder Evan Williams doing a keynote Q&A with Umair Haque. Williams may use the time to talk a bit about Twitter’s upcoming ad platform. Update: It’s actually an “At Platform” called At Anywhere — more here.

Interestingly enough, Twitter saw its first burst of popularity three years ago at this very conference.

Below find my live notes (paraphrased):

UH: Ev you have something pretty interesting you want to say today?

EW: Yeah, we want to announce something. We wanted to announce our new “At Platform” (undoubtedly to be spelled an @ Platform) – a way to integrate Twitter into any website. “At Anywhere” – basically this allows you to place the Twitter hovercards on any site. We have 13 sites we’re launching with including Amazon, ebay, Yahoo, Digg, Bing, Meebo, Salesforce.

UH: So what can you do with this?

EW: You can easily tweet from any page that is using this. Also, maybe you want talk to authors of posts without going to Twitter itself, you can just hover over their name and tweet them. Twitter is a very easy way to keep in touch.

UH: So this helps you contextualize information. But why would sites use this?

EW: A connection to users you didn’t have before – and it keeps people coming back. And it will result in more followers for a site. Also, hopefully more people who are your fans using twitter to talk about you or your content. And you can bring in users’ tweets talking about your site.

UH: So it’s a platform to juice up site’s networks and virility. But it’s an “At Platform” not an “Ad Platform”.

EW: Yeah, it’s about lowering the barrier for information.

UH: What makes 21st century businesses different? Like Twitter? The first principle to me is experimentation. Why are you willing to explore different possibilities?

EW: Experimentation lets you create value. “Whatever you assume when you start out, you’re wrong.” Most of the great businesses of our time have experimented. Like Google.

UH: So it’s about creating value, then figuring it out?

EW: Yes, it’s about creating experience for users and businesses. There is a ton of business use on Twitter today — it’s one of the biggest uses. We want to make that better, easier, faster.

UH: What is Twitter evolving to?

EW: What is Twitter has always been a tough question to answer. We think of it as an information network — different from a social network. It’s about getting info and also sharing. You can take advantage of Twitter without sharing anything about your life. We need to increase the signal-to-noise ratio.

UH: So better information, better connections, better choices.

EW: Yes.

UH: Experimentation is about iteration. So how does that happen at Twitter?

EW: We have a bunch of awesome people in the company now. We’ve grown very quickly over the past year. Our employee growth curve is almost like our user growth curve now. We have people on focused teams, like mobile, or internationalization. We’re worried about central thinking and slow processes. So we tell our teams to “go for it.”

UH: So what’s your role?

EW: I don’t get into the nuts and bolts of code, cause things would be a big mess. I spend most of my time thinking about the high level issues. And I think a lot about the company – how do we scale the company, about our culture, etc. How do we define the characteristics we want. I think there is a parallel between the service and the company — openness is huge, transparency.

UH: So openness is very important. Help us trace the arc of openness at Twitter.

EW: Yeah, it means a lot of things. We debated if openness or transparency. “A window is transparent, but a door is open.” The users have taken Twitter and morphed it into what they want it to be. Now developers are doing the same thing. Openness is really a survival technique.

I sit down with new employees when they start and go over 9 assumptions you should have about working at Twitter. One key one is assume there are more smart people outside the company than insides.

UH: What about giving the golden goose away? Why be so open?

EW: That was a big question for us – the deals with Bing and Google. These were the first guys we shared our full stream with. There’s a lot of debate about that. Because we don’t have a business model yet, so why give it away? But we went back to the principle of giving users the most value.

There are 50 million tweets a day, how do we show you the best ones for you? Right now, we don’t do a good enough job of that. But with these partnerships, we have more chances to do that.

UH: Was there a lot of internal debate about this?

EW: Yeah, there was a ton. But we decided it was good. And now we’ve expanded the deals – like with Yahoo. And a few weeks ago we talked about giving this data to thousands of others.

Now third party developers are building a lot of value. Like adding pictures to Twitter.

CoTweet and HootSuite are really interesting too. Twitter.com isn’t a good interface for doing customer support, but those guys are. CoTweet just got acquired by a company that wants to focus on that more.

We’d love to see much more focus on creating these deep experiences that create value.

UH: So experimentation and openness. Other companies want control, like Apple. How open are you guys?

EW: We’re pretty open – there is some control we need to employ because if we were infinitely open we’d be doing a disservice to users. Openness can work against you still. It has to be managed a lot. Having an open API makes it easier to make apps that will spam users. We send cease and desists everyday to companies making spam tools. We have to exert some control.

UH: I think shepherding is a good way to put it. So you had some interesting use recently – such as the earthquake in Chile.

EW: I got an email recently about the earthquake, thanking us for helping with the situation. This is very gratifying for us because we’ve always held it important for Twitter to reach the weakest signals in the world. We started out with a big focus on SMS – and it’s still really important to us. Because it reaches so many people. We have deals with 65 carriers around the world to send these SMS tweets.

We’re at the beginning. We’re seeing really strong growth in India where SMS is huge. And in the Middle East.

UH: I think this changing the world stuff is the future for entrepreneurs. It gets to the heart of the point about inclusiveness. So – what is an “active user”?

EW: To me it comes back to – is someone getting value out of Twitter? If they don’t have an account it’s hard to know, like people who search Google for tweets. In the beginning we put a lot of focus on telling the world or your friends and family what you’re doing. But now there is something interesting on Twitter for everyone – like the Flaming Lips being on Twitter, you can get updates on the band.

And as more people start getting information on Twitter, they’re more likely to get involved.

UH: Someone has started using Twitter inside the White House, right?

EW: Yeah, it’s really interesting that it’s from in the White House. It’s an official channel, but they’re using it a different type of way. It’s about reducing the walls between people with a lot of influence, and those who they influence. And that’s the most profound promise of the Internet. This is the wave I started on 10 years ago with blogging. It’s about the democracy of information. Anyone can put information on the web — that’s huge.

UH: Tweet Minister in the UK aggregates the tweets from members of parliament. This is re-wiring society in some ways. But we also have a counter-force – like state control of information.

EW: In some regions, yes, this is bad and hurting the web. But the Internet is a tidal wave that you will not be able to keep out. Like in China, who knows how long those firewalls will hold up – but not forever.

UH: Yes, there are many ways to get through the firewalls already. There’s a lot of pressure on them.

Let’s talk about “betterness.” I booked a trip to his five star resort in an exotic land. When I got there, it was a shack. The manager couldn’t do anything — so I put it on Twitter. Within 15 minutes the booking company called me, and in 20 minutes I got a new hotel. In a half an hour my vacation was fixed.

EW: That’s great. Our hope is that this is the norm, not a fluke. We have a bit of a dichotomy, because there is more everyday you want to search for. We don’t just want to maximize that, we hope to make Twitter more useful to you. We want to decrease time you spend on Twitter, not increase it.

Recently we went through a process to define our operating principles. The number one principle is “be a force for good.” Another principle is “pay attention.”

UH: David Pogue did a campaign against hidden charges from the carriers. It’s the same thing with the hotel operator and me. I know you’re a big fan of Warren Buffet – he also believes in creating real value.

EW: Yes, from a business perspective, Twitter needs to fundamentally be about helping people make better decisions. Or the help something happen that normally wouldn’t. Like the donations to Haiti through text message — we weren’t taking the money, but it spread virally through Twitter. People want to help each other out, we need to reduce the friction.

UH: Is that what you want to do with the new At Platform?

EW: Yes, totally. We’ll see what happens, the obvious stuff is more tweeting, but I think it’s a lowering of the friction as well.

UH: You ask yourself, how would i make Walmart better? Why ask yourself that?

EW: Because as we look at how businesses are using Twitter – we want our tool to help businesses get better.

The world is so often a black box where there is no communication. There’s a lack of dialogue and a lack of transparency. The promise of all these technologies is that this goes away. You close the loop.

UH: Outline for us your big picture goals.

EW: Fostering the open exchange of information. To be a force for good. The ease of exchange of information is important. Help out other people with something as small as a retweet. That’s our ambition.

UH: Google is all about archiving the world’s information. Yours is different — creating new information.

It’s all about advantage though – what’s your advantage.

EW: Our advantage will only come if everyone wins. We only do win-win deals. Because any deal where someone is losing is unsustainable. That’s why we haven’t turned on the revenue yet — there’s a lot of low-hanging fruit, but none of it is sustainable.

Creating an advantage for other people and not giving them a reason to work around you – that’s key.

UH: Is the Internet making a better media industry?

EW: I think there’s a huge shift going on – but it’s an ecosystem where everything is involved. This user-generated content just makes things richer. Blogging and traditional media work together. Twitter compliments traditional media. I was talking last night to some guys from CNN – it’s helped them change what they do. It’s a win-win.

UH: How will the At Platform speak to that?

EW: Hopefully these guys will us it to get the new out there.

UH: What makes you tick?

EW: There are two types of entrepreneurs. What drives me is creating things that didn’t exist before. Your product or service should be at the end of the sentence: “wouldn’t it be awesome if…”

It’s creating new stuff versus extracting from old stuff. There are people who look at money as the goal versus the teams. I create businesses to make new things. It’s a fuel for creating more things in the world. I’ve been lucky to stumble upon things that have helped change the world.

UH: Why focus on these things though?

EW: Largely luck. But maybe it’s what interests me. Twitter was a side project of Odeo – my cofounders came up with it. Blogging was a side project too at one point.

UH: If something is awesome, people will use it.

EW: Yes.

Also, helping others succeed is a sub principle of ours.

UH: Tell us one or two more of them.

EW: Be a force for good, pay attention — make things happen is another one. There’s also building a culture of trust.

UH: What are your big lessons to other entrepreneurs?

EW: Create something you want to exist in the world. Another is focus. Many people are trying to do a lot of things when they should be doing one thing. You may be wrong with whatever you’re trying out, but you’ll try other things.

A lot of the great companies are now coming from outside Silicon Valley. You don’t have to be there.

That’s a wrap.