Data Processing Performance Options

1

Posted by Lawrence Sinclair on 14 Sep 2009 at 04:19

Here are a few of my thoughts on technologies and approaches for achieving better data processing performance in the current technology landscape.

RDBMS
Using mySQL or another RDBMS, performance might be addressed with better indexing or by partitioning the data.
MAP-REDUCE NON-RELATIONAL SYSTEMS
A non-relational approach might be to useHadoopor one of its distributions (such asCloudera). This would allow processing to be distributed anywhere from 3 local machines, or a virtually unlimited (hundreds+++) number of machines on the cloud (such as Amazon EC2). But this is best suited for analytic and data processing tasks that can takes several minutes or hours.
THE BEST OF BOTH WORLDS?!
Somewhere in between these two systems isHadoopDBbyDaniel Abadiof Yale. It uses the Hadoop...

Coffee with East Agile

0

Posted by Lawrence Sinclair on 28 Aug 2009 at 23:33

If I were sitting down with you for a cup of coffee, this is what I'd tell you about East Agile, what we can do, and what it is like working with us. It is thirteen minutes long, but I think you will find it very enlightening and really help you get to know us.

Cheers,

-Lawrence Sinclair, CEO 

East Agile Intro Video <-- FULL 13 MINUTE VIDEO

(Download includes introduction, as well as discussions of our approach to analytics and software development)  

 

 

 

Or jump to the stream discussing our approach to agile development:

 

 

Learn about our approach to big data analytics:

 

  

Identifying Ruby on Rails Sites

6

Posted by Lawrence Sinclair on 15 Aug 2009 at 06:40

Can you tell if a site was implemented using Ruby on Rails? It is not always obvious. But there are clues that you can use. One of the East Agile developers shares the following advise on this topic.

Based on my experience about rails and web technologies, I have some rules to identify the technology behind a website:
- recognize by URL pattern
- view HTML source
- HTTP monitor by Firebug
- recognize it's not rails-based by understanding other platforms

Basically, I give you 2 simple approaches. To take advantage of these, you may need an understanding of some web technologies and you may need to install more tools.

1. URL pattern:
- rails-based website usually does not have those file extensions in URL: php, asp, aspx, do, jsp, ...
- Rails supports RESTful URL build-in. So URL in...

Micro-Multinationals

0

Posted by Lawrence Sinclair on 15 Aug 2009 at 06:27

Hal Varian was quoted recently talking about Micro-multinationals as a fairly recent business innovation he has noticed in Silicon Valley in particular - small businesses made up of people spread throughout the world with some key players in the Bay Area. We understand that mode of business well. And working with an East Agile development team is a great way to augment and enable that mode of operation.

Seehttp://news.cnet.com/8301-30684_3-10309375-265.html

Facebook Social Data

1

Posted by Lawrence Sinclair on 01 Aug 2009 at 02:54

Socialscore is a dashboard for Facebook users' social networks. The concept is to present meaningful metrics, information, and search capabilities to enable Facebook users to understand their social networks and social influence. The idea (IMHO) was rather cool, but we ended up with under a hundred users and the site was never very successful from a business perspective. However, it does remain a good demonstration of some of our programming capabilities.

I present Socialscore in the following video:

Mobile Platform Analytics

0

Posted by Lawrence Sinclair on 27 Jul 2009 at 11:36

Recently, we've been doing a lot of analytics work for clients with social networks and games on mobile platforms ranging from basic phones using SMS to iPhones. The focus in these cases involves discovering the drivers behind customer behavior and developing viable business models against a rapidly changing competitive landscape. Our onsite PhD analytics guru, Tim, has been really key in making this part of our business work, supported by the rest of the offsite team at East Agile. Tim carries some pretty powerful tools in his belt, which includes Hadoop on EC2, SAS, and some special joint-entropy optimization tools.

Oneforty.com becomes client

0

Posted by Lawrence Sinclair on 27 Jul 2009 at 11:24

After several weeks of initial development by Pivotal Labs in San Francisco, East Agile has provided some additional support and helped Laura Fitton's oneforty application during the transition to its own development team.

Lost Ben to Google

1

Posted by Lawrence Sinclair on 07 Jun 2009 at 13:27

Ben, one of our developers, has just left East Agile. He was poached by Google and will start work in their New York City office in a few weeks. That's a real loss for us.On the other hand, I'm rather pleased to see that the likes of Google think so well of our people.

Software Development using Economic Optimization

0

Posted by Lawrence Sinclair on 29 Apr 2009 at 23:43

Project Management tools can collect information about time required to produce specific product sub-components (features, user stories). Some agile software development tools, like Pivotal Tracker collect information about actual outcomes based on detailed descriptions of requirement. The tools also collect information about who was involved in developing those results.

Given information like this, it is quite possible to solve some interesting problems for larger organizations. For example, given a set of developers, in an environment with budget constraints, one can find the optimal price (cost) for development output units (velocity) given developer wages and marginal productivity. One can also look at this from a hedonic perspective where you look at the optimal allocation of...

Application in our Mojo Simulator

2

Posted by Lawrence Sinclair on 04 Apr 2009 at 18:19

The following links to a live example of code running in our Palm Pre Mojo Simulator. It is an extension of the simple RSS application in the O'Reily webOS book.

http://www.eastagile.com/uploads/news/

Please note that the simulator is only fully functional in Apple Safari version 4 beta since this is the only browser that supports the HTML5 which is used in Palm Pre applications.