If well designed, indexed and with good hardware it works great!

It works great because we usually know and can foresee the load on the system.
We have a stable user base and most often if we do online applications it’s B2B with a fairly predictable load.

But… Now companies want/have to start using social media and move closer to the end-user… Then everything changes!

We need to be able to scale quickly, have predictable performance when load increases and because not every project will generate money directly it can’t cost a fortune to build and run it.

The architecture of such a solution is a future post, for now I’m just focusing on the database.

So… Scale the normal db could work but maybe the load increases to quickly for new servers to be bought/allocated OR the application just get’s to large to be manageable with a normal cluster OR maybe it’s just not going to make economic sense to meet the demand with the increase in license cost (I know that you could use a OS db, but they might not scale very well), expensive hardware and so on…

So what can we do?

One solution… Non-relational databases or NoSQL as it’s popularly called.

It has many very interesting characteristics that potentially could solve many of the limitations of the relational database.

There are a few different approaches to the NoSQL and sometime the are used in conjunction where the major technologies are; Key/value, document-oriented, graph, tabular, object-oriented.

So what is it and how does it work… Instead of me copying what smarter people already wrote here is a short collection of great articles, blogs and other resources on the subject. Don’t forget to read the comments and updates for the most recent information!

Enjoy!

Articles & Blogs
CouchDB: A Case Study – “This is part 1 in a series of posts that describe our investigation into CouchDB as a solution to several database related performance issues facing the TextMe application.”
Is the Relational Database Doomed? – “Recently, a lot of new non-relational databases have cropped up both inside and outside the cloud. One key message this sends is, “if you want vast, on-demand scalability, you need a non-relational database”.”
Should you go Beyond Relational Databases? – “Relational databases, such as MySQL, PostgreSQL and various commercial products, have served us well for many years. Lately, however, there has been a lot of discussion on whether the relational model is reaching the end of its life-span, and what may come after it.”
Anti-RDBMS: A list of distributed key-value stores – “Perhaps you’re considering using a dedicated key-value or document store instead of a traditional relational database. Reasons for this might include…”
Neat ideas: Thrift, Thrudb – “Wow. The pace at which innovation happens on the web is truly breathtaking. Close on the heels of SimpleDB, The Third Rail guys have relased ThruDB, which I can only say is a fantastic idea. Simply put, ThruDB is a set of services built on top of Facebook’s Thrift, its’ cross-language services development framework, to provide highly scalable document-oriented data storage mechanism.”
Thrudb – faster, cheaper than SimpleDB – “Amazon’s recent announcement of SimpleDB generated a lot of buzz in the community – finally, a database service for our virtual infrastructure! Well, not so fast. SimpleDB is a far cry from the usual RDMBS we’re all used to, and it’s best described as a key-value store (BerkeleyDB, really), or a meta-data storage layer for your S3 objects. If you’re curious, Charles Ying and Todd Hoff provide a great overview, as well as a plethora of links for more information on SimpleDB.”

Products
Apache CouchDB – “Apache CouchDB is a document-oriented database that can be queried and indexed in a MapReduce fashion using JavaScript. CouchDB also offers incremental replication with bi-directional conflict detection and resolution.”
Truedb – “Thrudb is a set of simple services built on top of the Apache Thrift framework that provides indexing and document storage services for building and scaling websites. Its purpose is to offer web developers flexible, fast and easy-to-use services that can enhance or replace traditional data storage and access layers.”
Amazon SimpleDB – “Amazon SimpleDB is a highly available, scalable, and flexible non-relational data store that offloads the work of database administration. Developers simply store and query data items via web services requests, and Amazon SimpleDB does the rest.”
Google App Engine Data store – “The App Engine datastore is a schemaless object datastore, with a query engine and atomic transactions.”
Project Voldemort – “Voldemort is a distributed key-value storage system”

Read Ken Down’s post The argument for denormalization on the subject, it’s very interesting!

Morten Just, a user experience guy at Vodafone posted on his blog a smart way of using Google Drawings for wireframe’s…

Check it out!

What… Deliver what is most important to your customer.
How… Deliver it in time and on budget.

So… What can help you in terms of project management approaches to reach that?

Variables… What do we have to play with?

Scope, time, budget and quality… Thats pretty much it!

Are all equally flexible? NO!

Scope, time and budget are all very flexible but quality never is… Some (managers) can claim that it is but it is NOT!

I’m not saying that your application should be free from bugs but you shouldn’t purposely build it to less than the best of your ability.

So…
- We should deliver what the customer want on time, on budget and with high quality.
- Variables to play with are therefore scope, time and budget.

That all seems reasonable… BUT that is not always the case.

The missing factor… RISK!

The risk that WHAT might change if the TIME-frame is too long.
The risk that the SCOPE gets unmanageably large if the TIME-frame is too long.
The risk that the BUDGET gets “unattractive” if the SCOPE is to large due to the risk of not delivering on WHAT.

These are all well-known issues with old school waterfall models.
The waterfall model seem good in theory but lacks to deliver in real life.

So… We need to keep WHAT relevant and minimizing the risk of breaking time and budget.

The current answer to that… Agile software development

The Agile movement have done many iterations and are continuously evolving but to get practical… What is the most attractive approach today?!

Subjectively… I like the idea of combining several of them.

Lean software development for organizational principles.
Scrum for project management.
Extreme Programming (XP) for engineering practices.

Combining Scrum and XP is covered in the book Scrum and XP from the Trenches by Henrik Kniberg

But… as always, the hardest part is doing it!