Big Data

A wind-farm operator collects readings from thousands of turbine sensors many times a second, mixing numeric telemetry, maintenance text logs and photographs. Explain which Vs are challenging.

1. 01Volume

   Thousands of sensors sampled frequently produce an enormous, ever-growing quantity of readings - too much for a single server to hold or scan efficiently.

2. 02Velocity

   Readings arrive continuously and many times a second, so they must be ingested and processed as a real-time stream, not a nightly batch.

3. 03Variety

   Numeric telemetry, free-text logs and images together do not fit one relational schema - a mix of structured, semi-structured and unstructured data.

Result: All three Vs are present: the volume, velocity and variety together make this a big-data problem requiring distributed storage and processing rather than a single relational database.

A fraud team needs to find rings of accounts that transfer money to one another in circles. Explain why a graph model suits this better than relational tables.

1. 01The data is relationships

   What matters is the pattern of transfers between accounts - the edges - not just the accounts themselves.

2. 02Relational cost

   Finding a circular chain of transfers in relational tables needs the transactions table joined to itself repeatedly, once per hop - expensive and awkward for chains of unknown length.

3. 03Graph advantage

   In a graph model each account is a node and each transfer an edge; a circular ring is a cycle, found directly by traversing edges without repeated joins.

Result: A graph-based model represents the transfers as traversable edges, so detecting cycles (fraud rings) is a natural graph traversal - far more efficient than the many self-joins a relational database would require.

Show how MapReduce would count word occurrences in the two short documents 'the cat sat' and 'the cat ran', explaining each phase.

1. 01Map (in parallel)

   Each document is processed independently, emitting (word, 1) for each word. Doc 1: (the,1)(cat,1)(sat,1); Doc 2: (the,1)(cat,1)(ran,1).

2. 02Shuffle and sort

   Pairs are grouped by key: the -> [1, 1]; cat -> [1, 1]; sat -> [1]; ran -> [1].

3. 03Reduce (in parallel)

   For each key, sum the values: the -> 2, cat -> 2, sat -> 1, ran -> 1.

Result: The counts are the: 2, cat: 2, sat: 1, ran: 1. Because map runs independently per document and reduce independently per key, both phases parallelise across the cluster - the essence of processing big data.