DynamoDB is a perfect NoSQL DB for serverless applications. It is fast, reliable and scalable. However, things that you could do trivially in an SQL database, like fetching all items, require a bit more effort in DynamoDB. In this post, I will show you how to scan and get all items in a DynamoDB table using TypeScript and Node.js.

Because DynamoDB uses a paginated model for scan operations, you have to deal with the 1MB limit and pagination. This means that you have to make multiple requests to get all items in a table. The following code shows how to do this using AWS SDK v3 (I realize that a lot of projects are still relying on the older version — AWS SDK v2 — so please reach out and let me know if you want a v2 version of this example at some point)

import {AttributeValue, DynamoDBClient, ScanCommand} from "@aws-sdk/client-dynamodb";

const client = new DynamoDBClient({});

async function getDynamoDbItems(
    tableName: string,
    cursor?: Record<string, AttributeValue>
) {
    const items = await client.send(
        new ScanCommand({
            TableName: tableName,
            ExclusiveStartKey: cursor,
        })
    );
    return {
        items: items.Items ?? [],
        nextKey: items.LastEvaluatedKey,
    };
}

The above is a basic recursive function we have to call again and again with a different cursor until there are no more pages of data to retrieve from the DynamoDB table. The cursor is the LastEvaluatedKey returned from the previous scan operation. If it is not present, it means that we have reached the end of the table.

Here's an example of how we can use the above function to iterate the entire table:

// Start by defining the DynamoDB table name here
const TABLE_NAME = "my-table";

// This will include all the rows in the end
const result = [];

// Keep track of the current cursor. It's `undefined` at the beginning
let lastEvaluatedKey: Record<string, AttributeValue> | undefined = undefined;

do {
    // Fetch the next batch of items
    const {items, nextKey} = await getDynamoDbItems(
        TABLE_NAME,
        lastEvaluatedKey
    );

    // Push current page of results to the aggregator
    results.push(...items);

    // if present, set cursor for next iteration/recursion
    lastEvaluatedKey = nextKey;
} while (lastEvaluatedKey); // Run until there are no more pages

// Do something with `results` here

Happy recursive iterating!