Jorge Turbica / blog · Blog ← All articles ES · Leer en español
Cloud

DynamoDB now loads 100 million items from S3 in under an hour, into existing tables

Bulk put items loads S3 files into live DynamoDB tables with per-item conditions and no capacity hit. Cancelling doesn't undo what's written.

A hundred million 1 KB items in under an hour, without touching table capacity and without writing a single BatchWriteItem loop. That is AWS's illustrative figure for bulk put items, the DynamoDB feature available since November 5, 2026 that loads S3 files into a table that already exists. Until now, managed import from S3 only worked for brand-new tables.

What changes

DynamoDB already had Import from S3, but its documentation is explicit: it creates a new table when the import starts and does not support existing tables. Reloading a catalog, running a backfill or refreshing data every night meant building your own job around BatchWriteItem (25-item batches, UnprocessedItems retries, throttling handling) or reaching for Glue or EMR.

Bulk put items is a managed, asynchronous job:

  • Input: line-delimited DynamoDB JSON or Amazon Ion, optionally compressed with gzip or zstd, under an S3 prefix.
  • Per-record conditions: every line can carry its own ConditionExpression, with the same grammar as PutItem. One file can mix unconditional writes, insert-only writes (attribute_not_exists) and version-guarded ones. A record whose condition fails is not an error: it is skipped and logged in the report.
  • Separate capacity: writes run on dedicated system capacity and don't count toward your CloudWatch read and write metrics, so they won't trip your alarms or auto scaling.
  • Two tiers: Standard, the cheaper one, for loads with flexible timing; Accelerated, faster and billed at a higher rate, for migrations or recoveries with a deadline. Standard is available in every commercial Region and GovCloud; Accelerated in a subset.

How long it takes and what it costs

AWS publishes illustrative Accelerated-tier timings for 1 KB items, uniform keys and no GSI replication:

1 KB itemsEstimated time
10 millionunder 10 minutes
100 millionunder 1 hour
1 billion2 to 4 hours

On price, AWS only says it writes "at a lower price point" than equivalent per-request writes. The baseline to compare against: in us-east-1, on-demand mode charges $0.625 per million write request units. Loading 100 million 1 KB items with PutItem is 100 million WRUs, about $62.50 for the base table alone, and every GSI that projects those items adds its own writes. That is the ceiling bulk put items promises to undercut; check your Region's pricing table for each tier's exact rate.

How to run it

The command from the announcement, using the Accelerated tier and reporting both errors and failed conditions:

bash
aws dynamodb start-bulk-put-items \
  --table-name arn:aws:dynamodb:us-east-1:123456789012:table/MusicCollection \
  --input-s3-configuration '{"S3Bucket":"my-import-bucket","S3KeyPrefix":"daily/","InputFormat":"DYNAMODB_JSON"}' \
  --output-s3-configuration '{"S3Bucket":"my-report-bucket","S3KeyPrefix":"reports/","OutputType":["ERROR","CONDITION_CHECK_FAILURE"]}' \
  --performance-tier ACCELERATED

And an input record that is only written if the stored version is older:

json
{"Item": {"Artist": {"S": "The Ternaries"}, "SongTitle": {"S": "Blue"}, "Version": {"N": "12"}}, "ConditionExpression": "#v < :v", "ExpressionAttributeNames": {"#v": "Version"}, "ExpressionAttributeValues": {":v": {"N": "12"}}}

You track jobs with describe-bulk-operation and list-bulk-operations (90 days of history), and cancel-bulk-operation stops one. Permissions come down to two new actions: dynamodb:StartBulkOperation at account level and dynamodb:BulkPutItems on the table, plus read access to the input bucket, write access to the report bucket and access to the KMS keys. Calls are logged in CloudTrail, and starts are idempotent through a client token.

Cancelling undoes nothing, and other traps

The main trap is cancellation: there is no rollback. Cancel halfway and whatever was written stays in the table. That is why AWS recommends making every load safely re-runnable, with attribute_not_exists for inserts or a version comparison for refreshes. With that condition in place, re-running the same file after a failure overwrites nothing.

The second one is downstream. Imported items show up in DynamoDB Streams and Kinesis Data Streams as if they were application writes, and they also reach zero-ETL integrations. Loading 100 million items in an hour is a spike your stream consumers (Lambdas, replicas, search indexes) have to absorb, so size them before you run the Accelerated tier in production. GSIs and LSIs update automatically on isolated capacity: a heavily indexed table loads more slowly, but application latency is unaffected.

Finally, COMPLETED_WITH_ERRORS is not a success. It means some records were not written, most often because of malformed source data. The S3 report lists the object and position of each failure; have your pipeline read it and fail if it isn't empty.

BatchWriteItem and PutItem are still the tool for a handful of items in seconds. For nightly reloads, backfills and migrations into live tables, this feature replaces the batching script nearly every DynamoDB team maintains.

Sources

Jorge Turbica

Software Engineer and SRE. I write about the real challenges I run into in production.

See portfolio ↗